Get Services >  High Performance Computing >  How to Run Batch Job >  Using Checkpoint and Restart

The LSF software provides a basic checkpoint and restart library for use in compilation of programs that need to make use of the checkpoitnt and restart features of LSF. Checkpointing is a service whereby a 'snapshot' of a running job is saved to file, so that a job can be restarted from the snapshot, i.e. from the point when the job is checkpointed, instead of starting from scratch. This has two uses:

  • A long running job does not need to be re-run from scratch in the event of a system crash.
  • A job can be migrated to another system (of the same platform) without killing it and re-running it from scratch.

To enable LSF checkpoint and restart in a program, compile it to an object file instead of creating the executable straight away: After that, link it with the LSF linker, instead of the system default linker. LSF supplies 2 linkers, one for C and another for Fortran compilers.

C example:

   > cc -c myprog.c
   > ckpt_ld myprog.o -o myprog

Fortran example:

   > f77 -c myprog.f
   > ckpt_ld_f myprog.o -o myprog

The executable, myprog, is now ready to be submitted to LSF with checkpoint enabled.


Submitting a Job with Checkpointing Options

In order to use checkpointing, the bsub command has to be given an additional option: -k "[directory [period] ]" (if period is specified, quotes are required as the option takes in 2 arguments separated by a space). In this option, the specified directory is the checkpoint directory that is writable by the owner. We recommend /svutmp, failing which, try /lsftmp or $PWD (the current working directory). The period (in minutes) controls the frequency of checkpointing. As checkpointing is very resource intensive, it is recommended that a period greater than 30 minutes is chosen. The recommended period is 3 hours, otherwise, the compute system may just busy in doing chekcpointing for the program instead of running the program.

Example:

   > bsub -q linux64 -k "/svutmp 300" ./myprog

Or in your job submission script, add the following line:

   #BSUB -k "/svutmp 300"

To initiate a checkpoint manually:

   > bchkpnt JobId

where JobId is the id associated with the LSF job.

top