2Vh$\ddlZddlmZddlmZddlmZedGddeZy)N) keras_export)Callback) file_utilsz keras.callbacks.BackupAndRestorec\eZdZdZ d fd Zd dZdZd dZd dZdZ dZ d d Z xZ S) BackupAndRestoreaCallback to back up and restore the training state. `BackupAndRestore` callback is intended to recover training from an interruption that has happened in the middle of a `Model.fit` execution, by backing up the training states in a temporary checkpoint file, at the end of each epoch. Each backup overwrites the previously written checkpoint file, so at any given time there is at most one such checkpoint file for backup/restoring purpose. If training restarts before completion, the training state (which includes the `Model` weights and epoch number) is restored to the most recently saved state at the beginning of a new `Model.fit` run. At the completion of a `Model.fit` run, the temporary checkpoint file is deleted. Note that the user is responsible to bring jobs back after the interruption. This callback is important for the backup and restore mechanism for fault tolerance purpose, and the model to be restored from a previous checkpoint is expected to be the same as the one used to back up. If user changes arguments passed to compile or fit, the checkpoint saved for fault tolerance can become invalid. Example: >>> class InterruptingCallback(keras.callbacks.Callback): ... def on_epoch_begin(self, epoch, logs=None): ... if epoch == 4: ... raise RuntimeError('Interrupting!') >>> callback = keras.callbacks.BackupAndRestore(backup_dir="/tmp/backup") >>> model = keras.models.Sequential([keras.layers.Dense(10)]) >>> model.compile(keras.optimizers.SGD(), loss='mse') >>> model.build(input_shape=(None, 20)) >>> try: ... model.fit(np.arange(100).reshape(5, 20), np.zeros(5), epochs=10, ... batch_size=1, callbacks=[callback, InterruptingCallback()], ... verbose=0) ... except: ... pass >>> history = model.fit(np.arange(100).reshape(5, 20), np.zeros(5), ... epochs=10, batch_size=1, callbacks=[callback], ... verbose=0) >>> # Only 6 more epochs are run, since first training got interrupted at >>> # zero-indexed epoch 4, second training will continue from 4 to 9. >>> len(history.history['loss']) >>> 6 Args: backup_dir: String, path of directory where to store the data needed to restore the model. The directory cannot be reused elsewhere to store other files, e.g. by the `BackupAndRestore` callback of another training run, or by another callback (e.g. `ModelCheckpoint`) of the same training run. save_freq: `"epoch"`, integer, or `False`. When set to `"epoch"` the callback saves the checkpoint at the end of each epoch. When set to an integer, the callback saves the checkpoint every `save_freq` batches. Set `save_freq=False` only if using preemption checkpointing (i.e. with `save_before_preemption=True`). double_checkpoint: Boolean. If enabled, `BackupAndRestore` callback will save 2 last training states (current and previous). After interruption if current state can't be loaded due to IO error (e.g. file corrupted) it will try to restore previous one. Such behaviour will consume twice more space on disk, but increase fault tolerance. Defaults to `False`. delete_checkpoint: Boolean. This `BackupAndRestore` callback works by saving a checkpoint to back up the training state. If `delete_checkpoint=True`, the checkpoint will be deleted after training is finished. Use `False` if you'd like to keep the checkpoint for future usage. Defaults to `True`. ct|||_||_||_d|_d|_d|_|s td||_ tj|d|_ tj|d|_ |jdz|_|jdz|_|dk7r t!|t"std|dyy) Nrz"Empty `backup_dir` argument passedzlatest.weights.h5ztraining_metadata.jsonz.bkpepochztj|jrtj|j|jYd}~yd}~wwxYwN) _load_modelOSErrorrexistsrcopyrrrremove)rlogses ron_train_beginzBackupAndRestore.on_train_beginqs      $$T%<%<= OOD33T5G5G H  !B!BC5500""4#?#?@!!$">">?      s D C#DD c|jjstd|jdtj|j r|jj Y|jj js9|jj j|jj|jj|j tj|jrctj|jd5}tj|j}dddd}||j_yy#1swY!xYw)z6Get training state from temporary file and restore it.z]To use the BackupAndRestore callback, you model must be built before you call `fit()`. Model zJ is unbuilt. You can build it beforehand by calling it on a batch of data.Nrr )modelbuiltrrr#r optimizerbuildtrainable_variables load_weightsrFilejsonloadsread_initial_epoch)rftraining_metadatar s rr!zBackupAndRestore._load_modelszz %??    T// 0 $$0 ,,22 $$**4::+I+IJ JJ # #D$6$6 7   T99 :!=!=sC 9q$(JJqvvx$8! 9%g.E(-DJJ % ; 9 9s "$E&&E/cf|dz|_d|_|jdk(r|jyy)Nrr )rrr _save_model)rr r&s r on_epoch_endzBackupAndRestore.on_epoch_ends3#ai ! >>W $     %rcH|j|r|jyyr )_should_save_on_batchr:)rbatchr&s ron_train_batch_endz#BackupAndRestore.on_train_batch_ends!  % %e ,     -rctj|jstj|j|jrItj|j r*tj |j |j|jrItj|jr*tj |j|j|jj|j dtj|jd5}|j|jd}|jt!j"|dddy#1swYyxYw)aSaves the model. Args: epoch: the epoch this iteration is in. batch: the batch this iteration is in. `None` if the `save_freq` is set to `"epoch"`. logs: the `logs` dict passed in to `on_batch_end` or `on_epoch_end`. T)filepath overwritew)r r>N)rr#rmakedirsr rr$rrrr+ save_weightsr1rrwriter2dumps)rr6r7s rr:zBackupAndRestore._save_models  1    0  ! !j&7&78J8J&K OOD..0G0G H  ! !j&7&7  ( ('  OO,,d.O.O  ););tL __T993 ? 31,,..!  GGDJJ01 2  3 3 3s 0>E77Fc|jdk(ry||jkr|dz}n||jz }|xj|z c_||_|j|jk\rd|_yy)z?Handles batch-level saving logic, supports steps_per_execution.r Fr9rT)r rr)rr> add_batchess rr=z&BackupAndRestore._should_save_on_batchsq >>W $ D)) )!)K$"7"77K ,, ;, %  / /4>> A34D 0rc|jr@tj|jr tj|jyyyr )rrr#rrmtree)rr&s r on_train_endzBackupAndRestore.on_train_ends7  ! !j&7&7&H   doo .'I !r)r FTr ) __name__ __module__ __qualname____doc__r r(r!r;r?r:r=rL __classcell__)rs@rrrs>DR B".0 38 /rr)r2keras.src.api_exportrkeras.src.callbacks.callbackrkeras.src.utilsrrrrrVs5 -1&01I/xI/2I/r