BVh*dZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddl m!Z!dZ"dZ#e!dg d1dZ$ejJdde!dg d2dZ&e!dg d3dZ'e!dd4dZ(dZ)e!dd4d Z*d!Z+ejJdd"e!d#gd$Z,ejJdd%e!d&gd'Z-ejJdd(e!d)ge j\j^d*fd+Z0d,Z1d5d-Z2e!d.Gd/d0e3Z4y)6z@Checkpoint Manager and other utilities for managing checkpoints.N) text_format) saver_pb2)checkpoint_options)context)errors)ops)file_io)variable_scope) tf_logging) training_util)CheckpointState)compat) deprecation) tf_exportctjr|jStjj |S)z$Returns the numpy value of a tensor.)rexecuting_eagerlynumpyrget_default_sessionrun)tensors b/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/checkpoint/checkpoint_management.py _evaluater)s4  <<> " & &v ..cJ|d}tjj||S)a4Returns a filename for storing the CheckpointState. Args: save_dir: The directory for saving and restoring checkpoints. latest_filename: Name of the file in 'save_dir' that is used to store the CheckpointState. Returns: The path of the file that contains the CheckpointState proto. checkpoint)ospathjoin)save_dirlatest_filenames r_GetCheckpointFilenamer!0s$"O h 00rz%train.generate_checkpoint_state_proto)v1cL|g}|r|d|k7r'tjd||j||r)t|t|k7rt d|d|dt j j|st j j|s t j j||}t|D]H\}}t j j|r&t j j||||<Jt||||}|S)aGenerates a checkpoint state proto. Args: save_dir: Directory where the model was saved. model_checkpoint_path: The checkpoint file. all_model_checkpoint_paths: List of strings. Paths to all not-yet-deleted checkpoints, sorted from oldest to newest. If this is a non-empty list, the last element must be equal to model_checkpoint_path. These paths are also saved in the CheckpointState proto. all_model_checkpoint_timestamps: A list of floats, indicating the number of seconds since the Epoch when each checkpoint was generated. last_preserved_timestamp: A float, indicating the number of seconds since the Epoch when the last preserved checkpoint was written, e.g. due to a `keep_checkpoint_every_n_hours` parameter (see `tf.train.CheckpointManager` for an implementation). Returns: CheckpointState proto with model_checkpoint_path and all_model_checkpoint_paths updated to either absolute paths or relative paths to the current save_dir. Raises: ValueError: If `all_model_checkpoint_timestamps` was provided but its length does not match `all_model_checkpoint_paths`. z<%s is not in all_model_checkpoint_paths. Manually adding it.zKCheckpoint timestamps, if provided, must match checkpoint paths (got paths z and timestamps ))model_checkpoint_pathall_model_checkpoint_pathsall_model_checkpoint_timestampslast_preserved_timestamp) logginginfoappendlen ValueErrorrrisabsrelpath enumerater )rr&r'r(r)ipcoord_checkpoint_protos rgenerate_checkpoint_state_protor5@s< '!# $ $(== LLO&(%%&;<% . /,- .  &'F H II x 77==. / ggoo.CXN45E1 WW]]1 (*8(D"1%E+1!;&E7 9 rziUse `tf.train.CheckpointManager` to manage checkpoints rather than manually editing the Checkpoint proto.)date instructionsztrain.update_checkpoint_statec (t||||d||y)aUpdates the content of the 'checkpoint' file. This updates the checkpoint file containing a CheckpointState proto. Args: save_dir: Directory where the model was saved. model_checkpoint_path: The checkpoint file. all_model_checkpoint_paths: List of strings. Paths to all not-yet-deleted checkpoints, sorted from oldest to newest. If this is a non-empty list, the last element must be equal to model_checkpoint_path. These paths are also saved in the CheckpointState proto. latest_filename: Optional name of the checkpoint file. Default to 'checkpoint'. all_model_checkpoint_timestamps: Optional list of timestamps (floats, seconds since the Epoch) indicating when the checkpoints in `all_model_checkpoint_paths` were created. last_preserved_timestamp: A float, indicating the number of seconds since the Epoch when the last preserved checkpoint was written, e.g. due to a `keep_checkpoint_every_n_hours` parameter (see `tf.train.CheckpointManager` for an implementation). Raises: RuntimeError: If any of the model checkpoint paths conflict with the file containing CheckpointSate. F)rr&r'r save_relative_pathsr(r)N) update_checkpoint_state_internal)rr&r'r r(r)s rupdate_checkpoint_stater;s"H#1!;%&E79rz*__internal__.train.update_checkpoint_statecDt||}|rtjj|r!tjj ||}n|}g} |D]b} tjj| r0| j tjj | |R| j | dt ||| ||} nt |||||} || jk(rtd|ztj|tj| y)aUpdates the content of the 'checkpoint' file. This updates the checkpoint file containing a CheckpointState proto. Args: save_dir: Directory where the model was saved. model_checkpoint_path: The checkpoint file. all_model_checkpoint_paths: List of strings. Paths to all not-yet-deleted checkpoints, sorted from oldest to newest. If this is a non-empty list, the last element must be equal to model_checkpoint_path. These paths are also saved in the CheckpointState proto. latest_filename: Optional name of the checkpoint file. Default to 'checkpoint'. save_relative_paths: If `True`, will write relative paths to the checkpoint state file. all_model_checkpoint_timestamps: Optional list of timestamps (floats, seconds since the Epoch) indicating when the checkpoints in `all_model_checkpoint_paths` were created. last_preserved_timestamp: A float, indicating the number of seconds since the Epoch when the last preserved checkpoint was written, e.g. due to a `keep_checkpoint_every_n_hours` parameter (see `tf.train.CheckpointManager` for an implementation). Raises: RuntimeError: If any of the model checkpoint paths conflict with the file containing CheckpointSate. )r'r(r)z`Save path '%s' conflicts with path used for checkpoint state. Please use a different save path.N) r!rrr/r0r,r5r& RuntimeErrorr atomic_write_string_to_filerMessageToString) rr&r'r r9r(r)coord_checkpoint_filenamerel_model_checkpoint_pathrel_all_model_checkpoint_pathsr3ckpts rr:r:sJ5XO ww}}*+"$''// #+#8%'" '1 q &--bggooa.JK&--a0 1 +!#A(G!9  ;D +#=(G!9  ;D$"<"<< N,- ..  %%&?&1&A&A$&GIrztrain.get_checkpoint_statect|tjrtj|}d}t ||}d} t j |rt j|}t}tj|||jstd|ztjj|js/tjj||j|_ t!|j"D]R\}}tjj|r&tjj|||j"|<T|r|j1|S#t$j&$r]}t)j*dt-|j.|t)j*d|Yd}~|r|j1yyd}~wtj2$r]}t)j*dt-|j.|t)j*d|Yd}~|r|j1yyd}~wwxYw#|r|j1wwxYw)aReturns CheckpointState proto from the "checkpoint" file. If the "checkpoint" file contains a valid CheckpointState proto, returns it. Args: checkpoint_dir: The directory of checkpoints. latest_filename: Optional name of the checkpoint file. Default to 'checkpoint'. Returns: A CheckpointState if the state was available, None otherwise. Raises: ValueError: if the checkpoint read doesn't have model_checkpoint_path set. Nz%Invalid checkpoint state loaded from z%s: %sz%s: Checkpoint ignored) isinstancerPathLikefspathr!r file_existsread_file_to_stringr rMerger&r.rr/rr1r'rOpErrorr*warningtype__name__close ParseError) checkpoint_dirr rCr@f file_contentr2r3es rget_checkpoint_staterUs& ,YY~.N $4^5DF !4500 #%l  d d+  ' '@)*+ +WW]]455 6%'WW\\.262L2L&N"D;;<O$!Qww}}Q/1ww||NA/N$ ) )! ,O ggi +  OOHd1g..2 OO,.GH   ggi    OOHd1g..2 OO,.GH ggi   ggi sEC7E<9/E<<IAG'I'I=AI=IIII2cJ|tjjk(r|dzS|S)aReturns the pathname of a checkpoint file, given the checkpoint prefix. For V1 checkpoint, simply returns the prefix itself (the data file). For V2, returns the pathname to the index file. Args: prefix: a string, the prefix of a checkpoint. format_version: the checkpoint format version that corresponds to the prefix. Returns: The pathname of a checkpoint file, taking into account the checkpoint format version. .index)rSaverDefV2)prefixformat_versions r_prefix_to_checkpoint_pathr\5s(y)),,, H  -rztrain.latest_checkpointct||}|r|jrt|jtjj }t|jtjj }tj|stj|r |jStjd|jy)aFinds the filename of latest saved checkpoint file. Gets the checkpoint state given the provided checkpoint_dir and looks for a corresponding TensorFlow 2 (preferred) or TensorFlow 1.x checkpoint path. The latest_filename argument is only applicable if you are saving checkpoint using `v1.train.Saver.save` See the [Training Checkpoints Guide](https://www.tensorflow.org/guide/checkpoint) for more details and examples.` Args: checkpoint_dir: Directory where the variables were saved. latest_filename: Optional name for the protocol buffer file that contains the list of most recent checkpoint filenames. See the corresponding argument to `v1.train.Saver.save`. Returns: The full path to the latest checkpoint or `None` if no checkpoint was found. z&Couldn't match files for checkpoint %sN) rUr&r\rrXrYV1r get_matching_filesr*error)rQr rCv2_pathv1_paths rlatest_checkpointrcHs0 no >$ d((()C)C)2););)>)>@G()C)C)2););)>)>@G!!'*g.H.H/  ' '' mm<..0 rct|tjj}t j |ryt j |ryy)aChecks whether a V1 or V2 checkpoint exists with the specified prefix. This is an internal function to check if a checkpoint exists, since it takes into account the naming difference between V1 and V2 formats. Args: checkpoint_prefix: the prefix of a V1 or V2 checkpoint, with V2 taking priority. Typically the result of `Saver.save()` or that of `tf.train.latest_checkpoint()`, regardless of sharded/non-sharded or V1/V2. Returns: A bool, true if a checkpoint referred to by `checkpoint_prefix` exists. TF)r\rrXrYr r_)checkpoint_prefixpathnames rcheckpoint_exists_internalrgpsF((9(1(:(:(=(=?( ) !!"34  rz;Use standard file APIs to check for files with this prefix.ztrain.checkpoint_existsct|S)aChecks whether a V1 or V2 checkpoint exists with the specified prefix. This is the recommended way to check if a checkpoint exists, since it takes into account the naming difference between V1 and V2 formats. Args: checkpoint_prefix: the prefix of a V1 or V2 checkpoint, with V2 taking priority. Typically the result of `Saver.save()` or that of `tf.train.latest_checkpoint()`, regardless of sharded/non-sharded or V1/V2. Returns: A bool, true if a checkpoint referred to by `checkpoint_prefix` exists. )rg)res rcheckpoint_existsris& $$5 66rz*Use standard file utilities to get mtimes.ztrain.get_checkpoint_mtimescgfd}|D]7}t|tjj}||r0||9S)a^Returns the mtimes (modification timestamps) of the checkpoints. Globs for the checkpoints pointed to by `checkpoint_prefixes`. If the files exist, collect their mtime. Both V2 and V1 checkpoints are considered, in that priority. This is the recommended way to get the mtimes, since it takes into account the naming difference between V1 and V2 formats. Note: If not all checkpoints exist, the length of the returned mtimes list will be smaller than the length of `checkpoint_prefixes` list, so mapping checkpoints to corresponding mtimes will not be possible. Args: checkpoint_prefixes: a list of checkpoint paths, typically the results of `Saver.save()` or those of `tf.train.latest_checkpoint()`, regardless of sharded/non-sharded or V1/V2. Returns: A list of mtimes (in microseconds) of the found checkpoints. ctj|}|r5jtj|djdz yy)NrgeATF)r r_r,stat mtime_nsec)rffnamesmtimess rmatch_maybe_appendz1get_checkpoint_mtimes..match_maybe_appends@  ' ' 1F  mmGLL+66<=  r)r\rrXrY)checkpoint_prefixesrprerfros @rget_checkpoint_mtimesrrsY2 &/*)*;*3*<*<*?*?AH(#()* -rz8Use standard file APIs to delete files with this prefix.ztrain.remove_checkpointmetactt|||tjjk(rt|dzt|dzyt|y)aRemoves a checkpoint given by `checkpoint_prefix`. Args: checkpoint_prefix: The prefix of a V1 or V2 checkpoint. Typically the result of `Saver.save()` or that of `tf.train.latest_checkpoint()`, regardless of sharded/non-sharded or V1/V2. checkpoint_format_version: `SaverDef.CheckpointFormatVersion`, defaults to `SaverDef.V2`. meta_graph_suffix: Suffix for `MetaGraphDef` file. Defaults to 'meta'. rW.data-?????-of-?????N)_delete_file_if_existsmeta_graph_filenamerrXrY)recheckpoint_format_versionmeta_graph_suffixs rremove_checkpointrzsU"+->?A)"4"4"7"77,x78,/EEF,-rctj|D]} tj|y#tj$rt j d|YGwxYw)z"Deletes files matching `filespec`.zsHit NotFoundError when deleting '%s', possibly because another process/thread is also deleting/moving the same fileN)r r_ delete_filer NotFoundErrorr*rL)filespecrfs rrvrvsa,,X6LhL (#L   L oo ABJLLs2)AAcZtjdd|}dj||g}|S)zReturns the meta graph filename. Args: checkpoint_filename: Name of the checkpoint file. meta_graph_suffix: Suffix for `MetaGraphDef` file. Defaults to 'meta'. Returns: MetaGraph file name. z-[\d\?]+-of-\d+$.)resubr)checkpoint_filenamerybasenamesuffixed_filenames rrwrws3VV'-@ A(hh*;<= rztrain.CheckpointManagerceZdZdZ ddZedZedZedZedZ dZ d Z ed Z ed Z dd Zd ZdZy)CheckpointManageraNManages multiple checkpoints by keeping some and deleting unneeded ones. Example usage: ```python import tensorflow as tf checkpoint = tf.train.Checkpoint(optimizer=optimizer, model=model) manager = tf.train.CheckpointManager( checkpoint, directory="/tmp/model", max_to_keep=5) status = checkpoint.restore(manager.latest_checkpoint) while True: # train manager.save() ``` `CheckpointManager` preserves its own state across instantiations (see the `__init__` documentation for details). Only one should be active in a particular directory at a time. Nc ||_d|_||dkrtd|fz||_||_t |t jrt j|}||_ t jj|||_ ||_ || tdd|_||_||_t#|} t%j$} t'j(|_| d|_| dz |_y| j0|_| j2|_| |j.kr-t5j6d|j.| z fz| |_| j8} | j:} ~ | s|j.gt=| z} t?| | D]0\} }tA|| }||j.kDs"||j*| <2y)a Configure a `CheckpointManager` for use in `directory`. If a `CheckpointManager` was previously used in `directory`, its state will be restored. This includes the list of managed checkpoints and the timestamp bookkeeping necessary to support `keep_checkpoint_every_n_hours`. The behavior of the new `CheckpointManager` will be the same as the previous `CheckpointManager`, including cleaning up existing checkpoints if appropriate. Checkpoints are only considered for deletion just after a new checkpoint has been added. At that point, `max_to_keep` checkpoints will remain in an "active set". Once a checkpoint is preserved by `keep_checkpoint_every_n_hours` it will not be deleted by this `CheckpointManager` or any future `CheckpointManager` instantiated in `directory` (regardless of the new setting of `keep_checkpoint_every_n_hours`). The `max_to_keep` checkpoints in the active set may be deleted by this `CheckpointManager` or a future `CheckpointManager` instantiated in `directory` (subject to its `max_to_keep` and `keep_checkpoint_every_n_hours` settings). `CheckpointManager` can be also used for initializing the model if there is no checkpoints for restoring in `directory`. An example usage is: >>> import tempfile >>> tmp_dir = tempfile.mkdtemp() >>> checkpoint = tf.train.Checkpoint() >>> init_path = checkpoint.save(os.path.join(tmp_dir, 'init')) >>> def init_fn(): ... # Partially restore the checkpoint from `init_path`. ... checkpoint.restore(init_path) >>> manager = tf.train.CheckpointManager( ... checkpoint, ... directory=os.path.join(tmp_dir, 'ckpt'), ... max_to_keep=None, ... init_fn=init_fn) >>> # `restore_or_initialize` will call `init_fn` if there is no existing >>> # checkpoint in `directory`. >>> manager.restore_or_initialize() Args: checkpoint: The `tf.train.Checkpoint` instance to save and manage checkpoints for. directory: The path to a directory in which to write checkpoints. A special file named "checkpoint" is also written to this directory (in a human-readable text format) which contains the state of the `CheckpointManager`. max_to_keep: An integer, the number of checkpoints to keep. Unless preserved by `keep_checkpoint_every_n_hours`, checkpoints will be deleted from the active set, oldest first, until only `max_to_keep` checkpoints remain. If `None`, no checkpoints are deleted and everything stays in the active set. Note that `max_to_keep=None` will keep all checkpoint paths in memory and in the checkpoint state protocol buffer on disk. keep_checkpoint_every_n_hours: Upon removal from the active set, a checkpoint will be preserved if it has been at least `keep_checkpoint_every_n_hours` since the last preserved checkpoint. The default setting of `None` does not preserve any checkpoints in this way. checkpoint_name: Custom name for the checkpoint file. step_counter: A `tf.Variable` instance for checking the current step counter value, in case users want to save checkpoints every N steps. checkpoint_interval: An integer, indicates the minimum step interval between two checkpoints. init_fn: Callable. A function to do customized initialization if no checkpoints are in the directory. Raises: ValueError: If `max_to_keep` is not a positive integer. Nrz@Expected a positive integer or `None` for `max_to_keep`, got %d.zE`step_counter` should be passed if `checkpoint_interval` is not None.g?zWtime.time() returned a value %f seconds behind the last preserved checkpoint timestamp.)! _checkpoint_save_counter_assignr. _max_to_keep_keep_checkpoint_every_n_hoursrErrFrG _directoryrr_checkpoint_prefix_init_fn_last_checkpoint_step _step_counter_checkpoint_intervalrUtime collections OrderedDict _maybe_delete_latest_checkpoint_last_preserved_timestampr&r)r*rLr(r'r-zipmin)selfr directory max_to_keepkeep_checkpoint_every_n_hourscheckpoint_name step_countercheckpoint_intervalinit_fnrecovered_state current_clockall_timestamps all_pathsfilename timestamps r__init__zCheckpointManager.__init__s`"D $D;!#3   N  $D*GD')R[[)))I&iDO ggll9oFDDM&  >? ?#'d 'd 3D*95OIIKM$002D $d(5r'9d$ / E Ed'6'O'Od$ 77 7 /-- =? @ A*7&&FFn!<>FF001)2&X01X(>>? d  !D$5$5 5rct|jj\}}t|j|j |||j dy)z5Saves the `CheckpointManager`'s state in `directory`.T)r&r'r(r)r9N)rritemsr:rrcr)r filenames timestampss r _record_statezCheckpointManager._record_statesJ!3!3!9!9!;<Iz$ "44#,(2!%!?!? "rc|jS)aA common prefix for all checkpoints saved with this manager. For example, if `directory` (a constructor argument) were `"/tmp/tf-model"`, `prefix` would be `"/tmp/tf-model/ckpt"` and checkpoints would generally be numbered `"/tmp/tf-model/ckpt-1"`, `"/tmp/tf-model/ckpt-2"`, and so on. Each checkpoint has several associated files (e.g. `"/tmp/tf-model/ckpt-2.index"`). Returns: A string prefix. )rrs r_prefixzCheckpointManager._prefixs  " ""rc|jS)z)Returns the `tf.train.Checkpoint` object.)rrs rrzCheckpointManager.checkpoints   rc  jWtj}j/|jk(ry|r|jjzkry|_t j r*j j}|jdd ntj fd}tj|5j j}dddjjdd_ jj|}t|t j"st%j& |}dj(|fz}fd}|t+j,|g }nEt/j.|}|j0 |g|_n|j0j3|j j5|| } | S#1swYxYw) aCreates a new checkpoint and manages it. Args: checkpoint_number: An optional integer, or an integer-dtype `Variable` or `Tensor`, used to number the checkpoint. If `None` (default), checkpoints are numbered using `checkpoint.save_counter`. Even if `checkpoint_number` is provided, `save_counter` is still incremented. A user-provided `checkpoint_number` is not incremented even if it is a `Variable`. check_interval: An optional boolean. The argument is only effective when `checkpoint_interval` is passed into the manager. If `True`, the manager will only save the checkpoint if the interval between checkpoints is larger than `checkpoint_interval`. Otherwise it will always save the checkpoint unless a checkpoint has already been saved for the current step. options: Optional `tf.train.CheckpointOptions` object. This argument only works with TF2 checkpoint objects. For example, options = tf.saved_model.SaveOptions(experimental_io_device='/job:localhost') Returns: The path to the new checkpoint. It is also recorded in the `checkpoints` and `latest_checkpoint` properties. `None` if no checkpoint is saved. NcN|di|}j|j|S)z9Initialize the save counter if it has been newly created.)r initializer) next_creatorkwargsvsessions r_initializing_creatorz5CheckpointManager.save.._initializing_creator!s%  "6 " AMM"rF) read_value)sessglobal_step_tensorz%s-%dctj}|jvr j|=|j|<|_jj jyr)rrrrr) save_pathrrs r_record_and_sweep_statez7CheckpointManager.save.._record_and_sweep_state3si))+i d(( (   y )&/d# )d   kkm r)experimental_write_callbacks)options)rrrrrrr save_counter assign_addrrr variable_creator_scoperrrErintegral_typesr global_steprrCheckpointOptionscopyrr,_write) rcheckpoint_numbercheck_intervalr current_steprrrZrrrs ` @rsavezCheckpointManager.saves0   ,t112l # # / 455 5 l  & &)B)B BD#/d   "%%22la g'')g  0 01F G5''44 5 " " *$0$;$;A%$;$P! kk$++, & ')>)> ?'33+<>  &78 8F&"44(?'@g  '"g  - - 50G/H,,,334KL  '''@I [55s G99Hc,|jW|jj|j|jt |j |_|jS|j%|jtjdy)aRestore items in `checkpoint` from the latest checkpoint file. This method will first try to restore from the most recent checkpoint in `directory`. If no checkpoints exist in `directory`, and `init_fn` is specified, this method will call `init_fn` to do customized initialization. This can be used to support initialization from pretrained models. Note that unlike `tf.train.Checkpoint.restore()`, this method doesn't return a load status object that users can run assertions on (e.g. assert_consumed()). Thus to run assertions, users should directly use `tf.train.Checkpoint.restore()` method. Returns: The restored checkpoint path if the latest checkpoint is found and restored. Otherwise None. Nz?Customized initialization is done through the passed `init_fn`.) rrrestorerrrrrr*r+rs rrestore_or_initializez'CheckpointManager.restore_or_initializeVs{2 * t667 " " .%.t/A/A%B"  $ $$ }}  mmo ll KM rcR|jr|jjyy)z4Wait for any outstanding save or restore operations.N)rsyncrs rrzCheckpointManager.sync{s"  r)NrCNNN)NTN)rN __module__ __qualname____doc__rpropertyrrrcrrrrrrrrrrrrrs0.2% #'D3L   % %  #  #  +  +@& "  #  #  ^@#Jrr)NNN)NNNN)NNFNNr)rs)5rrros.pathrrrgoogle.protobufrtensorflow.core.protobufrtensorflow.python.checkpointrtensorflow.python.eagerrtensorflow.python.frameworkrrtensorflow.python.lib.ior tensorflow.python.opsr tensorflow.python.platformr r*tensorflow.python.trainingr /tensorflow.python.training.checkpoint_state_pb2r tensorflow.python.utilrr tensorflow.python.util.tf_exportrrr!r5 deprecatedr;r:rUr\rcrgrirrrXrYrzrvrwobjectrrrrrs0"G  '.;+.+,0<4K).6/ 1  678@DDH=A = 9= @ GI ./08<,0<@59 &91 I &9R 7B?AE599>EI>B GI@GIT '(7)7t& $%$&$N0 NP ()*7+P7$ =? ,-.'/?'T KM ()*090B0B0E0E(..+M.0L& $%ww&wr