BVh_YdZddlmZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z dd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZgdZeddZeddZeddZ d#dZed d$dZ edgdZ!dZ"dZ# d%dZ$d Z%d!Z&d"Z'y)&aTools to work with name-based checkpoints. While some of these symbols also work with the TF2 object-based checkpoints, they are not recommended for TF2. Please check `tensorflow/python/checkpoint` for newer utilities built to work with TF2 checkpoints. )abcN)checkpoint_management)distribute_lib)ops)io_ops)resource_variable_ops)variable_scope) variables)gfile) tf_logging)py_checkpoint_reader)saveable_object_util) tf_export)load_checkpoint load_variablelist_variablescheckpoints_iteratorinit_from_checkpointztrain.load_checkpointcbt|}|td|ztj|S)aReturns `CheckpointReader` for checkpoint found in `ckpt_dir_or_file`. If `ckpt_dir_or_file` resolves to a directory with multiple checkpoints, reader for the latest checkpoint is returned. Example usage: ```python import tensorflow as tf a = tf.Variable(1.0) b = tf.Variable(2.0) ckpt = tf.train.Checkpoint(var_list={'a': a, 'b': b}) ckpt_path = ckpt.save('tmp-ckpt') reader= tf.train.load_checkpoint(ckpt_path) print(reader.get_tensor('var_list/a/.ATTRIBUTES/VARIABLE_VALUE')) # 1.0 ``` Args: ckpt_dir_or_file: Directory with checkpoints file or path to checkpoint file. Returns: `CheckpointReader` object. Raises: ValueError: If `ckpt_dir_or_file` resolves to a directory with no checkpoints. zDCouldn't find 'checkpoint' file or checkpoints in given directory %s)_get_checkpoint_filename ValueErrorr NewCheckpointReader)ckpt_dir_or_filefilenames [/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/checkpoint_utils.pyrr.s@<&&6 7(  *,<= >>  1 1( ;;ztrain.load_variablecf|jdr|dd}t|}|j|S)aReturns the tensor value of the given variable in the checkpoint. When the variable name is unknown, you can use `tf.train.list_variables` to inspect all the variable names. Example usage: ```python import tensorflow as tf a = tf.Variable(1.0) b = tf.Variable(2.0) ckpt = tf.train.Checkpoint(var_list={'a': a, 'b': b}) ckpt_path = ckpt.save('tmp-ckpt') var= tf.train.load_variable( ckpt_path, 'var_list/a/.ATTRIBUTES/VARIABLE_VALUE') print(var) # 1.0 ``` Args: ckpt_dir_or_file: Directory with checkpoints file or path to checkpoint. name: Name of the variable to return. Returns: A numpy `ndarray` with a copy of the value of this variable. z:0N)endswithr get_tensor)rnamereaders rrrSs78 ]]4 9D + ,&   4  rztrain.list_variablesct|}|j}t|j}g}|D]}|j |||f|S)adLists the checkpoint keys and shapes of variables in a checkpoint. Checkpoint keys are paths in a checkpoint graph. Example usage: ```python import tensorflow as tf import os ckpt_directory = "/tmp/training_checkpoints/ckpt" ckpt = tf.train.Checkpoint(optimizer=optimizer, model=model) manager = tf.train.CheckpointManager(ckpt, ckpt_directory, max_to_keep=3) train_and_checkpoint(model, manager) tf.train.list_variables(manager.latest_checkpoint) ``` Args: ckpt_dir_or_file: Directory with checkpoints file or path to checkpoint. Returns: List of tuples `(key, shape)`. )rget_variable_to_shape_mapsortedkeysappend)rr" variable_mapnamesresultr!s rrrus_0 + ,&113, ""$ %% &.d MM4d+,-. -rc6tjd||tj|znd} tj|}|||k(r3|tj|z|kDrytj |ntjd||Sh)aWaits until a new checkpoint file is found. Args: checkpoint_dir: The directory in which checkpoints are saved. last_checkpoint: The last checkpoint path used or `None` if we're expecting a checkpoint for the first time. seconds_to_sleep: The number of seconds to sleep for before looking for a new checkpoint. timeout: The maximum number of seconds to wait. If left as `None`, then the process will wait indefinitely. Returns: a new checkpoint path, or None if the timeout was reached. z Waiting for new checkpoint at %sNzFound new checkpoint at %s)logginginfotimerlatest_checkpointsleep)checkpoint_dirlast_checkpointseconds_to_sleeptimeout stop_timecheckpoint_paths rwait_for_new_checkpointr7s$ ,,1>B'.':diikG#) +==nMO/_"D  499;1A#AI#M jj!" ll/A  rztrain.checkpoints_iteratorc#:Kd} t|||}|6|stjdy|rtjdyGtj}|}|||ztjz }|dkDrtj|w)aeContinuously yield new checkpoint files as they appear. The iterator only checks for new checkpoints when control flow has been reverted to it. This means it can miss checkpoints if your code takes longer to run between iterations than `min_interval_secs` or the interval at which new checkpoints are written. The `timeout` argument is the maximum number of seconds to block waiting for a new checkpoint. It is used in combination with the `timeout_fn` as follows: * If the timeout expires and no `timeout_fn` was specified, the iterator stops yielding. * If a `timeout_fn` was specified, that function is called and if it returns a true boolean value the iterator stops yielding. * If the function returns a false boolean value then the iterator resumes the wait for new checkpoints. At this point the timeout logic applies again. This behavior gives control to callers on what to do if checkpoints do not come fast enough or stop being generated. For example, if callers have a way to detect that the training has stopped and know that no new checkpoints will be generated, they can provide a `timeout_fn` that returns `True` when the training has stopped. If they know that the training is still going on they return `False` instead. Args: checkpoint_dir: The directory in which checkpoints are saved. min_interval_secs: The minimum number of seconds between yielding checkpoints. timeout: The maximum number of seconds to wait between checkpoints. If left as `None`, then the process will wait indefinitely. timeout_fn: Optional function to call after a timeout. If the function returns True, then it means that no new checkpoints will be generated and the iterator will exit. The function is called with no arguments. Yields: String paths to latest checkpoint files as they arrive. N)r4z8Timed-out waiting for a checkpoint (without timeout_fn).z#Timed-out waiting for a checkpoint.r)r7r,r-r.r0)r1min_interval_secsr4 timeout_fnr6new_checkpoint_pathstarttime_to_next_evals rrrsV/ 1:"  OP  :;  IIKE)O  11DIIK?1 jj"#) sBBztrain.init_from_checkpoint)v1cfd}tjr |dytjj|y)aReplaces `tf.Variable` initializers so they load from a checkpoint file. @compatibility(TF2) `tf.compat.v1.train.init_from_checkpoint` is not recommended for restoring variable values in TF2. To restore checkpoints in TF2, please use `tf.keras.Model.load_weights` or `tf.train.Checkpoint.restore`. These APIs use use an [object-based method of checkpointing] (https://www.tensorflow.org/guide/checkpoint#loading_mechanics), while `tf.compat.v1.init_from_checkpoint` relies on a more-fragile variable-name based method of checkpointing. There is no object-based equivalent of `init_from_checkpoint` in TF2. Please re-write your checkpoints immediately using the object-based APIs, see [migration guide] (https://www.tensorflow.org/guide/migrate#checkpoint_compatibility) for more details. You can load a name-based checkpoint written by `tf.compat.v1.train.Saver` using `tf.train.Checkpoint.restore` or `tf.keras.Model.load_weights`. However, you may have to change the names of the variables in your model to match the variable names in the name-based checkpoint, which can be viewed with `tf.train.list_variables(path)`. Another option is to create an `assignment_map` that maps the name of the variables in the name-based checkpoint to the variables in your model, eg: ``` { 'sequential/dense/bias': model.variables[0], 'sequential/dense/kernel': model.variables[1] } ``` and use `tf.compat.v1.train.init_from_checkpoint(path, assignment_map)` to restore the name-based checkpoint. After restoring, re-encode your checkpoint using `tf.train.Checkpoint.save` or `tf.keras.Model.save_weights`. @end_compatibility Values are not loaded immediately, but when the initializer is run (typically by running a `tf.compat.v1.global_variables_initializer` op). Note: This overrides default initialization ops of specified variables and redefines dtype. Assignment map supports following syntax: * `'checkpoint_scope_name/': 'scope_name/'` - will load all variables in current `scope_name` from `checkpoint_scope_name` with matching tensor names. * `'checkpoint_scope_name/some_other_variable': 'scope_name/variable_name'` - will initialize `scope_name/variable_name` variable from `checkpoint_scope_name/some_other_variable`. * `'scope_variable_name': variable` - will initialize given `tf.Variable` object with tensor 'scope_variable_name' from the checkpoint. * `'scope_variable_name': list(variable)` - will initialize list of partitioned variables with tensor 'scope_variable_name' from the checkpoint. * `'/': 'scope_name/'` - will load all variables in current `scope_name` from checkpoint's root (e.g. no scope). Supports loading into partitioned variables, which are represented as `'/part_'`. Assignment map can be a dict, or a list of pairs. The latter is necessary to initialize multiple variables in the current graph from the same variable in the checkpoint. Example: ```python # Say, '/tmp/model.ckpt' has the following tensors: # -- name='old_scope_1/var1', shape=[20, 2] # -- name='old_scope_1/var2', shape=[50, 4] # -- name='old_scope_2/var3', shape=[100, 100] # Create new model's variables with tf.compat.v1.variable_scope('new_scope_1'): var1 = tf.compat.v1.get_variable('var1', shape=[20, 2], initializer=tf.compat.v1.zeros_initializer()) with tf.compat.v1.variable_scope('new_scope_2'): var2 = tf.compat.v1.get_variable('var2', shape=[50, 4], initializer=tf.compat.v1.zeros_initializer()) # Partition into 5 variables along the first axis. var3 = tf.compat.v1.get_variable(name='var3', shape=[100, 100], initializer=tf.compat.v1.zeros_initializer(), partitioner=lambda shape, dtype: [5, 1]) # Initialize all variables in `new_scope_1` from `old_scope_1`. init_from_checkpoint('/tmp/model.ckpt', {'old_scope_1/': 'new_scope_1/'}) # Use names to specify which variables to initialize from checkpoint. init_from_checkpoint('/tmp/model.ckpt', {'old_scope_1/var1': 'new_scope_1/var1', 'old_scope_1/var2': 'new_scope_2/var2'}) # Or use tf.Variable objects to identify what to initialize. init_from_checkpoint('/tmp/model.ckpt', {'old_scope_1/var1': var1, 'old_scope_1/var2': var2}) # Initialize partitioned variables using variable's name init_from_checkpoint('/tmp/model.ckpt', {'old_scope_2/var3': 'new_scope_2/var3'}) # Or specify the list of tf.Variable objects. init_from_checkpoint('/tmp/model.ckpt', {'old_scope_2/var3': var3._get_variable_list()}) ``` Args: ckpt_dir_or_file: Directory with checkpoints file or path to checkpoint. assignment_map: Dict, or a list of key-value pairs, where keys are names of the variables in the checkpoint and values are current variables or names of current variables (in default graph). Raises: ValueError: If missing variables in current graph, or if missing checkpoints or tensors in checkpoints. ctSN)_init_from_checkpoint)_assignment_maprs rz&init_from_checkpoint..vs&;'(rN)rget_cross_replica_contextget_replica_context merge_call)rrDinit_from_checkpoint_fns`` rrrs:|(--/D!&&(33!rct|}t|}|j}t|tj r|j }d}t||D]\}}d}t|s"t|trtd|Dr|}n>tjj} | j|d}| t|| }|||vrt!d|d|d|t|rt|j#j%||sEt!d|j&d t)|j#d |d t)||d |j&} nd j+d |D} t-|||t/j0d| ||Od} d|vr|d|j3d} |j5dst!dj7| |t9}  D]B} | r| j;| dzsd| vr| d| j=d} | j?| Dt| D]} | tA| d} |dk7r| dd} |dk7r|| z} | j5dr| dd} | |vr't!d| d | tA| dzdd|d|d | j| d}| t| | }t-||| t/j0d| || y)z-See `init_from_checkpoint` for documentation.c |dS)Nr)pairs rrEz'_init_from_checkpoint..s $q'r)keyNc32K|]}t|ywrA) _is_variable.0vs r z(_init_from_checkpoint..s=A Q=szTensor z is not found in z checkpoint zShape of variable z (z%) doesn't match with shape of tensor z) from checkpoint reader.,c34K|]}|jywrAr!rQs rrTz(_init_from_checkpoint..s0qAFF0sz1Initialize variable %s from checkpoint %s with %s/zgAssignment map with scope only name {} should map to scope only {}. Should be 'scope/': 'other_scope/'.z/part_z in z) is not found in z checkpoint)!rrr$ isinstancerMappingitemsr%rPlistallvs_get_default_variable_store_varsget_collect_partitioned_variabler get_shapeis_compatible_withr!strjoin!_set_variable_or_list_initializerr,debugrindexrformatset startswithindexaddlen)rrD ckpt_filer"r(sort_keytensor_name_in_ckptcurrent_var_or_namevar store_varsvar_namescopesscope_variablesfull_tensor_names rrBrBs=&'78) + ,&113, ,#))+N"(28(3$OD.. C'(&- =)<= = c11399j NN. 5c +,?L  L 0 !1<    c }}11 , -/((C 0%s<8K+L'M  88880C00'Y8KL mmG.0CEf # #$%E&9&@&@&EF ) )# . 66%'7  nnXt, ;-h C#)#y:JK I 02B D+DsODrct|tjrtj|}t j |rt j|S|S)zHReturns checkpoint filename given directory or specific checkpoint file.)r\osPathLikefspathr IsDirectoryrr/)rs rrrsH "++.yy!12 '( 2 23C DD rc |jj}tj|j5tjd5t j ||g|g|g|d}t j|g}g}|jD]1\}} t j| |D]} |j| 3t|dk(sJ dddddddjgd} | |_ |j|j||_y#1swYRxYw#1swYVxYw)aOverrides given variable's initialization op. Sets variable initializer to assign op that initializes variable from tensor's value in the checkpoint. Args: variable: `tf.Variable` object. ckpt_file: string, full path of the checkpoint. tensor_name: Name of the tensor to load from the checkpoint. slice_spec: Slice specification for loading partitioned tensors. name: Name of the operation. z/cpu:0rWrrZN)restored_shapes)dtype base_dtyperdevicer restore_v2rop_list_to_dictr^saveable_objects_for_opr'rrrestore_initializer_op set_shapeshape_initial_value) variablers tensor_name slice_specr! base_type restore_opnames_to_saveablessaveable_objectsopsinit_ops r_set_checkpoint_initializerrs>"nn'')  zz(//" &CJJx$8 &""K=:, $HHIKJ.==xjI&,,.#b#;;BE#!"##  A %% % & & Q  ' ' d ' K'%( x~~&&(! & & & &s$D7 B D+D7+D4 0D77Ect|ttfrgd}|D]_}|j}| |j}n*||jk7rt d|d|jt ||||jayt |||dy)aOverrides initialization op of given variable or list of variables. Calls `_set_checkpoint_initializer` for each variable in the given list of variables. Args: variable_or_list: `tf.Variable` object or a list of `tf.Variable` objects. ckpt_file: string, full path of the checkpoint. tensor_name: Name of the tensor to load from the checkpoint. Raises: ValueError: if all objects in `variable_or_list` are not partitions of the same large variable. Nz)Slices must all be from the same tensor: z != rX)r\r_tuple_save_slice_info full_namerrspec)variable_or_listrsr slice_namerS slice_infos rrjrjs  4-0J N%%j  )) -- -$j&:&:<= =!!Y Z__MN 0)["Mrcdt|tjxstj|SrA)r\r Variableris_resource_variable)xs rrPrP.s* Q ** + 8  4 4Q 79rc|dz|vr:g}d}|d|zz|vr*|j||d|zz|dz }|d|zz|vr*|Sy)zEReturns list of `tf.Variable` that comprise the partitioned variable.z/part_0rz/part_%drZN)r')r!all_varsrwis rrere3sj I! C A a 8 + jj$a/011fa a 8 + J r)NrZN)rNN)checkpoint_initializer)(__doc__ collectionsrr~r.tensorflow.python.checkpointrtensorflow.python.distributertensorflow.python.frameworkrtensorflow.python.opsrrr rar tensorflow.python.platformr r r,tensorflow.python.trainingr !tensorflow.python.training.savingr tensorflow.python.util.tf_exportr__all__rrrr7rrrBrrrjrPrerLrrrs  >7+(76+,<;B6   "#!<$! '(+,!%$(?$)?$D +,-C!.C!LZDz&> &'TN>9 r