BVh*dZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlm Z ddl!m"Z"ddl!m#Z$ddl%m&Z&ddl'm(Z)ddl*m+Z+ddl*m,Z,ddl-m.Z.ddl-m/Z/ddl0m1Z1ddl2m3Z3ejhZ4ejjZ5ejlZ6ejnZ7ejpZ8ejrZ9ejtZ:da;ejxZ=dZ>dZ?d Z@Gd!d"ZAGd#d$eAZBd%ZCe3d&g'Gd(d)ZDe3d*g' d4d+ZE d5d,ZFd-ZGe3d.g' d6d/ZHd0ZIejejje jeDjeDj1d2ZP d7d3ZQy)8zSave and restore variables. Symbols in this file are deprecated. See replacements in tensorflow/python/training/trackable and tensorflow/python/training/saving. N)meta_graph_pb2) saver_pb2)trackable_object_graph_pb2)checkpoint_management)session)context) constant_op)device)errors) meta_graph)ops) array_ops)control_flow_ops) gen_io_ops)io_ops) string_ops) variables)gfile) tf_logging)metrics)base)py_checkpoint_reader) training_util)saveable_object)saveable_object_util)compat) tf_exportsaver_v1c0||kryt||z dzS)Nri@B)round)start_time_secondsend_time_secondss P/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/saver.py_get_duration_microsecondsr$Ls%**  #55@ AAcd}tjdj|}|D]}|tj|z }|S)z2Calculates filesize of checkpoint based on prefix.rz{}*)globformatrCalculateFileSize)prefixsizefilesfiles r#_get_checkpoint_sizer.SsI $ ))ELL( )%,dG % %d ++D, +r%ceZdZdZej Zej ZejZ ejZ e jjfdZdZdZdZdZdZdZd Z dd Zd Zd Z ddZ ddZy )BaseSaverBuilderzEBase class for Savers. Can be extended to create different Ops. c||_yN)_write_version)self write_versions r#__init__zBaseSaverBuilder.__init__ks 'Dr%cg}g}g}|D]d}|jD]S}|j|j|j|j|j|jUf|j t jjk(rtj||||S|j t jjk(rtj||||Std|j z)aCreate an Op to save 'saveables'. This is intended to be overridden by subclasses that want to generate different Ops. Args: filename_tensor: String Tensor. saveables: A list of BaseSaverBuilder.SaveableObject objects. Returns: An Operation that save the variables. Raises: RuntimeError: (implementation detail) if "self._write_version" is an unexpected value. )filename tensor_namestensors tensor_sliceszUnexpected write_version: )specsappendnametensor slice_specr3rSaverDefV1r_saveV2save_v2 RuntimeError)r4filename_tensor saveablesr9r:r;saveablespecs r#save_opzBaseSaverBuilder.save_opns$LGM....$DII&t{{#T__-..  i00333 \\"#% ''    2 2 5 5 5^^O\=#%% 58K8KK LLr%c ~g}|D]p}|jr tj|j}nd}tj|5|j |j |||dddr|S#1swY~xYw)a<Restore all tensors contained in saveables. By default, this issues separate calls to `restore_op` for each saveable. Subclasses may override to load multiple saveables in a single call. Args: filename_tensor: String Tensor. saveables: List of BaseSaverBuilder.SaveableObject objects. preferred_shard: Int. Shard to open first when loading a sharded file. restore_sequentially: Unused. Bool. If true, each restore is sequential. Returns: A list of Tensors resulting from reading 'saveable' from 'filename'. N)r rset_cpu0r extend restore_op)r4rGrHpreferred_shardrestore_sequentially all_tensorsrIr s r# bulk_restorezBaseSaverBuilder.bulk_restores$ KI %..x? ::f I OOOX G III I IIs #A;;B c g}|jD]M}|jtj||jg|j g|j gdO|S)aCreate ops to restore 'saveable'. This is intended to be overridden by subclasses that want to generate different Ops. Args: filename_tensor: String Tensor. saveable: A BaseSaverBuilder.SaveableObject object. preferred_shard: Int. Shard to open first when loading a sharded file. Returns: A list of Tensors resulting from reading 'saveable' from 'filename'. r)r<r=r restore_v2r>r@dtype)r4rGrIrPr:rJs r#rOzBaseSaverBuilder.restore_opsa G. nn   Odii[4??:K!ZZL **+ -.. Nr%c0tj|||S)zAppend sharding information to a filename. Args: filename_tensor: A string tensor. shard: Integer. The shard for the filename. num_shards: An int Tensor for the number of shards. Returns: A string tensor. )rsharded_filename)r4rGshard num_shardss r#rXz!BaseSaverBuilder.sharded_filenames  & &z JJr%cT|j||}tj|g|S)zAdd ops to save variables that are on the same shard. Args: filename_tensor: String Tensor. saveables: A list of SaveableObject objects. Returns: A tensor with the filename used to save. )rKrwith_dependencies)r4rGrHsaves r# _AddSaveOpszBaseSaverBuilder._AddSaveOpss) << 3D  - -tfo FFr%c tjd5tjt j |dt jdt jtjjd}t j||g}dddt|}g}g}t j|d}d} t|D]\} \} } | } tjtj| 5|j!| |} |j#| |j#|j%| | dddtj&|Dcgc]}|j(c}5tjtj| 5t+j,||d }tj&|g5tj.|cdddcdddcdddS#1swYvxYw#1swYPxYwcc}w#1swYnxYw dddn #1swYnxYwdddy#1swYyxYw) aAdd ops to save the params per shard, for the V2 format. Note that the sharded save procedure for the V2 format is different from V1: there is a special "merge" step that merges the small metadata produced from each device. Args: checkpoint_prefix: scalar String Tensor. Interpreted *NOT AS A FILENAME*, but as a prefix of a V2 checkpoint; per_device: A list of (device, BaseSaverBuilder.VarToSave) pairs, as returned by _GroupByDevices(). Returns: An op to save the variables, which, when evaluated, returns the prefix "" only and does not include the sharded spec suffix. CPUz^s3://.*z.partz _temp/partNrZr>T)delete_old_dirs)r r rwhererregex_full_matchr constantospathnormpath string_joinlen enumeraterrMrXr=r^control_dependenciesoprmerge_v2_checkpointsidentity)r4checkpoint_prefix per_device_SHARDED_SUFFIXtmp_checkpoint_prefixrZ sharded_savessharded_prefixesnum_shards_tensor last_devicerYr rHrXx merge_steps r#_AddShardedSaveOpsForV2z(BaseSaverBuilder._AddShardedSaveOpsForV2s0\ E 0!  % %&7 D   w '   rww// = >@o)44 o .0 0ZJM#,,ZlKK&/ &;L"" k ::*33F; <L001F1BD 01T--.> JK LLL ! !"?A144"? @ 7 ::*33K@ A7 44 /G  % %zl 3 7##$56 7 7 77 7 7+00LL #@ 7 7 7 777 7 7 7saBH AH-H*)I0/IH/4 I= IHH' /H8 4I< II II#c`|jtjjk(r|j ||St |}g}t j|d}t|D]Z\}\}}tj|5|j|||} |j|j| |ddd\tj|D cgc]} | jc} 5t!j"||cdddS#1swYxYwcc} w#1swYyxYw)a Add ops to save the params per shard. Args: filename_tensor: a scalar String Tensor. per_device: A list of (device, BaseSaverBuilder.SaveableObject) pairs, as returned by _GroupByDevices(). Returns: An op to save the variables. rZraN)r3rrArDrzrjr rerkr r rXr=r^rlrmrsharded_filespec) r4rGrqrZrtrvrYr rHrXrxs r#_AddShardedSaveOpsz#BaseSaverBuilder._AddShardedSaveOps8s& i00333  ) )/: FFZJM#,,ZlK&/ &;L"" ::f L00%1BDT--.> JKLLL ! !"?A144"? @M  ( (:K LMM LL #@MMs5DD3D$D $D-c|j||||}g}d} |D]} d} |reg} | jD]T} | j} | j}|j st j | }| j|V|| | t| jz}| t| jz } |j| j|| tj|d|iS)a&Add operations to restore saveables. Args: filename_tensor: Tensor for the path of the file to load. saveables: A list of SaveableObject objects. restore_sequentially: True if we want to restore variables sequentially within a shard. reshape: True if we want to reshape loaded tensors to the shape of the corresponding variable. preferred_shard: Shard to open first when loading a sharded file. name: Name for the returned op. Returns: An Operation that restores the variables. rNr>) rSr<r? get_shapeis_fully_definedrshaper=rjrestorergroup)r4rGrHrQreshaperPr>rR assign_opsidxrIshapesrJvrsaveable_tensorss r#_AddRestoreOpszBaseSaverBuilder._AddRestoreOpsRs,##OY$8:KJ C Df NN Dkk!++-%'')OOA&E --    %Ss8>>/B)BC S  c(()96BCD"  ! !: 9D 99r%c g}t|D]L\}\}}tj|5|j|j |||||ddddNt j |ddiS#1swYmxYw)aAdd Ops to restore variables from multiple devices. Args: filename_tensor: Tensor for the path of the file to load. per_device: A list of (device, SaveableObject) pairs, as returned by _GroupByDevices(). restore_sequentially: True if we want to restore variables sequentially within a shard. reshape: True if we want to reshape loaded tensors to the shape of the corresponding variable. Returns: An Operation that restores the variables. restore_shard)rPr>Nr> restore_all)rkr r r=rrr) r4rGrqrQrsharded_restoresrYr rHs r#_AddShardedRestoreOpsz&BaseSaverBuilder._AddShardedRestoreOpss &/ &; '"" ::f '   $ %$  & ''' '  ! !#3 H- HH''s 'A22A; c<tjd}|D]f}td|jD}t |dk7rt d|j z||jj|ht|jdS)a)Group Variable tensor slices per device. TODO(touts): Make sure that all the devices found are on different job/replica/task/cpu|gpu. It would be bad if 2 were on the same device. It can happen if the devices are unspecified. Args: saveables: A list of BaseSaverBuilder.SaveableObject objects. Returns: A list of tuples: (device_name, BaseSaverBuilder.SaveableObject) tuples. The list is sorted by ascending device_name. Raises: ValueError: If the tensors of a saveable are on different devices. cgSr2rr%r#z2BaseSaverBuilder._GroupByDevices..sr%c3ZK|]#}tj|j%ywr2)pydevcanonical_namer ).0rJs r# z3BaseSaverBuilder._GroupByDevices..s&H04%  t{{ +Hs)+z?All tensors of a saveable object must be on the same device: %sc |dS)Nrr)ts r#rz2BaseSaverBuilder._GroupByDevices..s AaDr%)key) collections defaultdictsetr<rj ValueErrorr>popr=sorteditems)r4rHrqrIcanonical_devices r#_GroupByDevicesz BaseSaverBuilder._GroupByDevicess"((4J:H8@HH  ! #24rQr8)_build_internal) r4rrrrrr>rQr8s r#buildzBaseSaverBuilder.builds5V   -&C 1  r%c tjs| r| s tdt|tst j |}t j|} |d}tj|d| D cgc]} | jc} 5}tj|xsddd} tj| dd } |r<|j| }| r|j| |}| r?|j| |||}n*| r|j!| | }| r|j#| | ||}dddtjrH| rj%nd }t'j( j%|d ||||j* Stj,}|j/}|D]O}|j1|D]9}t|t2j4s |j7|j8;Qt'j( j8j8j8||||j* Scc} w#1swY'xYw#t:$r|j=YwxYw) z5build() with option to only perform save and restore.z[save and restore operations need to be built together when eager execution is not enabled.Nrr]modelrr8)rr>Const)filename_tensor_namesave_tensor_namerestore_op_namerrrversion)rexecuting_eagerlyr isinstancedictrop_list_to_dictvalidate_and_slice_inputsr name_scopermrplaceholder_with_defaultrr}rr^rnumpyrrAr3get_default_graphget_all_collection_keysget_collectionrPartitionedVariableget_operation_by_namer>KeyError as_tensor)r4rrrrrr>rQr8 build_save build_restorerHrIrGrq save_tensorrOrgraphcheck_collection_listcollection_typeelements r#rz BaseSaverBuilder._build_internals  $ $ & 0= ? @@ ($ //?? $>>Ik f5>?? AJDH!::  gRj:o":: '3o )))4 //L+ 11/:2FQ* (()D+ **?I+?J*+JF  "0:**,   .446+!(E%%''##%e $;;=2 "/++O< "G !>!> ? "))',,7 " "  .33&++$//!(E%%''}@JJt "!  "s%7I BI+I I I<;I<)r)FF@NFr) FFrrNFrTT)__name__ __module__ __qualname____doc__rSaveSpecSaveableObjectrReferenceVariableSaveableVariableSaveableResourceVariableSaveablerrArDr6rKrSrOrXr^rzr}rrrrrrr%r#r0r0^s  % %("11.*CC1JJ#,#5#5#8#8(&MP>4 K GM7^M>&(' 0:dI:::*1!&3n$#"#4;+0&!%$(_'r%r0ceZdZdZdZy)BulkSaverBuilderz@SaverBuilder with support for bulk restoring multiple saveables.c:~g}|D]E}|jD]4}|j|j|j|jf6Gt |\}} } t jd5tj||| | cdddS#1swYyxYw)Nzcpu:0) r<r=r>r@rVzipr r rrU) r4rGrHrPrQ restore_specsrIrJnamesslicesdtypess r#rSzBulkSaverBuilder.bulk_restoreVs MG..G$dii$**EFGG /E66 G G   vv FGGGs /BBN)rrrrrSrr%r#rrSs H Gr%rc tjj}tj|}|r-t |dkDrt dj ||dStdd}|tj|||S)a&Returns the saver from SAVERS collection, or creates a default one. This method is used by other members of the training module, such as `Scaffold`, or `CheckpointSaverHook`. Returns: `Saver`. Raises: RuntimeError: If the SAVERS collection already has more than one items. rzgMore than one item in collection {}. Please indicate which one to use by passing it to the constructor.rT)r allow_empty) r GraphKeysSAVERSrrjrFr(Saveradd_to_collection)collection_keysaverssavers r#_get_saver_or_defaultrfs==''.   n -&  6{Q  O 6. ! ## !9 $ /% .%0 ,r%z train.Saver)v1ceZdZdZdddddddddddej j dddfdZdZdZ d Z d Z d Z d Z dd ZdZddZeddZedZdZdZdZ ddZ ddZdZeddZy)raSaves and restores variables. @compatibility(TF2) `tf.compat.v1.train.Saver` is not supported for saving and restoring checkpoints in TF2. Please switch to `tf.train.Checkpoint` or `tf.keras.Model.save_weights`, which perform a more robust [object-based saving](https://www.tensorflow.org/guide/checkpoint#loading_mechanics). ### How to Rewrite Checkpoints Please rewrite your checkpoints immediately using the object-based checkpoint APIs. You can load a name-based checkpoint written by `tf.compat.v1.train.Saver` using `tf.train.Checkpoint.restore` or `tf.keras.Model.load_weights`. However, you may have to change the names of the variables in your model to match the variable names in the name-based checkpoint, which can be viewed with `tf.train.list_variables(path)`. Another option is to create an `assignment_map` that maps the name of the variables in the name-based checkpoint to the variables in your model, eg: ``` { 'sequential/dense/bias': model.variables[0], 'sequential/dense/kernel': model.variables[1] } ``` and use `tf.compat.v1.train.init_from_checkpoint(path, assignment_map)` to restore the name-based checkpoint. After restoring, re-encode your checkpoint using `tf.train.Checkpoint.save` or `tf.keras.Model.save_weights`. See the [Checkpoint compatibility]( https://www.tensorflow.org/guide/migrate#checkpoint_compatibility) section of the migration guide for more details. ### Checkpoint Management in TF2 Use `tf.train.CheckpointManager` to manage checkpoints in TF2. `tf.train.CheckpointManager` offers equivalent `keep_checkpoint_every_n_hours` and `max_to_keep` parameters. To recover the latest checkpoint, ``` checkpoint = tf.train.Checkpoint(model) manager = tf.train.CheckpointManager(checkpoint) status = checkpoint.restore(manager.latest_checkpoint) ``` `tf.train.CheckpointManager` also writes a [`CheckpointState` proto] (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/training/checkpoint_state.proto) which contains the timestamp when each checkpoint was created. ### Writing `MetaGraphDef`s in TF2 To replace, `tf.compat.v1.train.Saver.save(write_meta_graph=True)`, use `tf.saved_model.save` to write the `MetaGraphDef` (which is contained in `saved_model.pb`). @end_compatibility See [Variables](https://tensorflow.org/guide/variables) for an overview of variables, saving and restoring. The `Saver` class adds ops to save and restore variables to and from *checkpoints*. It also provides convenience methods to run these ops. Checkpoints are binary files in a proprietary format which map variable names to tensor values. The best way to examine the contents of a checkpoint is to load it using a `Saver`. Savers can automatically number checkpoint filenames with a provided counter. This lets you keep multiple checkpoints at different steps while training a model. For example you can number the checkpoint filenames with the training step number. To avoid filling up disks, savers manage checkpoint files automatically. For example, they can keep only the N most recent files, or one checkpoint for every N hours of training. You number checkpoint filenames by passing a value to the optional `global_step` argument to `save()`: ```python saver.save(sess, 'my-model', global_step=0) ==> filename: 'my-model-0' ... saver.save(sess, 'my-model', global_step=1000) ==> filename: 'my-model-1000' ``` Additionally, optional arguments to the `Saver()` constructor let you control the proliferation of checkpoint files on disk: * `max_to_keep` indicates the maximum number of recent checkpoint files to keep. As new files are created, older files are deleted. If None or 0, no checkpoints are deleted from the filesystem but only the last one is kept in the `checkpoint` file. Defaults to 5 (that is, the 5 most recent checkpoint files are kept.) * `keep_checkpoint_every_n_hours`: In addition to keeping the most recent `max_to_keep` checkpoint files, you might want to keep one checkpoint file for every N hours of training. This can be useful if you want to later analyze how a model progressed during a long training session. For example, passing `keep_checkpoint_every_n_hours=2` ensures that you keep one checkpoint file for every 2 hours of training. The default value of 10,000 hours effectively disables the feature. Note that you still have to call the `save()` method to save the model. Passing these arguments to the constructor will not save variables automatically for you. A training program that saves regularly looks like: ```python ... # Create a saver. saver = tf.compat.v1.train.Saver(...variables...) # Launch the graph and train, saving the model every 1,000 steps. sess = tf.compat.v1.Session() for step in range(1000000): sess.run(..training_op..) if step % 1000 == 0: # Append the step number to the checkpoint name: saver.save(sess, 'my-model', global_step=step) ``` In addition to checkpoint files, savers keep a protocol buffer on disk with the list of recent checkpoints. This is used to manage numbered checkpoint files and by `latest_checkpoint()`, which makes it easy to discover the path to the most recent checkpoint. That protocol buffer is stored in a file named 'checkpoint' next to the checkpoint files. If you create several savers, you can specify a different filename for the protocol buffer file in the call to `save()`. NFrrct5ttjaddd| r |r tdt j r"t jd| td||_ ||_ ||_ ||_ ||_ ||_||_||_| |_d|_| |_d|_| |_| |_||_g|_g|_t j r*tj|jdzz|_n| s|j7|j r+|j9|j j:|_||_d|_y#1swY\xYw)azCreates a `Saver`. The constructor adds ops to save and restore variables. `var_list` specifies the variables that will be saved and restored. It can be passed as a `dict` or a list: * A `dict` of names to variables: The keys are the names that will be used to save or restore the variables in the checkpoint files. * A list of variables: The variables will be keyed with their op name in the checkpoint files. For example: ```python v1 = tf.Variable(..., name='v1') v2 = tf.Variable(..., name='v2') # Pass the variables as a dict: saver = tf.compat.v1.train.Saver({'v1': v1, 'v2': v2}) # Or pass them as a list. saver = tf.compat.v1.train.Saver([v1, v2]) # Passing a list is equivalent to passing a dict with the variable op names # as keys: saver = tf.compat.v1.train.Saver({v.op.name: v for v in [v1, v2]}) ``` Note: the newer `AutoTrackable` API is not supported by `Saver`. In this case, the `tf.train.Checkpoint` class should be used. The optional `reshape` argument, if `True`, allows restoring a variable from a save file where the variable had a different shape, but the same number of elements and type. This is useful if you have reshaped a variable and want to reload it from an older checkpoint. The optional `sharded` argument, if `True`, instructs the saver to shard checkpoints per device. Args: var_list: A list of `Variable`/`SaveableObject`, or a dictionary mapping names to `SaveableObject`s. If `None`, defaults to the list of all saveable objects. reshape: If `True`, allows restoring parameters from a checkpoint where the variables have a different shape. sharded: If `True`, shard the checkpoints, one per device. max_to_keep: Maximum number of recent checkpoints to keep. Defaults to 5. keep_checkpoint_every_n_hours: How often to keep checkpoints. Defaults to 10,000 hours. name: String. Optional name to use as a prefix when adding operations. restore_sequentially: A `Bool`, which if true, causes restore of different variables to happen sequentially within each device. This can lower memory usage when restoring very large models. saver_def: Optional `SaverDef` proto to use instead of running the builder. This is only useful for specialty code that wants to recreate a `Saver` object for a previously built `Graph` that had a `Saver`. The `saver_def` proto should be the one returned by the `as_saver_def()` call of the `Saver` that was created for that `Graph`. builder: Optional `SaverBuilder` to use if a `saver_def` was not provided. Defaults to `BulkSaverBuilder()`. defer_build: If `True`, defer adding the save and restore ops to the `build()` call. In that case `build()` should be called before finalizing the graph or using the saver. allow_empty: If `False` (default) raise an error if there are no variables in the graph. Otherwise, construct the saver anyway and make it a no-op. write_version: controls what format to use when saving checkpoints. It also affects certain filepath matching logic. The V2 format is the recommended choice: it is much more optimized than V1 in terms of memory required and latency incurred during restore. Regardless of this flag, the Saver is able to restore from both V2 and V1 checkpoints. pad_step_number: if True, pads the global step number in the checkpoint filepaths to some fixed width (8 by default). This is turned off by default. save_relative_paths: If `True`, will write relative paths to the checkpoint state file. This is needed if the user wants to copy the checkpoint directory and reload from the copied directory. filename: If known at graph construction time, filename used for variable loading/saving. Raises: TypeError: If `var_list` is invalid. ValueError: If any of the keys or values in `var_list` are not unique. RuntimeError: If eager execution is enabled and`var_list` does not specify a list of variables to save. @compatibility(eager) When eager execution is enabled, `var_list` must specify a `list` or `dict` of variables to save. Otherwise, a `RuntimeError` will be raised. Although Saver works in some cases when executing eagerly, it is fragile. Please switch to `tf.train.Checkpoint` or `tf.keras.Model.save_weights`, which perform a more robust object-based saving. These APIs will load checkpoints written by `Saver`. @end_compatibility NzgIf `var_list` is provided then build cannot be deferred. Either set defer_build=False or var_list=None.aSaver is deprecated, please switch to tf.train.Checkpoint or tf.keras.Model.save_weights for training checkpoints. When executing eagerly variables do not necessarily have unique names, and so the variable.name-based lookups Saver performs are error-prone.z\When eager execution is enabled, `var_list` must specify a list or dict of variables to saveF) _END_TIME_OF_LAST_WRITE_LOCK_END_TIME_OF_LAST_WRITEtimerrrloggingwarningrF _var_list_reshape_sharded _max_to_keep_keep_checkpoint_every_n_hours_name_restore_sequentially saver_def_builder _is_built _allow_empty _is_emptyr3_pad_step_number _filename_last_checkpoints_checkpoints_to_be_deleted_next_checkpoint_timer_check_saver_defr_save_relative_paths_object_restore_saver)r4var_listrrrrr>rQrbuilder defer_buildrr5pad_step_numbersave_relative_pathsr8s r#r6zSaver.__init__ss` &. ("&))+.x  ; <<  " oo    +, ,DNDMDM#D*GD'DJ!5DDNDMDN#DDN'D+DDND&(D#  " ))+;;dB B   jjl ~~  NN22d 3D"&D_..s E<<Fc~tjr td|j|jddy)Nz0Use save/restore instead of build in eager mode.Trr)rrrF_buildrr4s r#rz Saver.builds0  " K LLKK4tKDr%c,|j|||y)Nr )r r4checkpoint_pathrrs r# _build_eagerzSaver._build_eagersKKJmMr%c tjs|jryd|_|jrtjr|jt |j |_|jtj|_|js|jrd|_ ytdd|_ |jj|j|j|j|j |j"|j$|j&||| |_n|jr|j$rt)j*|jj,|j$|j_t)j*|jj.|j$|j_t)j*|jj0|j$|j_|j3tjs4t5j4|jj6dzz|_yy)zBuilds saver_def.NTzNo variables to saveF) rrrrr>rQr8rrr)rrrrrrr3rr_all_saveable_objectsrrrrrrrrrrr prepend_name_scoperrrrrrrrs r#r z Saver._builds  $ $ & dn >>W668  ()<)<=  "88: ^^   $. 12 2dn}}44 ..----''(,(K(Kzz#99"%5 'dn DJJ-0,B,B .. - -tzz-;dnn)(+(>(> .. ) )4::)7dnn%'*'='= .. ( ($**(6dnn$   $ $ & ))+DDtK K   'r%ct|jtjst d|jzt j so|jjs!t dt|jz|jjs!t dt|jzyy)Nz*saver_def must be a saver_pb2.SaverDef: %sz/saver_def must specify the save_tensor_name: %sz.saver_def must specify the restore_op_name: %s) rrrrArrrrstrrrs r#rzSaver._check_saver_defs dnni&8&8 9 C~~& ''  $ $ & ^^ , ,JT^^,-. . ^^ + +IT^^,-. ., 'r%c|\}}|S)zReturns the checkpoint filename given a `(filename, time)` pair. Args: p: (filename, time) pair. Returns: Checkpoint file name. r)r4pr>_s r#_CheckpointFilenamezSaver._CheckpointFilenamesGD! Kr%c|jjsy|jddD]2}||j|k(s|jj |4|jj |t j ft|j|jjkDr5|jj |jjdyy)z+Manages the list of the latest checkpoints.Nr) rrrrremover=rrjrr)r4latest_save_pathrs r#_RecordLastCheckpointzSaver._RecordLastCheckpoints >> % %  # #A &) T55a8 8 %%a()  !!#3TYY["AB 4 ! !"T^^%?%?? %%,,T-C-C-G-G-JK@r%c|jr|jjd}|d|jkD}|r-|xj|jjdzz c_y t j |j||jj|yy#t$r)}tjdt|Yd}~yd}~wwxYw)a(Deletes old checkpoints if necessary. `self._checkpoints_to_be_deleted` is going to contain checkpoints that are over `max_to_keep`. They are going to be deleted. If `keep_checkpoint_every_n_hours` was specified, keep an additional checkpoint every `N` hours. For example, if `N` is 0.5, an additional checkpoint is kept for every 0.5 hours of training; if `N` is 10, an additional checkpoint is kept for every 10 hours of training. Args: meta_graph_suffix: Suffix for `MetaGraphDef` file. Defaults to 'meta'. rrrNz Ignoring: %s) rrrrrrremove_checkpointrr Exceptionrrr)r4meta_graph_suffixr should_keepes r#_MaybeDeleteOldCheckpointsz Saver._MaybeDeleteOldCheckpointss &&  ) ) - -a 0aaD4555k  "" NN 8 84 ? A"0//  $ $Q ')?)?  '0A//0s*:B&& C/CCc|jS)zaGenerates a `SaverDef` representation of this saver. Returns: A `SaverDef` proto. )rrs r# as_saver_defzSaver.as_saver_def2s >>r%c>| |jS|jjj|rJ|jjj|r%|jjj|syt j }|j|jtj|j||_tj|j||_tj|j||_|S)zConverts this `Saver` to a `SaverDef` protocol buffer. Args: export_scope: Optional `string`. Name scope to remove. Returns: A `SaverDef` protocol buffer. N) rr startswithrrrrACopyFromr strip_name_scope)r4 export_scopers r#to_protozSaver.to_proto:s ^^ NN / / : :< H NN + + 6 6| D NN * * 5 5l C ""$I t~~&%(%9%9&& &6I"!$!5!5""L"2I # 4 4Y5N5N5A!CI r%ct||S)zReturns a `Saver` object created from `saver_def`. Args: saver_def: a `SaverDef` protocol buffer. import_scope: Optional `string`. Name scope to use. Returns: A `Saver` built from saver_def. rr>)r)r import_scopes r# from_protozSaver.from_protoUs 9< 88r%c@tfdjDS)zList of not-yet-deleted checkpoint filenames. You can pass any of the returned values to `restore()`. Returns: A list of checkpoint filenames, sorted from oldest to newest. c3@K|]}j|ywr2)r)rrr4s r#rz)Saver.last_checkpoints..ksL((+Ls)listrrs`r#last_checkpointszSaver.last_checkpointsbs LT5K5KL LLr%c|t|tsJ|Dcgc]}|tjfc}|_ycc}w)zDEPRECATED: Use set_last_checkpoints_with_time. Sets the list of old checkpoint filenames. Args: last_checkpoints: A list of checkpoint filenames. Raises: AssertionError: If last_checkpoints is not a list. N)rr5npinfr)r4r6ss r#set_last_checkpointszSaver.set_last_checkpointsms6 & -- -4DDaq"&&kDDDs9c6t|tsJ||_y)zSets the list of old checkpoint filenames and timestamps. Args: last_checkpoints_with_time: A list of tuples of checkpoint filenames and timestamps. Raises: AssertionError: If last_checkpoints_with_time is not a list. N)rr5r)r4last_checkpoints_with_times r#set_last_checkpoints_with_timez$Saver.set_last_checkpoints_with_time~s 0$ 77 77Dr%cg}|D]2} tj|g}|s|j ||df4|j |y#tj$rY\wxYw)aIRecovers the internal saver state after a crash. This method is useful for recovering the "self._last_checkpoints" state. Globs for the checkpoints pointed to by `checkpoint_paths`. If the files exist, use their mtime as the checkpoint timestamp. Args: checkpoint_paths: a list of checkpoint paths. rN)rget_checkpoint_mtimesr NotFoundErrorr=r>)r4checkpoint_pathscheckpoints_with_mtimesrmtimes r#recover_last_checkpointszSaver.recover_last_checkpointss|!+D%;;_j@} n<|jC|j>j@|j>jD| i} tj | } |rQ|jG| tIjJ| | |jL||jN|jQ|tj}t]j^t`tc| |td5t]jft`tcth||a4ddd|rbtIjj| |}tjs7|jljo5|jq||| ddd|j:ryt]jrt`tu | S#tRjTtRjVf$r6}tYjZ| stdj-|}|d}~wwxYw#1swYxYw#1swYxYw)a$ Saves variables. This method runs the ops added by the constructor for saving variables. It requires a session in which the graph was launched. The variables to save must also have been initialized. The method returns the path prefix of the newly created checkpoint files. This string can be passed directly to a call to `restore()`. Args: sess: A Session to use to save the variables. save_path: String. Prefix of filenames created for the checkpoint. global_step: If provided the global step number is appended to `save_path` to create the checkpoint filenames. The optional argument can be a `Tensor`, a `Tensor` name or an integer. latest_filename: Optional name for the protocol buffer file that will contains the list of most recent checkpoints. That file, kept in the same directory as the checkpoint files, is automatically managed by the saver to keep track of recent checkpoints. Defaults to 'checkpoint'. meta_graph_suffix: Suffix for `MetaGraphDef` file. Defaults to 'meta'. write_meta_graph: `Boolean` indicating whether or not to write the meta graph file. write_state: `Boolean` indicating whether or not to write the `CheckpointStateProto`. strip_default_attrs: Boolean. If `True`, default-valued attributes will be removed from the NodeDefs. For a detailed guide, see [Stripping Default-Valued Attributes](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md#stripping-default-valued-attributes). save_debug_info: If `True`, save the GraphDebugInfo to a separate file, which in the same directory of save_path and with `_debug` added before the file extension. This is only enabled when `write_meta_graph` is `True` Returns: A string: path prefix used for the checkpoint files. If the saver is sharded, this string ends with: '-?????-of-nnnnn' where 'nnnnn' is the number of shards created. If the saver is empty, returns None. Raises: TypeError: If `sess` is not a `Session`. ValueError: If `latest_filename` contains path components, or if it collides with `save_path`. RuntimeError: If save and restore ops weren't built. z;`build()` should be called before save if defer_build==TrueN checkpointz7*******************************************************z6TensorFlow's V1 checkpoint format has been deprecated.z3Consider switching to the more efficient V2 format:z7 `tf.train.Saver(write_version=tf.train.SaverDef.V2)`znow on by default.rz2'latest_filename' must not contain path componentsz%s-%d-z{:08d}z.'latest_filename' collides with 'save_path': 'z' and ''z'sess' must be a Session; %sTFr )save_dirmodel_checkpoint_pathall_model_checkpoint_pathslatest_filenamer )r#z1Parent directory of {} doesn't exist, can't save. api_label microseconds)strip_default_attrssave_debug_info)rOfilesize);rrrrrFr3rrArDrrrfrgsplitrras_strrintegral_typesr global_steprr(basenamerrSessionInterface TypeErrordirnamerrrrrunrrr update_checkpoint_state_internalr6rr&r FailedPreconditionErrorrAr IsDirectoryrAddCheckpointWriteDuration _SAVER_LABELr$rAddTrainingTimeSavedrmeta_graph_filenamer as_defaultexport_meta_graphRecordCheckpointSizer.)r4sess save_pathrWrMr#write_meta_graph write_staterQrR start_timecheckpoint_filesave_path_parentrKexcend_timercs r#r]z Saver.savesrJ >>'";";"=  G II$o i00333 ooOP ooNO ooKL ooOP oo*+ ooOP ww}}_%a( K LL i(I  V%:%: ;#//kB 9k"::o  %. 0LM!o   ) $ 7  i )* *  % % ' tW55 6 4t; <<wwy1 >>  $ $ &   $e  E"&.."A"A "&((nn--~~22OD#F !' .C D   $ $%: ;  @ @'$9)-)>)>-"&";";  =  ) )@  & & ( ZZ " " $ /  !"5- ! / /  ~~  "" '(=>@#"G,,f.B.B C  !12AHH# )) / /s11C#P,Q:R#Q71Q22Q7:RRc t|tjjdd|j||||||| S)aWrites `MetaGraphDef` to save_path/filename. Args: filename: Optional meta_graph filename including the path. collection_list: List of string keys to collect. as_text: If `True`, writes the meta_graph as an ASCII proto. export_scope: Optional `string`. Name scope to remove. clear_devices: Whether or not to clear the device field for an `Operation` or `Tensor` during export. clear_extraneous_savers: Remove any Saver-related information from the graph (both Save/Restore ops and SaverDefs) that are not associated with this Saver. strip_default_attrs: Boolean. If `True`, default-valued attributes will be removed from the NodeDefs. For a detailed guide, see [Stripping Default-Valued Attributes](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md#stripping-default-valued-attributes). save_debug_info: If `True`, save the GraphDebugInfo to a separate file, which in the same directory of filename and with `_debug` added before the file extension. Returns: A `MetaGraphDef` proto. T) add_shapesuse_pybind11_proto) r8 graph_defrcollection_listas_textr- clear_devicesclear_extraneous_saversrQrR)rer r as_graph_defr) r4r8rtrur-rvrwrQrRs r#rezSaver.export_meta_graph;sTD '')667 ..'!# 7/'  r%ctj}|jry| tdtj|}t j |std|ztjd| tjr|j|ddn<|j|jj|jj|it7j8t:t=|tj y#t j"$r} t%|}n##t j"$r t'|dwxYwtj(d t+||j,|j.||j0 |_|j0j3|| Yd}~d}~wt j4$r}t'|d d}~wwxYw)aRestores previously saved variables. This method runs the ops added by the constructor for restoring variables. It requires a session in which the graph was launched. The variables to restore do not have to have been initialized, as restoring is itself a way to initialize variables. The `save_path` argument is typically a value previously returned from a `save()` call, or a call to `latest_checkpoint()`. Args: sess: A `Session` to use to restore the parameters. None in eager mode. save_path: Path where parameters were previously saved. Raises: ValueError: If save_path is None or not a valid checkpoint. Nz%Can't load save_path when it is None.z0The passed save_path is not a valid checkpoint: zRestoring parameters from %sFTr z2a Variable name or other graph key that is missingzRestoring an object-based checkpoint using a name-based saver. This may be somewhat fragile, and will re-build the Saver. Instead, consider loading object-based checkpoints using tf.train.Checkpoint().)rrr names_to_keys cached_saver)rgrhz2a mismatch between the current graph and the graphrN)rrrrrurcheckpoint_exists_internalrinforrrr\rrrr rAobject_graph_key_mapping_wrap_restore_error_with_msgr"saver_from_object_based_checkpointrrrrInvalidArgumentErrorrAddCheckpointReadDurationrar$)r4rgrhrkrperrrzs r#rz Saver.restorels$J ~~  > ??y1 ; ;>--%11 $3d    ((di(HH  & &E ) C EEEs8>A%DG// D;:G; EA(GG/ G**G/c4tj|||y)zAdds a collection to MetaGraphDef protocol buffer. Args: meta_graph_def: MetaGraphDef protocol buffer. key: One of the GraphKeys or user-defined string. export_scope: Optional `string`. Name scope to remove. )r-N)r add_collection_def)meta_graph_defrr-s r#_add_collection_defzSaver._add_collection_defs!!,8r%)metar2)NNrTTFF)NNFNFFFF)rrrrrrArDr6rrr rrrr&r(r. staticmethodr2propertyr6r;r>rEr]rerrrr%r#rrsFV-4$)  &//22$#(_&BE M1Mf . L 0>6 9 9 M ME" 8A4# $ W#t"&(, %%)&+05,1(-/bJJX 8 8r%rztrain.import_meta_graphc $t|||fi|dS)aRecreates a Graph saved in a `MetaGraphDef` proto. This function takes a `MetaGraphDef` protocol buffer as input. If the argument is a file containing a `MetaGraphDef` protocol buffer , it constructs a protocol buffer from the file content. The function then adds all the nodes from the `graph_def` field to the current graph, recreates all the collections, and returns a saver constructed from the `saver_def` field. In combination with `export_meta_graph()`, this function can be used to * Serialize a graph along with other Python objects such as `QueueRunner`, `Variable` into a `MetaGraphDef`. * Restart training from a saved graph and checkpoints. * Run inference from a saved graph and checkpoints. ```Python ... # Create a saver. saver = tf.compat.v1.train.Saver(...variables...) # Remember the training_op we want to run by adding it to a collection. tf.compat.v1.add_to_collection('train_op', train_op) sess = tf.compat.v1.Session() for step in range(1000000): sess.run(train_op) if step % 1000 == 0: # Saves checkpoint, which by default also exports a meta_graph # named 'my-model-global_step.meta'. saver.save(sess, 'my-model', global_step=step) ``` Later we can continue training from this saved `meta_graph` without building the model from scratch. ```Python with tf.Session() as sess: new_saver = tf.train.import_meta_graph('my-save-dir/my-model-10000.meta') new_saver.restore(sess, 'my-save-dir/my-model-10000') # tf.get_collection() returns a list. In this example we only want # the first one. train_op = tf.get_collection('train_op')[0] for step in range(1000000): sess.run(train_op) ``` NOTE: Restarting training from saved `meta_graph` only works if the device assignments have not changed. Example: Variables, placeholders, and independent operations can also be stored, as shown in the following example. ```Python # Saving contents and operations. v1 = tf.placeholder(tf.float32, name="v1") v2 = tf.placeholder(tf.float32, name="v2") v3 = tf.math.multiply(v1, v2) vx = tf.Variable(10.0, name="vx") v4 = tf.add(v3, vx, name="v4") saver = tf.train.Saver([vx]) sess = tf.Session() sess.run(tf.global_variables_initializer()) sess.run(vx.assign(tf.add(vx, vx))) result = sess.run(v4, feed_dict={v1:12.0, v2:3.3}) print(result) saver.save(sess, "./model_ex1") ``` Later this model can be restored and contents loaded. ```Python # Restoring variables and running operations. saver = tf.train.import_meta_graph("./model_ex1.meta") sess = tf.Session() saver.restore(sess, "./model_ex1") result = sess.run("v4:0", feed_dict={"v1:0": 12.0, "v2:0": 3.3}) print(result) ``` Args: meta_graph_or_file: `MetaGraphDef` protocol buffer or filename (including the path) containing a `MetaGraphDef`. clear_devices: Whether or not to clear the device field for an `Operation` or `Tensor` during import. import_scope: Optional `string`. Name scope to add. Only used when initializing from protocol buffer. **kwargs: Optional keyed arguments. Returns: A saver constructed from `saver_def` in `MetaGraphDef` or None. A None value is returned if no variables exist in the `MetaGraphDef` (i.e., there are no variables to restore). Raises: RuntimeError: If called with eager execution enabled. @compatibility(eager) Exporting/importing meta graphs is not supported. No graph exists when eager execution is enabled. @end_compatibility r)'_import_meta_graph_with_return_elements)meta_graph_or_filervr1kwargss r#import_meta_graphrs*\ 11C1>  ;39 ;;< >>r%c tjr tdt|tj st j|}n|}t j|f|||d|\}}t|||}||fS)z@Import MetaGraph, and return both a saver and returned elements.Exporting/importing meta graphs is not supported when eager execution is enabled. No graph exists when eager execution is enabled.)rvr1return_elements) rrrFrr MetaGraphDefr read_meta_graph_file-import_scoped_meta_graph_with_return_elements&_create_saver_from_imported_meta_graph) rrvr1rrr imported_varsimported_return_elementsrs r#rr8s   / 00 &(C(C D445GHN'N>> %#)   *-) 11> @% ( ((r%c>|jdrW|}t|j}|r#|d}||}|jdt | }t |j |Stj|r t Stjdy)zFReturn a saver for restoring variable values to an imported MetaGraph.rrNr0)scopezHSaver not created because there are no variables in the graph to restore) HasFieldr5keysr>rjrrrrrr})rr1rr var_names sample_key sample_vars r#rrTs[) E]'')*IQ33% @@&&\: Wnll'( r%ztrain.export_meta_graphc  tjr|| tdtjd|||||||||| | | d | \} }| S)aReturns `MetaGraphDef` proto. Optionally writes it to filename. This function exports the graph, saver, and collection objects into `MetaGraphDef` protocol buffer with the intention of it being imported at a later time or location to restart training, run inference, or be a subgraph. Args: filename: Optional filename including the path for writing the generated `MetaGraphDef` protocol buffer. meta_info_def: `MetaInfoDef` protocol buffer. graph_def: `GraphDef` protocol buffer. saver_def: `SaverDef` protocol buffer. collection_list: List of string keys to collect. as_text: If `True`, writes the `MetaGraphDef` as an ASCII proto. graph: The `Graph` to export. If `None`, use the default graph. export_scope: Optional `string`. Name scope under which to extract the subgraph. The scope name will be striped from the node definitions for easy import later into new name scopes. If `None`, the whole graph is exported. graph_def and export_scope cannot both be specified. clear_devices: Whether or not to clear the device field for an `Operation` or `Tensor` during export. clear_extraneous_savers: Remove any Saver-related information from the graph (both Save/Restore ops and SaverDefs) that are not associated with the provided SaverDef. strip_default_attrs: Boolean. If `True`, default-valued attributes will be removed from the NodeDefs. For a detailed guide, see [Stripping Default-Valued Attributes](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/saved_model/README.md#stripping-default-valued-attributes). save_debug_info: If `True`, save the GraphDebugInfo to a separate file, which in the same directory of filename and with `_debug` added before the file extend. **kwargs: Optional keyed arguments. Returns: A `MetaGraphDef` proto. Raises: ValueError: When the `GraphDef` is larger than 2GB. RuntimeError: If called with eager execution enabled. @compatibility(eager) Exporting/importing meta graphs is not supported unless both `graph_def` and `graph` are provided. No graph exists when eager execution is enabled. @end_compatibility r) r8 meta_info_defrsrrtrurr-rvrwrQrRr)rrrFr export_scoped_meta_graph)r8rrsrrtrurr-rvrwrQrRrrrs r#rerels@  )*?*/*; / 00!99 !% !5-%   .! r%cdj||j}|j|j|j|S)NzRestoring from checkpoint failed. This is most likely due to {} from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error: {})r(message __class__node_defrm)rextra_verbiageerr_msgs r#rrs;%'-f^S[[&I  s||SVVW 55r%) proto_typer.r2c2tj|}|jtj}t j }|j|i}|jD],}|jD]}|j||j<.|S)zReturn name to key mappings from the checkpoint. Args: checkpoint_path: string, path to object-based checkpoint Returns: Dictionary mapping tensor names to checkpoint keys. ) rNewCheckpointReader get_tensor trackableOBJECT_GRAPH_PROTO_KEYrTrackableObjectGraphParseFromStringnodes attributescheckpoint_key full_name)rreaderobject_graph_stringobject_graph_protorznode attributes r#r~r~s 3 3O D&)))*J*JK2GGI$$%89- &&Dd__D +4+C+CmI''(DD r%c *| t|}|t j }| t }t|tstj|}tj|}t}|D].}|jD]}|j|j0t|j!} || z } | rm| |z } | j#|} tjddddj%t'| dj%t'| t)| fz|D]'}|jD]}||j|_)| t+|S|S#tj$rtd|zwxYw)aReturn a `Saver` which reads from an object-based checkpoint. This function validates that all variables in the variables list are remapped in the object-based checkpoint (or `names_to_keys` dict if provided). A saver will be created with the list of remapped variables. The `cached_saver` argument allows the user to pass in a previously created saver, so multiple `saver.restore()` calls don't pollute the graph when graph building. This assumes that keys are consistent, meaning that the 1) `checkpoint_path` checkpoint, and 2) checkpoint used to create the `cached_saver` are the same type of object-based checkpoint. If this argument is set, this function will simply validate that all variables have been remapped by the checkpoint at `checkpoint_path`. Note that in general, `tf.train.Checkpoint` should be used to restore/save an object-based checkpoint. Args: checkpoint_path: string, path to object-based checkpoint var_list: list of `Variables` that appear in the checkpoint. If `None`, `var_list` will be set to all saveable objects. builder: a `BaseSaverBuilder` instance. If `None`, a new `BulkSaverBuilder` will be created. names_to_keys: dict mapping string tensor names to checkpoint keys. If `None`, this dict will be generated from the checkpoint file. cached_saver: Cached `Saver` object with remapped variables. Returns: `Saver` with remapped variables for reading from an object-based checkpoint. Raises: ValueError if the checkpoint provided is not an object-based checkpoint. NotFoundError: If one of the variables in `var_list` can not be found in the checkpoint. This could mean the checkpoint or `names_to_keys` mapping is missing the variable. Nz0Checkpoint in %s not an object-based checkpoint.a# Existing variables not in the checkpoint: %s Variables names when this checkpoint was written which don't exist now: %s (%d variable name(s) did match) Could not find some variables in the checkpoint (see names above). Saver was attempting to load an object-based checkpoint (saved using tf.train.Checkpoint or tf.keras.Model.save_weights) using variable names. If the checkpoint was written with eager execution enabled, it's possible that variable names have changed (for example missing a '_1' suffix). It's also possible that there are new variables which did not exist when the checkpoint was written. You can construct a Saver(var_list=...) with only the variables which previously existed, and if variable names have changed you may need to make this a dictionary with the old names as keys.z, )r)r~r rArrrrrrrrrrr<addr>r intersectionjoinrrjr) rrrrzr{rH current_namesrIrJprevious_names missing_names extra_namesintersecting_namess r#rrsT(.?m..0H _ G Hd ##33H=H"<rs "   3.?>,+37.2++2,(,+,<D9;4=B)6-AA/GG99 );;);;-CC);;-y~~/ Br'r'jG'G&6 }o@ 8@ 8@ 8F ()*$)#'o>+o>f;@9=<@)80 ()*#$( $ $&*# #'$).3*/&+R+Rj6MM!! ^^ !*15/35948 [r%