BVhX\ ^dZddlZddlmZmZddlZddlmZddl m Z ddl m Z ddl mZddl mZdd lmZdd lmZdd lmZd efd Zde j4dej6defdZedgGddZdej<de j4d eeeeffdZ GddZ!y)z)hasattrrtype)objs Z/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/session_manager.py _maybe_namer s. [ sF 88O S ))sessr pathcxtjd}|r|j||j||y)z@Restores checkpoint values and SavedModel initializers if found.saved_model_initializersN)r get_collectionrunrestore)rr rsaved_model_init_opss r:_restore_checkpoint_and_maybe_run_saved_model_initializersr1s6++,FGHH !" --drztrain.SessionManager)v1c eZdZdZ ddej dej dej dej ddf d Z dd ed e jd ed ede e jeff dZ dd edej d e jd ed ede jf dZ dd ed e jd ed ede e jeff dZdedfd edee jfdZde jfdZde jde eeeffdZde jde eeeffdZde jde eeeffdZy)SessionManageraTraining helper that restores from checkpoint and creates session. This class is a small wrapper that takes care of session creation and checkpoint recovery. It also provides functions that to facilitate coordination among multiple training threads or processes. * Checkpointing trained variables as the training progresses. * Initializing variables on startup, restoring them from the most recent checkpoint after a crash, or wait for checkpoints to become available. ### Usage: ```python with tf.Graph().as_default(): ...add operations to the graph... # Create a SessionManager that will checkpoint the model in '/tmp/mydir'. sm = SessionManager() sess = sm.prepare_session(master, init_op, saver, checkpoint_dir) # Use the session to train the graph. while True: sess.run() ``` `prepare_session()` initializes or restores a model. It requires `init_op` and `saver` as an argument. A second process could wait for the model to be ready by doing the following: ```python with tf.Graph().as_default(): ...add operations to the graph... # Create a SessionManager that will wait for the model to become ready. sm = SessionManager() sess = sm.wait_for_session(master) # Use the session to train the graph. while True: sess.run() ``` `wait_for_session()` waits for a model to be initialized by other processes. N local_init_opready_opready_for_local_init_opgraphlocal_init_run_optionszdistribute_lib.RunOptionsc|tj}||_||_||_||_||_d|_||_||_ ||td|zyy)a_Creates a SessionManager. The `local_init_op` is an `Operation` that is run always after a new session was created. If `None`, this step is skipped. The `ready_op` is an `Operation` used to check if the model is ready. The model is considered ready if that operation returns an empty 1D string tensor. If the operation returns a non empty 1D string tensor, the elements are concatenated and used to indicate to the user why the model is not ready. The `ready_for_local_init_op` is an `Operation` used to check if the model is ready to run local_init_op. The model is considered ready if that operation returns an empty 1D string tensor. If the operation returns a non empty 1D string tensor, the elements are concatenated and used to indicate to the user why the model is not ready. If `ready_op` is `None`, the model is not checked for readiness. `recovery_wait_secs` is the number of seconds between checks that the model is ready. It is used by processes to wait for a model to be initialized or restored. Defaults to 30 seconds. Args: local_init_op: An `Operation` run immediately after session creation. Usually used to initialize tables and local variables. ready_op: An `Operation` to check if the model is initialized. ready_for_local_init_op: An `Operation` to check if the model is ready to run local_init_op. graph: The `Graph` that the model will use. recovery_wait_secs: Seconds between checks for the model to be ready. local_init_run_options: RunOptions to be passed to session.run when executing the local_init_op. local_init_feed_dict: Optional session feed dictionary to use when running the local_init_op. Raises: ValueError: If ready_for_local_init_op is not None but local_init_op is None NzgIf you pass a ready_for_local_init_op you must also pass a local_init_op , ready_for_local_init_op [%s]) r get_default_graph_local_init_op _ready_op_ready_for_local_init_op_graph_recovery_wait_secs_target_local_init_run_options_local_init_feed_dict ValueError)selfr#r$r%r&recovery_wait_secsr'local_init_feed_dicts r__init__zSessionManager.__init__wsf }##%e'DDN$;D!DK1DDL#9D !5D*}/D 8// 000E*rmasterr checkpoint_dircheckpoint_filename_with_pathr c||_tj}|r0t|jdr|jj t j|j|j|} |r |r td|r|s|s| dfS|rt| ||| dfSd} tj|} | r | jst|r^| |krYtjdt!j"|j$| |j$z } tj|} n| dfS| sg| jstt| || j|j'| j(| dfS)afCreates a `Session`, and tries to restore a checkpoint. Args: master: `String` representation of the TensorFlow master to use. saver: A `Saver` object used to restore a model. checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the dir will be used to restore. checkpoint_filename_with_path: Full file name path to the checkpoint file. wait_for_checkpoint: Whether to wait for checkpoint to become available. max_wait_secs: Maximum time to wait for checkpoints to become available. config: Optional `ConfigProto` proto used to configure the session. Returns: A pair (sess, is_restored) where 'is_restored' is `True` if the session could be restored, `False` otherwise. Raises: ValueError: If both checkpoint_dir and checkpoint_filename_with_path are set. _experimental_initialize_systemr&configzFCan not provide both checkpoint_dir and checkpoint_filename_with_path.FTrz'Waiting for checkpoint to be available.)r/r get_strategyrextendedr;rSessionr-r2rrget_checkpoint_statemodel_checkpoint_pathlogginginfotimesleepr.recover_last_checkpointsall_model_checkpoint_paths) r3r7r r8r9wait_for_checkpoint max_wait_secsr=strategyr wait_timeckpts r_restore_checkpointz"SessionManager._restore_checkpoints\>DL **,HGH--=?779 ??4<? 4++,T--- $99.IU{$44? eT//1 ""4#B#BC :rinit_opc |j|||||||\} } | s:|| s|j td|| j|| | r| | |j | \} }| stdt |d| d||j | \}}|s*tdt |d| d|jd|| S) a> Creates a `Session`. Makes sure the model is ready to be used. Creates a `Session` on 'master'. If a `saver` object is passed in, and `checkpoint_dir` points to a directory containing valid checkpoint files, then it will try to recover the model from checkpoint. If no checkpoint files are available, and `wait_for_checkpoint` is `True`, then the process would check every `recovery_wait_secs`, up to `max_wait_secs`, for recovery to succeed. If the model cannot be recovered successfully then it is initialized by running the `init_op` and calling `init_fn` if they are provided. The `local_init_op` is also run after init_op and init_fn, regardless of whether the model was recovered successfully, but only if `ready_for_local_init_op` passes. If the model is recovered from a checkpoint it is assumed that all global variables have been initialized, in particular neither `init_op` nor `init_fn` will be executed. It is an error if the model cannot be recovered and no `init_op` or `init_fn` or `local_init_op` are passed. Args: master: `String` representation of the TensorFlow master to use. init_op: Optional `Operation` used to initialize the model. saver: A `Saver` object used to restore a model. checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the dir will be used to restore. checkpoint_filename_with_path: Full file name path to the checkpoint file. wait_for_checkpoint: Whether to wait for checkpoint to become available. max_wait_secs: Maximum time to wait for checkpoints to become available. config: Optional `ConfigProto` proto used to configure the session. init_feed_dict: Optional dictionary that maps `Tensor` objects to feed values. This feed dictionary is passed to the session `run()` call when running the init op. init_fn: Optional callable used to initialize the model. Called after the optional `init_op` is called. The callable must accept one argument, the session being initialized. Returns: A `Session` object that can be used to drive the model. Raises: RuntimeError: If the model cannot be initialized or recovered. ValueError: If both checkpoint_dir and checkpoint_filename_with_path are set. r8r9rIrJr=zMModel is not initialized and no init_op or init_fn or local_init_op was given) feed_dictzCInit operations did not make model ready for local_init. Init op: z , init fn: z , error: z4Init operations did not make model ready. Init op: z, local_init_op: )rNr* RuntimeErrorr_try_run_local_init_opr _model_ready)r3r7rOr r8r9rIrJr=init_feed_dictinit_fnris_loaded_from_checkpointlocal_init_successmsgis_readys rprepare_sessionzSessionManager.prepare_sessionsz'+&>&> %&C/#'?'#D # % T-@-@-H@A A   N3  "99$?  3>w3G3:36 8 99 %%d+MHc   w $*=*=s D EE Krc H|j|||||||\}} |j|\} } | s|dfS|xs|} | stjd| | |dfS|j |\} } | stjd| | |dfStjd| || fS)aCreates a `Session`, recovering if possible. Creates a new session on 'master'. If the session is not initialized and can be recovered from a checkpoint, recover it. Args: master: `String` representation of the TensorFlow master to use. saver: A `Saver` object used to restore a model. checkpoint_dir: Path to the checkpoint files. The latest checkpoint in the dir will be used to restore. checkpoint_filename_with_path: Full file name path to the checkpoint file. wait_for_checkpoint: Whether to wait for checkpoint to become available. max_wait_secs: Maximum time to wait for checkpoints to become available. config: Optional `ConfigProto` proto used to configure the session. Returns: A pair (sess, initialized) where 'initialized' is `True` if the session could be recovered and initialized, `False` otherwise. Raises: ValueError: If both checkpoint_dir and checkpoint_filename_with_path are set. rQFzCRestoring model from %s did not make model ready for local init: %sz4Restoring model from %s did not make model ready: %szRestored model from %s)rNrTrCrDrU)r3r7r r8r9rIrJr=rrXrYrZrestoring_filer[s rrecover_sessionzSessionManager.recover_sessionasD'+&>&> %&C/#'?'#D ##99$? $ 5[#D'DN  ll &5[%%d+MHc  llI!3( 5[ LL)>: * **rInfc||_| td}t|} tj|j|j |}d}d}|j |\}}|r|j|\} }| r|S|j||j|jz } | dkrtjddd|fztjd||tj |j)aOCreates a new `Session` and waits for model to be ready. Creates a new `Session` on 'master'. Waits for the model to be initialized or recovered from a checkpoint. It's expected that another thread or process will make the model ready, and that this is intended to be used by threads/processes that participate in a distributed training configuration where a different thread/process is responsible for initializing or recovering the model being trained. NB: The amount of time this method waits for the session is bounded by max_wait_secs. By default, this function will wait indefinitely. Args: master: `String` representation of the TensorFlow master to use. config: Optional ConfigProto proto used to configure the session. max_wait_secs: Maximum time to wait for the session to become available. Returns: A `Session`. May be None if the operation exceeds the timeout specified by config.operation_timeout_in_ms. Raises: tf.DeadlineExceededError: if the session is not available after max_wait_secs. Nr`r<rz,Session was not ready after waiting %d secs.zGWaiting for model to be ready. Ready_for_local_init_op: %s, ready: %s)r/float_CountDownTimerrr@r-rTrU _safe_closesecs_remainingr.rDeadlineExceededErrorrCrDrErF) r3r7r=rJtimerr not_ready_msgnot_ready_local_msgrYr[remaining_ms_after_waits rwait_for_sessionzSessionManager.wait_for_sessions8DLElm M *E  __T\\V Ldm 040K0K 1-- "&"3"3D"9- + t    4#;#; ; 1 $** $ :m=M MO Oll=& 7 jj))*3 rrcD |jy#t$rYywxYw)zCloses a session without raising an exception. Just like sess.close() but ignores exceptions. Args: sess: A `Session`. N)close Exceptionr3rs rrdzSessionManager._safe_closes$  jjl    s  c0t|j|dS)aChecks if the model is ready or not. Args: sess: A `Session`. Returns: A tuple (is_ready, msg), where is_ready is True if ready and False otherwise, and msg is `None` if the model is ready, a `String` with the reason why it is not ready otherwise. zModel not ready)_readyr+ros rrUzSessionManager._model_readys $..$(9 ::rc0t|j|dS)a\Checks if the model is ready to run local_init_op. Args: sess: A `Session`. Returns: A tuple (is_ready, msg), where is_ready is True if ready to run local_init_op and False otherwise, and msg is `None` if the model is ready to run local_init_op, a `String` with the reason why it is not ready otherwise. zModel not ready for local init)rqr,ros r_model_ready_for_local_initz*SessionManager._model_ready_for_local_inits $//2 44rc |jw|j|\}}|r]tjd|j |j|j |j tjdyd|fSy)anTries to run _local_init_op, if not None, and is ready for local init. Args: sess: A `Session`. Returns: A tuple (is_successful, msg), where is_successful is True if _local_init_op is None, or we ran _local_init_op, and False otherwise; and msg is a `String` with the reason why the model was not ready to run local init. zRunning local_init_op.)rRoptionszDone running local_init_op.TNF)r*rsrCrDrr1r0)r3ris_ready_for_local_initrZs rrTz%SessionManager._try_run_local_init_ops} &%)%E%Ed%K"s  -. $$0J0J55  7 23cz r)NNNNNN)NNNF N) NNNNFryNNN)__name__ __module__ __qualname____doc__r OperationGraphr6str saver_libSaverrrr@boolrNr\r_rbrrkrdrUrsrTrrr"r"JsL)Z&* $/3<@A0]]A0 A0 #}} A0 YY A0:A0L $ +/ GG__G G &) G W__d "#GX $# +/ \\}}\__ \  \ &) \\B $ +/ @+@+__@+ @+ &) @+ W__d "#@+F!%E%L;+;+ ;+z goo $ ;w ;5x}9L3M ;4//4 T8C= !4"// T8C= !rr"opc|y |j|}|,|jtjk(s|jdk(rydj |Dcgc]}|j dc}}dd|zfScc}w#tj$rF}dt|vr"tjd |t||dt|fcYd}~Sd}~wwxYw) aChecks if the model is ready or not, as determined by op. Args: op: An op, either _ready_op or _ready_for_local_init_op, which defines the readiness of the model. sess: A `Session`. msg: A message to log to warning if not ready Returns: A tuple (is_ready, msg), where is_ready is True if ready and False otherwise, and msg is `None` if the model is ready, a `String` with the reason why it is not ready otherwise. Nrvrz, zutf-8FzVariables not initialized: uninitializedz%s : error [%s]) rdtypenpint32sizejoindecoderFailedPreconditionErrorrrCwarning)rrrZ ready_valueinon_initialized_varnameses rrqrq+s Z HHRLk  !2!2bhh!>   a  $(99(3 41QXXg  4$6 36NNNN 5  ) ) A &)3A7 CF] s5?A?A?A:, A?:A??C;C CCc$eZdZdZddgZdZdZy)rcz.A timer that tracks a duration since creation._start_time_secs_duration_secscDtj|_||_y)N)rErr)r3 duration_secss rr6z_CountDownTimer.__init__Ys IIKD'Drcv|jtj|jz z }td|S)Nr)rrErmax)r3diffs rrez_CountDownTimer.secs_remaining]s/   $))+0E0E"E FD q$<rN)rzr{r|r} __slots__r6rerrrrcrcTs6!#34)(rrc)"r}rEtypingrrnumpyrtensorflow.python.checkpointrtensorflow.python.clientrtensorflow.python.distributertensorflow.python.frameworkrr tensorflow.python.platformr rCtensorflow.python.trainingr r tensorflow.python.util.tf_exportr rrr@rrr"r~rrqrcrrrrsC ">,7.+<96**" //"+//9<2 %&']](]@& &$__& 4# &R  r