BVhCTdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZ ddlm!Z!ddl"m#Z#ddl$m%Z%e jLe jNfZ(e)Z*e%dgGddZ+ d>d Z,e%d!gd"ddddde*e*e*dddde*ddfd#Z-e%d$gGd%d&ej\'Z/e%d(gGd)d*e/Z0e%d+gGd,d-e/Z1Gd.d/Z2e%d0gGd1d2e2Z3e%d3gGd4d5e2Z4Gd6d7Z5Gd8d9e5Z6Gd:d;e5Z7Gd<d=e5Z8y)?z*A wrapper of Session API which runs hooks.N) config_pb2) checkpoint) graph_view)distribute_coordinator_context)errors)ops) array_ops)control_flow_ops) lookup_ops) resources) variables) tf_logging)summary)basic_session_run_hooks) coordinator) queue_runner)saver)session_manager)session_run_hook)function_utils) tf_exportztrain.Scaffold)v1ceZdZdZ ddZdZedZedZedZ edZ ed Z ed Z ed Z ed Zed ZedZedZy)Scaffoldax Structure to create or gather pieces commonly needed to train a model. When you build a model for training you usually need ops to initialize variables, a `Saver` to checkpoint them, an op to collect summaries for the visualizer, and so on. Various libraries built on top of the core TensorFlow library take care of creating some or all of these pieces and storing them in well known collections in the graph. The `Scaffold` class helps pick these pieces from the graph collections, creating and adding them to the collections if needed. If you call the scaffold constructor without any arguments, it will pick pieces from the collections, creating default ones if needed when `scaffold.finalize()` is called. You can pass arguments to the constructor to provide your own pieces. Pieces that you pass to the constructor are not added to the graph collections. The following pieces are directly accessible as attributes of the `Scaffold` object: * `saver`: A `tf.compat.v1.train.Saver` object taking care of saving the variables. Picked from and stored into the `SAVERS` collection in the graph by default. * `init_op`: An op to run to initialize the variables. Picked from and stored into the `INIT_OP` collection in the graph by default. * `ready_op`: An op to verify that the variables are initialized. Picked from and stored into the `READY_OP` collection in the graph by default. * `ready_for_local_init_op`: An op to verify that global state has been initialized and it is alright to run `local_init_op`. Picked from and stored into the `READY_FOR_LOCAL_INIT_OP` collection in the graph by default. This is needed when the initialization of local variables depends on the values of global variables. * `local_init_op`: An op to initialize the local variables. Picked from and stored into the `LOCAL_INIT_OP` collection in the graph by default. * `summary_op`: An op to run and merge the summaries in the graph. Picked from and stored into the `SUMMARY_OP` collection in the graph by default. You can also pass the following additional pieces to the constructor: * `init_feed_dict`: A session feed dictionary that should be used when running the init op. * `init_fn`: A callable to run after the init op to perform additional initializations. The callable will be called as `init_fn(scaffold, session)`. Nc F| t| ts tdd} | || j}| || j}| | j | || j }| || j}| || j}| | | j} | || j}| || j}_r fd_ nd_ |_ |_|_|_|_| _|_|_y)aPCreate a scaffold. Args: init_op: Optional op for initializing variables. init_feed_dict: Optional session feed dictionary to use when running the init_op. init_fn: Optional function to use to initialize the model after running the init_op. Will be called as `init_fn(scaffold, session)`. ready_op: Optional op to verify that the variables are initialized. Must return an empty 1D string tensor when the variables are initialized, or a non-empty 1D string tensor listing the names of the non-initialized variables. ready_for_local_init_op: Optional op to verify that the global variables are initialized and `local_init_op` can be run. Must return an empty 1D string tensor when the global variables are initialized, or a non-empty 1D string tensor listing the names of the non-initialized global variables. local_init_op: Optional op to initialize local variables. summary_op: Optional op to gather all summaries. Must return a scalar string tensor containing a serialized `Summary` proto. saver: Optional `tf.compat.v1.train.Saver` object to use to save and restore variables. May also be a `tf.train.Checkpoint` object, in which case object-based checkpoints are saved. This will also load some object-based checkpoints saved from elsewhere, but that loading may be fragile since it uses fixed keys rather than performing a full graph-based match. For example if a variable has two paths from the `Checkpoint` object because two `Model` objects share the `Layer` object that owns it, removing one `Model` may change the keys and break checkpoint loading through this API, whereas a graph-based match would match the variable through the other `Model`. copy_from_scaffold: Optional scaffold object to copy fields from. Its fields will be overwritten by the provided fields in this function. local_init_feed_dict: Optional session feed dictionary to use when running the local_init_op. Nz.copy_from_scaffold is not a Scaffold instance.c||S|SN)abs \/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/monitored_session.pyz#Scaffold.__init__..s1=aac|Srr)sessinit_fnselfs r!r"z#Scaffold.__init__..s74#6r#) isinstancer TypeErrorinit_opinit_feed_dict _user_init_fnready_opready_for_local_init_op local_init_oplocal_init_feed_dict summary_opr_init_fn_init_op_init_feed_dict _ready_op_ready_for_local_init_op_local_init_op_local_init_feed_dict _summary_op_saver) r'r*r+r&r-r.r/r1rcopy_from_scaffoldr0coalesces ` ` r!__init__zScaffold.__init__ds<\% *H 5HII8h"4"<"<=g 2 A ACn"4"B"BCg($6$?$?@h ( !#5#M#M!O}.@.N.NOm%&:&8&M&MOJ(:(E(EFju0667e!D6dmdmDM)DDN$;D!'D!5D!DDKr#c|j7d}tjdtjj ||_|j 7d}tjdtjj||_|j7d}tjdtjj||_|jBtjdtjjtj|_ |jBtjdtjjtj |_ |j"t%j&|_t)|j"t*j,rHt%j.t1j2|j"j5d |_n|j"j7tj8j;t=j>d |S) z5Creates operations if needed and finalizes the graph.ctjtjt j t j tjdS)Nsaved_model_initializers) r groupr global_variables_initializerr initialize_resourcesshared_resourcesrget_collectionrr#r!default_init_opz*Scaffold.finalize..default_init_opsG%%  2 2 4  * *9+E+E+G H   9 :< .default_ready_ops7  4 4 6  4 4 6!   r#r-ctjtjtjt j t jgdSrH)r rIr rJglobal_variablesr rKrDrr#r!default_ready_for_local_init_opz:Scaffold.finalize..default_ready_for_local_init_opsS  4 4**, .  4 4**, .!  r#r.r/r1T)var_listshardedzGraph was finalized.) r3rget_or_defaultr GraphKeysINIT_OPr5READY_OPr6READY_FOR_LOCAL_INIT_OPr7 LOCAL_INIT_OPdefault_local_init_opr9 SUMMARY_OPr merge_allr:training_saver_get_saver_or_defaultr(trackable_util CheckpointSaverrObjectGraphViewfrozen_saveable_objectsbuildget_default_graphfinalizelogginginfo)r'rFrLrOs r!rdzScaffold.finalizes }}< --i9N9N.=?dm ~~  ..z/2}}/E/E/?Adn $$,'/&=&= #S]]%J%J )'+d# "$33 3==66  ( (*d !00141I1I181B1BDd {{"88:dk$++~889"((--kk224dk  kk$$& LL'( Kr#c|jSr)r2r's r!r&zScaffold.init_fn ==r#c|jSr)r3rhs r!r*zScaffold.init_oprir#c|jSr)r5rhs r!r-zScaffold.ready_ops >>r#c|jSr)r6rhs r!r.z Scaffold.ready_for_local_init_ops  ( ((r#c|jSr)r7rhs r!r/zScaffold.local_init_ops   r#c|jSr)r8rhs r!r0zScaffold.local_init_feed_dicts  % %%r#c|jSr)r9rhs r!r1zScaffold.summary_op s   r#c|jSr)r:rhs r!rzScaffold.savers ;;r#c|jSr)r4rhs r!r+zScaffold.init_feed_dicts   r#ctj|}|r t|dkDr td|||dS|}|tj|||S)z-Get from cache or create a default operation.zMore than one item in the collection "%s". Please indicate which one to use by passing it to the tf.Scaffold constructor as: tf.Scaffold(%s=item to use)r)rrElen RuntimeErroradd_to_collection)arg_namecollection_keydefault_constructorelementsops r!rRzScaffold.get_or_defaultsl!!.1H X  *,:8 E E a[  B ~ NB/ Ir#ctjtjt j t jt jS)aReturns an op that groups the default local init ops. This op is used during session initialization when a Scaffold is initialized without specifying the local_init_op arg. It includes `tf.compat.v1.local_variables_initializer`, `tf.compat.v1.tables_initializer`, and also initializes local session resources. Returns: The default Scaffold local init op. ) r rAr local_variables_initializerr tables_initializerr rClocal_resourcesrr#r!rXzScaffold.default_local_init_op(sH  ! !--/%%'&&y'@'@'BC EEr#) NNNNNNNNNN)__name__ __module__ __qualname____doc__r=rdpropertyr&r*r-r.r/r0r1rr+ staticmethodrRrXrr#r!rr3s-`"'+!"&$(Qf:x       ) )   & &         EEr#rxd Tc *g}|r|j||r|jr|j|t|jjdvr]|j rd|j |j fz}nd}|rtjdd}d} |rtjdnd}| xs|} | r| r~| dkDry|jr'|jtj| | nF|rD|jtjtjj| || |r|dkDs |r|dkDr| r |jr)|jtj |||| nH|rF|jtj |||tjj| | |r|dkDs| r| dkDr|r|j"r*|jtj$|| ||| nI|rG|jtj$tjj||| ||| tj&d ||j)|||| }t+||| S)N)CollectiveAllReduceStrategyCollectiveAllReduceStrategyV1MultiWorkerMirroredStrategyz tmp_%s_%dtmpzCollective ops may deadlock with `save_checkpoints_secs` please use `save_checkpoint_steps` instead. Clearing `save_checkpoint_secs` and setting `save_checkpoint_steps` to 1000 now.izhCollective ops may run out of sync with`save_summaries_secs`, please use `save_summaries_steps` instead.r output_dir every_n_stepsscaffold save_steps save_secsrrrrsave_graph_defz all_hooks %r)configcheckpoint_dir max_wait_secssession_creatorhooksstop_grace_period_secs)extendis_chieftype _strategyr task_typetask_idrewarningshould_save_summaryappendrStepCounterHookospathjoinSummarySaverHookshould_checkpointCheckpointSaverHookrfrMonitoredSession)worker_contextrrrchief_only_hookssave_checkpoint_secssave_summaries_stepssave_summaries_secsrrlog_step_count_stepsrsave_checkpoint_steps summary_dirr all_hookstmpdirrs r!-_create_monitored_session_with_worker_contextr;s )  U.11 %&  (-LLn668N8NOOff oo=> "" oo89F-~+).BQ.F)) ! 1 1$4H JK  ! 1 1k6:0 23 !5!9 3a 7k)) ! 2 2-+$  &'  ! 2 2-+k6:  <= #7!#; $9A$=  ) ) # 7 70.!-  / 0  # 7 7 ^V40.!-  / 0 ,,~y)"22 #! 3#/ % 3 55r#ztrain.MonitoredTrainingSessioncN|tk(r|tk(rd}d}n|tk(rd}n |tk(rd}| tk(r|tk(rd} d}n|tk(rd}n | tk(rd} |xs t}tj}|rt ||||||||| | | | | ||S|s!t ||| | }t ||xsg| Sg}|r|j|t|||| }|xs|}|rc| r+| dkDr&|jtj|| |r|dkDs|r-|dkDr(|jtj|||| |r7|r|dkDs| r.| dkDr)|jtj|| ||| |r|j|t ||| S) aCreates a `MonitoredSession` for training. For a chief, this utility sets proper session initializer/restorer. It also creates hooks related to checkpoint and summary saving. For workers, this utility sets proper session creator which waits for the chief to initialize/restore. Please check `tf.compat.v1.train.MonitoredSession` for more information. @compatibility(TF2) This API is not compatible with eager execution and `tf.function`. To migrate to TF2, rewrite the code to be compatible with eager execution. Check the [migration guide](https://www.tensorflow.org/guide/migrate#1_replace_v1sessionrun_calls) on replacing `Session.run` calls. In Keras, session hooks can be replaced by Callbacks e.g. [logging hook notebook]( https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate/logging_stop_hook.ipynb) For more details please read [Better performance with tf.function](https://www.tensorflow.org/guide/function). @end_compatibility Args: master: `String` the TensorFlow master to use. is_chief: If `True`, it will take care of initialization and recovery the underlying TensorFlow session. If `False`, it will wait on a chief to initialize or recover the TensorFlow session. checkpoint_dir: A string. Optional path to a directory where to restore variables. scaffold: A `Scaffold` used for gathering or building supportive ops. If not specified, a default one is created. It's used to finalize the graph. hooks: Optional list of `SessionRunHook` objects. chief_only_hooks: list of `SessionRunHook` objects. Activate these hooks if `is_chief==True`, ignore otherwise. save_checkpoint_secs: The frequency, in seconds, that a checkpoint is saved using a default checkpoint saver. If both `save_checkpoint_steps` and `save_checkpoint_secs` are set to `None`, then the default checkpoint saver isn't used. If both are provided, then only `save_checkpoint_secs` is used. Default 600. save_summaries_steps: The frequency, in number of global steps, that the summaries are written to disk using a default summary saver. If both `save_summaries_steps` and `save_summaries_secs` are set to `None`, then the default summary saver isn't used. Default 100. save_summaries_secs: The frequency, in secs, that the summaries are written to disk using a default summary saver. If both `save_summaries_steps` and `save_summaries_secs` are set to `None`, then the default summary saver isn't used. Default not enabled. config: an instance of `tf.compat.v1.ConfigProto` proto used to configure the session. It's the `config` argument of constructor of `tf.compat.v1.Session`. stop_grace_period_secs: Number of seconds given to threads to stop after `close()` has been called. log_step_count_steps: The frequency, in number of global steps, that the global step/sec is logged. max_wait_secs: Maximum time workers should wait for the session to become available. This should be kept relatively short to help detect incorrect code, but sometimes may need to be increased if the chief takes a while to start up. save_checkpoint_steps: The frequency, in number of global steps, that a checkpoint is saved using a default checkpoint saver. If both `save_checkpoint_steps` and `save_checkpoint_secs` are set to `None`, then the default checkpoint saver isn't used. If both are provided, then only `save_checkpoint_secs` is used. Default not enabled. summary_dir: A string. Optional path to a directory where to save summaries. If None, checkpoint_dir is used instead. save_graph_def: Whether to save the GraphDef and MetaGraphDef to `checkpoint_dir`. The GraphDef is saved after the session is created as `graph.pbtxt`. MetaGraphDefs are saved out for every checkpoint as `model.ckpt-*.meta`. Returns: A `MonitoredSession` object. rNiX) rrrrrrrrrrrrr)rmasterrrr)rrrrrrrr) USE_DEFAULTrrget_current_worker_contextrWorkerSessionCreatorrrChiefSessionCreatorrrrrr)rrrrrrrrrrrrrrrrrrrs r!MonitoredTrainingSessionrst[(-@K-Ok){*{*k) {* +   #(1LLN. 8%)11/51#3% ''" *# %O 'kr5 77 ) %&'#   / -~+ 4q 8 ! 1 1$4H JK q &9&9A&= ! 2 2-+$  &'q &;&;a&? ! 5 5.,+  -.  U % 3 55r#ztrain.SessionCreatorc4eZdZdZej dZy)SessionCreatorzA factory for tf.Session.c6tdj|)Nz)create_session is not implemented for {}.)NotImplementedErrorformatrhs r!create_sessionzSessionCreator.create_sessionhs 3::4@ BBr#N)rrrrabcabstractmethodrrr#r!rrds!BBr#r) metaclassztrain.ChiefSessionCreatorc.eZdZdZ ddZdZdZy)rz+Creates a tf.compat.v1.Session for a chief.Ncp||_||_|xs t|_d|_||_||_y)aInitializes a chief session creator. Args: scaffold: A `Scaffold` used for gathering or building supportive ops. If not specified a default one is created. It's used to finalize the graph. master: `String` representation of the TensorFlow master to use. config: `ConfigProto` proto used to configure the session. checkpoint_dir: A string. Optional path to a directory where to restore variables. checkpoint_filename_with_path: Full file name path to the checkpoint file. N)_checkpoint_dir_checkpoint_filename_with_pathr _scaffold_session_manager_master_config)r'rrrrcheckpoint_filename_with_paths r!r=zChiefSessionCreator.__init__rs8"*D*GD'+DN DDLDLr#cL|jr |jStj|jj|jj |jj |jjtj|_|jSz!Gets or creates a SessionManager.)r/r0r-r.graph rsmSessionManagerrr/r0r-r.rrcrhs r!_get_session_managerz(ChiefSessionCreator._get_session_managerx   " ""--nn22!^^@@(( $ F F##% 'D   r#c t|jj|jj|j|jj |j |j|j|jj|jj|jjS)N)rrrrr*r+r&) rrdrprepare_sessionrrrrrr*r+r&rhs r!rz"ChiefSessionCreator.create_sessionsNN  $ $ & 6 6 nn""++&*&I&I||&&~~44&& 7 ((r#)NrNNNrrrrr=rrrr#r!rrns%3"-1 0 ! (r#rztrain.WorkerSessionCreatorc,eZdZdZ ddZdZdZy)rz,Creates a tf.compat.v1.Session for a worker.Ncb|xs t|_d|_||_||_||_y)aInitializes a worker session creator. Args: scaffold: A `Scaffold` used for gathering or building supportive ops. If not specified a default one is created. It's used to finalize the graph. master: `String` representation of the TensorFlow master to use. config: `ConfigProto` proto used to configure the session. max_wait_secs: Maximum time to wait for the session to become available. N)rrrrr_max_wait_secs)r'rrrrs r!r=zWorkerSessionCreator.__init__s0+DN DDLDL'Dr#cL|jr |jStj|jj|jj |jj |jjtj|_|jSrrrhs r!rz)WorkerSessionCreator._get_session_managerrr#c|jj|jj|j|j |j S)N)rr)rrdrwait_for_sessionrrrrhs r!rz#WorkerSessionCreator.create_sessionsJNN  $ $ & 7 7 T\\9L9L 8 NNr#)NrNirrr#r!rrs#4$ (( !Nr#rceZdZdZ ddZedZddZdZGddZ d Z d Z d Z d Z Gd deZddZdZdZy)_MonitoredSessionz5See `MonitoredSession` or `SingularMonitoredSession`.ctjj|_|xsg|_|jD]}|j t j}|s|r|j}|j|xs t|j||_ |rt|j|_ y|jj|_ y)aSets up a Monitored or Hooked Session. Args: session_creator: A factory object to create session. Typically a `ChiefSessionCreator` or a `WorkerSessionCreator`. hooks: An iterable of `SessionRunHook' objects. should_recover: A bool. Indicates whether to recover from `AbortedError` and `UnavailableError` or not. stop_grace_period_secs: Number of seconds given to threads to stop after `close()` has been called. rN)rrc finalized_graph_was_finalized_hooksbeginrrr_CoordinatedSessionCreatorr_coordinated_creator_RecoverableSession_sessr)r'rrshould_recoverrhrs r!r=z_MonitoredSession.__init__s !$ 5 5 7 A AD+2DK [[ggi4NNPN ~&668o!% ? ?'@+>+@kk5!@!7D&t'@'@Adj,,;;=djr#cX|jy|jjS)z,The graph that was launched in this session.N)_tf_sessrrhs r!rz_MonitoredSession.graphs% }}  ==?  r#Nc@|jj||||S)abRun ops in the monitored session. This method is completely compatible with the `tf.Session.run()` method. Args: fetches: Same as `tf.Session.run()`. feed_dict: Same as `tf.Session.run()`. options: Same as `tf.Session.run()`. run_metadata: Same as `tf.Session.run()`. Returns: Same as `tf.Session.run()`.  feed_dictoptions run_metadatarrun)r'fetchesrrrs r!rz_MonitoredSession.runs* ::>>!  ##r#ctj|}|dk7r|dk7rtdj||jj ||j dS)aRun ops using a step function. Args: step_fn: A function or a method with a single argument of type `StepContext`. The function may use methods of the argument to perform computations with access to a raw session. The returned value of the `step_fn` will be returned from `run_step_fn`, unless a stop is requested. In that case, the next `should_stop` call will return True. Example usage: ```python with tf.Graph().as_default(): c = tf.compat.v1.placeholder(dtypes.float32) v = tf.add(c, 4.0) w = tf.add(c, 0.5) def step_fn(step_context): a = step_context.session.run(fetches=v, feed_dict={c: 0.5}) if a <= 4.5: step_context.request_stop() return step_context.run_with_hooks(fetches=w, feed_dict={c: 0.1}) with tf.MonitoredSession() as session: while not session.should_stop(): a = session.run_step_fn(step_fn) ``` Hooks interact with the `run_with_hooks()` call inside the `step_fn` as they do with a `MonitoredSession.run` call. Returns: Returns the returned value of `step_fn`. Raises: StopIteration: if `step_fn` has called `request_stop()`. It may be caught by `with tf.MonitoredSession()` to close the session. ValueError: if `step_fn` doesn't have a single argument called `step_context`. It may also optionally have `self` for cases when it belongs to an object. ) step_context)r'rz`step_fn` may either have one `step_context` argument, or `self` and `step_context` arguments if it's an instance method. Got {} instead.N)run_with_hooks)rfn_args ValueErrorrr run_step_fnr)r'step_fnstep_fn_argumentss r!rz_MonitoredSession.run_step_fnsrN'..w7--2CH3  %%+V,=%> @@ :: ! !'4==?4 ! PPr#c2eZdZdZdZedZdZdZy)_MonitoredSession.StepContextaCControl flow instrument for the `step_fn` from `run_step_fn()`. Users of `step_fn` may perform `run()` calls without running hooks by accessing the `session`. A `run()` call with hooks may be performed using `run_with_hooks()`. Computation flow can be interrupted using `request_stop()`. c ||_||_y)zInitializes the `step_context` argument for a `step_fn` invocation. Args: session: An instance of `tf.compat.v1.Session`. run_with_hooks_fn: A function for running fetches and hooks. N)_session_run_with_hooks_fn)r'sessionrun_with_hooks_fns r!r=z&_MonitoredSession.StepContext.__init__Qsdm 1dr#c|jSr)rrhs r!r z%_MonitoredSession.StepContext.session[s ]]r#c&|j|i|S)z;Same as `MonitoredSession.run`. Accepts the same arguments.)rr'argskwargss r!rz,_MonitoredSession.StepContext.run_with_hooks_s $T $ $d 5f 55r#ctd)zExit the training loop by causing `should_stop()` to return `True`. Causes `step_fn` to exit by raising an exception. Raises: StopIteration z-step_fn has requested the iterations to stop.) StopIterationrhs r! request_stopz*_MonitoredSession.StepContext.request_stopcs I JJr#N) rrrrr=rr rrrr#r! StepContextrHs+26Kr#rcV|jduxs|jjSr)r should_stoprhs r!rz_MonitoredSession.should_stopms# ::  9!7!7!99r#c$|jyr)_close_internalrhs r!closez_MonitoredSession.closepsr#c|Srrrhs r! __enter__z_MonitoredSession.__enter__ss Kr#c`|tjtfvrd}|j||duSr)rOutOfRangeErrorrr)r'exception_typeexception_value tracebacks r!__exit__z_MonitoredSession.__exit__vs3&00-@@n( T !!r#ceZdZdZdZdZy),_MonitoredSession._CoordinatedSessionCreatorz Factory for _CoordinatedSession.cJ||_||_d|_d|_||_yr)_session_creatorrcoordtf_sess_stop_grace_period_secs)r'rrrs r!r=z5_MonitoredSession._CoordinatedSessionCreator.__init__s'-ddkdjdl%;d"r#c|jj|_tjg|_t jt jjr+tj|j|j |jD](}|j|j|j *tt|j|j|j |j S)zCreates a coordinated session.)clean_stop_exception_types)r%r%)r$rr&r Coordinatorr%rrErS QUEUE_RUNNERSrstart_queue_runnersrafter_create_session_CoordinatedSession_HookedSessionr')r'hooks r!rz;_MonitoredSession._CoordinatedSessionCreator.create_sessions**99;dl**bIdj  CMM77 8((dll$**M++<$ !!$,, ;< t{{ 3TZZ  & &((r#N)rrrrr=rrr#r!rr"}s*< (r#rc |s6|jD]'}|j|jj) |j t d|jj d|_d|j_d|j_|js#tjjyy#d|_d|j_d|j_|js#tjjwwxYw# |j t d|jj d|_d|j_d|j_|js#tjjww#d|_d|j_d|j_|js#tjjwwxYwxYw)NzSession is already closed.) rendrr&rrurr%rrrc_unsafe_unfinalize)r'rrs r!rz!_MonitoredSession._close_internals7  3A %%))11 2 3 7 :: 9: :  ,0!!)*.!!'((   ! 4 4 6) ,0!!)*.!!'((   ! 4 4 6) 7 :: 9: :  ,0!!)*.!!'((   ! 4 4 6) ,0!!)*.!!'((   ! 4 4 6)s18D"1CAD"H $1F.AH .AHH c2|jjduS)zhReturn True if the monitored session is closed. For tests only. Returns: A boolean. Nrr&rhs r! _is_closedz_MonitoredSession._is_closeds  $ $ , , 44r#c.|jjS)zReturn underlying tf.compat.v1.Session object. Warning: accessing the returned object in user code is likely to cause races or "flaky tests". Returns: A tf.compat.v1.Session object. r5rhs r!rz_MonitoredSession._tf_sesss  $ $ , ,,r#rNNNr)rrrrr=rrrrrrrrr rrrr6rrr#r!rrsl= '* !>F ! ! #(6Qp#K#KJ:"(>(27"5 -r#rztrain.MonitoredSessionc*eZdZdZ dfd ZxZS)ra Session-like object that handles initialization, recovery and hooks. Example usage: ```python saver_hook = CheckpointSaverHook(...) summary_hook = SummarySaverHook(...) with MonitoredSession(session_creator=ChiefSessionCreator(...), hooks=[saver_hook, summary_hook]) as sess: while not sess.should_stop(): sess.run(train_op) ``` Initialization: At creation time the monitored session does following things in given order: * calls `hook.begin()` for each given hook * finalizes the graph via `scaffold.finalize()` * create session * initializes the model via initialization ops provided by `Scaffold` * restores variables if a checkpoint exists * launches queue runners * calls `hook.after_create_session()` Run: When `run()` is called, the monitored session does following things: * calls `hook.before_run()` * calls TensorFlow `session.run()` with merged fetches and feed_dict * calls `hook.after_run()` * returns result of `session.run()` asked by user * if `AbortedError` or `UnavailableError` occurs, it recovers or reinitializes the session before executing the run() call again Exit: At the `close()`, the monitored session does following things in order: * calls `hook.end()` * closes the queue runners and the session * suppresses `OutOfRange` error which indicates that all inputs have been processed if the monitored_session is used as a context How to set `tf.compat.v1.Session` arguments: * In most cases you can set session arguments as follows: ```python MonitoredSession( session_creator=ChiefSessionCreator(master=..., config=...)) ``` * In distributed setting for a non-chief worker, you can use following: ```python MonitoredSession( session_creator=WorkerSessionCreator(master=..., config=...)) ``` See `MonitoredTrainingSession` for an example usage based on chief or worker. Note: This is not a `tf.compat.v1.Session`. For example, it cannot do following: * it cannot be set as default session. * it cannot be sent to saver.save. * it cannot be sent to tf.train.start_queue_runners. @compatibility(TF2) This API is not compatible with eager execution and `tf.function`. To migrate to TF2, rewrite the code to be compatible with eager execution. Check the [migration guide](https://www.tensorflow.org/guide/migrate#1_replace_v1sessionrun_calls) on replacing `Session.run` calls. In Keras, session hooks can be replaced by Callbacks e.g. [logging hook notebook]( https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate/logging_stop_hook.ipynb) For more details please read [Better performance with tf.function](https://www.tensorflow.org/guide/function). @end_compatibility Args: session_creator: A factory object to create session. Typically a `ChiefSessionCreator` which is the default one. hooks: An iterable of `SessionRunHook' objects. Returns: A MonitoredSession object. c4tt| ||d|y)NTrr)superrr=)r'rrr __class__s r!r=zMonitoredSession.__init__s& D* 5 +7r#)NNr)rrrrr= __classcell__r>s@r!rrsUp $&)77r#rztrain.SingularMonitoredSessionc8eZdZdZ dfd ZdZxZS)SingularMonitoredSessionaO Session-like object that handles initialization, restoring, and hooks. Please note that this utility is not recommended for distributed settings. For distributed settings, please use `tf.compat.v1.train.MonitoredSession`. The differences between `MonitoredSession` and `SingularMonitoredSession` are: * `MonitoredSession` handles `AbortedError` and `UnavailableError` for distributed settings, but `SingularMonitoredSession` does not. * `MonitoredSession` can be created in `chief` or `worker` modes. `SingularMonitoredSession` is always created as `chief`. * You can access the raw `tf.compat.v1.Session` object used by `SingularMonitoredSession`, whereas in MonitoredSession the raw session is private. This can be used: - To `run` without hooks. - To save and restore. * All other functionality is identical. Example usage: ```python saver_hook = CheckpointSaverHook(...) summary_hook = SummarySaverHook(...) with SingularMonitoredSession(hooks=[saver_hook, summary_hook]) as sess: while not sess.should_stop(): sess.run(train_op) ``` Initialization: At creation time the hooked session does following things in given order: * calls `hook.begin()` for each given hook * finalizes the graph via `scaffold.finalize()` * create session * initializes the model via initialization ops provided by `Scaffold` * restores variables if a checkpoint exists * launches queue runners Run: When `run()` is called, the hooked session does following things: * calls `hook.before_run()` * calls TensorFlow `session.run()` with merged fetches and feed_dict * calls `hook.after_run()` * returns result of `session.run()` asked by user Exit: At the `close()`, the hooked session does following things in order: * calls `hook.end()` * closes the queue runners and the session * suppresses `OutOfRange` error which indicates that all inputs have been processed if the `SingularMonitoredSession` is used as a context. @compatibility(TF2) This API is not compatible with eager execution and `tf.function`. To migrate to TF2, rewrite the code to be compatible with eager execution. Check the [migration guide](https://www.tensorflow.org/guide/migrate#1_replace_v1sessionrun_calls) on replacing `Session.run` calls. In Keras, session hooks can be replaced by Callbacks e.g. [logging hook notebook]( https://github.com/tensorflow/docs/blob/master/site/en/guide/migrate/logging_stop_hook.ipynb) For more details please read [Better performance with tf.function](https://www.tensorflow.org/guide/function). @end_compatibility cTt|||||}tt|||d|y)aCreates a SingularMonitoredSession. Args: hooks: An iterable of `SessionRunHook' objects. scaffold: A `Scaffold` used for gathering or building supportive ops. If not specified a default one is created. It's used to finalize the graph. master: `String` representation of the TensorFlow master to use. config: `ConfigProto` proto used to configure the session. checkpoint_dir: A string. Optional path to a directory where to restore variables. stop_grace_period_secs: Number of seconds given to threads to stop after `close()` has been called. checkpoint_filename_with_path: A string. Optional path to a checkpoint file from which to restore variables. )rrrrrFr<N)rr=rBr=) r'rrrrrrrrr>s r!r=z!SingularMonitoredSession.__init__csB.*%&C EO  "D2 5 37r#c"|jS)z/Returns underlying `TensorFlow.Session` object.)rrhs r! raw_sessionz$SingularMonitoredSession.raw_sessions ==?r#)NNrNNrN)rrrrr=rEr?r@s@r!rBrB!s+>B"&)-1!7Fr#rBcTeZdZdZdZedZedZdZdZ dZ dZ d Z y ) _WrappedSessionapWrapper around a `tf.compat.v1.Session`. This wrapper is used as a base class for various session wrappers that provide additional functionality such as monitoring, coordination, and recovery. In addition to the methods exported by `SessionInterface` the wrapper provides a method to check for stop and never raises exceptions from calls to `close()`. cP||_t|jt|_y)zCreates a `_WrappedSession`. Args: sess: A `tf.compat.v1.Session` or `_WrappedSession` object. The wrapped session. N)rr(rG_wrapped_is_stoppable)r'r%s r!r=z_WrappedSession.__init__sDJ!+DJJ!HDr#c.|jjSr)rrrhs r!rz_WrappedSession.graphs ::  r#c.|jjSr)rsess_strrhs r!rLz_WrappedSession.sess_strs ::  r#c|jry|jr(|jxr|jjSy)zReturn true if this session should not be used anymore. Always return True if the session was closed. Returns: True if the session should stop, False otherwise. T) _check_stoprrIrrhs r!rz_WrappedSession.should_stops;   zz  ' ' DDJJ,B,B,DD r#cy)zHook for subclasses to provide their own stop condition. Returns: True if the session should stop, False otherwise. Frrhs r!rNz_WrappedSession._check_stops r#c|jr# |jjd|_yy#t$r }tjd|Yd}~-d}~wwxYw#d|_wxYw)NzAn error occurred when attempting to close the session. This may be due to a preemption in a connected worker or parameter server. Error: %s)rr_PREEMPTION_ERRORSreerrorr'es r!rz_WrappedSession.closese zz   B  >?@ B BB  s&1 AAAAA A&c:|jj|i|Srrr s r!rz_WrappedSession.runs 4::>>4 *6 **r#cZ|xs |j}|tj||Sr)rrr)r'rrErs r!rz_WrappedSession.run_step_fns+$/txxN $00nM NNr#N) rrrrr=rrrLrrNrrrrr#r!rGrGsO I      +Or#rGc0eZdZdZdZdZdZddZdZy) raA wrapped session that recreates a session upon certain kinds of errors. The constructor is passed a SessionCreator object, not a session. Calls to `run()` are delegated to the wrapped session. If a call raises the exception `tf.errors.AbortedError` or `tf.errors.UnavailableError`, the wrapped session is closed, and a new one is created by calling the factory again. cZ||_tj||jy)zCreate a new `_RecoverableSession`. The value returned by calling `sess_creator.create_session()` will be the session wrapped by this recoverable session. Args: sess_creator: A 'SessionCreator' to be wrapped by recoverable. N) _sess_creatorrGr=_create_session)r' sess_creators r!r=z_RecoverableSession.__init__s%&DT4#7#7#9:r#c |jjS#t$r }tjd|Yd}~nd}~wwxYwH)NaAn error was raised while a session was being created. This may be due to a preemption of a connected worker or parameter server. A new session will be created. This error may also occur due to a gRPC failure caused by high memory or network bandwidth usage in the parameter servers. If this error occurs repeatedly, try increasing the number of parameter servers assigned to the job. Error: %s)rYrrQrerfrSs r!rZz#_RecoverableSession._create_sessionsO  %!!0022  %  !#$ % % % s AAAc |jr|jjSy#t$rE}tjd||j |j |_Yd}~yd}~wt$rYywxYw)NTaAn error was raised while considering whether the session is complete. This may be due to a preemption in a connected worker or parameter server. The current session will be closed and a new session will be created. This error may also occur due to a gRPC failure caused by high memory or network bandwidth usage in the parameter servers. If this error occurs repeatedly, try increasing the number of parameter servers assigned to the job. Error: %sF)rrNrQrerfrrZ ExceptionrSs r!rNz_RecoverableSession._check_stopsu zz%%''  ll !" # jjl'')dj  s%) B;A22 BBNc |js|j|_|jj||||S#t$r7}t j d||j d|_Yd}~nd}~wwxYw)NrAn error was raised. This may be due to a preemption in a connected worker or parameter server. The current session will be closed and a new session will be created. This error may also occur due to a gRPC failure caused by high memory or network bandwidth usage in the parameter servers. If this error occurs repeatedly, try increasing the number of parameter servers assigned to the job. Error: %s)rrZrrQrerfr)r'rrrrrTs r!rz_RecoverableSession.runs zz++-$*zz~~ % ' '    !#$ %    s?A B -A>>Bc6 |js|j|_|jj}|jj|||S#t$r7}t j d||jd|_Yd}~nd}~wwxYw)Nr`)rrZrrrQrerfr)r'rrErrTs r!rz_RecoverableSession.run_step_fn-s zz++-$*zz%%g{NKK    !#$ %    sAA B -BBr9) rrrrr=rZrNrrrr#r!rrs  ; %4.r#rc*eZdZdZddZdZdZdZy)r.aA wrapped session that works with a `tf.Coordinator`. Calls to `run()` are delegated to the wrapped session. If a call raises an exception, the exception is reported to the coordinator. In addition, after each call to `run()` this session ask the coordinator if the session should stop. In that case it will join all the threads registered with the coordinator before returning. If the coordinator was requested to stop with an exception, that exception will be re-raised from the call to `run()`. cLtj||||_||_y)aCreate a new `_CoordinatedSession`. Args: sess: A `tf.compat.v1.Session` object. The wrapped session. coord: A `tf.train.Coordinator` object. stop_grace_period_secs: Number of seconds given to threads to stop after `close()` has been called. N)rGr=_coordr')r'r%r%rs r!r=z_CoordinatedSession.__init__Qs#T4(DK#9D r#cj|jj|jjSr)rdraise_requested_exceptionrrhs r!rNz_CoordinatedSession._check_stop^s' KK))+ ;; " " $$r#c&|jj |jj|jd tj |y#t $rYywxYw# tj |w#t $rYwwxYwxYw)NT)rignore_live_threads)rdrrr'rGrr^rhs r!rz_CoordinatedSession.closeesKK   kk!%!=!="$ d#     d#    sA'A)A A&%A&)B+BB B  B B  Bc |jj|i|S#t$rt$r:} |jj #t$rt$r|dwxYwd}~wwxYwr)rrrQr^rdrf)r'rroriginal_exceptions r!rz_CoordinatedSession.runssy TZZ^^T ,V ,,     --/    + d*+ s&A*A  A% A""A%%A*Nr8)rrrrr=rNrrrr#r!r.r.Cs  :%  r#r.c6eZdZdZdZdZd dZdZdZdZ y) r/aA _WrappedSession that calls hooks during calls to run(). The list of hooks to call is passed in the constructor. Before each call to `run()` the session calls the `before_run()` method of the hooks, which can return additional ops or tensors to run. These are added to the arguments of the call to `run()`. When the `run()` call finishes, the session calls the `after_run()` methods of the hooks, passing the values returned by the `run()` call corresponding to the ops and tensors that each hook requested. If any call to the hooks, requests stop via run_context the session will be marked as needing to stop and its `should_stop()` method will now return `True`. cLtj||||_d|_y)zInitializes a _HookedSession object. Args: sess: A `tf.compat.v1.Session` or a `_WrappedSession` object. hooks: An iterable of `SessionRunHook' objects. FN)rGr=r _should_stop)r'r%rs r!r=z_HookedSession.__init__s#T4(DKDr#c|jS)See base class.)rmrhs r!rNz_HookedSession._check_stops   r#Nc 2|jr tdd|i}tjtj|||j }|xst j}|j||||}|xst j}tj|||||}|jD]3}|j|tj||vr||nd||5|jxs |j |_|dS)roz,Run called even after should_stop requested.caller) original_argsr )rrrrN)resultsrr)rrurSessionRunContextSessionRunArgsrr RunOptions_call_hook_before_run RunMetadatarGrr after_runSessionRunValuesrmstop_requested) r'rrrractual_fetches run_contextoutputsr0s r!rz_HookedSession.runs  G HH(N"44&55gyI K0..0G**;+4g?I ;:#9#9#;L!! ! "#G * nn   + +'+wgdmD' )**))G[-G-GD 8 r#ci}|jD]}|j|}||j|j||<|jr8|j ||jd|j |j|j s|j||j |s|S|s|S|j ||d|j ||S)z7Calls hooks.before_run and handles requests from hooks.z Same tensor is fed by two hooks.z0Same tensor is fed by a SessionRunHook and user.)r before_runrr_raise_if_feeds_intersectsupdater_merge_run_options)r'r} fetch_dictuser_feed_dictr hook_feedsr0requests r!rwz$_HookedSession._call_hook_before_runsJ < ,g   ?? &$__*T      ) )*g6G6G*L N   G-- . ??  ! !'7?? ; <    ## :<n% r#ct|jt|jz}|r#t|dztt |zy)Nz Conflict(s): )setkeysrustrlist)r'feeds1feeds2message intersections r!rz)_HookedSession._raise_if_feeds_intersectssJv{{}%FKKM(::L #33c$|:L6MM NNr#cVt|j|j|_t|j|j|_t|j|j|_t|j|j|_|j j j|j j |j jxs|j j|j _|jxs |j|_ y)aMerge two instances of RunOptions into the first one. During the merger, the numerical fields including trace_level, timeout_in_ms, inter_op_thread_pool are set to the larger one of the two. The boolean value is set to the logical OR of the two. debug_tensor_watch_opts of the original options is extended with that from the incoming one. Args: options: The options to merge into. incoming_options: The options to be merged into the first argument. N) max trace_level timeout_in_msinter_op_thread_pooloutput_partition_graphs debug_optionsdebug_tensor_watch_optsrreset_disk_byte_usage"report_tensor_allocations_upon_oom)r'rincoming_optionss r!rz!_HookedSession._merge_run_optionssg113C3O3OPG 5 5 0 > >@G#&w'C'C'7'L'L$NG &)''00'2G# 1188&&>>@ 33 =&&<< / 22 <;; .r#r9) rrrrr=rNrrwrrrr#r!r/r/s' !F8O =r#r/) NNNNNNNrrrNNT)9rrrtensorflow.core.protobufrtensorflow.python.checkpointrr]rtensorflow.python.distributertensorflow.python.frameworkrrtensorflow.python.opsr r r r r tensorflow.python.platformrretensorflow.python.summaryrtensorflow.python.trainingrrrrr[rrrtensorflow.python.utilr tensorflow.python.util.tf_exportr AbortedErrorUnavailableErrorrQobjectrrrrABCMetarrrrrrBrGrr.r/rr#r!rs, 1 /E3G.++2,++<->23><716))6+B+BCh   !DEDE"DET  m5` /01   $$# %!u52u5p %&'Bs{{B(B *+,2(.2(-2(j +,-'N>'N.'NTk-k-\ '()`7(`7*`7F /01f0f2fRHOHOVj/jZ@/@FA=_A=r#