BVhSFdZddlZddlZddlZddlZddlmZddlmZ ddl m Z ddl m Z e dGdd Ze d g Gd d ejZy)z9Coordinator to help multiple threads stop when requested.N)errors) tf_logging)compat) tf_exportztrain.CoordinatorceZdZdZddZdZddZdZdZe jdZ dd Z d Z dd Zed Zd Zy) Coordinatora A coordinator for threads. This class implements a simple mechanism to coordinate the termination of a set of threads. #### Usage: ```python # Create a coordinator. coord = Coordinator() # Start a number of threads, passing the coordinator to each of them. ...start thread 1...(coord, ...) ...start thread N...(coord, ...) # Wait for all the threads to terminate. coord.join(threads) ``` Any of the threads can call `coord.request_stop()` to ask for all the threads to stop. To cooperate with the requests, each thread must check for `coord.should_stop()` on a regular basis. `coord.should_stop()` returns `True` as soon as `coord.request_stop()` has been called. A typical thread running with a coordinator will do something like: ```python while not coord.should_stop(): ...do some work... ``` #### Exception handling: A thread can report an exception to the coordinator as part of the `request_stop()` call. The exception will be re-raised from the `coord.join()` call. Thread code: ```python try: while not coord.should_stop(): ...do some work... except Exception as e: coord.request_stop(e) ``` Main code: ```python try: ... coord = Coordinator() # Start a number of threads, passing the coordinator to each of them. ...start thread 1...(coord, ...) ...start thread N...(coord, ...) # Wait for all the threads to terminate. coord.join(threads) except Exception as e: ...exception that was passed to coord.request_stop() ``` To simplify the thread implementation, the Coordinator provides a context handler `stop_on_exception()` that automatically requests a stop if an exception is raised. Using the context handler the thread code above can be written as: ```python with coord.stop_on_exception(): while not coord.should_stop(): ...do some work... ``` #### Grace period for stopping: After a thread has called `coord.request_stop()` the other threads have a fixed time to stop, this is called the 'stop grace period' and defaults to 2 minutes. If any of the threads is still alive after the grace period expires `coord.join()` raises a RuntimeError reporting the laggards. ```python try: ... coord = Coordinator() # Start a number of threads, passing the coordinator to each of them. ...start thread 1...(coord, ...) ...start thread N...(coord, ...) # Wait for all the threads to terminate, give them 10s grace period coord.join(threads, stop_grace_period_secs=10) except RuntimeError: ...one of the threads took more than 10s to stop after request_stop() ...was called. except Exception: ...exception that was passed to coord.request_stop() ``` Nc|tjf}t||_t j |_t j|_d|_ d|_ t|_ y)a%Create a new Coordinator. Args: clean_stop_exception_types: Optional tuple of Exception types that should cause a clean stop of the coordinator. If an exception of one of these types is reported to `request_stop(ex)` the coordinator will behave as if `request_stop(None)` was called. Defaults to `(tf.errors.OutOfRangeError,)` which is used by input queues to signal the end of input. When feeding training data from a Python iterator it is common to add `StopIteration` to this list. NF) rOutOfRangeErrortuple_clean_stop_exception_types threadingLock_lockEvent _stop_event_exc_info_to_raise_joinedset_registered_threads)selfclean_stop_exception_typess V/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/coordinator.py__init__zCoordinator.__init__|sa")$*$:$:#< ',-G'HD$!DJ (D#DDL #uDcft|tr|d}n|}t||jrd}|S)aCheck if the exception indicated in 'ex' should be ignored. This method examines `ex` to check if it is an exception that should be reported to the users. If yes, it returns `ex` as is, otherwise it returns None. The code returns None for exception types listed in `_clean_stop_exception_types`. Args: ex: None, an `Exception`, or a Python `exc_info` tuple as returned by `sys.exc_info()`. Returns: ex or None. N) isinstancer r )rexex2s r_filter_exceptionzCoordinator._filter_exceptions6""e qEc c#t778 b Irc|j5|j|}|jr4t|tr|\}}}||t j \}}}||jjs |r|jt|tr6tjdtj|d|||_ nLtjdt|tj|t j |_ t|jdk7s|jdr|jds t!d|jz|jj#dddy#t $rt j |_ YGwxYw#1swYyxYw) a Request that the threads stop. After this is called, calls to `should_stop()` will return `True`. Note: If an exception is being passed in, in must be in the context of handling the exception (i.e. `try: ... except Exception as ex: ...`) and not a newly created one. Args: ex: Optional `Exception`, or Python `exc_info` tuple as returned by `sys.exc_info()`. If this is the first call to `request_stop()` the corresponding exception is recorded and re-raised from `join()`. Nz!Error reported to Coordinator: %sr)exc_infoz%Error reported to Coordinator: %s, %srzHex must be a tuple or sys.exc_info must return the current exception: %s)rr rrr sysr"ris_setrlogginginfor as_str_anytypelen ValueErrorr)rr_ ex_instances r request_stopzCoordinator.request_stops -  ! !" %b  b%  ![!  ^"lln ![!     $ $ & $))1 E " LL<**2a51"$ &')D # LL@b**2. 0'*llnD #$))*a/))!,))!, 7"++,-- [--N 7), d%  7O--s0EF4F )F4 "F1.F40F11F44F=c|j5d|_d|_|jj r|jj dddy#1swYyxYw)zcClears the stop flag. After this is called, calls to `should_stop()` will return `False`. FN)rrrrr%clearrs r clear_stopzCoordinator.clear_stopsS !dl $d   "  !!!s AAA"c6|jjS)zSCheck if stop was requested. Returns: True if a stop was requested. )rr%r1s r should_stopzCoordinator.should_stops    " " $$rc#lK dy#|jtjYyxYww)aContext manager to request stop when an Exception is raised. Code that uses a coordinator must catch exceptions and pass them to the `request_stop()` method to stop the other threads managed by the coordinator. This context handler simplifies the exception handling. Use it as follows: ```python with coord.stop_on_exception(): # Any exception raised in the body of the with # clause is reported to the coordinator before terminating # the execution of the body. ...body... ``` This is completely equivalent to the slightly longer code: ```python try: ...body... except: coord.request_stop(sys.exc_info()) ``` Yields: nothing. N)r)r.r$r"r1s rstop_on_exceptionzCoordinator.stop_on_exceptions+>+ + 3<<>*s4 4&14c8|jj|S)zWait till the Coordinator is told to stop. Args: timeout: Float. Sleep for up to that many seconds waiting for should_stop() to become True. Returns: True if the Coordinator is told stop, False if the timeout expired. )rwait)rtimeouts r wait_for_stopzCoordinator.wait_for_stop)s     ))rc||j5|jj|dddy#1swYyxYw)zQRegister a thread to join. Args: thread: A Python thread to join. N)rradd)rthreads rregister_threadzCoordinator.register_thread5s4 + ""6*+++s2;c|j5| |j}n$|jjt|}t |}dddt d|Dr6|j ds% t d|Dr|j ds%d}t d|DrR|dk\rMtj|||z}d|z}tt||d}t d|Dr|dk\rM|Dcgc]}|js|j!}}|j5d|_ t|_|jr|j\}}}||rE|r&tj d d j#|nt%d d j#|zdddy#1swYsxYwcc}w#1swYyxYw) aWait for threads to terminate. This call blocks until a set of threads have terminated. The set of thread is the union of the threads passed in the `threads` argument and the list of threads that registered with the coordinator by calling `Coordinator.register_thread()`. After the threads stop, if an `exc_info` was passed to `request_stop`, that exception is re-raised. Grace period handling: When `request_stop()` is called, threads are given 'stop_grace_period_secs' seconds to terminate. If any of them is still alive after that period expires, a `RuntimeError` is raised. Note that if an `exc_info` was passed to `request_stop()` then it is raised instead of that `RuntimeError`. Args: threads: List of `threading.Threads`. The started threads to join in addition to the registered threads. stop_grace_period_secs: Number of seconds given to threads to stop after `request_stop()` has been called. ignore_live_threads: If `False`, raises an error if any of the threads are still alive after `stop_grace_period_secs`. Raises: RuntimeError: If any thread is still alive after `request_stop()` is called and the grace period expires. Nc3<K|]}|jywNis_alive.0ts r z#Coordinator.join..g ,qajjl ,g?gMbP?c3<K|]}|jywrArBrDs rrGz#Coordinator.join..prHrIgTz2Coordinator stopped with threads still running: %s )rrunionrlistanyr:timesleepmaxminrCnamerrr&r'join RuntimeError) rthreadsstop_grace_period_secsignore_live_threadsstop_wait_secsrF stragglersr,r-s rrUzCoordinator.join>s>  ****00W>W g  ,G , ,T5G5G5L   ,G , ,T5G5G5LN ,G , ,1G31N jj .>)n3~/EFNn  ,G , ,1G31N#*:QQZZ\!&&:J:  $dl!$d  33;   ,,Kxx + -Bhhz"#$ $ $ $A:; $ $s$?F=G G 8ArUpropertyr]r_rrrrsp]~%>4;z !%!+!+F *+7:$L$\  rrztrain.LooperThread)v1cReZdZdZ dfd Zed dZdZdZdZ dZ xZ S) LooperThreada}A thread that runs code repeatedly, optionally on a timer. This thread class is intended to be used with a `Coordinator`. It repeatedly runs code specified either as `target` and `args` or by the `run_loop()` method. Before each run the thread checks if the coordinator has requested stop. In that case the looper thread terminates immediately. If the code being run raises an exception, that exception is reported to the coordinator and the thread terminates. The coordinator will then request all the other threads it coordinates to stop. You typically pass looper threads to the supervisor `Join()` method. c8t|tstd|ztt|d|_||_||_||_ |jr|xsd|_ |xsi|_ n|s|r td|jj|y)aCreate a LooperThread. Args: coord: A Coordinator. timer_interval_secs: Time boundaries at which to call Run(), or None if it should be called back to back. target: Optional callable object that will be executed in the thread. args: Optional arguments to pass to `target` when calling it. kwargs: Optional keyword arguments to pass to `target` when calling it. Raises: ValueError: If one of the arguments is invalid. z*'coord' argument must be a Coordinator: %sTrhz@'args' and 'kwargs' argument require that you also pass 'target'N) rrr+superrkrdaemon_coord_timer_interval_secs_target_args_kwargsr>)rcoordtimer_interval_secstargetargskwargs __class__s rrzLooperThread.__init__s e[ ) CeK LL ,&(DKDK 3DDL ||:2dj\rdl  ' ((KK%rcFt|||||}|j|S)axStart a LooperThread that calls a function periodically. If `timer_interval_secs` is None the thread calls `target(args)` repeatedly. Otherwise `target(args)` is called every `timer_interval_secs` seconds. The thread terminates when a stop of the coordinator is requested. Args: coord: A Coordinator. timer_interval_secs: Number. Time boundaries at which to call `target`. target: A callable object. args: Optional arguments to pass to `target` when calling it. kwargs: Optional keyword arguments to pass to `target` when calling it. Returns: The started thread. )rvrwrx)rkstart)rtrurvrwrxloopers rloopzLooperThread.loops(&%!4V$!')F LLN Mrcn|jj5|j|jF|jj s|j |jj s+nt j }|jj|t j z sP||jz }|j |jj|t j z sP|jdddy#1swYyxYwrA) ror6 start_looprpr4run_looprPr: stop_loop)rnext_timer_times rrunzLooperThread.runs  & & (  oo " " *++))+ --/++))+))+++++Odiik,IJ T66 6/ --/++++Odiik,IJ nn   sA!D+=BD+D++D4cy)zCalled when the thread starts.Nrhr1s rrzLooperThread.start_looprcy)zCalled when the thread stops.Nrhr1s rrzLooperThread.stop_looprrcj|jr'|j|ji|jyy)z+Called at 'timer_interval_secs' boundaries.N)rqrrrsr1s rrzLooperThread.run_loops) ||dllDJJ/$,,/r)NNN)NN) rarbrcrdr staticmethodr}rrrr __classcell__)rys@rrkrks? DH&<.   0rrk)rdrer$r rPtensorflow.python.frameworkrtensorflow.python.platformrr&tensorflow.python.utilr tensorflow.python.util.tf_exportrrThreadrkrhrrrss@  .<)6 yy yz  #$%a09##a0&a0r