BVh4dZddlZddlmZddlmZddlmZ ddl m Z ddl m Z ddl m Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZiZdZe j8dddZdZdZdZ y)z>TPU specific APIs to be used in conjunction with TPU Strategy.N) config_pb2)session)cluster_resolver)context) def_function) monitoring)device)errors)ops) tf_logging)topology)tpu)compat)localz/tensorflow/tpu/worker_addressz;The worker address that the coordinator/client connects to.addressc  |&t|tjr t|ds t dt j dtjjtjjtjd|~tjrbtjjtjj }|j"dj%|j"|d}t'||sJt)j*|j,}|t.vrt j0d|t j d||t2vrdj%|j5tjrt7j8d fd }t7j:}|r*t j0d t7j<d  t?jtAjB5|}dddtjD |t7j<| tjjOjQ}n/t?jRs|jU} |jW} tYjZd} | r)| j\j_| jat?jbje5tgjh| | 5} | jktAjl}ddddddnKt?jtAjB5tAjld }|cdddSt j dtojp} |js|| t.|<ttjwdjy|j{| S#1swYxYw#tFjH$r(}tGjJddd tM|zd}~wwxYw#|t7j<|wwxYw#1swY?xYw#1swYxYw#1swYxYw)a`Implementation for tpu.experimental.initialize_tpu_system. Kept separate to avoid tpu_oss code duplication. Initialize the TPU devices. Args: cluster_resolver: A tf.distribute.cluster_resolver.TPUClusterResolver, which provides information about the TPU cluster. tpu_cluster_resolver_cls: a reference to tf.distribute.cluster_resolver.TPUClusterResolver so that an instance of it can be initialized if cluster_resolver is None. Returns: The tf.tpu.Topology object for the topology of the TPU cluster. If called inside tf.function, it returns the serialized topology object instead. Raises: RuntimeError: If running inside a tf.function. NotFoundError: If no TPU devices found in eager mode. TypeError: If tpu_cluster_resolver_cls is not tf.distribute.cluster_resolver.TPUClusterResolver. Ntpu_hardware_featureRtpu_cluster_resolver_cls is not tf.distribute.cluster_resolver.TPUClusterResolver.z6Deallocate tpu buffers before initializing tpu system.{}/replica:0/task:0rz|TPU system %s has already been initialized. Reinitializing the TPU can cause previously created variables on TPU to be lost.zInitializing the TPU system: %sF autographc4tjddS)NF)job compilation_failure_closes_chipstpu_cancellation_closes_chips)rinitialize_systemrsW/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu_strategy_util.py _tpu_init_fnz0initialize_tpu_system_impl.._tpu_init_fnrs!  " "+0(-//zIt looks like tf.function behavior was disabled, perhaps using tf.config.run_functions_eagerly. tf.tpu.experimental.initialize_tpu_system requires tf.function to work. This primitive will override the disable.z9TPUs not found in the cluster. Failed in initialization: Tallow_soft_placementconfigtarget)rrz!Finished initializing TPU system.) serializedr)> issubclasscluster_resolver_libClusterResolverhasattr TypeErrorlogginginfor _clear_cachesclear_kernel_cachegccollectexecuting_eagerlyr DeviceSpec from_string device_namerformat isinstanceras_text_tpu_INITIALIZED_TPU_SYSTEMSwarning_LOCAL_MASTERS get_job_namerfunctionfunctions_run_eagerlyrun_functions_eagerlyr r_tpu_system_device_name async_waitr InvalidArgumentError NotFoundErrorstr_initialize_logical_devicesnumpy#executing_eagerly_outside_functionsmaster cluster_specr ConfigProto cluster_defCopyFromas_cluster_defGraph as_default session_libSessionrunrr Topologyset_tpu_topology_tpu_worker_addressget_cellset get_master)rtpu_cluster_resolver_cls curr_devicetpu_namer run_eagerlyoutputeserialized_topologyrJrKsession_configsess tpu_topologyrs @rinitialize_tpu_system_implre+s 2%Z 4 D D. +-C D  > ??  ,,GH //!!# //&&(**, #  "%%11'//2C2O2OPk  $#**;??;/3 $&> ?? ? ^^,11 2( )) OO '(02  ,,0(;^# & &'7'D'D'F GC  U+/,/446K oo = ((/ 8 ::c11#6 7     **;7 OO113 ,,.  2 2 4  $ $ &F#002L++FN  )),*E*E*GH    !@   nV D@"hhs'<'<'>?@@@ C//4 5!11E; ! !! ,,23"".AB,##$78'38$ y)--.>.I.I.KL ]    & &   E F    **;7 !@@@@!!sl(Q-Q Q-S'$S SS! Q*%Q--R(#R##R((R++SS SS!S+c*tjS)zReturns all currently initialized tpu systems. Returns: A dictionary, with tpu name as the key and the tpu topology as the value. )r;copyr!rget_initialized_tpu_systemsris " & & ((r!cl |&t|tjr t|ds t dd |~t j rbtjjt j j}|jdj|j |d}t||sJtj|j }|t"vrt%j&d|zt%j(d|t j r0|t*vrdj|j- t/j0d  fd }t/j2}|r*t%j&d t/j4d t7jt9j: 5|ddd|t/j4| t%j(d t j j=t j j?nt7j@s|jC}|jE}tGjHd }|r)|jJjM|jOt7jPjS5tUjV||5} | jYt9jZddddddn t]dt%j(d|t"vrt"|=yy#1swYxYw#|t/j4|wwxYw#1swYmxYw#1swYexYw)aWImplementation for tpu.experimental.shutdown_tpu_system. Kept separate to avoid tpu_oss code duplication. Shuts down the TPU devices. This will clear all caches, even those that are maintained through sequential calls to tf.tpu.experimental.initialize_tpu_system, such as the compilation cache. Args: cluster_resolver: A tf.distribute.cluster_resolver.TPUClusterResolver, which provides information about the TPU cluster. tpu_cluster_resolver_cls: a reference to tf.distribute.cluster_resolver.TPUClusterResolver so that an instance of it can be initialized if cluster_resolver is None. Raises: RuntimeError: If no TPU devices found for eager execution or if run in a tf.function. TypeError: If tpu_cluster_resolver_cls is not tf.distribute.cluster_resolver.TPUClusterResolver. NrrrrzDYou are shutting down a TPU system %s that has not been initialized.z Shutting down the TPU system: %sFrc2tjy)Nr)rshutdown_systemrsr_tpu_shutdown_fnz2shutdown_tpu_system_impl.._tpu_shutdown_fns c"r!zIt looks like tf.function behavior was disabled, perhaps using tf.config.run_functions_eagerly. tf.tpu.experimental.shutdown_tpu_system requires tf.function to work. This primitive will override the disable.zClearing out eager cachesTr"r$zinitialize_tpu_system is not supported within tf.functions. You should call initialize_tpu_system outside of your tf.function. z"Finished shutting down TPU system.)/r(r)r*r+r,rr3r r4r5r6rr7r8rr9r:r;r-r<r.r=r>rr?r@rAr rrBr/r0rIrJrKrrLrMrNrOrPrQrRrSrTrl RuntimeError) rr[r\r]rmr^rJrKrbrcrs @rshutdown_tpu_system_implros4%Z 4 D D. +-C D  > ?? #  "%%11'//2C2O2OPk  $#**;??;/3 $&> ?? ? ^^,11 2( -- OO#%-./ ,,18<  ~% " ( ()9)F)F)H IcU+#,# 446K oo = ((/8 ::c11#6 7  **;7 LL,- OO##% OO((*  2 2 4  $ $ &F#002L++FN  )),*E*E*GH    !(   nV D( $$&'(((  ]   ,,34 )) **9  **;7 ! ((((sH5(NM4%N6N*$N2N*4M>9NNN' #N**N3)!__doc__r1tensorflow.core.protobufrtensorflow.python.clientrrR-tensorflow.python.distribute.cluster_resolverrr)tensorflow.python.eagerrrrtensorflow.python.frameworkr r r tensorflow.python.platformr r-tensorflow.python.tpur rtensorflow.python.utilrr;r= StringGaugerWrerirorhr!rrzsrE /;b+0...+<*%)-j,,$A9N M`)m+r!