BVh9\dZddlmZddlZddlZddlZddlZddlZddlZddl m Z dZ ddl m Z ddlmZe j"Ze j$d dd e j$d dd d ZdZdZdZdZdZdZdZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$Gdd Z%y#e$rdZ YtwxYw)!zCloud TPU Client.)futuresN)flagsT) discovery)clientFruntime_oom_exitz,Exit the script when the TPU runtime is OOM. hbm_oom_exitz(Exit the script when the TPU HBM is OOM.KUBE_GOOGLE_CLOUD_TPU_ENDPOINTS TPU_CONFIG,TPU_NAMETPU_API_DISCOVERY_URLGCE_METADATA_IPGCE_METADATA_HOST8470Zzhttp://{}:8475/requestversionc>tjjS)zA wrapper function around datetime.datetime.utcnow. This function is created for unit testing purpose. It's not easy to do StubOutWithMock with datetime.datetime package. Returns: datetime.datetime )datetimeutcnowS/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/client/client.py_utcnowr5s    ! ! ##rcHtjjtSN)osenvironget#_DISCOVERY_SERVICE_URL_ENV_VARIABLErrr_environment_discovery_urlrAs ; <>"=>. ::n %% rc||r|SttfD])}|tjvstj|cSyr)_GKE_ENV_VARIABLE_DEFAULT_ENV_VARIABLErr)tpues r _get_tpu_namerMns= J 4 5aBJJ ZZ] rcHt|tr|jdS|S)Nzutf-8) isinstancebytesdecode)ss rr,r,xs 5 88G  (rceZdZdZ ddZdZdZdZdZdZ d Z d Z d Z d Z d ZdZdZdZdZdZdZdZdZdZddZddZy)ClientzClient for working with the Cloud TPU API. This client is intended to be used for resolving tpu name to ip addresses. It's recommended to use this library as a contextlib to utilize all functionality. Nct|tr+|s tdt|dk7r t d|d}t |}|St }|r<|jd}|xs|jd}|xs|jd}n tdt||_ |jjd  |_ ||_ d|_ d|_d|_d|_|jrz|d k7r||_ |rt||_nt#d |_|rt||_n$t#d }|j%d d|_t'xs||_yy)Nz#At least one TPU must be specified.r4z>Using multiple TPUs in a single session is not yet implementedr tpu_node_nameprojectzonez(Please provide a TPU Name to connect to.r3defaultzproject/project-idz instance/zone/)rOlist ValueErrorr<NotImplementedErrorrMrGrr,_tpur:_use_api_service _credentials_project_zone_discovery_urlr1r9r) selfrKrXrW credentialsservice discovery_urltpu_node_config zone_paths r__init__zClient.__init__sa#t >?? SQ! LN N Fc  C {,.o !!/2;_00;2**62CDD DI ,,Y77DMDMDDMDJD }}  !'  ) 12FG d^ -o> __S)"- 68IMdrc d|zS)z&Return the structured Symptom message.z Symptom: r)rfmsgs r _symptom_msgzClient._symptom_msgs  rc||syt|D]}|ddk7r |djdd}tjj|d}t |z }|tj t ksntj|jd j|jy y) z)Check if a runtime OOM event is reported.F symptomType OUT_OF_MEMORY createTime.r%Y-%m-%dT%H:%M:%Ssecondsza recent runtime OOM has occurred ~{} seconds ago. The model script will terminate automatically. To prevent future OOM events, please consider reducing the model size. To disable this behavior, set flag --runtime_oom_exit=false when starting the script.T reversedr9rstrptimer timedelta_OOM_EVENT_COOL_TIME_SECloggingwarningroformatrwrfsymptomssymptomoom_datetime_str oom_datetime time_diffs r _oom_eventzClient._oom_events  H%  ? 2 .44S9!<&&//0@0CEl)l*i X''0HI I   !&!2!23  5 6 rc||syt|D]}|ddk7r |djdd}tjj|d}t |z }|tj t ksntj|jd j|jy y) z%Check if a HBM OOM event is reported.FrqHBM_OUT_OF_MEMORYrsrtrrurvza recent HBM OOM has occurred ~{} seconds ago. The model script will terminate automatically. To prevent future HBM OOM events, please consider reducing the model size. To disable this behavior, set flag --hbm_oom_exit=false when starting the script.Trxrs r_hbm_oom_eventzClient._hbm_oom_events  H%  #6 6 .44S9!<&&//0@0CEl)l*i X''0HI I   !&!2!23  5 6 rcH|jr |jSts td|j}||dk(rtj j }|jr$tjdd||jdStjdd|dS)aCreates a new Cloud TPU API object. This works around an issue where the underlying HTTP connection sometimes times out when the script has been running for too long. Other methods in this object call this method to get a new API object whenever they need to communicate with the Cloud API. Raises: RuntimeError: If the dependent Python packages are missing. Returns: A Google Cloud TPU API object. z_Missing runtime dependency on the Google API client. Run `pip install cloud-tpu-client` to fix.rYrKv1F)rgdiscoveryServiceUrlcache_discovery)rgr) ra_GOOGLE_API_CLIENT_INSTALLED RuntimeErrorrbrGoogleCredentialsget_application_defaultrerbuild)rfrgs r _tpu_servicezClient._tpu_services }} ]] ' F GG##KkY6,,DDFk  __  !"11 !!__ ;GGrcTd|jd|jd|jS)z)Returns the full Cloud name for this TPU.z projects/z /locations/z/nodes/)rcrdr_rfs r _full_namezClient._full_name s! tzz499 ..rc0|j} |jjjj |j }|j S#t$r }td|jd|d}~wwxYw)z:Returns the TPU metadata object from the TPU Get API call.)namez)Could not lookup TPU metadata from name 'zY'. Please doublecheck the tpu argument in the TPUClusterResolver constructor. Exception: N) rprojects locationsnodesrrexecute Exceptionr]r_)rfrhrrLs r_fetch_cloud_tpu_metadataz Client._fetch_cloud_tpu_metadatas!GF     & & ( . . 0 4 4$//:K 4 La YY[ F 7;yy!E FFFsAA,, B5BBc^|jr!|j}|j|Syr)r`rr)rfkeymetadatas r_get_tpu_propertyzClient._get_tpu_propertys) }}//1h \\#  rcd|_y)NT)_openrs r __enter__zClient.__enter__$s DJrc ~~~yrr)rftypevalue tracebacks r__exit__zClient.__exit__'s  eYrc|j}|j}|r|dvrytjr|j |rytj r|j |ryy)zReturns true if the TPU is in a state where training should eventually resume. If false the TPU is in a unrecoverable state and should be recreated. ) TERMINATED PREEMPTEDFT)staterFLAGSrrrr)rfrrs r recoverablezClient.recoverable*s[ JJLE}}H 55    DOOH$=     3 3H =  rc$|jdS)z%Return Cloud TPU Symptoms of the TPU.rrrs rrzClient.symptoms9s  ! !* --rc$|jdS)zReturn state of the TPU.rrrs rrz Client.state=s  ! !' **rc$|jdS)zReturn health of the TPU.healthrrs rrz Client.healthAs  ! !( ++rc|jstj|jdd} tj j |}tj j|}tj|j}|jdS|jdS#tjj$r}|j}|dk(rYd}~y|d}~wwxYw)z"Return runtime version of the TPU.rr7currentVersionNtensorflowVersion)r`_VERSION_SWITCHER_ENDPOINTrnetwork_endpointsr(r)r*r+rDrEr-rerror HTTPErrorcoder)rfurlr/r0version_detailsrL status_codes rruntime_versionzClient.runtime_versionEs == & - -  "1 %k 2 4c nn$$S)~~%%c***TYY[1""#344  ! !"5 66 \\ # #ff # ' sA1B::C4C/-C//C4c$|jdS)z#Return accelerator type of the TPU.acceleratorTyperrs raccelerator_typezClient.accelerator_typeYs  ! !"3 44rc|jS)zPReturn if the Cloud TPU API is available, if not certain features will not work.)r`rs r api_availablezClient.api_available]s ==rc|jS)zFReturn the name of the tpu, or the ip address if name is not provided.)r_rs rrz Client.nameas 99rctdS)zNReturn the local ip address of the Google Cloud VM the workload is running on.z instance/network-interfaces/0/ip)r1rs r get_local_ipzClient.get_local_ipes $%G HHrc|jstt|jS|j }|j ddk7r+t d|jd|j ddd|vr|dS|d|dd gS) zReturn a list of tpu endpoints.rREADYzTPU "z" is not yet ready; state: ""networkEndpointsr7r8r6)r`r\rAr_rrr)rfresponses rrzClient.network_endpointsis == 7 B CC--/H||G' ))X\\'%:< ==X% ( ))$[18F;KL MMrctj|z}|jdk7rtjd|j |j |jtj|z|kDrt d|j ztj||jdk7rtjd|j y)a?Wait for TPU to become healthy or raise error if timeout reached. Args: timeout_s (int): The timeout in seconds for waiting TPU to become healthy. interval (int): The interval in seconds to poll the TPU for health. Raises: RuntimeError: If the TPU doesn't become healthy by the timeout. HEALTHYzFWaiting for TPU "%s" with state "%s" and health "%s" to become healthyz0Timed out waiting for TPU "%s" to become healthyzTPU "%s" is healthy.N)timerr}r~rrrsleep)rf timeout_sintervaltimeouts rwait_for_healthyzClient.wait_for_healthywsiikI%G ++-9 $ oo / ))+tzz|T[[]4 x ' ) > LN N jj ++-9 $ OO*DIIK8rcfd}|j}tjt|5}|j ||}|D]}|s|j  dddy#1swYyxYw)aConfigure TPU software version. Args: version (string): Version of software to configure the TPU with. restart_type (string): Restart behaviour when switching versions, defaults to always restart. Options are 'always', 'ifNeeded'. c|d}tdzj|}tjj |d} tjj |y#tj j$rJ}|j}|dk(rtdjtdj|d}~wwxYw) zConfigure individual TPU worker. Args: worker: A dict with the field ipAddress where the configure request will be sent. r7z/{}?restartType={}r)datarzTensorflow version {} is not available on Cloud TPU, try a previous nightly version or refer to https://cloud.google.com/tpu/docs/release-notes for the latest official version.zFailed to configure worker {}N) rrr(r)r*r+rrrr)workerr@rr/rLr restart_typeversions rconfigure_workerz6Client.configure_tpu_version..configure_workers+&j '*> > F F g| -c NN " "3S " 1c Ns# \\ # # Nff # -.4VG_ > > 9@@LM M NsA##C ACC ) max_workersN)rrThreadPoolExecutorr<mapresult)rfrrrworkersexecutorresultsrs `` rconfigure_tpu_versionzClient.configure_tpu_versionsnN0$$&G  # #G = -w7g&  --/sA/A//A8)NNNrYNN)i)always)__name__ __module__ __qualname____doc__rlrorrrrrrrrrrrrrrrrrrrrrrrrTrT~s$! 0Jd,,"GH. F .+,7(5I N9.(rrT)&r concurrentrrrDr}rrr(abslrrgoogleapiclientr oauth2clientr ImportErrorr DEFINE_boolrIrC_ENDPOINTS_SEPARATORrJrr"r!r;r|rrrr$r1rArGrMr,rTrrrrs  #''!  $d@B.$<>6*"&=#!2&9#< $=" xx'!&'s B!!B+*B+