BVh^ dZddlZddlZddlmZmZmZmZmZm Z m Z m Z ddl m Z ddlZddlmZddlmZddlmZddlmZdd lmZddlmZdd lmZdd lmZdd lm Z dd lm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddlm&Z&ddl'm(Z(ddl'm)Z)ddl'm*Z*ddl'm+Z+ddl'm,Z,ddl'm-Z-ddl.m/Z0ddl.m1Z1ddl.m2Z2ddl.m3Z3ddl.m4Z4ddl.m5Z5ddl6m7Z7dd l8m9Z:dd!l;mZ>dd$l;m?Z?dd%l;m@Z@dd&lAmBZBeCd'd(gZDd)ZEd*ZFd+ZGe4jrZ9d,ee d-e fd.ZHeBd/g0 dmd1eejd,ee d2eJd3eeJd-e:jf d4ZL dnd1ejd,ee d-e%jfd5ZNeBd6g0dnd,ee d-e%jfd7ZOejd8e%jd9e>jd:e>jd-eJfd;ZReBdejZTeBd?Gd@dAejdAdBdCgZVeBdDg0e?j dodEedFefdGeeee:jdHee2jdIee0jdJee dKeedLeeTdMeeVd-eefdNZZdOZ[dGee:jdPeee&jdLeTd-e eeeeejffdQZ^ dndRZ_ dpdEedFefdGeeee:jdHee2jdIee0jdJee dSeJdKeedLeeTdMeeVd-eee:jfdTZ`dUedVeJd-e eee:jee%jeeffdWZadUedVeJd-e eee:jee%jeeffdXZb dqdEedFefdGeeeee:jdYecdZeeecd[e eJeeJfd\eeecdHee2jdIee0jdJee dMeeVd-e e%jee:jffd]ZdeBd^g0e?j dqdEedFefdGeee:jdYecdZeeecd[e eJeeJfd\eeecdHee2jdIee0jdJee dMeeVd-ee:jfd_ZeeBd`g0e?j drdEedFefdGeeeee:jdYecdHee2jdIee0jdJee dMeeVfdaZfeBdbg0e?j dsdEedFefdGeeeee:jdHee2jdIee0jdJee dMeeVd-efdcZgeCgddZhd-eJfdeZiGdfdge+jZkdhe%jfdiZm dtdEedFefdGeee:jdHee2jdIee0jdJee d-ee:jf djZndke%jfdlZoy)uz Library of TPU helper functions.N)AnyCallableIterableListOptionalTextTupleUnion)logging)xla)attr_value_pb2)dynamic_padding_pb2)tpu_embedding_configuration_pb2)tf2)auto_control_deps)composite_tensorconfig) constant_op)dtypes) func_graph)function)ops) tensor_shape) array_ops)array_ops_stack)cond)control_flow_ops)math_ops)variable_scope)device_assignment) tensor_tracer)tpu_feed) tpu_function) tpu_name_util)tpu_replication)tpu_ops)core)compat)nest)object_identity)traceback_utils)variable_utils) tf_export Placeholder VarHandleOp_post_device_rewrite_tpu_compilation_status_pivot_for_clusterjobreturnc|yd|zS)z;Returns the device name for the TPU_SYSTEM device of `job`.z/device:TPU_SYSTEM:0z/job:%s/device:TPU_SYSTEM:0)r4s I/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu.py_tpu_system_device_namer9Js[ ! (3 ..ztpu.initialize_system)v1embedding_config compilation_failure_closes_chipstpu_cancellation_closes_chipsc |dn|j}d}||rd}nd}tjt|5t j ||}| |cdddStj |g5t j|}dddtj g5tj|d cdddcdddS#1swYIxYw#1swYnxYw dddy#1swYyxYw) aInitializes a distributed TPU system for use with TensorFlow. Args: embedding_config: If not None, a `TPUEmbeddingConfiguration` proto describing the desired configuration of the hardware embedding lookup tables. If embedding_config is None, no hardware embeddings can be used. job: The job (the XXX in TensorFlow device specification /job:XXX) that contains the TPU devices that will be initialized. If job=None it is assumed there is only one job in the TensorFlow flock, and an error will be returned if this assumption does not hold. compilation_failure_closes_chips: Set the configuration whether we want to close TPU chips when there is a compilation failure. tpu_cancellation_closes_chips: Set the configuration whether we want to close TPU chips when a TPU execution is cancelled. If the value is None, the behavior will be determined by the command line flag `tpu_cancellation_closes_chips` for the TPU worker. WARNING: this argument only applies to TFRT TPU runtime. Returns: A serialized `TopologyProto` that describes the TPU system. Note: the topology must be evaluated using `Session.run` before it can be used. Nr)r=r>rtpu_init_identityname) SerializeToStringrdevicer9r'configure_distributed_tpucontrol_dependenciesconfigure_tpu_embeddingridentity)r<r4r=r> config_string"tpu_cancellation_closes_chips_enumtopologyembedding_inits r8initialize_systemrPRs 8*12#557()$".$+,(+,( zz)#./D00)I&HH  DD ! !8* -M66mLnM ! !>"2 3D   /B CDDDDMMDDDDDDsAC:#C:9CC:.C$ C:C! C:$C- )C::Dc|j}tjt|5t j |cdddS#1swYyxYw)aVInitializes a distributed TPU Embedding system for use with TensorFlow. The following two are equivalent: 1. initialize_system() with embedding_config. 2. initialize_system() without embedding_config, then initialize_system_for_tpu_embedding(). initialize_system() should not be called with embedding_config if initialize_system_for_tpu_embedding() is meant to be called later. Args: embedding_config: a `TPUEmbeddingConfiguration` proto describing the desired configuration of the hardware embedding lookup tables. job: The job (the XXX in TensorFlow device specification /job:XXX) that contains the TPU devices that will be initialized. If job=None it is assumed there is only one job in the TensorFlow flock, and an error will be returned if this assumption does not hold. Returns: A no-op. rN)rFrrGr9r'rJ)r<r4rLs r8#initialize_system_for_tpu_embeddingrRsK0#446- zz)#./A  * *- @AAAs AAztpu.shutdown_systemctjt|5tj}ddd|S#1swYSxYw)aQShuts down a running a distributed TPU system. Args: job: The job (the XXX in TensorFlow device specification /job:XXX) that contains the TPU devices that will be shutdown. If job=None it is assumed there is only one job in the TensorFlow flock, and an error will be returned if this assumption does not hold. N)rrGr9r'shutdown_distributed_tpu)r4rTs r8shutdown_systemrUsD zz)#./B&??AB !!B !!s >Aopresource_readsresource_writesc|jdk(r&|s|r!|j|jyyd}t||xs||S)zGReplaces TPUReplicatedInput outputs with its inputs in resource_inputs.TPUReplicatedInputTFcg}g}|D]R}|jjdk(s|j||j|jjT|D]}|j ||j ||xs|S)zEReplaces handles in `resource_inputs` with their unreplicated inputs.rZ)rVtypeappendextendinputsdiscardupdate)resource_inputs to_removeto_addresourcets r8#replace_with_unreplicated_resourceszJtpu_replicated_input_resolver..replace_with_unreplicated_resourcessI F#*   1 1" hkk(()*!a !6"  Yr:)r\clearbool)rVrWrXrgs r8tpu_replicated_input_resolverrjs`WW$$    1.AC1/B DDr:ztpu.PaddingSpecceZdZdZdZdZy) PaddingSpecz:Represents the type of padding policies for tpu.replicate.rrAN)__name__ __module__ __qualname____doc__AUTO POWER_OF_TWOr7r:r8rlrlsB $,r:rlztpu.XLAOptionsc(eZdZdZ dfd ZxZS) XLAOptionsa{XLA compilation options. Attributes: use_spmd_for_xla_partitioning: Boolean. Whether to use XLA's SPMD partitioner instead of MPMD partitioner when compiler partitioning is requested. enable_xla_dynamic_padder: Boolean. Whether to enable XLA dynamic padder infrastructure to handle dynamic shapes inputs inside XLA. True by default. Disabling this may cause correctness issues with dynamic shapes inputs, as XLA will just assume the inputs are with padded shapes. However users can optionally set it to False to improve device time if masking is already handled in the user side. c.tt| |||SN)superrt__new__)clsuse_spmd_for_xla_partitioningenable_xla_dynamic_padder __class__s r8rxzXLAOptions.__new__s# S )#/L*C EEr:)TT)rmrnrorprx __classcell__r|s@r8rtrts -1(,EEr:rtrzr{z tpu.replicate computation.r_ infeed_queuer!rEmaximum_shapes padding_spec xla_optionsc .t||||||||dS)aBuilds a graph operator that runs a replicated TPU computation. Example for the basic usage that `inputs` has static shape: ```python def computation(x): x = x + 1 return tf.math.reduce_mean(x) x = tf.convert_to_tensor([1., 2., 3.]) y = tf.convert_to_tensor([4., 5., 6.]) tf.compat.v1.tpu.replicate(computation, inputs=[[x], [y]]) ``` If the `inputs` has dynamic shapes and you would like to automatically bucketize the inputs to avoid XLA recompilation. See the advanced example below: ```python def computation(x): x = x + 1 return tf.math.reduce_mean(x) # Assume input tensors in two replicas `x` and `y` both have dynamic shape # ([None, 2]). tf.compat.v1.tpu.replicate( computation, inputs=[x, y], maximum_shapes=[tf.TensorShape([None, None])], padding_spec=tf.compat.v1.tpu.PaddingSpec.POWER_OF_TWO) ``` Args: computation: A Python function that builds the computation to replicate. inputs: A list of lists of input tensors or `None` (equivalent to `[[]]`), indexed by `[replica_num][input_num]`. All replicas must have the same number of inputs. Each input can be a nested structure containing values that are convertible to tensors. Note that passing an N-dimension list of compatible values will result in a N-dimension list of scalar tensors rather than a single Rank-N tensors. If you need different behavior, convert part of inputs to tensors with `tf.convert_to_tensor`. infeed_queue: If not `None`, the `InfeedQueue` from which to append a tuple of arguments as inputs to computation. device_assignment: If not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. Uses a default device assignment if `None`. The `DeviceAssignment` may be omitted if each replica of the computation uses only one core, and there is either only one replica, or the number of replicas is equal to the number of cores in the TPU system. name: (Deprecated) Does nothing. maximum_shapes: A nested structure of tf.TensorShape representing the shape to which the respective component of each input element in each replica should be padded. Any unknown dimensions (e.g. tf.compat.v1.Dimension(None) in a tf.TensorShape or -1 in a tensor-like object) will be padded to the maximum size of that dimension over all replicas. The structure of `maximum_shapes` needs to be the same as `inputs[0]`. padding_spec: An enum specified by `tpu.PaddingSpec`. This describes the padding policy when the `inputs` to `tpu.replicate` is dynamic. One usage is to enable automatic bucketizing on the inputs by setting the value to `tpu.PaddingSpec.POWER_OF_TWO`, which can help to reduce the recompilation in the XLA side. xla_options: An instance of `tpu.XLAOptions` which indicates the options passed to XLA compiler. Use `None` for default options. Returns: A list of outputs, indexed by `[replica_num]` each output can be a nested structure same as what computation() returns with a few exceptions. Exceptions include: 1) None output: a NoOp would be returned which control-depends on computation. 2) Single value output: A tuple containing the value would be returned. 3) Operation-only outputs: a NoOp would be returned which control-depends on computation. TODO(b/121383831): Investigate into removing these special cases. Raises: ValueError: If all replicas do not have equal numbers of input tensors. ValueError: If the number of inputs per replica does not match the number of formal parameters to `computation`. ValueError: If the static `inputs` dimensions don't match with the values given in `maximum_shapes`. ValueError: If the structure of inputs per replica does not match the structure of `maximum_shapes`. )rrrrA)split_compile_and_replicate)rr_rr!rErrrs r8 replicaters4D %  #  ! ""r:cNtj|tj}tj|tj|dzz }tj |}tj |dz|}tj|tj}|S)zCeil input `x` to power of `n`.g?)rcastrfloat32logceilpowint32)xnlognxresults r8_ceil_to_pow_of_nrnspmmAv~~&! ,,q/HLLS1 1% -- % <<C '& == .& -r: padded_shapesc g}g}g}t|D]$\}}t|D]\}jj} |dk(rN|jg|j| |jt j | dt n;t| D]\} } | | ||| k7sd||| <t| ||||<tj} | jjttjd||j| 'g} |D];}| jtj t#j$|d=g}g}g}t|D]\}}|jg|jgt'|dz }t|D]G\}|||}jj} ||}t)||r|t| D]\} } ||| s|dk(r?|dz }t+j,}||_| |_||_|j|||jtj4|| t6j8gt|j:D]\} } ||| rkd}| j<t| j<|}n;tj>| || |}|t@jBk(r tE|d}d||| z g}nddg}j|jjGr2tIjHtjJdfd fd }ntjL}|jjttjd||j|4||jJt'|}tO|D]} || jQ|| ||fS) aPad all input tensors given padded_shapes. The real shape tensors will be concatenated with the padded original inputs. Args: inputs: The original inputs. padded_shapes: A list of padded shapes for each input. If an entry is None, no padding is performed. padding_spec: An enum specified by `tpu.PaddingSpec`. This describes the padding policy when the `inputs` to `tf.tpu.replicate` is dynamic. One usage is to enable automatic bucketizing on the inputs by setting the value to `tpu.PaddingSpec.POWER_OF_TWO`, which can help to reduce the recompilation in the XLA side. Returns: The padded inputs and a PaddingMap list which maps the padded input dimension to the real shape argument index. rF)dtypeTbaxisrArBc0tjSrv)rpad) input_tensorpaddingssr8z _pad_all_input..simmL(;r:cSrvr7)rsr8rz _pad_all_input..slr:)) enumerate get_shapeas_listr]np full_likerimaxrshaperV _set_attr_POST_DEVICE_REWRITE_ATTRr AttrValuer reduce_maxrstacklenanydynamic_padding PaddingMap arg_index shape_indexpadding_arg_indexrrrdimsvaluemaximumrlrrris_fully_definedrconstantrranger^)r_rrmaximum_static_shapes need_paddinginput_shape_tensorscore_idxinputs_per_coreidx input_shapeisreal_input_shapershapes_per_input padded_inputs real_shapes padding_mapsreal_shape_idxinput_shape_tensor padded_shape padding_mapminimum_dynamic_dim_size max_dim_sizepadding padded_input num_replicasrrs @@r8_pad_all_inputrxsp2,#,V#48h&78\ **,446k Q""2&$$[1BLLe4HIk* (DAq Y!4S9!<<#'L a  (&))>s)C&Ec"#6## #  " "T *,#%%&67%88*.-NO112BC!LNN-+,#,V#4?5hr)A-N&7;5\.s3H= **,446k"3'l \#  L$<k* DDAq # q !1}!n+668k&)k#()k%.>?G!fG //' "' #*  ! ! # 4 4 6  &;"$, #|X>, !! %  $ $t , . h&&|4h&&|4w;5 ?5B]#,  ,a!KN+,  $$r:ct|tjr&tt j |d}|f|zS|S)aiFor an input, replaced the input by a tuple if the input is composite. If `maybe_composite` is not composite, return the parameter `non_composite_output` otherwise return a tuple which consists of the value of the parameter `composite_output` the same number of times as there are components of the composite tensor. This is useful for computing a mask when flattening nested data with `expand_composites=True`. For example ```python nest.flatten(data, expand_composites=True) ``` and ```python nest.flatten(nest.map( data, lambda x: _flatten_and_filter_composite(x, False, True))) ``` will have the same length and second will be True if the tensor in the first is derived from a expanding a composite tensor. Args: maybe_composite: A value to test for being a composite tensor. non_composite_output: The value to return when `maybe_composite` is not a composite. composite_output: the value to fill the output tuple with if `maybe_composite` is a composite. Returns: `non_composite_output` or a tuple with multiple copies of `composite_output`. Texpand_composites) isinstancerCompositeTensorrr*flatten)maybe_compositenon_composite_outputcomposite_outputnum_componentss r8_flatten_and_filter_compositers?L!1!A!ABoNON   // r:use_tpuc T<~|ggn|}|xs t}i} |R|jj|jj j d} |j | d<tj| d<tjrtjdt|tstdt|td|Dr%td|D cgc] } t| c} t!|} | d k(rgSt#d | D]} t%j&|d ||  t)j*|}|D cgc]} t%j| d }} t%jt%j,d |d }g}|D]K} |j/| Dcgc].}|t1j2d nt5j6|0c}M|d Dcgc]}|j8}}t!|d }t!|}t#| D]x} t!|| |k7r(t;dj=|| t!|| || Dcgc]}|j8}}||k7s^t;dj=|| |t?j@|||}|o|/td|d|d D cgc]} | jBc} d|td|d|d D cgc]} | jBc} dd|jDd|d}|r|r t;dt%j&|d |dt%jtGt%j|d t%j|Dcgc]\}}tI||c}}}|Dcgc]}|tKjL|nd}}t%j&|d |d|}tO|||\}}|rd }tjd|d tQ|dd| d<|jR| d<t5jT}g}t#d t!|d D]U} t#| Dcgc] }|||  } }|j/tWjX| dj=| Wt|tZj\r|j_d|jBz}!n|j_d }!tajb|!d!z}"|"jetftijjtmjn|!"tqjr|!| |"#}# |#jutWjvd1| |d$| }$tyjz| 5t5j||$g5|r|j~rD]}%||%jj}&tj||%j|%j||%j||%j<||%jj|&t|D cgc]+\} }tj|d%j=| -}} }tG||D]:\} }'|r|'s | jjed&tijjd '<tG||d D( cgc] \}(} | dn|( })}(} t%j|d |)d|d (})|7|j| |jD]}*|)j/|*tj}+|+j},|+j<z.split_compile_and_replicate..s>ZdE] + +>s!#zGtpu.replicate() inputs must be a list of lists/tuples, received types: rrATrct|ddS)NFT)rrs r8rz-split_compile_and_replicate..s-a=r:z`Replicas must have the same number of inputs. Replica 0 had {} inputs, replica {} had {} inputs.zdReplicas must have matching input types. Replica 0 had input types {}, replica {} had input types {}zOSupplied computation cannot be called with the specified inputs. You specified z inputs: z, but the computation needs  zand z: additional inputs from infeed, but the computation needs Fz9Dynamic input shapes are not supported with infeed queues) check_typesz&TPU has inputs with dynamic shapes: %sstep_marker_locationSTEP_MARK_AT_ENTRYrzzinput{}rDcluster_clusterz/pivot)r)rErpivot)rrzreplicated_input_{}_tpu_input_identityr) structure flat_sequencerc|jdd}|d|d<tjd|| ||g|i|S||g|i|S)z)Variables on TPU have a few restrictions. partitionerNz~Partitioned variables are not supported on TPU. Got `partitioner` that is %s for variable %s. Setting `partitioner` to `None`.)getr warning)getterrEargskwargsrsaved_custom_getters r8 custom_getterz2split_compile_and_replicate..custom_getterNsnjj5  ""&&  //12=t E  &.t.v. .$VTCDCFC Cr:z= 2.0 detected. Tensor Tracer v1 is not enabled.c3FK|]}tj|ywrv)r)as_bytesrrs r8rz.split_compile_and_replicate..sKAV__Q/Ks!host_compute_corecompilation_statuszshard_%dzoutput{}zoutput_%d_shard_%dr7)hrtrN serializedcore_assignmentrtolistrrget_soft_device_placementr inforr TypeErrorr\rrrr*assert_same_structurer-convert_variables_to_tensors map_structurer]rrrconvert_to_tensorr ValueErrorformatr check_function_argument_countrEnumber_of_tuple_elementsziprr TensorShapergetattrrzget_default_graphr'tpu_replicated_inputr FuncGraph unique_namerno_opr_PIVOT_FOR_CLUSTERr rr)rr&TPUReplicateContextEntertpu_replicate_metadatar$tpu_shard_contextrIr{rrtf2xlaset_dynamic_dimension_sizerr set_shaperrrKrVpack_sequence_asset_number_of_shardsgenerate_dequeue_opr get_variable_scope use_resourcerset_use_resourceset_custom_getteris_flat_postprocess_flat_outputs_postprocess_non_flat_outputsr" TensorTracer is_enabledrenabledwarn trace_tpu ExitResultreport_unsupported_operationsExitHostComputeCorerr^tpu_compilation_result_TPU_COMPILATION_STATUS_ATTRgrouptpu_replicated_output)=rr_rr!rErrrrmetadata_kwargsrrrper_replica_inputflat_inputs_with_nones is_composite flat_inputsrflat_input_types input_arityflat_input_aritytypes arg_errordynamic_shape_inputsyflat_maximum_shapesrunpadded_inputsrgraphflat_replicated_inputsreplicareplicas cluster_namercontextmetadatarr composite replicatedcomputation_inputsrfvscopesaved_use_resourceroutputsneed_spmd_partitioningoutputs_is_flatoutput_tensors control_deps pack_templatettr attr_valuecompile_statusrVreplicated_outputsys replica_outsrs= @r8rr's F >B4v&+z|+/"  & & 1 1 3  - - 5 5 7 > > @ O //+,-3,L,L,N/() %%' LL"# FD !  $V ~/ 00>v>>  178#DI89 ; <<V,Q I L !5avay&)45  6 6v >& &  ll$= d00=vayJK, + #c $%9 Q#2G2G2JJ(31~6!agg66F1I+)*   8a 6!9~$ !!' QF1I!G II*!n -QWW -E -   GGMv+QH7 88 8//; .)  &-y&)1LQ!&&1L0MN''0k 3 44  &-y&)1LQ!&&1L0MQ P 667844=; @ AA   E GG vay.eL,,fQi0n57 8 Q 'q! , 89 % ()}   #$> {1~/B+02"O ./B/;!=K! ll;VAYG,3)+?-A/()//12    !% C A' (1a7<\7JKG G$Q'KHK!!$$ 9++A. 011 z++, $$Z%**%<=L$$Y/L  lX&= >%//$ **V__\-JKM  / / l% A'v2 MMO--G!7G6EGH  ' ' TE// ;TE +"G"G' OK.{/D/DEKK+'-'H'H()>)>?))()F)FG(I !## % !!6!6 7 A A+ N O 67 a   Q%:%A%A!%D E  4lC ;,!Y$y $$...'11D9 ; ;FI$& p=App%& n22n< 7n??o o 2o>p p" rOrPc| t}tj|d}tj|d}|tjfz }d} |r2|Dcgc]&}t |t jr|n||(}}nXt jtd5|Dcgc]&}t |t jr|n||(}}ddd|Dcgc]}t |t js|!}}|Dcgc]}t |t jr|!}}|||zk7r tdt|d kDr|dd t|z }g}|D]} | |jd|rWtj| }|j j#d t%j&d |j|qt j| jr | jn td5tj| }|j j#d t%j&d |j|ddd|||fScc}wcc}w#1swYxYw#t$r}td|d}~wwxYwcc}wcc}w#1swYXxYw) aValidates non-flat outputs, add backs device assignments and other attrs. Args: outputs: Output from `computation` inside `tpu.rewrite`. need_spmd_partitioning: Whether XLA SPMD partitioning is needed. Returns: - Tensors extracted from outputs. - Operations extracted from outputs. - A pack template for use with nest.pack_sequence_as to pack the tensors. NFrTc4|dStj|Srv)rr rs r8rz+_postprocess_flat_outputs..sAID33H3H3Kr:rz_TPU function return values must all either be Operations or convertible to Tensors. Got error: zYTPU functions must return zero-or more Tensor values followed by zero or more Operations.rA_tpu_output_identityr)rr*rrrrr OperationrGr( Exceptionr rr]rrKrVrr r) rOrPrT maybe_convertoeoutput_operationsrRnew_output_tensorsrfs r8r&r&s, _gG ,,w%@- LLD 9' $$& (('K-3!S]]+!q1A Ag ::d1g   As}}-A=3C C   #*JQZ3==-IqJJ&K!jCMM.JAK.K !222  # $$  a!"=1s+<'=#=>M %ay%    Q addnn+^-E-E-MN" ::!((ahhQ 8%   q ! -~/G/G$/OP!!!$ %%% . ==g    3  ../S 2 333 KK8%%syJ+I;!J)J.+JJJ'J1J1J61J6AJ;;JJJ J J.J))J.;K cXtj|d}t|D]S\}}|d||<t|tj rt d|jd t j|}|r^tj|}|jjdtjdtj|||<t j |j r |j n t#d 5tj|}|jjdtjdtj|||<dddV|g|fS#t$r}t d|dd}~wwxYw#1swYxYw) aValidates non-flat outputs, add backs device assignments and other attrs. Args: outputs: Output from `computation` inside `tpu.rewrite`. need_spmd_partitioning: Whether XLA SPMD partitioning is needed. Returns: - Tensors extracted from outputs. - An empty Operations list because Operations are not allowed in non-flat outputs. - A pack template for use with nest.pack_sequence_as to pack the tensors. TrNztpu.rewrite does not support Operation as return value in non-flat output structure. You can set returned Operations as control dependencies of returned Tensors so Operations are triggered when Tensors are evaluated. Operation found: ""z`TPU function return values must all either be Operations or convertible to Tensors. Got error: "r]rr)r*rrrrr^r rEr r_rrKrVrr rrGr()rOrP flat_outputsrrarbs r8r'r's$g>, %$0dayl1o!S]]#  678ffXQ @ AA 7   "a   Q addnn+^-E-E-MN!**1-l1o ::!((ahhQ 80   q ! -~/G/G$/OP#,,Q/ Q 00?$0N r7 ""3 7  1121 6 777"00s%&E?AF? FFFF) num_shardsinput_shard_axesoutputs_from_all_shardsoutput_shard_axesc |dkrtd||gn|}t|tstdt ||D cgc]} t j | }} |dgt|z}t|t|k7r$tdt|dt|d|rSt||D cgc]\} } tj| || } } } t| D cgc] } t| }} ngg|z}t|||||| \}}t|dt jr||dgfSt|d}|dg|z}|t|k7rtd |d t|d t|tr|g|z}|t|k7rtd |dt|dg}t||t|D]\} }} |rt| dj}|duxr|jdk(}|j!|rt#j$t| ntj&t| | }|j!| d||fScc} wcc} } wcc} w)a Shards `computation` for parallel execution. `inputs` must be a list of Tensors or None (equivalent to an empty list), each of which has a corresponding split axis (from `input_shard_axes`). Each input is split into `num_shards` pieces along the corresponding axis, and computation is applied to each shard in parallel. Tensors are broadcast to all shards if they are lexically captured by `computation`. e.g., x = tf.constant(7) def computation(): return x + 3 ... = shard(computation, ...) If `outputs_from_all_shards` is true, the outputs from all shards of `computation` are concatenated back together along their `output_shard_axes`. Otherwise, each output is taken from an arbitrary shard. Inputs and outputs of the computation must be at least rank-1 Tensors. Args: computation: A Python function that builds a computation to apply to each shard of the input. inputs: A list of input tensors or None (equivalent to an empty list). Each input tensor has a corresponding shard axes, given by `input_shard_axes`, which must have size divisible by `num_shards`. num_shards: The number of shards. input_shard_axes: A list of dimensions along which to shard `inputs`, or `None`. `None` means "shard all inputs along dimension 0". If not `None`, there must be one dimension per input. outputs_from_all_shards: Boolean or list of boolean. For each output, if `True`, outputs from all shards are concatenated along the corresponding `output_shard_axes` entry. Otherwise, each output is taken from an arbitrary shard. If the argument is a boolean, the argument's value is used for each output. output_shard_axes: A list of dimensions along which to concatenate the outputs of `computation`, or `None`. `None` means "concatenate all outputs along dimension 0". If not `None`, there must be one dimension per output. Ignored if `outputs_from_all_shards` is False. infeed_queue: If not `None`, the `InfeedQueue` to use to augment the inputs of `computation`. device_assignment: If not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. Uses a default device assignment if `None`. The `DeviceAssignment` may be omitted if each shard of the computation uses only one core, and there is either only one shard, or the number of shards is equal to the number of cores in the TPU system. name: (Deprecated) Does nothing. xla_options: An instance of `tpu.XLAOptions` which indicates the options passed to XLA compiler. Use `None` for default options. Returns: A tuple of (compile op, [output tensors]). Raises: ValueError: If num_shards <= 0 ValueError: If len(input_shard_axes) != len(inputs) ValueError: If len(output_shard_axes) != len(outputs from `computation`) rz0num_shards must be a positive integer. Received NzAtpu.shard()'s inputs must be a list of Tensors or None. Received zKLength of input_shard_axes must be equal to the number of inputs. Received z inputs and z input_shard_axes.rrr!rErzMLength of output_shard_axes must be equal to the number of outputs. Received z outputs and z output_shard_axes.zSLength of outputs_from_all_shards must be equal to the number of outputs. Received z outputs and z outputs_from_all_shards.)r rrrr\rr rrrsplitrr^rirndimsr]rrconcat)rr_rhrirjrkrr!rErrr split_inputsrtransposed_inputs compile_oprO num_outputsresults all_shardsr is_scalars r8split_compile_and_shardrx\sR1_  ::,G II2V& FD !  $V ~/ 00/5 5C ! !! $ 5& 5sS[([C()) ,,/K= ,-..@B CC -v68 T1 :D18L8 +.|*<=Qa==z)3)  *g CMM*  | ## GAJ+k)C)** --8M: 1233FH II'.67+EC/00  (M & ' ((A C DD '"#46M#&= 2 tZdjjet#:)9i nn o++DG4$++DG$?BnnQqT  W Y 68 >sI-"I2/I8z tpu.shardc 2t||||||||||  dS)a Shards `computation` for parallel execution. `inputs` must be a list of Tensors or None (equivalent to an empty list), each of which has a corresponding split axis (from `input_shard_axes`). Each input is split into `num_shards` pieces along the corresponding axis, and computation is applied to each shard in parallel. Tensors are broadcast to all shards if they are lexically captured by `computation`. e.g., x = tf.constant(7) def computation(): return x + 3 ... = shard(computation, ...) TODO(phawkins): consider adding support for broadcasting Tensors passed as inputs. If `outputs_from_all_shards` is true, the outputs from all shards of `computation` are concatenated back together along their `output_shard_axes`. Otherwise, each output is taken from an arbitrary shard. Inputs and outputs of the computation must be at least rank-1 Tensors. Args: computation: A Python function that builds a computation to apply to each shard of the input. inputs: A list of input tensors or None (equivalent to an empty list). Each input tensor has a corresponding shard axes, given by `input_shard_axes`, which must have size divisible by `num_shards`. num_shards: The number of shards. input_shard_axes: A list of dimensions along which to shard `inputs`, or `None`. `None` means "shard all inputs along dimension 0". If not `None`, there must be one dimension per input. outputs_from_all_shards: Boolean or list of boolean. For each output, if `True`, outputs from all shards are concatenated along the corresponding `output_shard_axes` entry. Otherwise, each output is taken from an arbitrary shard. If the argument is a boolean, the argument's value is used for each output. output_shard_axes: A list of dimensions along which to concatenate the outputs of `computation`, or `None`. `None` means "concatenate all outputs along dimension 0". If not `None`, there must be one dimension per output. Ignored if `outputs_from_all_shards` is False. infeed_queue: If not `None`, the `InfeedQueue` to use to augment the inputs of `computation`. device_assignment: If not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. Uses a default device assignment if `None`. The `DeviceAssignment` may be omitted if each shard of the computation uses only one core, and there is either only one shard, or the number of shards is equal to the number of cores in the TPU system. name: (Deprecated) Does nothing. xla_options: An instance of `tpu.XLAOptions` which indicates the options passed to XLA compiler. Use `None` for default options. Returns: A list of output tensors. Raises: ValueError: If num_shards <= 0 ValueError: If len(input_shard_axes) != len(inputs) ValueError: If len(output_shard_axes) != len(outputs from `computation`) ) r_rhrirjrkrr!rErrA)rx) rr_rhrirjrkrr!rErs r8shardrzs:T ! '5))   ! " "r:ztpu.batch_parallelc &t|||||||S)aShards `computation` along the batch dimension for parallel execution. Convenience wrapper around shard(). `inputs` must be a list of Tensors or None (equivalent to an empty list). Each input is split into `num_shards` pieces along the 0-th dimension, and computation is applied to each shard in parallel. Tensors are broadcast to all shards if they are lexically captured by `computation`. e.g., x = tf.constant(7) def computation(): return x + 3 ... = shard(computation, ...) The outputs from all shards are concatenated back together along their 0-th dimension. Inputs and outputs of the computation must be at least rank-1 Tensors. Args: computation: A Python function that builds a computation to apply to each shard of the input. inputs: A list of input tensors or None (equivalent to an empty list). The 0-th dimension of each Tensor must have size divisible by `num_shards`. num_shards: The number of shards. infeed_queue: If not `None`, the `InfeedQueue` from which to append a tuple of arguments as inputs to `computation`. device_assignment: If not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. Uses a default device assignment if `None`. The `DeviceAssignment` may be omitted if each shard of the computation uses only one core, and there is either only one shard, or the number of shards is equal to the number of cores in the TPU system. name: (Deprecated) Does nothing. xla_options: An instance of `tpu.XLAOptions` which indicates the options passed to XLA compiler. Use `None` for default options. Returns: A list of output tensors. Raises: ValueError: If `num_shards <= 0` )rhrr!rEr)rz)rr_rhrr!rErs r8batch_parallelr|Us'j  )  r:z tpu.rewritec4t||dn|g||||dS)aRewrites `computation` for execution on a TPU system. Args: computation: A Python function that builds a computation to apply to the input. If the function takes n inputs, 'inputs' should be a list of n tensors. `computation` may return a list of operations and tensors. Tensors must come before operations in the returned list. The return value of `rewrite` is a list of tensors corresponding to the tensors from the output of `computation`. All `Operation`s constructed during `computation` will be executed when evaluating any of the returned output tensors, not just the ones returned. inputs: A list of input tensors or `None` (equivalent to an empty list). Each input can be a nested structure containing values that are convertible to tensors. Note that passing an N-dimension list of compatible values will result in a N-dimension list of scalar tensors rather than a single Rank-N tensors. If you need different behavior, convert part of inputs to tensors with `tf.convert_to_tensor`. infeed_queue: If not `None`, the `InfeedQueue` from which to append a tuple of arguments as inputs to `computation`. device_assignment: if not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. May be omitted for a single-core computation, in which case the core attached to task 0, TPU device 0 is used. name: (Deprecated) Does nothing. xla_options: An instance of `tpu.XLAOptions` which indicates the options passed to XLA compiler. Use `None` for default options. Returns: Same data structure as if computation(*inputs) is called directly with some exceptions for correctness. Exceptions include: 1) None output: a NoOp would be returned which control-depends on computation. 2) Single value output: A tuple containing the value would be returned. 3) Operation-only outputs: a NoOp would be returned which control-depends on computation. TODO(b/121383831): Investigate into removing these special cases. Nrmr)r)rr_rr!rErs r8rewriter~s6d nd6()    !  ""r:)ReadVariableOpAssignVariableOpAssignAddVariableOpAssignSubVariableOpr0Variable VariableV2c8tj}|r|j}|r t|try|j }|r t|t jr |j}n(t|tjr |j}ny|ry)z6Check if it is currently under `_TPUInferenceContext`.TF) rr_get_control_flow_contextr_TPUInferenceContext outer_contextr _FuncGraph _outer_graphrr outer_graph)rCrHs r8under_tpu_inference_contextrs   !% --/G  G1 2%%g %,,-  e E:// 0e   r:cVeZdZdZd dedeffd ZdZdZdZ dZ e d Z xZ S) rzA `ControlFlowContext` for nodes inside a TPU inference computation. The primary role of `_TPUInferenceContext` is to indicate the mode of operation and possibly sanity check operators inside a tpu.rewrite_for_inference() computation. rE check_opscFtt| ||_||_yrv)rwr__init___name _check_ops)selfrErr|s r8rz_TPUInferenceContext.__init__s .0DJDOr:c&|j|yrv_AddOpInternalrrVs r8AddOpz_TPUInferenceContext.AddOpr:c|jr8|jtvr&td|jd|jd|j r|j j |yy)NzOperation of type z (z) is not supported on the TPU for inference. Execution will fail if this op is used in the graph. Make sure your variables are using variable_scope.)rr\_DENYLISTED_INFERENCE_OPSNotImplementedErrorrE_outer_context AddInnerOprs r8rz#_TPUInferenceContext._AddOpInternalsi 277&??  rwwir"''3F F GG  $$R(r:cX|}|jr|jj|}|Srv)rAddValue)rvalrs r8rz_TPUInferenceContext.AddValue s+ F ""++C0f Mr:c&|j|yrvrrs r8rz_TPUInferenceContext.AddInnerOprr:cyrvr7)rs r8 grad_statez_TPUInferenceContext.grad_states r:)T)rmrnrorprrirrrrrpropertyrr}r~s@r8rrsC 4 D )   r:rrCcZtd|jDs tdy)aValidates whether rewrite_for_inference() 'worked' for variables. The rewrite_for_inference() method is supposed to append GuaranteeConstOps after ReadVariableOps, but this mechanism works only if you are using tf.compat.v1.get_variable() to create and access variables in your tpu computation. This validation method can be called immediately after calling tpu.rewrite_for_inference() to check whether GuaranteeConstOps where added to the graph. Typical usages: tpu.validate_inference_rewrite_for_variables( tf.compat.v1.get_default_graph()) tpu.validate_inference_rewrite_for_variables(sess.graph) Args: graph: The graph which needs to be validated. Raises: RuntimeError: if validation failed. c3:K|]}|jdk(yw)GuaranteeConstN)r\rs r8rz;validate_inference_rewrite_for_variables..,s HAQVV' ' HszNo GuaranteeConst ops found in the graph after running tpu.rewrite_for_inference(...). Please check that you are using tf.get_variable() to create and access variables in your tpu computation.N)rget_operations RuntimeError)rCs r8(validate_inference_rewrite_for_variablesrs3* H1E1E1G H H    Ir:c8dfd}t|||||S)aRewrites `computation` for inference on a TPU system. Other than 'rewriting' the computation to run on a TPU, if using variables in your computation, it moves the ReadVariableOps outside the TPU computation, and adds GuaranteeConst ops just after the ReadVariableOps. This mechanism works only if you are using tf.compat.v1.get_variable() to create and access variables in your tpu computation. You can validate whether this worked, by calling validate_inference_rewrite_for_variables() method immediately after this method to check whether GuaranteeConstOps where added to the graph. Args: computation: A Python function that builds a computation to apply to the input. If the function takes n inputs, 'inputs' should be a list of n tensors. If the function returns m outputs, rewrite will return a list of m tensors. inputs: A list of input tensors or `None` (equivalent to an empty list). infeed_queue: If not `None`, the `InfeedQueue` from which to append a tuple of arguments as inputs to `computation`. device_assignment: if not `None`, a `DeviceAssignment` describing the mapping between logical cores in the computation with physical cores in the TPU topology. May be omitted for a single-core computation, in which case the core attached to task 0, TPU device 0 is used. name: The name of the operator. Returns: A list of output tensors. ctjd5tj||g|i||dzcdddS#1swYyxYw)Nz/GuaranteeConstrD)rrIrguarantee_const)rrErrs r8guarantee_const_getterz5rewrite_for_inference..guarantee_const_getterVsR ! !$ 'H  & &  ' ' 'd5F.FHHHHs $AA cttjjd} |j t j }|j}|j}|j|jd|i|}|j||j||j|S#|jwxYw)z1Execute computation under `_TPUInferenceContext`.rewrite_for_inferencerDc|jSrv)rG)rVs r8rzDrewrite_for_inference..wrapped_computation..fs 299r:) rrrrrr r!rcaching_devicer$set_caching_devicer/) rrrHrMprev_custom_getterprev_caching_devicerrrs r8wrapped_computationz2rewrite_for_inference..wrapped_computation[s"  " " $ 0 01H IKG mmo002f!//"11 56  45D+F+f 12  34 lln Mllns B C C)r_rr!rE)r~)rr_rr!rErrs` @r8rr4s-DH ,  )   r: prune_graphc|g|jjDcgc]}|c}zD]}t|tjs|j D]}|j tvrd}|jD]}|jsd}n|r@tjd|j|j |jtjycc}w)aPrunes unconnected ops as listed in _UNCONNECTED_OPS_TO_PRUNE. Args: prune_graph: A tensorflow graph from which we wish to prune unconnected ops as listed in _UNCONNECTED_OPS_TO_PRUNE. In general, these ops should have no inputs and no consumers. These can often be left behind due to graph construction rewiring (for instance TF-Hub). While they never execute, they will cause XLA compile to fail so we strip them from XLA compile by removing the tpu_replicate attribute. FTzGPruning OP %s of type %s from XLA Compile due to it being disconnected.N) _functionsvaluesrrGraphrr\_UNCONNECTED_OPS_TO_PRUNErO consumersr rrE _clear_attrr&_TPU_REPLICATE_ATTR)rfrCrVoutputs_consumedoutputs r8prune_unconnected_ops_from_xlarzs}''..0  a rs  ' NNN;4OY!.98.3.20+4+1&2*0L/*./1-6)'2216   >?28)/$/D/ &'(JN-148 6D}FFG6D $6D'+6D$,D> 6D  6D)6DvA#==A $A ]]A: $%& "$ "3== "' "11 D  D#55 D%66 D<@ D2 DF  !"$,,# EK<'#*EE4  !!7;37JN$(*.(,h"#s(#h" T$z0012 3h"8//0h" 5 F FG h" 4. h" SM h";'h"*%h"26ch""!h"V@% Z&& '@%!9!9:;@%@% 4S ?D!;!;< <= @%H48)\7;37JN$(*.(,R.#s(#R. T$z0012 3R.8//0R. 5 F FG R. 4. R.  R.SMR.;'R.*%R. $z  !"R.j ^> ^> ^> 4**+ ,d3==.A49 LM^>B<# <# <# 4**+ ,d3==.A49 LM<#BAE,07;-137JN(,_#s(#_ T$x (9(9:;< =__tCy) _ #4d#34 _  S * _8//0_ 5 F FG_ 4._*%_ s}}d:#4#455 6_D {m!!15,07;-137JN(,R"#s(#R" T*++, -R"R"tCy) R" #4d#34 R"  S * R"8//0R" 5 F FGR" 4.R"*%R"26j6G6G1HR""R"j #$%!!AE37JN(,:#s(#: T$x (9(9:;< =::8//0 :  5 F FG : 4. :*%:"&:z }o!!AE37JN(, 6"#s(#6" T$x (9(9:;< =6"8//06" 5 F FG 6" 4. 6" *% 6" 25 6""6"x !T$%+AA%PCII>1537JN B#s(#B T*++, -B8//0B 5 F FG B 4. B %)):):$; BL<