AVhLdZddlZddlZddlZddlZddlZddlZddlmZddl Z ddl m Z ddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlm Z ddl!m"Z#dd l$mZ%ddl&m'Z'ddl&m(Z(ddl&m)Z)ddl*m+Z+ddl*m,Z-ddl*m.Z.ddl*m/Z/ddl*m0Z0ddl*m1Z1ddl*m2Z2ddl*m3Z3ddl*m4Z4dd l5m6Z6dd!l5m7Z7dd"l5m8Z8dd#l5m9Z9dd$l5m:Z;dd%lm?Z?dd'l@mAZBdd(l@mCZCdd)l@mDZDdd*l@mEZEdd+lFmGZGdd,lHmIZIdd-lHmJZJdd.lHmKZKdd/lHmLZLd0ZMd1ZNd2ZOejd3ZQd4ZRd5ZSeKjd6g7Gd8d9ejZUeKjd:g7eIjd:Gd;dejZYGd?d@ejZ[dAZ\e/je/je/je/je/je/jfZcGdBdCejZedDZfy)Ez TPU Strategy.N)logging)ag_ctx)api) xla_sharding)cross_device_ops) device_util)distribute_lib)distribute_utils) input_lib) input_util) numpy_dataset) reduce_util)tpu_replicated_variable)tpu_util) tpu_values)values)tpu_cluster_resolver)context) def_function)function) constant_op)device) device_spec)dtypes)indexed_slices)ops) sparse_tensor) tensor_shape) tensor_util) array_ops)control_flow_ops)math_ops)resource_variable_ops) variables) ragged_tensor) save_contextdevice_assignment)tpu)tpu_hardware_feature) training_loop)tpu_ops) deprecation)nest) tf_export) tf_inspectFchtxr+tjxrtj S)z8Whether to batch variable initialization in tf.function.)/_EXPERIMENTAL_TPU_BATCH_VARIABLE_INITIALIZATIONrexecuting_eagerlyr&in_save_contextY/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/distribute/tpu_strategy.py$enable_batch_variable_initializationr9Ns16-  # # %-**, ,r7c#Ktjrdytj5ddddy#1swYyxYwwN)r#executing_eagerly_outside_functions init_scoper6r7r8maybe_init_scoper>Ws6,,.       s.A > A AA ctjrqt|tjsVt|t j s;t|r$t|jtjs tdyyyy)z/Validate the function passed into strategy.run.zTPUStrategy.run(fn, ...) does not support pure eager execution. please make sure the function passed into `strategy.run` is a `tf.function` or `strategy.run` is called inside a `tf.function` if eager behavior is enabled.N) rr4 isinstancerFunctionrConcreteFunctioncallable__call__NotImplementedError)fns r8validate_run_functionrG`sq!\223X667 2,:bkk<3H3HI  % &&J84"r7c di}i}|r,|jDcic]\}}|s||}}}|r,|jDcic]\}}|r||}}}g}d}ttj|jj D]\} } | dk(r| j dk(r| jtjjk(r|j| j ^| jtjjk(r| }| jtjjk(s|stfd|Dr-tdt|dt!fd|Dd |||fcSg} d } t|D]\} } | r6|| |k\r td t|| kr td | ||| <d } D|$| |k\r| r td| j| jt|| kr td | ||| <|rt#j$|fi|| |fS|||fScc}}wcc}}w)a|Inspects arguments to partially apply any DistributedVariable. This avoids an automatic cast of the current variable value to tensor. Note that a variable may be captured implicitly with Python scope instead of passing it to run(), but supporting run() keeps behavior consistent with MirroredStrategy. Since positional arguments must be applied from left to right, this function does some tricky function inspection to move variable positional arguments into kwargs. As a result of this, we can't support passing Variables as *args, nor as args to functions which combine both explicit positional arguments and *args. Args: fn: The function to run, as passed to run(). args: Positional arguments to fn, as passed to run(). kwargs: Keyword arguments to fn, as passed to run(). Returns: A tuple of the function (possibly wrapped), args, kwargs (both possibly filtered, with members of args possibly moved to kwargs). If no variables are found, this function is a noop. Raises: ValueError: If the function signature makes unsupported use of *args, or if too many arguments are passed. cntj|}|xrt|dtjSNr)r.flattenr@rDistributedVariable)xflats r8is_distributed_varz:_maybe_partial_apply_variables..is_distributed_vars+ <<?D  CJtAw(B(BCCr7Nrselfc3.K|] }|ywr;r6.0arOs r8 z1_maybe_partial_apply_variables..s?Q-a0?zWMixing Variables and positional-only parameters not supported by TPUStrategy. Received z& DistributedVariables in **kwargs and c3.K|] }|ywr;r6rRs r8rUz1_maybe_partial_apply_variables..sD! 21 5DrVz" in *args, expected zero for both.FzTPUStrategy.run() cannot handle Variables passed to *args. Either name the function argument, or capture the Variable implicitly.zBToo many positional arguments passed to call to TPUStrategy.run().TzTPUStrategy.run() cannot handle both Variables and a mix of positional args and *args. Either remove the *args, or capture the Variable implicitly.)items enumerater0 signature parametersrnamekind ParameterPOSITIONAL_OR_KEYWORDappendVAR_POSITIONALPOSITIONAL_ONLYany ValueErrorlensum functoolspartial)rFargskwargs var_kwargs nonvar_kwargskvpositional_argsindex_of_star_argsip star_argshave_seen_var_argrTrOs @r8_maybe_partial_apply_variablesru{s<D *- #)<<>K41a5G5J!Q$KJKA/A!/D1M/ ,,R0;;BBDEda Av!&&F"vv%%;;;QVV$ :''66 6 :''77 7 s?$?? %%(_$56DtDDEF' '  v ;>)o,da!  'A1C,C   _  " P  ()j#$  'A1C,C )* *   1   _  " P  +,mOA&'9,<   R .: . = HH T6 [LsII IIzdistribute.TPUStrategy)v1cTeZdZdZ dfd Zd dZedZdZdZ dZ xZ S) TPUStrategyV2aSynchronous training on TPUs and TPU Pods. To construct a TPUStrategy object, you need to run the initialization code as below: >>> resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') >>> tf.config.experimental_connect_to_cluster(resolver) >>> tf.tpu.experimental.initialize_tpu_system(resolver) >>> strategy = tf.distribute.TPUStrategy(resolver) While using distribution strategies, the variables created within the strategy's scope will be replicated across all the replicas and can be kept in sync using all-reduce algorithms. To run TF2 programs on TPUs, you can either use `.compile` and `.fit` APIs in `tf.keras` with TPUStrategy, or write your own customized training loop by calling `strategy.run` directly. Note that TPUStrategy doesn't support pure eager execution, so please make sure the function passed into `strategy.run` is a `tf.function` or `strategy.run` is called inside a `tf.function` if eager behavior is enabled. See more details in https://www.tensorflow.org/guide/tpu. `distribute_datasets_from_function` and `experimental_distribute_dataset` APIs can be used to distribute the dataset across the TPU workers when writing your own training loop. If you are using `fit` and `compile` methods available in `tf.keras.Model`, then Keras will handle the distribution for you. An example of writing customized training loop on TPUs: >>> with strategy.scope(): ... model = tf.keras.Sequential([ ... tf.keras.layers.Dense(2, input_shape=(5,)), ... ]) ... optimizer = tf.keras.optimizers.SGD(learning_rate=0.1) >>> def dataset_fn(ctx): ... x = np.random.random((2, 5)).astype(np.float32) ... y = np.random.randint(2, size=(2, 1)) ... dataset = tf.data.Dataset.from_tensor_slices((x, y)) ... return dataset.repeat().batch(1, drop_remainder=True) >>> dist_dataset = strategy.distribute_datasets_from_function( ... dataset_fn) >>> iterator = iter(dist_dataset) >>> @tf.function() ... def train_step(iterator): ... ... def step_fn(inputs): ... features, labels = inputs ... with tf.GradientTape() as tape: ... logits = model(features, training=True) ... loss = tf.keras.losses.sparse_categorical_crossentropy( ... labels, logits) ... ... grads = tape.gradient(loss, model.trainable_variables) ... optimizer.apply_gradients(zip(grads, model.trainable_variables)) ... ... strategy.run(step_fn, args=(next(iterator),)) >>> train_step(iterator) For the advanced use cases like model parallelism, you can set `experimental_device_assignment` argument when creating TPUStrategy to specify number of replicas and number of logical devices. Below is an example to initialize TPU system with 2 logical devices and 1 replica. >>> resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') >>> tf.config.experimental_connect_to_cluster(resolver) >>> topology = tf.tpu.experimental.initialize_tpu_system(resolver) >>> device_assignment = tf.tpu.experimental.DeviceAssignment.build( ... topology, ... computation_shape=[1, 1, 1, 2], ... num_replicas=1) >>> strategy = tf.distribute.TPUStrategy( ... resolver, experimental_device_assignment=device_assignment) Then you can run a `tf.add` operation only on logical device 0. >>> @tf.function() ... def step_fn(inputs): ... features, _ = inputs ... output = tf.add(features, features) ... ... # Add operation will be executed on logical device 0. ... output = strategy.experimental_assign_to_logical_device(output, 0) ... return output >>> dist_dataset = strategy.distribute_datasets_from_function( ... dataset_fn) >>> iterator = iter(dist_dataset) >>> strategy.run(step_fn, args=(next(iterator),)) `experimental_spmd_xla_partitioning` enables the experimental XLA SPMD feature for model parallelism. This flag can reduce the compilation time and HBM requirements. When running in this mode, every input tensor must either be partitioned (via `strategy.experimental_split_to_logical_devices`) or fully replicated (via `strategy.experimental_replicate_to_logical_devices`) to all logical devices. And calling `strategy.experimental_assign_to_logical_device` will result in a ValueError in this mode. ct|t||||tjj dj dtjj dj |jjtjj dj |jjd|_ y)aKSynchronous training in TPU donuts or Pods. Args: tpu_cluster_resolver: A `tf.distribute.cluster_resolver.TPUClusterResolver` instance, which provides information about the TPU cluster. If None, it will assume running on a local TPU worker. experimental_device_assignment: Optional `tf.tpu.experimental.DeviceAssignment` to specify the placement of replicas on the TPU cluster. experimental_spmd_xla_partitioning: If True, enable the SPMD (Single Program Multiple Data) mode in XLA compiler. This flag only affects the performance of XLA compilation and the HBM requirement of the compiled TPU program. Ceveat: if this flag is True, calling `tf.distribute.TPUStrategy.experimental_assign_to_logical_device` will result in a ValueError. )r(use_spmd_for_xla_partitioningV2 TPUStrategy num_workersnum_replicas_per_workerTN super__init__ TPUExtendedr distribution_strategy_gaugeget_cellset#distribution_strategy_replica_gaugeextended num_hostsnum_replicas_per_host%_enable_packed_variable_in_eager_mode)rPrexperimental_device_assignment"experimental_spmd_xla_partitioning __class__s r8rzTPUStrategyV2.__init__Ys* G  <*L  ..77=AA-P66??s4==22366??!##&3t}}'J'J#K26D.r7ct|t|||\}}}tj|t j }|xst j}|jj||||S)a3Run the computation defined by `fn` on each TPU replica. Executes ops specified by `fn` on each replica. If `args` or `kwargs` have `tf.distribute.DistributedValues`, such as those produced by a `tf.distribute.DistributedDataset` from `tf.distribute.Strategy.experimental_distribute_dataset` or `tf.distribute.Strategy.distribute_datasets_from_function`, when `fn` is executed on a particular replica, it will be executed with the component of `tf.distribute.DistributedValues` that correspond to that replica. `fn` may call `tf.distribute.get_replica_context()` to access members such as `all_reduce`. All arguments in `args` or `kwargs` should either be nest of tensors or `tf.distribute.DistributedValues` containing tensors or composite tensors. Example usage: >>> resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') >>> tf.config.experimental_connect_to_cluster(resolver) >>> tf.tpu.experimental.initialize_tpu_system(resolver) >>> strategy = tf.distribute.TPUStrategy(resolver) >>> @tf.function ... def run(): ... def value_fn(value_context): ... return value_context.num_replicas_in_sync ... distributed_values = ( ... strategy.experimental_distribute_values_from_function(value_fn)) ... def replica_fn(input): ... return input * 2 ... return strategy.run(replica_fn, args=(distributed_values,)) >>> result = run() Args: fn: The function to run. The output must be a `tf.nest` of `Tensor`s. args: (Optional) Positional arguments to `fn`. kwargs: (Optional) Keyword arguments to `fn`. options: (Optional) An instance of `tf.distribute.RunOptions` specifying the options to run `fn`. Returns: Merged return value of `fn` across replicas. The structure of the return value is the same as the return value from `fn`. Each element in the structure can either be `tf.distribute.DistributedValues`, `Tensor` objects, or `Tensor`s (for example, if running on a single replica). rGru autograph tf_convert autograph_ctxcontrol_status_ctxr RunOptionsrtpu_runrPrFrirjoptionss r8runzTPUStrategyV2.runsm`"5b$GBf   b-"B"B"D EB4224G == T67 ;;r7c.|jjS)a[Returns the cluster resolver associated with this strategy. `tf.distribute.TPUStrategy` provides the associated `tf.distribute.cluster_resolver.ClusterResolver`. If the user provides one in `__init__`, that instance is returned; if the user does not, a default `tf.distribute.cluster_resolver.TPUClusterResolver` is provided. r_tpu_cluster_resolverrPs r8cluster_resolverzTPUStrategyV2.cluster_resolvers == . ..r7c|jjr td|jjjd}|dks||k\rtdj ||t j||dS)ayAdds annotation that `tensor` will be assigned to a logical device. This adds an annotation to `tensor` specifying that operations on `tensor` will be invoked on logical core device id `logical_device_id`. When model parallelism is used, the default behavior is that all ops are placed on zero-th logical device. ```python # Initializing TPU system with 2 logical devices and 4 replicas. resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') tf.config.experimental_connect_to_cluster(resolver) topology = tf.tpu.experimental.initialize_tpu_system(resolver) device_assignment = tf.tpu.experimental.DeviceAssignment.build( topology, computation_shape=[1, 1, 1, 2], num_replicas=4) strategy = tf.distribute.TPUStrategy( resolver, experimental_device_assignment=device_assignment) iterator = iter(inputs) @tf.function() def step_fn(inputs): output = tf.add(inputs, inputs) # Add operation will be executed on logical device 0. output = strategy.experimental_assign_to_logical_device(output, 0) return output strategy.run(step_fn, args=(next(iterator),)) ``` Args: tensor: Input tensor to annotate. logical_device_id: Id of the logical core to which the tensor will be assigned. Raises: ValueError: The logical device id presented is not consistent with total number of partitions specified by the device assignment or the TPUStrategy is constructed with `experimental_spmd_xla_partitioning=True`. Returns: Annotated tensor with identical value as `tensor`. zCannot assign a tensor to a logical device in SPMD mode. To disable SPMD, Please construct the TPUStrategy with `experimental_spmd_xla_partitioning=False`rz`logical_core_id` to assign must be lower then total number of logical devices per replica. Received logical device id {} but there are only total of {} logical devices in replica.Tuse_sharding_op)r_use_spmd_for_xla_partitioningrd _tpu_devicesshapeformatr assign_device)rPtensorlogical_device_idnum_logical_devices_per_replicas r8%experimental_assign_to_logical_devicez3TPUStrategyV2.experimental_assign_to_logical_devices\ }}33  7 88 '+mm&@&@&F&Fq&I#A<< 56? ? ?>>  %%+V,K,@,@&B CCii 45==O   V_d KKr7c0tj|dS)aAdds annotation that `tensor` will be replicated to all logical devices. This adds an annotation to tensor `tensor` specifying that operations on `tensor` will be invoked on all logical devices. ```python # Initializing TPU system with 2 logical devices and 4 replicas. resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') tf.config.experimental_connect_to_cluster(resolver) topology = tf.tpu.experimental.initialize_tpu_system(resolver) device_assignment = tf.tpu.experimental.DeviceAssignment.build( topology, computation_shape=[1, 1, 1, 2], num_replicas=4) strategy = tf.distribute.TPUStrategy( resolver, experimental_device_assignment=device_assignment) iterator = iter(inputs) @tf.function() def step_fn(inputs): images, labels = inputs images = strategy.experimental_split_to_logical_devices( inputs, [1, 2, 4, 1]) # model() function will be executed on 8 logical devices with `inputs` # split 2 * 4 ways. output = model(inputs) # For loss calculation, all logical devices share the same logits # and labels. labels = strategy.experimental_replicate_to_logical_devices(labels) output = strategy.experimental_replicate_to_logical_devices(output) loss = loss_fn(labels, output) return loss strategy.run(step_fn, args=(next(iterator),)) ``` Args: tensor: Input tensor to annotate. Returns: Annotated tensor with identical value as `tensor`. Tr)r replicate)rPrs r8)experimental_replicate_to_logical_devicesz7TPUStrategyV2.experimental_replicate_to_logical_devicesis\  ! !&$ ??r7)NNFr6NN) __name__ __module__ __qualname____doc__rrpropertyrrrr __classcell__rs@r8rxrxsIcL%).227%6N8>> resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') >>> tf.config.experimental_connect_to_cluster(resolver) >>> tf.tpu.experimental.initialize_tpu_system(resolver) >>> strategy = tf.distribute.experimental.TPUStrategy(resolver) While using distribution strategies, the variables created within the strategy's scope will be replicated across all the replicas and can be kept in sync using all-reduce algorithms. To run TF2 programs on TPUs, you can either use `.compile` and `.fit` APIs in `tf.keras` with TPUStrategy, or write your own customized training loop by calling `strategy.run` directly. Note that TPUStrategy doesn't support pure eager execution, so please make sure the function passed into `strategy.run` is a `tf.function` or `strategy.run` is called inside a `tf.function` if eager behavior is enabled. ctjdt| t |||t j jdjdt jjdj|jjt jjdj|jjd|_ y) aPSynchronous training in TPU donuts or Pods. Args: tpu_cluster_resolver: A tf.distribute.cluster_resolver.TPUClusterResolver, which provides information about the TPU cluster. device_assignment: Optional `tf.tpu.experimental.DeviceAssignment` to specify the placement of replicas on the TPU cluster. z`tf.distribute.experimental.TPUStrategy` is deprecated, please use the non-experimental symbol `tf.distribute.TPUStrategy` instead.r'r{r|r}r~TN)rwarningrrrr rrrrrrrr)rPrr(rs r8rzTPUStrategy.__init__s OO KL G  / ..77=AA-P66??s4==22366??!##&3t}}'J'J#K26D.r7ct|t|||\}}}tj|t j }|xst j}|jj||||S)zSee base class.rrs r8rzTPUStrategy.runsj"5b$GBf   b-"B"B"D EB4224G == T67 ;;r7c.|jjS)alReturns the cluster resolver associated with this strategy. `tf.distribute.experimental.TPUStrategy` provides the associated `tf.distribute.cluster_resolver.ClusterResolver`. If the user provides one in `__init__`, that instance is returned; if the user does not, a default `tf.distribute.cluster_resolver.TPUClusterResolver` is provided. rrs r8rzTPUStrategy.cluster_resolvers == . ..r7)NNr) rrrrrrrrrrs@r8r|r|s/0%)!%6F <  /  /r7r|cBeZdZdZ dfd ZedZddZxZS) TPUStrategyV1z)TPU distribution strategy implementation.ct|t||||tjj dj dtjj dj |jjtjj dj |jjd|_ y)aInitializes the TPUStrategy object. Args: tpu_cluster_resolver: A tf.distribute.cluster_resolver.TPUClusterResolver, which provides information about the TPU cluster. steps_per_run: Number of steps to run on device before returning to the host. Note that this can have side-effects on performance, hooks, metrics, summaries etc. This parameter is only used when Distribution Strategy is used with Keras. device_assignment: Optional `tf.tpu.experimental.DeviceAssignment` to specify the placement of replicas on the TPU cluster. Currently only supports the usecase of using a single core within a TPU cluster. V1r|r}r~TNr)rPr steps_per_runr(rs r8rzTPUStrategyV1.__init__s$ G[ "M3DFG..77=AA-P66??s4==22366??!##&3t}}'J'J#K26D.r7c.|jjS)z0DEPRECATED: use .extended.steps_per_run instead.) _extendedrrs r8rzTPUStrategyV1.steps_per_runs >> ' ''r7ct|t|||\}}}tj|t j }|xst j}|jj||||S)aRun `fn` on each replica, with the given arguments. Executes ops specified by `fn` on each replica. If `args` or `kwargs` have "per-replica" values, such as those produced by a "distributed `Dataset`", when `fn` is executed on a particular replica, it will be executed with the component of those "per-replica" values that correspond to that replica. `fn` may call `tf.distribute.get_replica_context()` to access members such as `all_reduce`. All arguments in `args` or `kwargs` should either be nest of tensors or per-replica objects containing tensors or composite tensors. Users can pass strategy specific options to `options` argument. An example to enable bucketizing dynamic shapes in `TPUStrategy.run` is: >>> resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='') >>> tf.config.experimental_connect_to_cluster(resolver) >>> tf.tpu.experimental.initialize_tpu_system(resolver) >>> strategy = tf.distribute.experimental.TPUStrategy(resolver) >>> options = tf.distribute.RunOptions( ... experimental_bucketizing_dynamic_shape=True) >>> dataset = tf.data.Dataset.range( ... strategy.num_replicas_in_sync, output_type=dtypes.float32).batch( ... strategy.num_replicas_in_sync, drop_remainder=True) >>> input_iterator = iter(strategy.experimental_distribute_dataset(dataset)) >>> @tf.function() ... def step_fn(inputs): ... output = tf.reduce_sum(inputs) ... return output >>> strategy.run(step_fn, args=(next(input_iterator),), options=options) Args: fn: The function to run. The output must be a `tf.nest` of `Tensor`s. args: (Optional) Positional arguments to `fn`. kwargs: (Optional) Keyword arguments to `fn`. options: (Optional) An instance of `tf.distribute.RunOptions` specifying the options to run `fn`. Returns: Merged return value of `fn` across replicas. The structure of the return value is the same as the return value from `fn`. Each element in the structure can either be "per-replica" `Tensor` objects or `Tensor`s (for example, if running on a single replica). rrs r8rzTPUStrategyV1.runskf"5b$GBf   b-"B"B"D EB4224G == T67 ;;r7)NNNr) rrrrrrrrrrs@r8rrs/1%)!!%6< ( (9|_ t=j>|_!|j dddfD]} t3j8| }|j@jE|g|j@|j1| |jBjE|g|jB|j1|||_#d|_$d|_%dg|_&tOjPr#tSjTtNjV| |_,||_-d|_.|j dd|jLd f}|D]'}tOj^|sd|_.d|_0yd|_0ycc}wcc}w) Nrz device:TPU:dtyper)replica logical_corejobTF)1rrtpu_cluster_resolver_libTPUClusterResolverweakrefWeakKeyDictionary_tpu_function_cacherget_tpu_system_metadata _tpu_metadata_device_assignmentdevicesr\rarrayobjectrr DeviceSpecV2 from_stringrrange num_replicasnum_cores_per_replicar`r canonicalize tpu_deviceget_host_for_device _host_device collections OrderedDict_device_input_worker_devices_host_input_worker_devices setdefaultr_require_static_shapes(experimental_enable_get_next_as_optional_logical_device_stackrr4atexitregister async_wait_use_var_policyr_using_custom_deviceis_custom_device_enable_data_reorder)rPcontainer_strategyrrr(rzdtpu_devices_flatjob_name tpu_devices replica_idreplica_devicesrr host_devicerrs r8rzTPUExtended.__init__Zs9 G'(#5HHLm '88:D!5D33KKMD/D**22Omqvv6MOO  ((( )1A3 )9d))556Fq6IJNNhk/<<= ,*!"3"I"IJ &L  &&#..(#/"/$% & & ?+ ,((;f=d#778I8I!8LQ8OPD )4(?(?(AD%&1&=&=&?D#''1-G 33J?k ''22;C '' 4;;JG %%00bA %%k299+F G'D"&D48D1"#D  " oog(() =Found tensor {} with spec {}. TPUStrategy does not support distributed datasets with device prefetch when using sparse or ragged tensors. If you intend to use sparse or ragged tensors, please pass a tf.distribute.InputOptions object with experimental_fetch_to_device set to False to your dataset distribution function.) r@rPerReplicaSpec_component_specsr. flatten_with_joined_string_pathsrSparseTensorSpecr%RaggedTensorSpecrdrtype)rP element_specspecspathrs r8 _check_speczTPUExtended._check_specs, 5 56!22l  1 1, ?E ? d D=99(99; < % &,VD$t*%= ? ? ?r7c \|r2|jtjjk(r t d| |j r|j |jtj||j||j|j||jS)NzgInputReplicationMode.PER_REPLICA is only supported in `experimental_distribute_datasets_from_function`.)rr replica_order)experimental_replication_moder InputReplicationMode PER_REPLICArEr1r=r:r get_distributed_datasetr2rrr)rPrrs r8 _experimental_distribute_datasetz,TPUExtended._experimental_distribute_dataset&sG99++778  >  '>> w++,  - - (   "!77--/  r7c |r2|jtjjk(r t d|j |}g}|j }t|D]3}|jtj|||j5tj||||j||j}| |jr|j!|j"|S)NzInputReplicationMode.PER_REPLICA is only supported in `experimental_distribute_datasets_from_function` of tf.distribute.MirroredStrategyr!)rr?)r@r rArBrEr2r}rr`r$rr &get_distributed_datasets_from_functionrrr1r=r:)rP dataset_fnrrr(r}rqdistributed_datasets r8"_distribute_datasets_from_functionz.TPUExtended._distribute_datasets_from_function:sG99++778  . // ++G4MN++K ; <N77)#99;<< %KK   "--/ '>> *778 r7c g}t|jD]7}|j|tj||j9t j |dS)NT) always_wrap)rrr`r ValueContextr regroup)rPvalue_fnper_replica_valuesrs r8-_experimental_distribute_values_from_functionz9TPUExtended._experimental_distribute_values_from_functionYskD667M  >..z/3/I/IK LMM  # #$6D IIr7c" |i}tj|}tj fd t j j _ fd}t|tsJ|jz}t jj5jdk(r|}ntj|||}ddd`t!j" _t|tra|Dcgc]}t|tj&r|!}}t)|jz} t+| D cgc] } || d|  }} ng}t- | S#1swYxYwcc}wcc} w)Nc|}tjj}|rCtj|g5|Dcgc]}t j |c}cdddS|Scc}w#1swYyxYw)zSingle step on the TPU device.N)r.rKlast_step_outputsrcontrol_dependenciesr identity)inputs fn_resultflat_last_step_outputsfctxrFs r8run_fnz?TPUExtended._experimental_run_steps_on_iterator..run_fnls|S&/i#||C,A,AB   % %yk 2 I1GHA)$$Q'H I II I IsA6A1#A61A66A?c~j}g}tjD]-fd}|jt j ||f/t j|jt jj}t|dtrt j|}|S)z%The rewritten step fn running on TPU.c0tj|Sr;)r select_replica)rMrs r8zUTPUExtended._experimental_run_steps_on_iterator..rewrite_fn..s#3#B#B $r7rz)r( xla_optionsr)get_nextrrr`r. map_structurer)rr XLAOptionsrr@listrK) riper_replica_inputsreplicate_inputsr^replicate_outputsrmulti_worker_iteratorr[rPs @r8 rewrite_fnzCTPUExtended._experimental_run_steps_on_iterator..rewrite_fns 099;d8893*!3!3 ."0!2 33 --   33nn4&D&DF G %a($ / LL):; r7r)r.rKr MultiStepContextrget_default_graph_get_control_flow_context_outer_control_flow_contextr@rerrrrr+repeatr!grouprun_op Operationrer_set_last_step_outputs) rPrFri iterationsinitial_loop_valuesrjrhrMlast_step_tensor_outputs output_numrqrZr[s ``` @@r8#_experimental_run_steps_on_iteratorz/TPUExtended._experimental_run_steps_on_iteratords",,':;  $ $ &C" 99; $: )4 00 0-0J0JJ D%% &F  q &L)00Z1DF F (!''(9:CJ#T*'"jCMM.J!""/0D4N4NNj;@;L"67 "1=j= 1"" "$3 89 JEFF""s'/E;F.FF ;Fcpt|j5||i|cdddS#1swYyxYwr;)_TPUReplicaContextr)rPrFrirjs r8_call_for_each_replicaz"TPUExtended._call_for_each_replicas7 D446 7!   !!!s,5c#K|jjd}||k\rtdj|||jj | t jdn5tjtj|5dddd|jjy#1swY$xYw#|jjwxYww)9Places variables and ops on the specified logical device.rz]`logical_device_id` not in range (was {}, but there are only {} logical devices per replica).N) rrrdrrr`renclosing_tpu_contextrrr)corepop)rPrrs r8experimental_logical_devicez'TPUExtended.experimental_logical_devices'+&7&7&=&=a&@#;;  **0&!@+B CC  %%&78'  ' ' ) 1 ZZ!23 4      $$&     $$&s7AC,AC CC &C,C C C))C,cBtj|jy)zExperimental method added to be used by Estimator. This is a private method only to be used by Estimator. Other frameworks should directly be calling `tf.tpu.experimental.initialize_tpu_system` N)rinitialize_tpu_systemrrs r8_experimental_initialize_systemz+TPUExtended._experimental_initialize_systems 2243M3MNr7c  |jddrdi|S|jdd}| |fi|S|jdd}|!jddjdf nWt|tj r1t j|j5di|cdddS|j jj\ fd  fd fd  fd } fd }js&trjr d kDr|}n|}d |d<njr d kDr }n }tjj|tj tj"fi|}js!trt%|dj&|S#1swYxYw)z?Create a TPUMirroredVariable. See `DistributionStrategy.scope`.skip_mirrored_creatorFcustom_tpu_variable_creatorN colocate_withrcJd}g}tD]\}}tj|5|dk(r,|d}t5t |r|n|}ddd|dkDr+|dj j dd}d||fz|d<||d<tjtj5di|}dddttjrJ|j|ddd|S#1swYxYw#1swYNxYw#1swYxYw)Returns a list of `tf.Variable`s. The list contains `number_replicas` `tf.Variable`s and can be used to initialize a `TPUMirroredVariable`. Args: **kwargs: the keyword arguments for creating a variable Nr initial_value:%s/replica_%d/r\r6)rYrrr>rCr\splitr device_policyDEVICE_PLACEMENT_SILENTr@rTPUMirroredVariabler`) rjr value_listrqrvar0namernr next_creators r8_create_mirrored_tpu_variableszDTPUExtended._create_variable.._create_mirrored_tpu_variabless;mjG$$!Q ZZ]  !V"?3M"# 4192!mo&3 4U!!}))//4Q7H.1 =F6N$1& !$$W%D%DE '&v&A ' :#A#AB BB   A -  0% 4 4 ' '%  s<DDA D6 D ?5DD D D DD" c|d}t5t|r|n|}dddg}t D]}g}t D]N}tj j ||5||d< di|}ddd|j Pdj|d|}tj||}|j ||S#1swYxYw#1swYixYw)Returns a list of `TPUReplicatedVariable`s. The list consists of `num_replicas` `TPUReplicatedVariable`s and can be used to initialize a `TPUMirroredVariable`. Each `TPUReplicatedVariable` contains a list of `tf.Variable`s which are replicated to `num_cores_per_replica` logical cores to enable XLA SPMD compilation. Args: **kwargs: the keyword arguments for creating a variable rN{}/r:{}r\r$r\r6) r>rCrrrrr`rrTPUReplicatedVariable) rjrmirrored_replicated_var_listrreplicated_var_list logic_core_idrn replica_nametpu_replicated_varrrrrPs r8)_create_mirrored_tpu_replicated_variableszOTPUExtended._create_variable.._create_mirrored_tpu_replicated_variables#s_-m  .+3 ,  - .&("l+ @* "#89 (Mzz$++J7 FG '&3F? #&v&A '  $ $Q '  ( !''v C 4JJ) > %++,>? @*)%.. ' 'sC3C)C&)C2 ctjdi|}jj|t |dj|S)N _lazy_scoper6)rTPUUninitializedVariablelazy_variable_trackeradd_uninitialized_varsetattr)rjuninitialized_variablerPs r8uninitialized_variable_creatorzDTPUExtended._create_variable..uninitialized_variable_creatorIsI'@@J6J   66 $mT5O5OP ##r7c z|jddd i|Sg}d}tD]g\}}tj|5|dk(r`|jdd}t 5|9t |r|}tj ||jdd}ddd|dkDr+|djjdd}d||fz|d<||d<|jdd|dj|d<|jd d|dj|d <tjtj5 d i|}dddttj rJ|j#|dddj|S#1swYxYw#1swYOxYw#1swYxYw) rrNrrrrrr\rr6)getrYrrr>rCconvert_to_tensorr\rrrrrrr@rrr`) rjrrrqrrrnrrrs r8,_create_uninitialized_mirrored_tpu_variableszRTPUExtended._create_variable.._create_uninitialized_mirrored_tpu_variablesRs OT * 2-777jmG$"$!Q ZZ]!  !V"JJ=M!# *M*"//- # 5 5!GT)B!  U!!}))//4Q7H.1 =F6N$1& ! ZZ & .$_5;;F7O ZZ & .$_5;;F7O$$W%D%DE 9.88A 9 :#A#AB BB   A C! ! "F=  2 9 9;! ! s="F0)rCrrrrrrrr`rrr)rjrrrrrrrrnrrrrrrPrs r87_create_uninitialized_mirrored_tpu_replicated_variablesz]TPUExtended._create_variable.._create_uninitialized_mirrored_tpu_replicated_variablessjj$'ejj$'ejj$7m  8B6BB  <  $ m $)OM//5 -%2& ! ]$_5;;F7O ]$_5;;F7O< &("l+ @* "#89 (Mzz$++J7 FG 9.88A 9  $ $Q ' (!''v C 4JJ)   %++,>? @*)=<<* 9 9s AE2 E#E #E, rT!experimental_batch_initializationrr6)rrrr@r r,rr_devicesrrr9rr create_mirrored_variablerTPU_VARIABLE_CLASS_MAPPINGTPU_VARIABLE_POLICY_MAPPINGrr)rPrrjrrrr real_creatormirrored_variablerrrrrrs`` @@@@@@r8_create_variablezTPUExtended._create_variableszz)51  #F ##"(**%t##. ( @ @@JJ5M!!!T%?%?%C"CDg M=#=#= > ::m** +&%f%&&&&g*.*;*;*A*A'L'#J *L$2h0*0*d  $ $)M)O , ,1F1JN C 48f 01  , ,1F1J@ 5 (AA   "3344     $ $)M)O 0J0JK u&&s .GG cft|ddstj|_|jS)N_lazy_variable_tracker)getattrrLazyVariableTrackerrrs r8rz!TPUExtended.lazy_variable_trackers, 414 8$,$@$@$Bd!  & &&r7c:fd}tjd|S)Nctj}jjD]+}t j |5||i|||<ddd-t jj|S#1swY[xYwr;) rrrkeysrrrPerWorkerResourcer)rrirj host_to_tablerrPs r8lookup_creatorz;TPUExtended._resource_creator_scope..lookup_creators!--/m::??AE+ ZZ $ E'3T'DV'D- $ E EE % %d&>&>&@- PP E Es  BB StaticHashTable)rresource_creator_scope)rPrs` r8_resource_creator_scopez#TPUExtended._resource_creator_scopesQ  % %&7 HHr7cjttjsStj}ttjr4j (tfdj j D}tjtkrtj||}nltj|dt|}ttt|tdz D]*}tj|g|||tzdz z|},|j||}|S)Nc3TK|]}jj|!ywr;_packed_variable on_devicerSrvalues r8rUz8TPUExtended._gather_to_implementation..s*3  * *1 -3%(axisr) r@rDistributedValuesrerLrrre_XLA_OP_BY_OP_INPUTS_LIMITr concatr_broadcast_output)rPr destinationsrrroutputrqs ` r8_gather_to_implementationz%TPUExtended._gather_to_implementations eV55 6 lell#J ""$(-(>(>(J3))1133j 5<<66 6f 00 1>f/Z/!35!!! Hz!A(B$BQ$FG G  # #L& 9F Mr7cxtj|}t|dk(rstj|d}tj|j }||k7r5t j|5tj|}ddd|S|Stj||}|S#1swY|SxYw)Nrr) cross_device_ops_libget_devices_fromrerrrrrr rUsimple_broadcast)rPrrrdest_canonicalhost_canonicals r8rzTPUExtended._broadcast_outputs"33LAG 7|q"// ;n"//0A0ABn > ) ZZ ' .%%f-& . M6M$44V\Jf M  . Ms 5B//B9crttjstjrt j |tjjk(r$tjd|jz n,|tjjk7rtd|dtj Sttjs"t#j$|||jSj}ttj&r4j((t+fdj(j,D}t/jt0krtj2|}nht5j6|d|dj8}t;dt/|t0D]$}|tj2|||t0zz }&|tjjk(r|dt/|z z}|j=||}|S)Ng?z `reduce_op`=z[ is not supported. Currently we only support ReduceOp.SUM and ReduceOp.MEAN in TPUStrategy.c3TK|]}jj|!ywr;rrs r8rUz)TPUExtended._reduce_to..0s*3  * *1 -3rrr)r@rrr is_tf_typerr~rReduceOpMEANr" scalar_mulrSUMrEr,cross_replica_sumrreduce_non_distributed_valuerLrrrreradd_nr zeros_likerrr)rP reduce_oprrrrrrqs ` r8 _reduce_tozTPUExtended._reduce_tos5&223u%  + + - 9 k**// /##R(B(B%BUK  ,,00 0!9+&E EF F & &u -- eV55 6 " > > UL$*D*DFFJ ""$(-(>(>(J3))1133j 5<<66~~j)f##JqMA9L9LMfQJ)CDO!(..Aa2L.L!MNNOK((--- c*o%&f  # #L& 9F Mr7ct|tjst|tjsJt j |r ||g|i|S||g|i|fS|j}|/tjs|r ||g|i|S||g|i|fSg}g}|%|jD]} |j|| fn.|jD]} |j| | jf!|jtj j"k7rQ|j$tj&j(k7r*t+j,|t+j,|t/|D]\} } | d} | d} d| z} t1j| 5t3j4| 5t1j6| 5|j|| gt+j8| |it+j8| |dddddddddt+j:|||S#1swY2xYw#1swY6xYw#1swYxYw)Nrrz update_%d)r@rTPUVariableMixinr#BaseResourceVariablerr~rrr4rr`rrsynchronization variables_libVariableSynchronizationON_READ aggregationVariableAggregationNONEr assert_mirroredrYrr UpdateContext name_scoper^update_regroup)rPvarrFrirjrp packed_varupdatesvalues_and_devicesrrrqvalue_and_devicer\s r8_updatezTPUExtended._updateFsn c:66 7: " 7 7<99 9%%'3 #''''3(((**%%Jg&?&?&A *.t.v..:///11G&&8&!!:v"678::9%!!5%,,"789 }DDLLL =<<AAA&&t,&&v.();< > q!e"f 1_d ::f >  ' ' *> >>$ >  u ='66q$? =!00F; = > >>>> >  * *4% @@>>>>>>s=)I"?IAI I!I" I II I""I+ ct|tjst|tjsJ|j Sr;)r@rrr#r read_value)rPrs r8read_varzTPUExtended.read_varts< c:66 7: " 7 7<99 9 >> r7c|Sr;r6)rPrs r8value_containerzTPUExtended.value_containerys Lr7c~t|ttfr|StjJt |j Dcgc]}|}}tj|dd|j }|dS|Scc}w)Nrconcat_dimensionsplit_dimension split_count) r@floatintrr~rrr, all_to_all)rPrrrbroadcast_tensorresults r8 _broadcast_tozTPUExtended._broadcast_to|s &5#,' m%%'3*/0J0J*KLQ&LL!! 00 2fAY MMs A:c |j|jjStt t |jj Dcgc]}|jj|c}Scc}wr;)rrrrerrrr)rPrs r8rzTPUExtended.num_hostssw &    ) )) s"4#:#:#G#GHJ++77:JK LLJs"A; c|j|jjS|jj|jjz}t |jj |Sr;)rrnum_of_cores_per_hostrminr)rPmax_models_per_hosts r8rz!TPUExtended.num_replicas_per_hostsc &    5 55 --CC22HHI t&&335H IIr7cr|j|jjS|jjSr;)rr num_coresrrs r8rz!TPUExtended._num_replicas_in_syncs2 &    ) ))  " " / //r7cy)NFr6rs r8experimental_between_graphz&TPUExtended.experimental_between_graphs r7cyNTr6rs r8experimental_should_initz$TPUExtended.experimental_should_init r7cyrr6rs r8should_checkpointzTPUExtended.should_checkpointrr7cyrr6rs r8should_save_summaryzTPUExtended.should_save_summaryrr7cTt|jdd|jdfS)Nr)rrrrs r8worker_deviceszTPUExtended.worker_devicess( ""1d&@&@&D#DE FFr7c|jSr;)rrs r8parameter_deviceszTPUExtended.parameter_devicess   r7cTtj|jjS)z7Return the `tf.tpu.experimental.HardwareFeature` class.)r*HardwareFeaturerrs r8r*z TPUExtended.tpu_hardware_features& / / ""77 99r7c|jSr;)r)rPvar_lists r8non_slot_deviceszTPUExtended.non_slot_devicess   r7cR~tj|j5tjd5||i|}|r|cdddcdddSt j |j|cdddcdddS#1swYnxYwdddy#1swYyxYwr;)rrrr rr.rc_local_results)rPrrFrirjrpr s r8_update_non_slotzTPUExtended._update_non_slots D%% &?(D(DT(J?4"6"f ??? !!$"5"5v> ???????s.B B BB5 BB BB&cP~~~|r!|j|j|yyr;)CopyFrom_update_config_proto)rPsession_config cluster_spec task_typetask_ids r8 _configurezTPUExtended._configures, id77GHr7ctj|}d|_|jj }|r)|j j |j|Sr)copydeepcopyisolate_session_staterr. cluster_defr+as_cluster_def)rP config_protoupdated_configr.s r8r,z TPUExtended._update_config_protosR]]<0N+/N(--::j@} t5jB||jD|| | } dddd } tGdtHr| dd<dtGdtJjLrdgtO z} n? Dcgc]4}tj0d| tj|6} }tjP| S#1swYxYwcc}w) z3TF Function used to replicate the user computation.rz8`TPUStrategy.run` is called with [args: %s] [kwargs: %s]Ncpt|5|i|d<ddddS#1swYdSxYw)z>Wraps user function to provide replica ID and `Tensor` inputs.replica_id_in_sync_grouprN)rz)r replica_argsreplica_kwargsrFr strategys r8 replicated_fnzNTPUExtended._tpu_function_creator..tpu_function..replicated_fn sD : N :,9.9&) :ay :ays (5rrzKinput tensor {} to TPUStrategy.run() has unknown rank, which is not allowedr`)r(maximum_shapes padding_specracb|Dcgc]}t|tjr|!c}Scc}wr;)r@rrr)rMos r8r_zITPUExtended._tpu_function_creator..tpu_function..?s!OA*Q 2NaOOs,,))rvlogrrr`rconstantrint32r r^&experimental_enable_dynamic_batch_sizer.rKrrrrankrndimrdrr TensorShapepack_sequence_as&experimental_bucketizing_dynamic_shaper) PaddingSpec POWER_OF_TWOscopeexperimental_xla_optionsrdrrrr@rerrrrerM)rirjrGrgrqrHflattened_list input_tensorrP maximum_shaperIrarh filter_opsrr rFrrPrFs @r8 tpu_functionz7TPUExtended._tpu_function_creator..tpu_functions ll1M! tf X223:!  ! !!6<< 8  , ,Q 5  , ,Q 7 9 ::  7 7>  %66O#..*.*M*M;O MM  "55)%# % %Pj F1It $vay)q   jCMMB!FS):%;;,   ! !&)Z V8L-M N   % %&7 881 % %( s/AJ5%9K5J>)rr4rrrr)rPrFrr]rFs``` @r8r=z!TPUExtended._tpu_function_creatorsv  "rT-E-E'E  % %b ))'')HL9\  "!**<8l%1dr" r7cy)zAWhether this strategy indicates working in multi-worker settings.Fr6rs r8_in_multi_worker_modez!TPUExtended._in_multi_worker_modeRs r7c|Sr;r6)rPrCs r8_get_local_replica_idz!TPUExtended._get_local_replica_id[s ##r7)NNNFr;)NNNN)7rrrrrrrrr rA PER_WORKERr)r/r2r=rDrIrPrxr{ contextlibcontextmanagerrrrrrrrrrrrrr rrrrrrrrr!r*r&r)r1r,r;rr=r_rarrs@r8rrWs&  $) d&L)4VE9&::EEJ$ : ?(>JHL^@! ''&OkZ ' ' I> /b,A\ , L L  J  J 0 0          G G   9 9 ?!%" I  Wr$r7rct|tr |dk\r|S||zStjt j |d|||zSrJ)r@rr where_v2r" greater_equal)rrPs r8_make_axis_nonnegativerh_sOc qy k D[   tQ'  t  r7c<eZdZdZddZedZdZdZd dZ y) rzz+Replication Context class for TPU Strategy.cHtjj|||y)NrB)r ReplicaContextr)rPrFrCs r8rz_TPUReplicaContext.__init__}s$!!** h1I+Kr7ctj||j}tj|j }|t jdfS|jj|fSrJ) r require_replica_context _strategyrconstant_valuerCr)rrr)rPdsrs r8rz_TPUReplicaContext.devicess]**40 B++D,I,IJJhhqk^kk((4 66r7cL|jjj|S)r})rFrr)rPrs r8rz._TPUReplicaContext.experimental_logical_devices == ! ! = =>O PPr7c t|tr$t|}||xx|jzcc<|St j t jt j|||tjz|}|Sr;) r@rrerr rfr"equalrr)rP value_shape value_rankr output_shapes r8 _compute_all_gather_output_shapez3_TPUReplicaContext._compute_all_gather_output_shapesv*c"+&l4D555  '' .. 3T : 44 4 l r7Nc~tj|D]'}t|tjst dfd}tj|Dcgc] }|||}}tj ||Scc}w)Nz)all_gather does not support IndexedSlicescttj|}|jj+t j|}t j|}nl|jj}|jj }t j|}t t|D]}|| ||||<t||}t|trdg|dzz}j||<nKt jtjtj |dz|jd}j|||}|j t"vrt j$j&j}t j(||}tj*||j }t j,|||z} j/t0j2j4| } t j(| |St j,||} t j6| |} t9j:| ||j} t j(j&dg} t j6| jg} t9j:| ddj} tj<t j$| jd}t j>| ||}t j(||S)Nrrrr) rrrrPr as_listrrerhr@rrrfr"rsrwr&_DTYPES_SUPPORTED_BY_CROSS_REPLICA_SUMone_hotrCrcast expand_dims all_reducerrrrr,rargmaxgather)rrrurtvalue_shape_tensorrqreplica_broadcast_shapervreplica_id_maskgathered_valuerVunordered_outputconcat_replica_idxla_to_replica_context_idreplica_context_to_xla_idsorted_with_extra_dimrPs r8_all_gather_tensorz9_TPUReplicaContext.all_gather.._all_gather_tensors##E*e    !^^E* ooe, [[%% kk))+ &__U3s;'( 3A ^ #/2KN 3$D* 5d J $#$#a"8(,(A(A%"+"4"4 NN8>>*Q,7 >  % % # :: z4)l > >#++  ) )4+D+DF#++ 46"--E"..ud;oM  $ $n6  >>&&u48(?@"-- ! 11 3&--  ) )A30%NN  9 9:<$+$6$6 11 %3!%-OO   7"77 9%!!* 0 0 7d!D  !6 EEr7r)r.rKr@r IndexedSlicesrErS)rPrrexperimental_hintsrnrtyss` r8 all_gatherz_TPUReplicaContext.all_gathers \\% O A~33 4!"MNNOYFv59LL4G Hq QT * HB H   ++ IsB)rr;) rrrrrrrrrwrr6r7r8rzrzxs03K  7  7Q b,r7rzctj|j|}|jj D]-\}}||}|t j |||<&|d||</|j|y)z0Sets the last step outputs on the given context.Nr)r.rSrS_last_step_outputs_reduce_opsrXr PerReplicars)rZrvlast_step_tensor_outputs_dictr\rrs r8rsrss#'"7"7 5#7::@@B6odI *4 0F,2,=,=f,E#D)-31I#D)6:;r7)grrrrcr3rgrabslrnumpyr tensorflow.python.autograph.corerr tensorflow.python.autograph.implrr+tensorflow.python.compiler.xla.experimentalrtensorflow.python.distributerrrr r r r r rrrrr-tensorflow.python.distribute.cluster_resolverrrtensorflow.python.distribute.v1rtensorflow.python.eagerrrrtensorflow.python.frameworkrrrrrrrrrrtensorflow.python.opsr r!r"r#r$rtensorflow.python.ops.raggedr%tensorflow.python.saved_modelr&tensorflow.python.tpur(device_assignment_libr)r*r+tensorflow.python.tpu.opsr,tensorflow.python.utilr-r.r/r0rr3r9rdr>rGruStrategyrxdeprecated_endpointsr| StrategyV1rStrategyExtendedV1rrhbfloat16float16float32float64rNuint32r{rkrzrsr6r7r8rs  D=DQ4792364@13/jE+0,3;3.6+543+2*7<66L%6/-.',- 27/   &6tn-"5d@N++d@6d@N :rB!!!"GHQ/.))Q/ICQ/h>?@b