BVhdZddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z dd l mZdd lmZdd lmZd Zd ZdZGddeZGddeZy)z;Helper library for handling infeed between hosts and TPUs. N) xla_sharding)dtypes)ops) tensor_shape) array_ops) tpu_name_util) tpu_sharding)tpu_ops)nestc |tj|Stj|}|g}tj|jj }tj ||\}}tt||||D]\}\}}} } | dkr|dkDr~|dz} tj | | \} } | g| z| gz}t|| kr|dg| t|z zz }g}|D])}|jtj|||+|}n.|Dcgc]#}tj|t| |%}}tj|}|Scc}w)aOPartitions or replicates the input tensor. The ops inside this function are placed on the host side. Args: tensor: The input tensor which will be partitioned or replicated. dims: A list of integer describes how to partition the input tensor. Returns: An iterator of `Tensor`s or a list of partitioned tensors. r)num_or_size_splitsaxis)r) itertoolsrepeatnparrayshapeas_listdivmod enumerateziplenappendrsplitintr flatten)tensordimsoutput shape_list quotients remaindersrquotient remainderdim original_size ceil_rationum_full_slots left_overr new_outputxs N/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tpu_feed.pypartition_or_replicate_on_hostr.#s~ \   F ## $$ 8&xx ,,./*))J5)Z9B )Zz2:4"5d 5Xy#} ax1}ais3$/ ! !;Xt MMr9ceZdZdZ ddZdZedZedZdZ edZ d Z ed Z ed Z d Zed ZdZdZdZdZddZ ddZ ddZdZdZ ddZy) InfeedQueuezA helper object to build a device infeed queue. The InfeedQueue builds the host-side and device-side Ops to enqueue and dequeue elements, respectively, and ensures that their types and shapes match. Ncd|_d|_d|_|dn||_|d|_n||_|5| t |}n'| t |}n| t |}n t d|dkrt d|dt|Dcgc]}tjc}|_ ||j|nd|_ ||j|nd|_||j||j!ycc}w) aCreates a new InfeedQueue with the given configuration. The configuration need not be fully specified at creation since it can be modified subsequently by methods that set the values explicitly or infer them from the shapes of inputs. Args: number_of_tuple_elements: the number of Tensors fed atomically through the queue, must be present unless it can be inferred from other arguments. tuple_types: if not None, a list of types of the elements of the queue. tuple_shapes: if not None, a list of shapes of the elements of the queue. shard_dimensions: if not None, a list of dimensions on which the elements of the queue should be sharded during automatic parallelization. number_of_partitions: if > 1, the infeed dequeue shape will contain the full shape that includes all partitions and add corresponding XLA annotation on the infeed dequeue op. In this case, the infeed is still data parallel that feeds per-core batch size to each core while the XLA computation may be partitioned. As XLA requires infeed dequeue shape to be per-replica shape, thus we need number_of_partitions here to calculate the per-replica unpartitioned shape. name: the name of the queue. Raises: ValueError: if number_of_tuple_elements <= 0; or number_of_tuple_arguments, tuple_types, tuple_shapes, and shard_dimensions are all None; or the length of tuple_types, tuple_shapes, or shard_dimensions is not equal to number_of_tuple_elements; or any element of shard_dimensions can't be converted to a Dimension. TypeError: if any element of tuple_types or tuple_shapes can't be converted to a dtype or TensorShape, respectively. FNr@r zHnumber of tuple elements cannot be inferred from InfeedQueue constructorrznumber_of_tuple_elements z must be > 0)_frozen_generated_enqueue_ops_generated_dequeue_op_name_number_of_partitionsr ValueErrorranger ShardingPolicy_sharding_policiesset_tuple_types _tuple_typesset_tuple_shapes _tuple_shapesset_shard_dimensions _validate)selfnumber_of_tuple_elements tuple_types tuple_shapesshard_dimensionsnumber_of_partitionsname_s r-__init__zInfeedQueue.__init__s=PDL"'D!&D"&,DDJ##$d #7d '  #&{#3  ##&|#4  '#&'7#8    1$ 23K2LM%% &&055M/N*+ ##%D ;'d L)d#  01NNs Dc|j:t|j|jD]\}}|j |}yy)zChecks that the configuration is self-consistent. Raises: ValueError: if the shapes and sharding policies don't match. N)rTrrJrNget_sharded_shape)rQpolicyrrXs r-rPzInfeedQueue._validatesK  $ !8!8$:L:LM,/65  $ $U +,%r9c,t|jS)z1Returns the number of InfeedQueue tuple elements.)rrJrQs r-rRz$InfeedQueue.number_of_tuple_elementss t&& ''r9c|jS)z4Returns the types of the InfeedQueue tuple elements.)rLr^s r-rSzInfeedQueue.tuple_typess   r9c t||jk7r$tdt|d|j|jrQt |j |D]7\}}||k7s tdt|j dt|y |Dcgc]}tj|c}|_ycc}w#t$r)}tdt|dt||d}~wwxYw)aSets the type of each element of the queue. tuple_types must be a list of length self.number_of_tuple_elements, and each element must be convertible to a dtype. Args: tuple_types: the types of each queue element. Raises: ValueError: if tuple_types is not of length self.number_of_tuple_elements. TypeError: if an element of tuple_types cannot be converted to a dtype. ztuple_types is , but must be a list of length zcTrying to update InfeedQueue with frozen configuration with an incompatible type. Frozen types are z, updated types are zF, but must be a list of elements each convertible to dtype: got error N) rrRrGstrrBrrLras_dtype TypeError)rQrSfrozenupdatedtes r-rKzInfeedQueue.set_tuple_typess  ;4888  C ,-.112 4  ||"4#4#4kB7 67 W 5589J9J5K4LM##&{#3"467 77N9DEAV__Q/EENc+./0==@VH FGLM NNs*C C;C C C;$C66C;c|jS)z5Returns the shapes of the InfeedQueue tuple elements.)rNr^s r-rTzInfeedQueue.tuple_shapess   r9c *t||jk7r$tdt|d|j |Dcgc]}t j |}}|jrQt|j|D]7\}}||k7s tdt|jdt|n||_ |jycc}w#tt f$r)}t dt|dt||d}~wwxYw)aSets the shape of each element of the queue. tuple_shapes must be a list of length self.number_of_tuple_elements, and each element must be convertible to a TensorShape. Args: tuple_shapes: the shapes of each queue element. Raises: ValueError: if tuple_shapes is not of length self.number_of_tuple_elements. TypeError: if an element of tuple_shapes cannot be converted to a TensorShape. ztuple_shapes is razL, but must be a list of elements each convertible to TensorShape: got error NzeTrying to update InfeedQueue with frozen configuration with an incompatible shape. Frozen shapes are z, updated shapes are ) rrRrGrbras_shaperdrBrrNrP)rQrTrrhrerfs r-rMzInfeedQueue.set_tuple_shapes s4  <D999  S./0112 4 @LMul++E2MlM  ||"4#5#5|D& 67 W 7T''())>\"#%& &&(dNN#N  "  S./0A VH  s)CCCCD)$D  Dc|jS)z@Returns the sharding policies of the InfeedQueue tuple elements.)rJr^s r-sharding_policieszInfeedQueue.sharding_policies5s  " ""r9cT|jDcgc]}|jc}Scc}w)zGets the shard dimension of each tuple element. Returns: A list of length number_of_tuple_elements, where each list entry is the shard dimension of that tuple element or None if the shard dimension has not been set. )rJshard_dimension)rQr\s r-rUzInfeedQueue.shard_dimensions:s$261H1H IvF " " II Is%ct||jk7r$tdt|d|jt |j |D]\}}|j ||jy)akSets the shard_dimension of each element of the queue. shard_dimensions must be a list of length self.number_of_tuple_elements, and each element must be convertible to a Dimension compatible with self.tuple_shapes. Args: shard_dimensions: the dimensions of each queue element. Raises: ValueError: if shard_dimensions is not of length self.number_of_tuple_elements; or an element of shard_dimensions cannot be converted to a Dimension; or an element of shard_dimensions is a Dimension that is out of range for the corresponding tuple element shape. zshard_dimensions is raN)rrRrGrbrrJset_shard_dimensionrP)rQrUr\ dimensions r-rOz InfeedQueue.set_shard_dimensionsFs"  = == -c2B.C-DE..2.K.K-LN OO"4#:#: 0; or the policies have been frozen and number_of_shards was already set to something else. N)rJset_number_of_shardsset_number_of_partitionsrFrP)rQrtr\s r-rvz InfeedQueue.set_number_of_shardshsJ))B !!"23 %%d&@&@AB NNr9c6t||jk7r%tdt|d|jd|j |Dcgc]}|j c}|j |Dcgc]}|jc}ycc}wcc}w)aSets the shapes and types of the queue tuple elements. input_tensors is a list of Tensors whose types and shapes are used to set the queue configuration. Args: input_tensors: list of Tensors of the same types and shapes as the desired queue Tuple. Raises: ValueError: if input_tensors is not a list of length self.number_of_tuple_elements input_tensors is z, but should be a list of z TensorsN)rrRrGrbrMrrKdtype)rQ input_tensorsrgs r-$set_configuration_from_input_tensorsz0InfeedQueue.set_configuration_from_input_tensorsxs =T::: *3}+=*>?$$($A$A#B(L MMM:q177:;=9a!''9:;9s B6Bc F|jsd|_t|}|j||D]>}t||jk7st dt |d|jt|jDcgc]}|Dcgc]}||jc}!}}}t|j|Dcgc]\}}|j|}}}|j|td|jD]M}t|d||D]6\} } | j| jk7s tdt |dO|j!|dDcgc]}|jc}ycc}wcc}}wcc}}wcc}w)aSets the shapes and types of the queue tuple elements. input_tensors is a list of lists of Tensors whose types and shapes are used to set the queue configuration. The length of the outer list is the number of shards required, and each inner list is the tuple of Tensors to use to determine the types and shapes of the corresponding shard. This method depends on the shard dimension, and calling it freezes the shard policy. Args: input_tensors: list of lists of Tensors. The outer list length corresponds to the desired number of shards, and each inner list is the size and shape of the desired configuration of the corresponding shard. Raises: ValueError: if any inner list is not a list of length self.number_of_tuple_elements; or the inner lists do not combine to form a consistent unsharded shape. TypeError: if the types of the Tensors in the inner lists do not match. NryzX but must be a list of lists, where each inner list has length number_of_tuple_elements=r rz-types of the tuple elements of input_tensors z are not consistent)rBrNrrvrRrGrbrHrrrJget_unsharded_shaperMrtrzrdrK) rQr{rtrgisharded_shapesr\sunsharded_shapest1t2s r-,set_configuration_from_sharded_input_tensorsz8InfeedQueue.set_configuration_from_sharded_input_tensorss( << d=)./ I Q400 0M 234((,(E(E'F HI II %T%B%BCE!./tzz/ENE t66G VQ ""1% *+ 1d++ ,:-*M!,<=:(2r 88rxx =]#$$79: :::  =+;)rQ tpu_device full_namerr\r dequeue_op partitionss r-generate_dequeue_opzInfeedQueue.generate_dequeue_ops, KKM !!#*=*=*? K LL!%Dtzz)I #4#5#5t7N7NO UF &&v'?'?'FGN ::m((4 5M11$$^)M MM//""> Kj !!Q&  #4#5#5t7N7NO UF &&sU[['89AACJ 7z: NN# MMs5*F #F7F&F#c $d||fz}|Dcgc]}|j}}||Dcgc]}|j} }td|jD](} | d| | k7st d|dt | dt j|d5tj||||cdddSt j|5tj||||cdddScc}wcc}w#1swYyxYw#1swYyxYw) aGenerate a host-side Op to enqueue a tuple to the queue. If device is None the inputs are all required to have the same device specification, and the enqueue Op is colocated with inputs[0]. Otherwise the enqueue Op is placed on 'device'. Args: inputs: a list of Tensors with the types and shapes of the tuple elements. name_prefix: the base name for the Op. index: the shard index, used to uniquify the Op name. device: device to place the Op on, or None if it should be colocated with the inputs. tpu_ordinal: ordinal of the TPU device on the host to use for infeed if device is a CPU device. Should be set to -1 if device is a TPU device. Returns: An Op corresponding to a shard of infeed enqueued at the host, suitable for use within a replicated block. Raises: ValueError: if device is None and inputs do not all have the same device specification. %s/%dNr rzinput devices for shard z are z, but should all be the sameinputsrrWdevice_ordinal) rrrHrRrGrbr colocate_withr infeed_enqueue_tuple) rQr name_prefixindexr tpu_ordinalrrgrdevicesrs r-_generate_enqueue_opz InfeedQueue._generate_enqueue_ops-<;..I% &!agg &F & ~#)*a*g*Q556!! 1: #(uS\NC  ! !!   VAY '(++& ((( ::f (++& ((('* ((((s"C0C5C: D:DDc |j||j|jrtjs t dd|_|d}d|j z}t|t|jDcgc]+\}}|j||||||r||nd-c}}Scc}}w)aGenerates the host-side Ops to enqueue the shards of a tuple. sharded_inputs is a list, one for each shard, of lists of Tensors. sharded_inputs[i] is the tuple of Tensors to use to feed shard i of the queue. Returns the host-side Ops that must be run to enqueue the sharded tuple. The Op for shard i is colocated with the inputs for shard i. Implicitly freezes the queue configuration if it is not already frozen. If the configuration has already been frozen, and is not compatible with the types and shapes of sharded_inputs, an error will be raised. Args: sharded_inputs: a list of lists of Tensors. The length of the outer list determines the number of shards. Each inner list indicates the types and shapes of the tuples in the corresponding shard. tpu_ordinal_function: if not None, a function that takes the shard index as input and returns the ordinal of the TPU device the shard's infeed should be placed on. tpu_ordinal_function must be set if the inputs are placed on CPU devices. placement_function: if not None, a function that takes the shard index as input and returns the host device where the enqueue op should be placed on. Returns: A list of host-side Ops, one for each shard, that when executed together will enqueue a full-size element of infeed. Raises: ValueError: if the queue configuration has previously been frozen and the shapes of the elements of sharded_inputs are not compatible with the frozen configuration; or if the shapes of the elements of sharded_inputs don't form a consistent unsharded tuple; or if the elements of a tuple have different device constraints. TypeError: if the queue configuration has previously been frozen and the types of the elements of sharded_inputs are not compatible with the frozen configuration; or if the types of the elements of sharded_inputs don't form a consistent unsharded tuple. 2Can't generate two enqueue Ops from the same queueTNcyN)rs r-z2InfeedQueue.generate_enqueue_ops..nsr9 %s/enqueue)rr) rrrCrrrGrErrHrtr)rQsharded_inputstpu_ordinal_functionplacement_functionrshardrs r-generate_enqueue_opsz InfeedQueue.generate_enqueue_ops<sX 55nEKKM ""3+>+>+@ K LL"&D#-+K".%8M8M2NO  UE !!   ,U30B%e, " N  s 0B>cd|dz zS)Nz/task:%d/device:CPU:0rrQrs r-_default_placement_functionz'InfeedQueue._default_placement_function}s "eai 00r9c |dzS)Nrrrs r-_default_ordinal_functionz%InfeedQueue._default_ordinal_functions 19r9c | |j}||j}nfd}fd}||}||}|j||j|jrt j s tdd|_d|jz}|jdk(r|Dcgc]}|g} }ned} t||jt|jD cgc]+\}} } | ||j| jd|| fz -} } }} t|jD cgc]} | Dcgc]}||  c}}} }d |jz}t|t|jD cgc]'\}} |j||| || ||  )c} }Scc}wcc} } }wcc}wcc}} wcc} }w) a| POORLY-PERFORMING ON MULTI-HOST SYSTEMS. Generates the host-side Ops to enqueue a tuple. This method performs poorly because it takes an entire input on a single host, splits it, and distributes it to all of the cores. It is present only to simplify tutorial examples. inputs is a list of Tensors to use to feed the queue. Each input is split into self.number_of_shards shards. Returns an Op for each shard to enqueue the shard. The Op for shard i is placed on device placement_function(i). Implicitly freezes the queue configuration if it is not already frozen. If the configuration has already been frozen, and is not compatible with the types and shapes of inputs, an error will be raised. Args: inputs: a list of Tensors which indicates the types and shapes of the queue tuple. device_assignment: if not `None`, a TPU `DeviceAssignment`. If device_assignment is not `None`, but `placement_function` and `ordinal_function` are None, then `device_assignment` will be used to place infeeds on the first k TPU shards, where k is the number of shards in the queue. If all three are `None`, then default placement and ordinal functions are used. placement_function: if not None, a function that takes the shard index as input and returns a device string indicating which device the shard's infeed should be placed on. If placement_function and tpu_ordinal_function are None, inputs are sharded round-robin across the devices in the system. tpu_ordinal_function: if not None, a function that takes the shard index as input and returns the ordinal of the TPU device the shard's infeed should be placed on. If placement_function and tpu_ordinal_function are None, inputs are sharded round-robin across the devices in the system. Returns: A list of host-side Ops, one for each shard, that when executed together will enqueue a full-size element of infeed. Raises: ValueError: if the queue configuration has previously been frozen and the shapes of the elements of inputs are not compatible with the frozen configuration. TypeError: if the queue configuration has previously been frozen and the types of the elements of inputs are not compatible with the frozen configuration. c(j|SN)replica) host_devicerdevice_assignments r-_placement_function_from_mapzWInfeedQueue.split_inputs_and_generate_enqueue_ops.._placement_function_from_map ,,U,;;r9c(j|Sr)rrs r-_ordinal_function_from_mapzUInfeedQueue.split_inputs_and_generate_enqueue_ops.._ordinal_function_from_maprr9rTz%s/splitr ctj|5tj||||cdddS#1swYyxYw)NrrW)rrrr)inp num_shardsrrWs r-split_fnzCInfeedQueue.split_inputs_and_generate_enqueue_ops..split_fns;   s # Hjt$G H H Hs 9Arrr)rr)rrr|rrCrrrGrErtrrJrHrRror)rQrrrrrrsplit_name_prefixrtransposed_sharded_inputsrr\rrrrrs ` r-%split_inputs_and_generate_enqueue_opsz1InfeedQueue.split_inputs_and_generate_enqueue_opssl  #!==  %#==<<  #9  %9--f5KKM ""3+>+>+@ K LL"&D"TZZ/ !4:";SC5";";H'*&$2I2I*/0M0M*N'P ## #sFE ##))/77  9## %T%:%:;=%>? Qx?=N=+K".%8M8M2NO  UE !!   %e,,U3 " 5 )#<#?= s*( F0&0F52 G; F< G,G<G)NNNNNNrr)NN)NNN)__name__ __module__ __qualname____doc__rYrPpropertyrRrSrKrTrMrmrUrOrtrvr|rrrrrrrrrr9r-r@r@xs)- $$( L\ , ( (  "NH  'R # #  J  J0 7 7 ;(1>f4,Od#'') 3(n15.2<B1?C?CAE jr9r@cFeZdZdZ dfd ZddZdZdZdZxZ S) _PartitionedInfeedQueueaA helper object to build a device infeed queue with input partition. Args: number_of_tuple_elements: the number of Tensors fed atomically through the queue, must be present unless it can be inferred from other arguments. device_assignment: A TPU `DeviceAssignment` which is used to place all the partitions to different TPU infeed queues. host_id: The id of the host machine. input_partition_dims: A nested list/tuple of integers. Each inner list/tuple describes how to partition the corresponding input tensor. tuple_types: If not None, a list of types of the elements of the queue. tuple_shapes: If not None, a list of shapes of the elements of the queue. name: The name of the queue. chtt| ||dd|dn|||_||_||_y)NPartitionedInfeedQueue)rRrSrTrUrW)superrrY_input_partition_dims_host_id_device_assignment) rQrRrhost_idinput_partition_dimsrSrTrW __class__s r-rYz _PartitionedInfeedQueue.__init__sJ !41!9)- %4 2A "6DDM/Dr9c|j|jrtjs t dd|_d|j z}t |j|jDcgc]\}}|j|}}}tjtj|5tj|j||}dddt!|j"Scc}}w#1swY%xYw)aGenerate TPU dequeue ops. Args: tpu_device: The TPU device ordinal where the infeed instruction should be placed. Returns: A list of Outputs corresponding to a partition of infeed dequeued into XLA, suitable for use within a replicated block. Raises: ValueError: if the types or shapes of the tuple elements have not been set; or if a dequeue op has already been generated. rTrrN)rrDrrrGrErrNrJr[rrrr rrLr>r)rQrrrr\rvaluess r-rz+_PartitionedInfeedQueue.generate_dequeue_ops KKM !!#*=*=*? K LL!%Dtzz)I #4#5#5t7N7NO UF   'N M&&z2 3K++""> KfK 7** ,,KKs5C<;#DD c |j|t|}t|d}t|j|k(sJg}t|D]}||}t j ||j}t |||D cgc] \}} t|j|| "} }} |jj|jd|} t|jjD]} |jj| | } tj| 5|jj!| | }g}| D]"}t#|d}||j%|$|rP|j%t'j(||Dcgc]}|j*c}dj-|| |ddd|Scc} }wcc}w#1swYxYw)aGenerates the host-side Ops to enqueue the partitioned inputs. sharded_inputs is a list, one for each replica, of lists of Tensors. sharded_inputs[i] is the tuple of Tensors to use to feed replica i. sharded_inputs[i][j] is partitioned by self._input_partition_dims[j]. For example, if sharded_inputs[i][j] is a 2-D Tensor: [[A, B, C, D], [E ,F, G, H]] self._input_partition_dims[j] is [2, 4]. sharded_inputs[i][j] will be partitioned and flattened into: [A, B, C, D, E, F, G, H] and fed into the logical core ids: [0, 1, 2, 3, 4, 5, 6, 7] respectively. Args: sharded_inputs: a list of lists of Tensors. The length of the outer list determines the number of shards. Each inner list indicates the types and shapes of the tuples in the corresponding shard. Returns: A list of host-side Ops, one for each shard, that when executed together will enqueue a full-size element of infeed. Raises: ValueError: if the queue configuration has previously been frozen and the shapes of the elements of sharded_inputs are not compatible with the frozen configuration; or if the shapes of the elements of sharded_inputs don't form a consistent unsharded tuple; or if the elements of a tuple have different device constraints; or if the partition dims are invalid. TypeError: if the queue configuration has previously been frozen and the types of the elements of sharded_inputs are not compatible with the frozen configuration; or if the types of the elements of sharded_inputs don't form a consistent unsharded tuple. r)task_id logical_core)rrNzenqueue/replica_{0}/input_{1}r)rrrrHr flatten_up_toriter._check_dims_and_partition_or_replicate_on_hostrlookup_replicasrnum_cores_per_replicarrrrnextrr rrformat)rQrnumber_of_replicasrR enqueue_ops replica_indexflattened_inputsinputs_part_dims_flatr,rinputs_parted_iters replica_idrrordinal infeed_inputsitinput_for_devices r-rz,_PartitionedInfeedQueue.generate_enqueue_ops2sJ 55nE^,">!#45 t)) *.F FF FK12(- ' 6"001A151K1KM^M:24a tBB1dK L**::--a;11>@j 7 7 M MN-,((44\5;ZZ  -++77 |8='-' 5b#B~ +""#34 5    ,,(-:;AGG;8??%|5#* , - - --#(-R KB< - -s* %G33G&'4G&G!.G&!G&&G/ c*|ytj|}|dkjr td|j dk(ry|j |j j k7r/tdj||j j |jd|jjk7r3tdj|jj||jjy)apChecks that input partition dims are valid for the `Tensor`. Args: tensor: Input tensor for partitioning. dims: A list of integer describes how to partition the input tensor. Raises: ValueError: If the tensor can't be partitioned by dims or the num_cores_per_replica doesn't match the number of partitions(dims.prod()). Nr z&All input partition dims must be >= 1.zuThe product of each input partition dim should equal to num_cores_per_replica. (dim = {}, num_cores_per_replica = {})rzInput partition dims must have the same number of dimensions as the `Tensor` to be partitioned. (tensor shape = {}, input partition dims = {}).) rranyrGr3rrrrrrassert_is_fully_definedrQrrs r-_check_input_partition_dimsz3_PartitionedInfeedQueue._check_input_partition_dimss | 88D>D q~~ ? @@ yy{a  yy{d--CCC  &t66LLM OO zz!} ***  ""(&)=)=)?"F HH  LL((*r9c>|j||t||S)amChecks dims and partitions or replicates the input tensor. The ops inside this function are placed on the host side. Args: tensor: The input tensor which will be partitioned or replicated. dims: A list of integer describes how to partition the input tensor. Returns: An iterator of `Tensor`s or a list of partitioned tensors. )rr.rs r-rzF_PartitionedInfeedQueue._check_dims_and_partition_or_replicate_on_hosts! $$VT2 )&$ 77r9)NNNNr) rrrrrYrrrr __classcell__)rs@r-rrs2 &%) 0$,objectr@rrr9r-rs[ D.+4+/.-'+\$0 Mw &w tJ8kJ8r9