AVhdZddlZddlmZmZmZmZmZmZm Z m Z ddl Z ddl mZddl mZddl mZddl mZddl mZdd l mZdd l mZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddl m!Z!ddl m"Z"ddl#m$Z$e ejJejLdejNfZ(ee(ge(fZ)e$dGddejTZ+ejXdZ-dZ.dee/e(fdeejJdeeedej`fdZ1d Z2d!Z3d"Z4d#Z5d$Z6d%Z7d&Z8d'Z9d(Z:d)Z;d*Ze?Z@d-e ejjeje+jejfdejjfd.ZEd/d0dee/fd1ZFd2ZGd3ZH dAdejfd4ZId5eejd6eejdeejfd7ZJd8eKd9ej`deejfd:ZLd;e(deej`fd<ZMd;e(d9ej`de(fd=ZNdee/e(fd9ej`dee/e(ffd>ZOd?ZPd@ZQy)BzStructured Tensors.N)CallableDictListMappingOptionalSequenceTupleUnion) constant_op)dtypes)extension_type)ops)tensor) tensor_shape) type_spec) array_ops) check_ops)control_flow_ops)math_ops)dynamic_ragged_shape)ragged_factory_ops) ragged_tensor) RowPartition)compat)nest) tf_exportStructuredTensorzexperimental.StructuredTensorc eZdZUdZeeefed<ejed<dZe ee efZ deeefdejfdZ ed1dZedejd dfd Ze d2d Ze d3deeefd ededeej,d df dZ d1dee e eed ffded dfdZdeedfdeee e eeffded dfdZdZdZe dZ!e dZ"e dZ#e dZ$dZ%dej,d dfdZ&dZ'dZ(d Z)d!Z*d"Z+d#Z,d$Z-d%Z.ed4d&Z/ed'Z0ed(Z1ed)Z2ed*Z3ed+Z4ed,Z5d-Z6d.Z7Gd/d0Z8y )5raA multidimensional collection of structures with the same schema. A **`StructuredTensor`** is a multi-dimensional collection of ***structures*** with the same ***schema***, where: * A ***schema*** is a collection of fields, each of which has a name and type. * A ***structure*** maps each field in the schema to a tensor value (which could be a nested StructuredTensor). As an important special case, a 1D `StructuredTensor` encodes a 2D table, where columns are heterogeneous `Tensor`s, and rows are the aligned elements in each of those `Tensor`s. Internally, StructuredTensors use a "field-major" encoding: for each leaf field, there is a single tensor that stores the value of that field for all structures in the `StructuredTensor`. ### Examples >>> # A scalar StructuredTensor describing a single person. >>> s1 = tf.experimental.StructuredTensor.from_pyval( ... {"age": 82, "nicknames": ["Bob", "Bobby"]}) >>> s1.shape TensorShape([]) >>> s1["age"] >>> # A vector StructuredTensor describing three people. >>> s2 = tf.experimental.StructuredTensor.from_pyval([ ... {"age": 12, "nicknames": ["Josaphine"]}, ... {"age": 82, "nicknames": ["Bob", "Bobby"]}, ... {"age": 42, "nicknames": ["Elmo"]}]) >>> s2.shape TensorShape([3]) >>> s2[0]["age"] ### Field Paths A *field path* is a tuple of field names, specifying the path to a nested field. _fields _ragged_shapeztf.StructuredTensorfields ragged_shapec ||_||_yN)rr )selfr!r"s b/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/structured/structured_tensor.py__init__zStructuredTensor.__init__ssDL%Dc t|tsJ|t|tjsJ||!t|tj sJ||t|t sJ|t|t||||S)aPrivate constructor -- use factory methods to create StructuredTensors. This constructor builds a `StructuredTensor` from the given attributes, performing minimal validation. Args: fields: A dictionary mapping from string to `Tensor`, `RaggedTensor`, or `StructuredTensor`. (This dict is not copied, so the caller must ensure that it does not get mutated via leaked references.) shape: `tf.TensorShape` with statically known rank. nrows: scalar integer `tf.Tensor`, or `None` if `shape.rank==0`. row_partitions: tuple of `RowPartition`s, with length `shape.rank-1`. internal: ignored argument. Returns: a StructuredTensor. r!r") isinstancedictr TensorShaperTensortupler_dynamic_ragged_shape_init)clsr!shapenrowsrow_partitionsinternals r& _old_initzStructuredTensor._old_initxs& fd #+V+ # e\55 6== 6 =Jufmm<CeC <  !Z05&7G8FG 7 /u0>@ AAr(returncti|S)zCreates a `StructuredTensor` with no fields and ragged_shape. Args: ragged_shape: the shape of the structured tensor. Returns: a StructuredTensor with no fields and ragged_shape. r*)r)r1r"s r& from_shapezStructuredTensor.from_shapes 2L AAr(Nc tj|}|j}| tdt |t s!t dt|jz|dkr |r td|dk(r | td|1t|}t|td|dz k7rtd |dkrd }t |}tjdd |j5t||||}|jdkDr|j!|jdz }|j#D]\}}t |t$st d |t&j)|std |zt+|||<t |j#D cgc]\} } | t-| |fc} } }|||cdddScc} } w#1swYyxYw)aCreates a `StructuredTensor` from a dictionary of fields. Args: fields: A dictionary mapping from string to `Tensor`, `RaggedTensor`, or `StructuredTensor`, providing the values for individual fields in each structure. If `shape.rank > 0`, then every tensor in `fields` must have the same shape in the first `shape.rank` dimensions; and that shape must be compatible with `shape`; and `result[i1...iN][key] = fields[key][i1...iN]` (where `N==shape.rank`). shape: A `TensorShape`: static information about the shape of the `StructuredTensor`. Must have a known `rank`. Defaults to scalar shape (i.e. `rank=0`). nrows: scalar integer tensor containing the number of rows in this `StructuredTensor`. Should only be specified if `shape.rank > 0`. Default value is inferred from the `fields` values. If `fields` is empty, then this must be specified. row_partitions: A list of `RowPartition`s describing the (possibly ragged) shape of this `StructuredTensor`. Should only be specified if `shape.rank > 1`. Default value is inferred from the `fields` values. If `fields` is empty, then this must be specified. validate: If true, then add runtime validation ops that check that the field values all have compatible shapes in the outer `shape.rank` dimensions. Returns: A `StructuredTensor`. Examples: >>> tf.experimental.StructuredTensor.from_fields({'x': 1, 'y': [1, 2, 3]}) >>> tf.experimental.StructuredTensor.from_fields( ... {'foo': [1, 2], 'bar': [3, 4]}, shape=[2]) N.StructuredTensor's shape must have known rank.z#fields must be a dictionary, got %sz1row_partitions must be None or [] if shape.rank<2rz#nrows must be None if shape.rank==0z(len(row_partitions) must be shape.rank-1rz%Unexpected type for key in `fields`: z'Field name %r is not currently allowed.r*)ras_shaperank ValueErrorr+r, TypeErrortype__name__r/lenmaxr name_scopevaluesr0_with_num_row_partitionsitemsstr_FIELD_NAME_REmatch"_convert_to_structured_field_value_replace_row_partitions) r1r!r2r3r4validater@keyvaluekvs r& from_fieldszStructuredTensor.from_fieldssf  ! !% (E ::D | G HH fd # ;6l++, -- axN J KK qyU& < ==!^,n ^ Atax 0 0CDD n &\F 0&--/ B4(~Ne a..uzzA~> 6*#u#s#A#GH H##C(DsJK K8?s %+\\^5!Aq21nEF566U 3%44 5!44s.B9G"'G?G"G""G+r@rPdtypec |s tdt|ts td|dkr td|jDcic]\}}|t |}}}| t |dd}t ||}t|||}|dkDr|j|dz }|j}|jDcic]\}}|t||}}}t||Scc}}wcc}}w)aCreates a `StructuredTensor` from a nonempty dictionary of fields. Note that if the shape dtype is not specified, the shape dtype will be inferred from any fields that have a shape dtype. If fields differ, then int64 will be preferred to int32, because coercing from int32 to int64 is safer than coercing from int64 to int32. If there are no ragged fields, then it will be int64 by default, but this will be changed to int32 in the future. Args: fields: A dictionary mapping from string to `Tensor`, `RaggedTensor`, or `StructuredTensor`, providing the values for individual fields in each structure. If `rank > 0`, then every tensor in `fields` must have the same shape in the first `rank` dimensions. Cannot be empty. rank: The rank of the resulting structured tensor. validate: If true, then add runtime validation ops that check that the field values all have compatible shapes in the outer `rank` dimensions. dtype: If specified, then forces dtype of the shape to be this. Returns: A `StructuredTensor`. Examples: >>> tf.experimental.StructuredTensor.from_fields_and_rank( ... {'x': 1, 'y': [1, 2, 3]}, 0) >>> StructuredTensor.from_fields_and_rank({'foo': [1, 2], 'bar': [3, 4]}, ... 1) zMust provide at least one fieldzrank must be an integerrzrank must be nonnegativeNr=r*) rAr+intrJrN_find_shape_dtype_fields_with_dtype_shape_from_fieldsrI_row_partitionsrOr) r1r!r@rPrVrSrTr2new_rps r&from_fields_and_rankz%StructuredTensor.from_fields_and_ranks Z  8 99 dC 0 11 ax 1 22?E||~5;a -a 00F }d3e  .F vtU 3E ax,,TAX6e  " "F>s C,C2updatescV|jDcgc]\}}t||f}}}t|}tdt |D]A}||d}||dz d}|dt ||k(s(t dj |||jd||Scc}}w)a Creates a new `StructuredTensor` with the updated fields. If this `StructuredTensor` is a scalar, and `k` is the `FieldName` being updated and `v` the new value, then: ``` result[k] = v # If (k, v) is in updates and v is a FieldValue result[k] = f(self[k]) # If (k, f) is in updates and f is a FieldFn result[k] = self[k] # If k is in self.field_names but not in updates ``` If this `StructuredTensor` has rank `N` and shape `[D1...DN]`, then each FieldValue `v` in `updates` must have shape `[D1...DN, ...]`, that is, prefixed with the same shape as the `StructuredTensor`. Then the resulting `StructuredTensor` will have: ``` result[i1...iN][k] = v[i1...iN] # (k, v) in updates result[i1...iN][k] = f(self.field_value(k))[i1...iN] # (k, f) in updates result[i1...iN][k] = self[i1...iN][k] # k not in updates ``` Note that `result.shape` is always equal to `self.shape` (but the shapes of nested StructuredTensors may be changed if they are updated with new values). Args: updates: A dictionary mapping `FieldName` to either a `FieldValue` to be used to update, or a `FieldFn` that will transform the value for the given `FieldName`. `FieldName` can be a string for a direct field, or a sequence of strings to refer to a nested sub-field. `FieldFn` is a function that takes a `FieldValue` as input and should return a `FieldValue`. All other fields are copied over to the new `StructuredTensor`. New `FieldName` can be given (to add new fields), but only to existing `StructuredTensor`, it won't automatically create new nested structures -- but one can create a whole `StructureTensor` sub-structure and set that into an existing structure. If the new value is set to `None`, it is removed. validate: If true, then add runtime validation ops that check that the field values all have compatible shapes in the outer `shape.rank` dimensions. Returns: A `StructuredTensor`. Raises: `ValueError`: If the any of the `FieldName` keys points to non-existent sub-structures, if parent and child nodes are updated, if shapes change, if a delete update is given for a non-existent field, or if a `FieldFn` transforming function is given for a `FieldName` that doesn't yet exist. Examples: >>> shoes_us = tf.experimental.StructuredTensor.from_pyval([ ... {"age": 12, "nicknames": ["Josaphine"], ... "shoes": {"sizes": [8.0, 7.5, 7.5]}}, ... {"age": 82, "nicknames": ["Bob", "Bobby"], ... "shoes": {"sizes": [11.0, 11.5, 12.0]}}, ... {"age": 42, "nicknames": ["Elmo"], ... "shoes": {"sizes": [9.0, 9.5, 10.0]}}]) >>> def us_to_europe(t): ... return tf.round(t * 2.54 + 17.0) # Rough approximation. >>> shoe_sizes_key = ("shoes", "sizes") >>> shoes_eu = shoes_us.with_updates({shoe_sizes_key: us_to_europe}) >>> shoes_eu.field_value(shoe_sizes_key) r=rNz`StructuredTensor.with_updates` does not allow both parent and child nodes to be updated: parent={}, child={}. If needed you can update child nodes in the parent update value.r>)rJ_normalize_field_name_to_tuplesortedrangerErAformat_with_updates_impl)r%r_rPnamerR updates_itemsi prev_names r& with_updateszStructuredTensor.with_updates@sR)0 9$u5T:EB9M9=)M 1c-( ) " 1 a dA&q)i os9~ ) + ==CV4>!" " "  " "2}h ??9sB% error_prefix.c  t|jdttdtffd dtdtt t fdt f fd }|D]-\}}|r|ds tdj |t|dk(rJ|d}|6|vr td j |j|y||||<|d}|dd }|vr-td jt|z ||} t| ts7td jt|z |t| ||fg} | j|fz| |}||<0 tj!|j"|j$|j'| S#t$r'} d} rdj| } t| | d } ~ wwxYw)z0Recursive part of `with_updates` implementation.rfr7c"t|fzSr$)rK)rfrks r& name_fullpathz:StructuredTensor._with_updates_impl..name_fullpaths ' ((r(rRc|t|r/|vr tdj|||}|S)Nz`StructuredTensor.with_updates` cannot update the field {} because a transforming function was given, but that field does not already exist.)callablerArd)rfrRrn new_fieldss r& apply_valuez8StructuredTensor._with_updates_impl..apply_valuesK % z !((.}T/B(CE Ej&' lr(rz>`StructuredTensor.with_updates` does not allow empty names {}.r=NzQ`StructuredTensor.with_updates` cannot delete field {} because it is not present.z]`StructuredTensor.with_updates` cannot create new sub-field {} if parent field {} is not set.z`StructuredTensor.with_updates` cannot create new sub-field {} if parent structure {} is not a `StructuredTensor` that can contain sub-structures -- it is a `{}`.)r2r4r3rPz&`StructuredTensor.with_updates` failedz{} for field {})r,rrrKr _FieldValue_FieldFnrArdrEpopr/r+rrCrerUr2r4r3)r%rkr_rPrrrfrRprefixsuffix current_value one_updateemsgrnrqs ` @@r&rez#StructuredTensor._with_updates_implsG dll#J)HSM)c) # eK,4-5'6 :E *# e a &t,-/ / TaAw =  #006}T7J0KM M .. (u5*T aab  #228&t,mF.C3EF F#6* -)9:<>> d = [ ... {'docs': [{'tokens':[1, 2]}, {'tokens':[3]}]}, ... {'docs': [{'tokens':[7]}]}] >>> st = tf.experimental.StructuredTensor.from_pyval(d) >>> st2 =st.promote(('docs','tokens'), 'docs_tokens') >>> st2[0]['docs_tokens'] >>> st2[1]['docs_tokens'] Args: source_path: the path of the field or substructure to promote; must have length at least 2. new_name: the name of the new field (must be a string). Returns: a modified structured tensor with the new field as a child of the grandparent of the source_path. Raises: ValueError: if source_path is not a list or a tuple or has a length less than two, or new_name is not a string, or the rank of source_path is unknown and it is needed. znew_name is not a stringz#source_path must be a list or tupler<z)source_path must have length at least twoN)r+rKrAlistr/rErrj)r%rnew_namegrandparent_path new_fieldnew_paths r&promotezStructuredTensor.promotes6 h $ 1 22 kD%= 1 < == ;! B CC"3B'$$[2BCI8+-H   h 2 33r(c.|jjS)z@The rank of this StructuredTensor. Guaranteed not to be `None`.r r@r%s r&r@zStructuredTensor.rank4s    " ""r(c6|jjS)zThe static shape of this StructuredTensor. The returned `TensorShape` is guaranteed to have a known rank, but the individual dimension sizes may be unknown. Returns: `tf.TensorShape` r _to_tensor_shapers r&r2zStructuredTensor.shape9s    . . 00r(c|jS)z"Deprecated form of row_partitions.)r4rs r&r\z StructuredTensor._row_partitionsFs   r(cV|jdkry|jjS)a A tuple of `RowPartition`s defining the shape of this `StructuredTensor`. When `self.rank <= 1`, this tuple will be empty. When `self.rank > 1`, these `RowPartitions` define the shape of the `StructuredTensor` by describing how a flat (1D) list of structures can be repeatedly partitioned to form a higher-dimensional object. In particular, the flat list is first partitioned into sublists using `row_partitions[-1]`, and then those sublists are further partitioned using `row_partitions[-2]`, etc. The following examples show the row partitions used to describe several different `StructuredTensor`, each of which contains 8 copies of the same structure (`x`): >>> x = {'a': 1, 'b': ['foo', 'bar', 'baz']} # shape = [] (scalar) >>> s1 = [[x, x, x, x], [x, x, x, x]] # shape = [2, 4] >>> tf.experimental.StructuredTensor.from_pyval(s1).row_partitions (tf.RowPartition(row_splits=[0 4 8]),) >>> s2 = [[x, x], [x, x], [x, x], [x, x]] # shape = [4, 2] >>> tf.experimental.StructuredTensor.from_pyval(s2).row_partitions (tf.RowPartition(row_splits=[0 2 4 6 8]),) >>> s3 = [[x, x, x], [], [x, x, x, x], [x]] # shape = [2, None] >>> tf.experimental.StructuredTensor.from_pyval(s3).row_partitions (tf.RowPartition(row_splits=[0 3 3 7 8]),) >>> s4 = [[[x, x], [x, x]], [[x, x], [x, x]]] # shape = [2, 2, 2] >>> tf.experimental.StructuredTensor.from_pyval(s4).row_partitions (tf.RowPartition(row_splits=[0 2 4]), tf.RowPartition(row_splits=[0 2 4 6 8])) >>> s5 = [[[x, x], [x]], [[x, x]], [[x, x], [x]]] # shape = [3, None, None] >>> tf.experimental.StructuredTensor.from_pyval(s5).row_partitions (tf.RowPartition(row_splits=[0 2 3 5]), tf.RowPartition(row_splits=[0 2 3 5 7 8])) Note that shapes for nested fields (such as `x['b']` in the above example) are not considered part of the shape of a `StructuredTensor`, and are not included in `row_partitions`. If this `StructuredTensor` has a ragged shape (i.e., if any of the `row_partitions` is not uniform in size), then all fields will be encoded as either `RaggedTensor`s or `StructuredTensor`s with these `RowPartition`s used to define their outermost `self.rank` dimensions. Returns: A `tuple` of `RowPartition` objects with length `self.rank - 1` (or `0` if `self.rank < 2`) r<r>)r@r _as_row_partitionsrs r&r4zStructuredTensor.row_partitionsMs(l yy1}     0 0 22r(c@|jdk(ry|jdS)aThe number of rows in this StructuredTensor (if rank>0). This means the length of the outer-most dimension of the StructuredTensor. Notice that if `self.rank > 1`, then this equals the number of rows of the first row partition. That is, `self.nrows() == self.row_partitions[0].nrows()`. Otherwise `self.nrows()` will be the first dimension of the field values. Returns: A scalar integer `Tensor` (or `None` if `self.rank == 0`). rN)r@r rs r&r3zStructuredTensor.nrowss# yyA~    a  r(c||jjk(r|Stt|j||jj |S)Nr*)r rVrrZr with_dtype)r%rVs r&with_shape_dtypez!StructuredTensor.with_shape_dtypesJ ""((( k !$,,6''2259 ;;r(cTtj|d}td|DS)z1True if all fields are composed of eager tensors.T)expand_compositesc3PK|]}t|tj ywr$)r+r EagerTensor).0ts r& z-StructuredTensor._is_eager..s?!z!S__-?s$&)rflattenall)r%tensorss r& _is_eagerzStructuredTensor._is_eagers#ll448G ?w? ??r(cHt|jjS)z;Returns the string field names for this `StructuredTensor`.)r/rkeysrs r& field_nameszStructuredTensor.field_namess ""$ %%r(ct|ttfrG|}|D]>}t|tst dj |||j |}@|S|j|S)aReturns the tensor value for the specified field or path. If `field_name` is a `string`, then it names a field directly owned by this `StructuredTensor`. If this `StructuredTensor` has shape `[D1...DN]`, then the returned tensor will have shape `[D1...DN, V1...VM]`, where the slice `result[d1...dN]` contains the field value for the structure at `self[d1...dN]`. If `field_name` is a `tuple` of `string`, then it specifies a path to a field owned by nested `StructuredTensor`. In particular, `struct.field_value((f1, f2, ..., fN))` is equivalent to `struct.field_value(f1).field_value(f2)....field_value(fN)` Args: field_name: `string` or `tuple` of `string`: The field whose values should be returned. Returns: `Tensor`, `StructuredTensor`, or `RaggedTensor`. Raises: KeyError: If the given field_name is not found. zField path {} not found in {})r+rr/rKeyErrorrdr~r)r% field_namerRfs r&r~zStructuredTensor.field_valuesw0*tUm,e%!%!128??$ ! !!!!$ % l << ##r(ct|tr t|}nt|ts|f}|s|S|jdk(r|j |S|j |S)a<Returns the specified piece of this StructuredTensor. * If `struct_tensor` is scalar (i.e., a single structure), then `struct_tensor[f]` returns the value of field `f` (where `f` must be a string). * If `struct_tensor` is non-scalar (i.e., a vector or higher-dimensional tensor of structures), `struct_tensor[i]` selects an element or slice of the tensor using standard Python semantics (e.g., negative values index from the end). `i` may have any of the following types: * `int` constant * `string` constant * scalar integer `Tensor` * `slice` containing integer constants and/or scalar integer `Tensor`s #### Multidimensional indexing `StructuredTensor` supports multidimensional indexing. I.e., `key` may be a `tuple` of values, indexing or slicing multiple dimensions at once. For example, if `people` is a vector of structures, each of which has a vector- valued `names` field, then `people[3, 'names', 0]` is equivalent to `people[3]['names'][0]`; and `people[:, 'names', :]` will return a (possibly ragged) matrix of names, with shape `[num_people, num_names_per_person]`. Args: key: Indicates which piece of the StructuredTensor to return. Returns: A `Tensor`, `StructuredTensor`, or `RaggedTensor`. r)r+rr/r@_scalar_getitem_tensor_getitem)r%rQs r& __getitem__zStructuredTensor.__getitem__s^B#t #Jc U # Fc  k yyA~  ! !# &&  ! !# &&r(ctdtrydjjdj[djLt fd|j jD}tj||jStdtjs td|j djddS)Nrc3PK|]\}}||jddfyw)r=Nrrrr~rQs r&rz3StructuredTensor._scalar_getitem..s5K1[ !8!8QR!ABKs#&zJKey for indexing a StructuredTensor must be a string or a full slice (':')r=)r+slicestartstopstepr,rrJrrUr2rbytes_or_text_typesrAr)r%rQr!s ` r&rz StructuredTensor._scalar_getitems3q65!c!fll&: A A 3K59\\5G5G5IKKf  ) )&$** == A : : ; 6 77 <<A  + +CG 44r(c|j}t|krtfd|jj D}|j j }tD]\}}t|tr+|j|j |j;d||<At|ttjfrd||<g| t!dt!d|z|Dcgc] }|dk7s | }}t"j%||St|t&j(s t!d|j|j+d||dzdzScc}w)Nc3JK|]\}}||jfywr$rrs r&rz3StructuredTensor._tensor_getitem..s/O5Z$[%<% > ?".9Aba9l9  ) )*l CCD 6#=#= >OPP \\#d) $ 0 0Udc$()n1L MM:s 4 E0?E0ct|jj}d|D}d|D}dj|}d|d|jdS)Nc3\K|]$\}}|t|jddf&yw) z N)rKreplacerrSrTs r&rz,StructuredTensor.__repr__..+s) Mdaq#a&..'789 Ms*,c3FK|]\}}dj||yw)z"{}": {}N)rdrs r&rz,StructuredTensor.__repr__..,s! 9$!Qj1% 9s!z , z()rbrrJjoinr2)r%r! dict_reprs r&__repr__zStructuredTensor.__repr__)sR DLL&&( )F Mf MF 9& 9F""6*I!*4:: 78r(c|js tdi}|jjD]\}}t |t j r|j}t |tjr|j}nKt |tjr|j}n t |tr|j}|||<t!|j"dkDrk|s$t%|j&|j)St+t-|j/t-|j1|j2S|S)aReturns this StructuredTensor as a nested Python dict or list of dicts. Converts this `StructuredTensor` to a nested python value: * `StructTensors` with `rank=0` are converted into a dictionary, with an entry for each field. Field names are used as keys and field values are converted to python values. In particular: * Scalar Tensor fields are converted to simple values (such as `int` or `float` or `string`) * Non-scalar Tensor fields and RaggedTensor fields are converted to nested lists of simple values. * StructuredTensor fields are converted recursively using `to_pyval`. * `StructTensors` with `rank>0` are converted to nested python `list`s, containing one dictionary for each structure (where each structure's dictionary is defined as described above). Requires that all fields are Eager tensors. >>> tf.experimental.StructuredTensor.from_fields( ... {'a': [1, 2, 3]}, [3]).to_pyval() [{'a': 1}, {'a': 2}, {'a': 3}] Note that `StructuredTensor.from_pyval(pyval).to_pyval() == pyval`. Returns: A nested Python dict or list of dicts. z>   H JJF **,  e E3?? +  E2:: &  e]77 8  e- . fSk  4:: 5d6I6I6:jjlD D - v{{} tFMMO4diiAAmr(c(|j||dS)aZConstructs a StructuredTensor from a nested Python structure. >>> tf.experimental.StructuredTensor.from_pyval( ... {'a': [1, 2, 3], 'b': [[4, 5], [6, 7]]}) }, shape=())> Note that `StructuredTensor.from_pyval(pyval).to_pyval() == pyval`. Args: pyval: The nested Python structure that should be used to create the new `StructuredTensor`. typespec: A `StructuredTensor.Spec` specifying the expected type for each field. If not specified, then all nested dictionaries are turned into StructuredTensors, and all nested lists are turned into Tensors (if rank<2) or RaggedTensors (if rank>=2). Returns: A `StructuredTensor`. r> _from_pyval)r1pyvaltypespecs r& from_pyvalzStructuredTensor.from_pyvalqs2 ??5(B //r(ct|tr|j|||St|ttfr@t }t ||}||j|||||S|j|||S|j|||S)a(Helper function for from_pyval. Args: pyval: The nested Python structure that should be used to create the new `StructuredTensor`. typespec: A `StructuredTensor.Spec` specifying the expected type for each field. If not specified, then all nested dictionaries are turned into StructuredTensors, and all nested lists are turned into Tensors (if rank<2) or RaggedTensors (if rank>=2). path_so_far: the path of fields that led here (for error messages). Returns: A `StructuredTensor`. ) r+r, _from_pydictrr/set!_pyval_find_struct_keys_and_depth_from_pylist_of_dict_from_pylist_of_value_from_pyscalar)r1rr path_so_farrr@s r&rzStructuredTensor._from_pyvals"%   eX{ ;; ED%= ) Ud .ud ;d ''tT8(35 5((+FF   x ==r(c|$tfd|jD}n|j}|jt |t j r&|jdk(rt|tk(stdd|d|tfd|jD}t j|dd S) zEConverts python dictionary `pyval` to a StructuredTensor with rank=0.c3VK|] \}}|j|d|fzf"ywr$r)rrSrTr1rs r&rz0StructuredTensor._from_pydict..s61A4t1CDE1s&)r Value at  does not match typespec:  vs c3\K|]#\}}|j|||fzf%ywr$r)rrSrTr1 field_specsrs r&rz0StructuredTensor._from_pydict..s:1A;q>;!;MNO1s),r>Fr!r2rP) r,rJ_shape _field_specsr+rSpecr@rrArU)r1rrrr! spec_shapers` ` @r&rzStructuredTensor._from_pydicts1"'++-11f??j))k#3#8#89oo"s5zS5E'E%uh89 91"'++-11f  ' 'vR% ' PPr(ctd|D}|D]}t||d|Ltjdg|z}|j D]\} } |j | d|| fz|| <!n|j } t|tjrt|t| z rtd|d|d||j}|j|krtd|d|d|| j D]/\} } |j |j| g| || fz|| <1 |s|tj||Stj!||d S#t"$r} td || d} ~ wwxYw) z?Converts python list `pyval` to a StructuredTensor with rank>1.c3$K|]}|gf ywr$r>)rrQs r&rz8StructuredTensor._from_pylist_of_dict..s,3),r=Nrrrz* does not match typespec (rank mismatch): FrError parsing path )r,_pyval_update_fieldsrr-rJrrr+rrrrArr@get_from_pylist_of_empty_dictrU Exception)r1rrr@rrr!childr2rQtargetrspecexcs r&rz%StructuredTensor._from_pylist_of_dicts,t, ,F-5&!,-&&v}5e!<<>J-3oofdK3&4HIs J$$k($4$9$9: v;[) )%uh89 9ooe d '2E8EF F$**,=+3oo JJsB {cV';=s =J ( ::5$GG  ) )uu*66 J +? @cIJs8E*E** F3FFc|dk(rtjiddS|dk(r&t|}|f}tji||S|dkDrutjt |}t|}t jt|gdg|dz zz}tji||j|Sy) z=Converts a pylist of empty dictionaries to StructuredTensors.rr>Frr=)r!r2r3N)r!r2r4r3) rrUrErconstant_dicts_to_zerosrr-_nested_row_partitions)r1rr@r3r2 ragged_zeross r&rz+StructuredTensor._from_pylist_of_empty_dicts qy  ) )2 ) NN %jehe  ) )5 ) NN '001GHl%je&&E |v7J'KLe  ) )%<< * r(c | tj|St |t j r\ tj||j}|jj|jstd|d|d||St |tjrS tj||j|j|j|j |jdzdSt |t"j$r>t'|}|td|d|d||j)|t+|||Std|d|d|#t$r}td||d}~wwxYw#t$r}td||d}~wwxYw#t$r}td||d}~wwxYw)zEConverts python list `pyval` to a Tensor or RaggedTensor with rank>1.Nrrrrr=)rV ragged_rankrow_splits_dtype inner_shape)rrrrAr+r TensorSpecr rVr2is_compatible_withrRaggedTensorSpec_dtype _ragged_rank_row_splits_dtyperrr_pyval_empty_list_depthrr)r1rrrrr empty_ranks r&rz&StructuredTensor._from_pylist_of_valuesL!**511 Hf// 0L%%eX^^<^^ . .v|| <%x89 9 m Hm<< =L!** // --%77 (=(=(A(BC E E H.33 4*51j  %x89 9''suj((35 5 #Xu6 77?LKABKL LKABKLLKABKLsGE( F*AF(( F1FF F%F  F%( G1GGc@| tj|St |t j r|jjdk(std|d|d|tj||jS#t$r}td||d}~wwxYw)z1Converts python scalar value `pyval` to a Tensor.Nrrrrr) r rrrAr+rrr2r@rV)r1rrrrs r&rzStructuredTensor._from_pyscalarsL##E**6#4#45nn!!Q&%x89 9 ! !% 88LKABKLsB B BBct|ts td|jjdk(rt d|jzt ||S)aPartitions the outer dimension of this StructuredTensor. Returns a new `StructuredTensor` with the same values as `self`, where the outer dimension is partitioned into two (possibly ragged) dimensions. Requires that this StructuredTensor have an outer dimension (i.e., `self.shape.rank > 0`). >>> st = tf.experimental.StructuredTensor.from_pyval( ... [{'foo': 12}, {'foo': 33}, {'foo': 99}]) >>> partition = RowPartition.from_row_lengths([2, 0, 1]) >>> st.partition_outer_dimension(partition) }, shape=(3, None))> Args: row_partition: A `RowPartition`. Returns: A `StructuredTensor` with rank `values.rank + 1`. z%row_partition must be a RowPartition.rz"Shape %s must have rank at least 1)r+rrBr2r@rA_partition_outer_dimension)r% row_partitions r&partition_outer_dimensionz*StructuredTensor.partition_outer_dimension.sL. m\ 2 = >> zz! ;djjH II %dM ::r(ctj||jjdd}tj||jjdd}||kst d||fzt |||S)a.Merges outer_axis...inner_axis into a single dimension. Returns a copy of this RaggedTensor with the specified range of dimensions flattened into a single dimension, with elements in row-major order. >>> st = tf.experimental.StructuredTensor.from_pyval( ... [[{'foo': 12}, {'foo': 33}], [], [{'foo': 99}]]) >>> st.merge_dims(0, 1) Args: outer_axis: `int`: The first dimension in the range of dimensions to merge. May be negative (to index from the last dimension). inner_axis: `int`: The last dimension in the range of dimensions to merge. May be negative (to index from the last dimension). Returns: A copy of this tensor, with the specified dimensions merged into a single dimension. The shape of the returned tensor will be `self.shape[:outer_axis] + [N] + self.shape[inner_axis + 1:]`, where `N` is the total number of slices in the merged dimensions. outer_axisz rank(self)) axis_name ndims_name inner_axiszDExpected outer_axis (%d) to be less than or equal to inner_axis (%d))rget_positive_axisr2r@rA _merge_dims)r%rrs r& merge_dimszStructuredTensor.merge_dimsKs4,,  !J ,,  !J  # ),6 +CD EE tZ 44r(ceZdZdZdZedZedejddfdZ e de jfdZ e deeej"ffdZe de jfd Ze d Zy ) StructuredTensor.SpeczA spec for StructuredTensor.c |jJyr$)r rs r& __validate__z"StructuredTensor.Spec.__validate__ws    ++ +r(c :d}|jD]p\}}t|}|td|d|j}|||krtd|d|d|d|j |}||}`|j |}rt j||S)z:Creates a spec of a StructuredTensor with fields and rank.NCannot convert spec of .Rank of field  is , but must be at least r r)rJ$_dynamic_ragged_shape_spec_from_specrAr@ _truncate _merge_withrr) r1r!r@r2rSrTfield_shape_untruncateduntruncated_rank field_shapes r&_from_fields_and_rankz+StructuredTensor.Spec._from_fields_and_rankzseLLN 1&1a"Fq"I " *4QCq9: :277  (-=-D^A3d3C2DE337&;< <-77= =%##K0% 1 " " " GGr(r2r7c0tj|iS)z.Creates the spec of an empty StructuredTensor.r%)rr)r1r2s r& _from_shapez!StructuredTensor.Spec._from_shapes  " " " CCr(c6|jjSr$rrs r&rzStructuredTensor.Spec._shapes    0 0 22r(c|jSr$)rrs r&rz"StructuredTensor.Spec._field_specss \\r(c|jSr$)rrs r&r2zStructuredTensor.Spec.shapes [[r(c.|jjSr$rrs r&r@zStructuredTensor.Spec.ranks    $ $$r(N)rD __module__ __qualname____doc__r classmethodr,rDynamicRaggedShaper.propertyrr-rrrKrTypeSpecrr2r@r>r(r&rrts&,HH$D(;;D DD3 0033d3 (:(:#:;|//%%r(r)F)r>NNF)FNr$)9rDr3r4r5rrKrs__annotations__rr7r r FieldNamer'r6r6r9rUrXboolrr DTyper^rrtrjr rrerrr8r@r2r\r4r3rrrr~rrrrrrrrrrrrrrrr>r(r&rr5s#*V 3 # $$%888 "(C#&')&WS+%56&1DD& AA8 B-@@ B B B!% W4W4r &* @?c;&'@? @? @? fll # @? 0B @?@?H%*W@ E+x2M,N!NOW@!W@.@W@rS#U38_S#"&uYk>F?G9H.H(I#JS#$(S#-?S#jJ4&4X # #  1  1    73 73r!$;FLL;5G;@& $N+'Z 5N688t004>>:QQ JJ@$%7%7N 9 9,;:'5R2%2%r(z^[a-zA-Z][a-zA-Z0-9_]*$cht|tjtjt fr|Stj |rtj|St|tjr|S tj|S#ttf$r}td|z|d}~wwxYw)z@Converts `value` to a Tensor, RaggedTensor, or StructuredTensor.z)Unexpected type for value in `fields`: %rN)r+rr.rrr is_ragged"convert_to_tensor_or_ragged_tensorr ExtensionTyperconvert_to_tensorrArB)rRrzs r&rNrNs : : 7 88 E  . . ;(H J0  E  (();< <.[s DUc&)n $ Dr=)r,rZrErrcr)rrHdepth value_slicerfs ` @r&rrGs + aZ D&! "" fQi.% DeAs6{.C D DD Df  't[%!)D  s#A=c|st|Dcgc]}ic}St|dd|djd}|dj}tt|dz Dcgc]}|||||dzc}Scc}wcc}w)aReturns a python list of empty dicts from the given row partitions. Args: row_partitions: The row-partitions describing the ragged shape of the result. nrows: The number of rows in the outermost row-partition. (Or if `len(row_partitions)==0`, then the number of empty dicts to return.) Returns: A nested python list whose leaves (if any) are empty python dicts. r=Nrr})rcr row_splitsrE)r4r3_rHsplitsrhs r&rrbs e %1B %% 3qrN1-88:2>@F A  ) ) +F5:3v;?5K LF6!9VAE] + LL & Ms B+Bct|tr |j|jyt|tt fr5d}|D],}t ||}|||dz}||dzk7s#td|Sy)aFinds the keys & depth of nested dictionaries in `pyval`. Args: pyval: A nested structure of lists, tuples, and dictionaries. keys: (output parameter) A set, which will be updated with any keys that are found in the nested dictionaries. Returns: The nesting depth of dictionaries in `pyval`, or `None` if `pyval` does not contain any dictionaries. Raises: ValueError: If dictionaries have inconsistent depth. rNr=z"Inconsistent depth of dictionaries)r+r,updaterrr/rrA)rrrmr child_depths r&rrwstKK  %$' EA5eTBk  =/% kAo %?@ @ A L r(cTt|tttfs t d|j D]A\}}t d|D]}|d} |jt|tr||ngCt|ttfr|D]}t|||dzyy)aAppend the field values from `pyval` to `fields`. Args: pyval: A python `dict`, or nested list/tuple of `dict`, whose value(s) should be appended to `fields`. fields: A dictionary mapping string keys to field values. Field values extracted from `pyval` are appended to this dictionary's values. depth: The depth at which `pyval` should be appended to the field values. z*Expected dict or nested list/tuple of dictr=r}N) r+r,rr/rArJrcappendr)rr!rmrQrrqrs r&rrs ED$. / A BB||~AmsF 1e_bzf MM 5$ 7%*R@A e}%55&%!)45&r(ct|tr<|sy|Dcgc] }t|}}td|Dryt |dzSycc}w)zFind the max depth for nested empty lists. Args: pyval: A nested python list. Returns: The maximum depth of empty lists in `pyval`, or None if `pyval` contains anything other than nested empty lists. r=c3$K|]}|du ywr$r>)rrms r&rz*_pyval_empty_list_depth..s -U5D= -rN)r+rr anyrF)rrTdepthss r&r r sTt  27 8Q%a( 8F 8 -f --  [1_  9sAc t|tjss|St|tjr;tjj t |jdddSt|tsJtfd|jjD}tj||j|jtt|j t#dzS)aUpdates `value` to use `new_partitions` as its (outer) row partitions. This is used to ensure that all fields in a `StructuredTensor` use identical `RowPartition` objects for the shared dimensions. In particular, `StructuredTensor.from_fields` first merges all of the row partitions from any fields, and then replaces the outer row partitions of all fields with the merged row partitions (using this function). Args: value: A `Tensor`, `RaggedTensor`, or `StructuredTensor`. new_partitions: A list of row-partitions that should be used by `value`. Must be equivalent to `value`'s current row partitions. Returns: A value that is equivalent to `value`, where outer row partitions have been replaced by `new_partitions`. r=Nr)rHrc3@K|]\}}|t|fywr$)rO)rrSrTnew_partitionss r&rz*_replace_row_partitions..s*; 1a1!^DE;rl)r!r2r3r4)r+rr.rr_from_row_partitionrOrHrr,rrJr6r2r3r/r4rE)rRr~rqs ` r&rOrOs$v}}%^ L%334  % % 9 9&u||^AB5GH$Q' : )) e- .. .;$)MM$7$7$9;;J  % %kkkkm^, e""3~#6#789: & ;;r(cLjdu}t|tjrq|sot j j jgt j|jddgd}t j||St|tjtjfr tjj|St|tsJj}j}t!j"||gj%|jdd}t'fd|j(j+D}tj-||j f|j.zS)aPartitions the outer dimension of `value` using `row_partitions`. Examples: >>> partition = RowPartition.from_row_lengths([2, 0, 1]) >>> _partition_outer_dimension(tf.constant([1, 2, 3]), partition) >>> struct_value = tf.experimental.StructuredTensor.from_pyval( ... [{'x': 1}, {'x': 2}, {'x': 3}]) >>> _partition_outer_dimension(struct_value, partition) }, shape=(3, None))> Args: value: Tensor, RaggedTensor, or StructuredTensor row_partition: RowPartition Returns: A value with the same type as `value`, where `result.rank = value.rank + 1`. NrMr=r)axisc3@K|]\}}|t|fywr$)r)rrSrTrs r&rz-_partition_outer_dimension..s*7q!0MBC7rl)rer+rr.rconcatr3r2rVreshaperrrrrSstatic_uniform_row_lengthrr- concatenater,rrJr6r4)rRrr? new_shaper3ncolsr2r!s ` r&rrsy2..0D8)v}}%i       * * , . )<)< =ab A C I   UI ..%&--)C)CDE  % % 9 9 }  e- .. .  & &E  3 3E  $ $e&+&- ..9k%++ab/.J 7 % 3 3 577F  % %}**, 5/// 11r(cdksJt|tjtjfrtj |St|t sJtfd|jjD}|jj}t ||S)zDMerges `outer_axis...inner_axis` of `value` into a single dimension.c3BK|]\}}|t|fywr$)r)rrSrTrrs r&rz_merge_dims..!s+7q!k!Z<=7s) r+rr.rrrrr,rrJr r)rRrrr!r"s `` r&rrs j   }'A'ABC  # #E:z BB e- .. . 7 % 3 3 577F&&22J L FL 11r(rct|tjr |jStj jj |Sr$)r+rrr rr7 _from_spec)rs r&r&r&+s> &++,     2 2 7 7 B B4 HHr(rfr;ct|tr|fSt|tr t|St|tsJ|S)zEFieldName can be given also as string, this normalizes it to a tuple.)r+rKrr/)rfs r&rara6s<c 7Nd ; D%  +r(c`t|try|Dcgc] }t|c}Scc}w)z(Replaces dictionaries zeros in a pylist.r)r+r,r)rxs r&rr@s(t &+ ,/!  ,, ,s+ctt|tr|j||Stj|||S)a)Merges outer_axis...inner_axis into a single dimension. If outer == inner, this is a NOOP. If inner < outer, then this fials. If inner >= source.shape.rank, then the behavior is undefined. Args: source: a tensor, ragged tensor, or structured tensor. outer: a python int, indicating the first dimension to compress (must be nonnegative). inner: a python int, indicating the first dimension to keep (of the tail) (must be nonnegative). Returns: source with outer_axis...inner_axis merged into a single dimension. )r+rrr)sourceouterinners r&rrGs7"()   UE **  # #FE5 99r(c&t|tr |jStj||}t|t j rtjg|St|tjr|Std|d|d)zHExtension of DynamicRaggedShape.from_tensor to support StructuredTensor.rM)r4rzExpected shape tf.shape(z<) to return a Tensor or a DynamicRaggedShape. Instead, got: r!) r+rr rshape_v2rr.rr7rB)fieldrVr2s r&!_dynamic_ragged_shape_from_tensorr^s'(      UU 3%v}}%  2 2u ..%-@@A L,UG477 u-et4(6FGk :  : +A3/ 0c9:s< AAArct|tjr|jjSt|t r|j jSyr$)r+rr_row_partitionrVrr )rs r&rDrDsD}112    % %%'(    $ $$ r(ct|tjr|j|St|tr|j |S|Sr$)r+rrwith_row_splits_dtyperr)rrVs r&_field_with_shape_dtypersD}112  & &u --'(  ! !% (( ,r(c j|jDcic]\}}|t||c}}Scc}}wr$)rJr)r!rVrSrTs r&rZrZs-=C\\^ L6Aq! $Q . . LL Ls/ct|tsJ|t|tjsJ||1t|tj st|t sJ||t|tsJ||j}| tdt|||}t||}d}|jr/tjj|j!|}|dk(r4tjjt#j$d|St'|t)|||}|dk(rYtj*|d}||}|,t'|tjj|g|}| t-d|S|r+t'|tjj/||}| t-d|S) z2Produce a DynamicRaggedShape for StructuredTensor.Nr;rr)rr=zMMust specify `nrows`, a fully specified `shape`, or have `fields` if `rank=1`zPMust specify row_partitions, a fully specified shape, or have fields if rank > 1)r+r,rr-rr.rXr/r@rBrYrZis_fully_definedrr7_from_inner_shaperrzerosrr[dimension_valuerAfrom_row_partitions)r!r2r3r4r@rVr alt_values r&r0r0s FD !)6) ! E<33 4;e; 4 *UFMM:j S?   :n.3$5E6DE 5 $ \ D EE FE> :% fe ,& &  ! 4 4 F F uG&F QY  2 2 D DE* ,, (:64(O P& QY,,U1X6Ie #   1 1 C CgU D $%f~ 7 88 M !//CC % D )*F  ^ 2 33 -r(c t|ts td|jD]}t|trtd|j D]'}t|t jrtdtjjjtj|dtj}|j }| td|j#D]k\}}t%|}|t'd|d|j }|||krt'd|d |d |d|j)|}|j+|}mt,j|| S) z/A placeholder for the old StructuredTensorSpec.z!field_specs must be a dictionary.z2field_specs must be a dictionary with string keys.z6field_specs must be a dictionary with TypeSpec values.rr;r r!r"r#r$r%)r+r,rBrrKrHrr9rr7r_from_tensor_shaperr?r rFr@rJr&rAr'r(r)r2rrSrTr@r)r*r+s r&StructuredTensorSpecrs K & 7 88    La a  J KKL    Pa a++ , N OOP  1 1 6 6 I IE"Av|| 5% $ \ D EE!!# +fq!B1E& 015 66.33$)9D)@ s$/?.@A004vQ8 99)33D9K   k *E +   UK  HHr(r$)Rr5retypingrrrrrrr r rrtensorflow.python.frameworkr r r rrrrtensorflow.python.opsrrrrtensorflow.python.ops.raggedrrr*tensorflow.python.ops.ragged.row_partitionrtensorflow.python.utilrr tensorflow.python.util.tf_exportrr.rrArsrtBatchableExtensionTypercompilerLrNrKr=rYrVr_rXrYrarrrrr rOrrobject_structured_tensor_factory_keyr7rrrr&rarrrrrXr[rDrrZr0rr>r(r&rs RRR3.6+.41++2*=;6C)'6 MM  !  [M;. / *+p%~<<p%,p%l#56$" C$ %.6v}}.EXl349?D"=J,8 (6M*>5.,#;L.1b 2"(I $77<<..0@0E0E!!" #I,,11 I#-:0A.AAA"$778$778"556 <<$%9%L%LM$khv||.D;#)<<4?MwsK'78M$llM/6sK7G/HM 4pIr(