BVhdZddlZddlZddlZddlmZddlm Z ddl m Z ddl m ZddlmZe j"Ze j&Ze j*Ze j.Ze j2Ze j6Ze j:Ze j>Z e jBZ"e jFZ$e jJZ&dZ'd Z(d Z)d Z*Gd d ejVZ,Gdde-Z.e.Z/dZ0d4dZ1dZ2dZ3dZ4dZ5dZ6dZ7dZ8dZ9dZ: d5dZ; d5dZdZ?d4d Z@d4d!ZA d7d"ZB d7d#ZCd$ZDd%ZEd&ZFd'ZGd8d(ZHd8d)ZId*ZJ d5d+ZK d5d,ZL d6d-ZM d5d.ZN d5d/ZOd0ZPd1ZQd2ZRd3ZSy)9aUtility methods for handling nests. This module encapsulates different semantics of handling nests by the public tf.nest APIs and internal tf.data APIs. The difference in semantics exists for historic reasons and reconciliation would require a non-backwards compatible change. The implementation of the different semantics use a common utility to avoid / minimize further divergence between the two APIs over time. N)pywrap_tensorflow) tf_logging) _pywrap_utils)collections_abc)CustomNestProtocolzThe two structures don't have the same sequence type. Input structure has type {input_type}, while shallow structure has type {shallow_type}.zThe two structures don't have the same sequence length. Input structure has length {input_length}, while shallow structure has length {shallow_length}.zThe input_tree has fewer items than the shallow_tree. Input structure has length {input_size}, while shallow structure has length {shallow_size}.zThe shallow_tree's keys are not a subset of the input_tree's keys. The shallow_tree has the following keys that are not in the input_tree: {}.ceZdZdZdZdZy)Modalitya7Modality/semantic used for treating nested structures. - Modality.CORE follows tensorflow_core/tf.nest semantics. The following collection types are recognized by `tf.nest` as nested structures: * `collections.abc.Sequence` (except `string` and `bytes`). This includes `list`, `tuple`, and `namedtuple`. * `collections.abc.Mapping` (with sortable keys). This includes `dict` and `collections.OrderedDict`. * `collections.abc.MappingView` (with sortable keys). * [`attr.s` classes](https://www.attrs.org/). Any other values are considered **atoms**. Not all collection types are considered nested structures. For example, the following types are considered atoms: * `set`; `{"a", "b"}` is an atom, while `["a", "b"]` is a nested structure. * [`dataclass` classes](https://docs.python.org/library/dataclasses.html) * `tf.Tensor` * `numpy.array` - Modality.DATA follows tf.data's nest semantics. This modality makes two changes: 1. It removes support for lists as a level of nesting in nested structures. 2. It adds support for `SparseTensorValue` as an atomic element. The motivation for this change is twofold: 1. It seems more natural for lists to be treated (e.g. in Dataset constructors) as tensors, rather than lists of (lists of...) tensors. 2. This is needed because `SparseTensorValue` is implemented as a `namedtuple` that would normally be flattened and we want to be able to create sparse tensor from `SparseTensorValue's similarly to creating tensors from numpy arrays. COREDATAN)__name__ __module__ __qualname____doc__r r P/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/util/nest_util.pyr r Ls&P $ $rr ceZdZgZdZdZy) _DotStringcyN.rselfs r__str__z_DotString.__str__| rcyrrrs r__repr__z_DotString.__repr__rrN)r r r __slots__rrrrrrrys)rrc|tjk(r t|S|tjk(r t |St dj |)aqReturns true if its input is a nested structure. For Modality.CORE refer to [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest) for the definition of a nested structure. Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] structure: the value to test. Returns: True if the input is a nested structure. -Unknown modality used {} for nested structure)r r _tf_core_is_nestedr _tf_data_is_nested ValueErrorformat)modality structures r is_nestedr'sM i ((8== i (( 7>>xH rc.tj||S)aReturns True iff `instance` is a `namedtuple`. Args: instance: An instance of a Python object. strict: If True, `instance` is considered to be a `namedtuple` only if it is a "plain" namedtuple. For instance, a class inheriting from a `namedtuple` will be considered to be a `namedtuple` iff `strict=False`. Returns: True if `instance` is a `namedtuple`. )r IsNamedtuple)instancestricts r is_namedtupler,s  # #Hf 55rct|rtttt||t |}|t j k(r t j |j}n|}|D] }|||< |St|r|ttt||t |}t|dds4tjtjdj|d |fd|DSt|r t!|St#|s t%|r@t'|t(j*rt |j,}n t |}||St/|r0t1|dk(sJ|j2}|j5|dSt7|r$t1|dk(sJ|j5|dSt'|t8rt;t!||St'|t(j*r%t |t;|j,|St'|t<r.|j?d}|jA|tC|St ||S#t$r*}tdjt |||d}~wwxYw) a_Converts the sequence `args` to the same type as `instance`. Args: instance: an instance of `tuple`, `list`, `namedtuple`, `dict`, `collections.OrderedDict`, or `composite_tensor.Composite_Tensor` or `type_spec.TypeSpec`. args: items to be converted to the `instance` type. Returns: `args` with the type of `instance`. __supported_by_tf_nest__FzPMapping types may not work well with tf.nest. Prefer using MutableMapping for {}c3,K|] }||f ywNr).0keyresults r z sequence_like..sB#C-BszError creating an object of type {} like {}. Note that it must accept a single positional argument representing an iterable of key-value pairs, in addition to self. Cause: {}Nr)"_is_mutable_mappingdictzip_tf_core_sortedtype _collections defaultdictdefault_factory _is_mappinggetattrr log_first_nWARNr$ TypeError_is_mapping_viewlistr, _is_attrs isinstance_wrapt ObjectProxy __wrapped___is_composite_tensorlen _type_spec_from_components _is_type_specrange sequence_liker__tf_flatten____tf_unflatten__tuple) r*args instance_typedr3errspecmetadatar4s @rrPrPss" #oh/6 7FNM 000  " "8#;#; >   D  a ))X t9>>  $ $T!W --(E" h ..(F../ 4>-(<($ I   ()/tH~x(M  s7J K%J??Kct|jd}d|D}|Dcgc]}|t||fc}Scc}w)a<Returns a list of (name, value) pairs from an attrs instance. TODO(b/268078256): check if this comment is valid, and if so, ensure it's handled in the function below. The list will be sorted by name. Args: obj: an object. Returns: A list of (attr_name, attr_value) pairs, sorted by attr_name. __attrs_attrs__c34K|]}|jywr1)name)r2as rr5z#_get_attrs_items.. s&1&s)r? __class__)objattrs attr_names attr_names r_get_attrs_itemsrds? #--!2 3%&&*@J K99gc9- . KK Ks<cf t|jS#t$r tdwxYw)HReturns a sorted list of the dict keys, with error if keys not sortable.z,nest only supports dicts with sortable keys.)sortedkeysrB)dict_s rr9r9s6D %**,  D B CCDs0c~ tt|S#t$r}td|jd}~wwxYw)rfz4nest only supports dicts with sortable keys. Error: N)rgrDrBmessage)ries r_tf_data_sortedrmsA $u+    >qyykJ s <7<c#K|tjk(rt|Ed{y|tjk(rt |Ed{yt dj |7F7!w)zYield elements of `iterable` in a deterministic order. Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] iterable: an iterable. Yields: The iterable elements in a deterministic order. Nr )r r _tf_core_yield_valuer _tf_data_yield_valuer#r$r%iterables r yield_valuers"s`#H---8== #H--- 7>>xH  .-s!"A/A+&A/ A- A/-A/c#:Kt|D] \}}| ywr1)_tf_core_yield_sorted_items)rr_vs rroro6s#)(3 da G scr|tjk(r t|Stdj |)Nr )r r rur#r$rqs ryield_sorted_itemsry;s4  &x 00 7>>xH rc#RKt|trt|D]}|yt|tk(rt|D]}|yt|t t jfrt|D] }|||f yt|rt|D]}|yt|r"|jD]}|t||fyt|r6|j}|j j"|j%|fyt'|r%|j j"|j(fyt|t*r9|j-d}t|tsJt|Ed{yt|D]}|y7w)aYield (key, value) pairs for `iterable` in a deterministic order. For Sequences, the key will be an int, the array index of a value. For Mappings, the key will be the dictionary key. For objects (e.g. namedtuples), the key will be the attribute name. In all cases, the keys will be iterated in sorted order. Args: iterable: an iterable. Yields: The iterable's (key, value) pairs, in order of sorted keys. r/N)rFrD enumerater:rSr7_collections_abcMappingr9rErdr,_fieldsr?rJrL value_typer _to_componentsrN_component_specsrrQ)rritemr3field type_specflat_components rruruDs $(# j H~(# j(T#3#;#;<= x( #  * jX!!, 78U+ ++,H%##I    ' ')A)A()K KKX    & &(A(A AA(./,,.q1N ne ,, ,((((# j)sFF' F% F'c#Kt|tjrt|D] }|| y|jj dk(r|yt |rt|D] \}}| yt|tr0|jd}t|tsJ|Ed{y|D]}|y7w)zYield elements of `iterable` in a deterministic order. Args: iterable: an iterable. Yields: The iterable elements in a deterministic order. SparseTensorValuer/N) rFr|r}rmr_r rErdrrQrS)rrr3rvattrrvalues rrprpys*223 x( SM ""&99 N#H-4 j(./,,.q1N ne ,, , ksB.C0C1Cc|tjk(rt||||y|tjk(rt |||yt dj |)a<Asserts that two structures are nested in the same way. For Modality.CORE refer to [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest) for the definition of a structure. Note the method does not check the types of atoms inside the structures. Examples: * These atom vs. atom comparisons will pass: >>> tf.nest.assert_same_structure(1.5, tf.Variable(1, tf.uint32)) >>> tf.nest.assert_same_structure("abc", np.array([1, 2])) * These nested structure vs. nested structure comparisons will pass: >>> structure1 = (((1, 2), 3), 4, (5, 6)) >>> structure2 = ((("foo1", "foo2"), "foo3"), "foo4", ("foo5", "foo6")) >>> structure3 = [(("a", "b"), "c"), "d", ["e", "f"]] >>> tf.nest.assert_same_structure(structure1, structure2) >>> tf.nest.assert_same_structure(structure1, structure3, check_types=False) >>> import collections >>> tf.nest.assert_same_structure( ... collections.namedtuple("bar", "a b")(1, 2), ... collections.namedtuple("foo", "a b")(2, 3), ... check_types=False) >>> tf.nest.assert_same_structure( ... collections.namedtuple("bar", "a b")(1, 2), ... { "a": 1, "b": 2 }, ... check_types=False) >>> tf.nest.assert_same_structure( ... { "a": 1, "b": 2, "c": 3 }, ... { "c": 6, "b": 5, "a": 4 }) >>> ragged_tensor1 = tf.RaggedTensor.from_row_splits( ... values=[3, 1, 4, 1, 5, 9, 2, 6], ... row_splits=[0, 4, 4, 7, 8, 8]) >>> ragged_tensor2 = tf.RaggedTensor.from_row_splits( ... values=[3, 1, 4], ... row_splits=[0, 3]) >>> tf.nest.assert_same_structure( ... ragged_tensor1, ... ragged_tensor2, ... expand_composites=True) * These examples will raise exceptions: >>> tf.nest.assert_same_structure([0, 1], np.array([0, 1])) Traceback (most recent call last): ... ValueError: The two structures don't have the same nested structure >>> tf.nest.assert_same_structure( ... collections.namedtuple('bar', 'a b')(1, 2), ... collections.namedtuple('foo', 'a b')(2, 3)) Traceback (most recent call last): ... TypeError: The two structures don't have the same nested structure For Modality.DATA, nested structures are treated differently than Modality.CORE. Please refer to class Modality's documentation above to read up on these differences. Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] nest1: an atom or a nested structure. nest2: an atom or a nested structure. check_types: - For Modality.CORE: if `True` (default) types of structures are checked as well, including the keys of dictionaries. If set to `False`, for example a list and a tuple of objects will look the same if they have the same size. Note that namedtuples with identical name and fields are always considered to have the same shallow structure. Two types will also be considered the same if they are both list subtypes (which allows "list" and "_ListWrapper" from trackable dependency tracking to compare equal). `check_types=True` only checks type of sub-structures. The types of atoms are not checked. - For Modality.DATA: if `True` (default) types of sequences should be same as well. For dictionary, "type" of dictionary is considered to include its keys. In other words, two dictionaries with different keys are considered to have a different "type". If set to `False`, two iterables are considered same as long as they yield the elements that have same structures. expand_composites: Arg only valid for Modality.CORE. If true, then composite tensors such as `tf.sparse.SparseTensor` and `tf.RaggedTensor` are expanded into their component tensors. Raises: ValueError: If the two structures do not have the same number of atoms or if the two structures are not nested in the same way. TypeError: If the two structures differ in the type of sequence in any of their substructures. Only possible if `check_types` is `True`. r N)r r _tf_core_assert_same_structurer _tf_data_assert_same_structurer#r$)r%nest1nest2 check_typesexpand_compositess rassert_same_structurersSB"5%>OP8== "5%= 7>>xH rc,t|}t|} tj||||y#ttf$rS}t t d|}t t d|}t|t |d|d|d}~wwxYw)NctSr1_DOTrvs rz0_tf_core_assert_same_structure..rctSr1rrs rrz0_tf_core_assert_same_structure..rrz Entire first structure: z Entire second structure: )boolrAssertSameStructurer#rBstr_tf_core_map_structurer:)rrrrrlstr1str2s rrr s [!+,- %% uk#4 i  %ne< =D %ne< =D $q' q64  s1BABBc2tj|||yr1)rAssertSameStructureForData)rrrs rrrs**5%Ercg}|xst}t|D]P}||r-t|||||\}}|j||||}8|j|||dz }R||fS)aHelper function for pack_sequence_as. Args: structure: structure to mimic. flat: Flattened values to output substructure for. index: Index at which to start reading from flat. is_nested_fn: Function used to test if a value should be treated as a nested structure. sequence_fn: Function used to generate a new structure instance. Returns: The tuple (new_index, child), where: * new_index - the updated index into `flat` having processed `structure`. * packed - the subset of `flat` corresponding to `structure`, having started at `index`, and packed into the same nested format. Raises: ValueError: if `structure` contains more atoms than `flat` (assuming indexing starts from `index`). r/)rPro!_tf_core_packed_nest_with_indicesappend) r&flatindex is_nested_fn sequence_fnpackeds new_indexchilds rrr!s0 &,}+  * aA: T5, i mmK5)*e mmDK  qje  rcg}t|D]T}t|r.t|||\}}|jt |||}<|j|||dz }V||fS)aHelper function for pack_nest_as. Args: structure: Substructure (tuple of elements and/or tuples) to mimic flat: Flattened values to output substructure for. index: Index at which to start reading from flat. Returns: The tuple (new_index, child), where: * new_index - the updated index into `flat` having processed `structure`. * packed - the subset of `flat` corresponding to `structure`, having started at `index`, and packed into the same nested format. Raises: ValueError: if `structure` contains more elements than `flat` (assuming indexing starts from `index`). r/)rpr"!_tf_data_packed_nest_with_indicesrrP)r&rrrrrrs rrrHsu& &  *a!:1dEJi mmM!U+,e mmDK  qje rc|tjk(r t||S|tjk(r t |St dj |)akFlattens a nested structure. - For Modality.CORE: refer to [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest) for the definition of a structure. If the structure is an atom, then returns a single-item list: [structure]. This is the inverse of the `nest.pack_sequence_as` method that takes in a flattened list and re-packs it into the nested structure. In the case of dict instances, the sequence consists of the values, sorted by key to ensure deterministic behavior. This is true also for OrderedDict instances: their sequence order is ignored, the sorting order of keys is used instead. The same convention is followed in `nest.pack_sequence_as`. This correctly repacks dicts and OrderedDicts after they have been flattened, and also allows flattening an OrderedDict and then repacking it back using a corresponding plain dict, or vice-versa. Dictionaries with non-sortable keys cannot be flattened. Users must not modify any collections used in nest while this function is running. Examples: 1. Python dict (ordered by key): >>> dict = { "key3": "value3", "key1": "value1", "key2": "value2" } >>> tf.nest.flatten(dict) ['value1', 'value2', 'value3'] 2. For a nested python tuple: >>> tuple = ((1.0, 2.0), (3.0, 4.0, 5.0), 6.0) >>> tf.nest.flatten(tuple) [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] 3. For a nested dictionary of dictionaries: >>> dict = { "key3": {"c": (1.0, 2.0), "a": (3.0)}, ... "key1": {"m": "val1", "g": "val2"} } >>> tf.nest.flatten(dict) ['val2', 'val1', 3.0, 1.0, 2.0] 4. Numpy array (will not flatten): >>> array = np.array([[1, 2], [3, 4]]) >>> tf.nest.flatten(array) [array([[1, 2], [3, 4]])] 5. `tf.Tensor` (will not flatten): >>> tensor = tf.constant([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]]) >>> tf.nest.flatten(tensor) [] 6. `tf.RaggedTensor`: This is a composite tensor thats representation consists of a flattened list of 'values' and a list of 'row_splits' which indicate how to chop up the flattened list into different rows. For more details on `tf.RaggedTensor`, please visit https://www.tensorflow.org/api_docs/python/tf/RaggedTensor. with `expand_composites=False`, we just return the RaggedTensor as is. >>> tensor = tf.ragged.constant([[3, 1, 4, 1], [], [5, 9, 2]]) >>> tf.nest.flatten(tensor, expand_composites=False) [] with `expand_composites=True`, we return the component Tensors that make up the RaggedTensor representation (the values and row_splits tensors) >>> tensor = tf.ragged.constant([[3, 1, 4, 1], [], [5, 9, 2]]) >>> tf.nest.flatten(tensor, expand_composites=True) [, ] Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] structure: an atom or a nested structure. Note, numpy arrays are considered atoms and are not flattened. expand_composites: Arg valid for Modality.CORE only. If true, then composite tensors such as `tf.sparse.SparseTensor` and `tf.RaggedTensor` are expanded into their component tensors. Returns: A Python list, the flattened version of the input. Raises: TypeError: The nest is or contains a dict with non-sortable keys. r )r r _tf_core_flattenr _tf_data_flattenr#r$)r%r&rs rflattenrgsQ@ I'8 998== I && 7>>xH rcN|dgSt|}tj||S)z=See comments for flatten() in tensorflow/python/util/nest.py.N)rrFlatten)r&rs rrrs/ 6M,-   y*; <>> structure = { "key3": "", "key1": "", "key2": "" } >>> flat_sequence = ["value1", "value2", "value3"] >>> tf.nest.pack_sequence_as(structure, flat_sequence) {'key3': 'value3', 'key1': 'value1', 'key2': 'value2'} 2. For a nested python tuple: >>> structure = (('a','b'), ('c','d','e'), 'f') >>> flat_sequence = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0] >>> tf.nest.pack_sequence_as(structure, flat_sequence) ((1.0, 2.0), (3.0, 4.0, 5.0), 6.0) 3. For a nested dictionary of dictionaries: >>> structure = { "key3": {"c": ('alpha', 'beta'), "a": ('gamma')}, ... "key1": {"e": "val1", "d": "val2"} } >>> flat_sequence = ['val2', 'val1', 3.0, 1.0, 2.0] >>> tf.nest.pack_sequence_as(structure, flat_sequence) {'key3': {'c': (1.0, 2.0), 'a': 3.0}, 'key1': {'e': 'val1', 'd': 'val2'}} 4. Numpy array (considered a scalar): >>> structure = ['a'] >>> flat_sequence = [np.array([[1, 2], [3, 4]])] >>> tf.nest.pack_sequence_as(structure, flat_sequence) [array([[1, 2], [3, 4]])] 5. tf.Tensor (considered a scalar): >>> structure = ['a'] >>> flat_sequence = [tf.constant([[1., 2., 3.], [4., 5., 6.]])] >>> tf.nest.pack_sequence_as(structure, flat_sequence) [] 6. `tf.RaggedTensor`: This is a composite tensor thats representation consists of a flattened list of 'values' and a list of 'row_splits' which indicate how to chop up the flattened list into different rows. For more details on `tf.RaggedTensor`, please visit https://www.tensorflow.org/api_docs/python/tf/RaggedTensor. With `expand_composites=False`, we treat RaggedTensor as a scalar. >>> structure = { "foo": tf.ragged.constant([[1, 2], [3]]), ... "bar": tf.constant([[5]]) } >>> flat_sequence = [ "one", "two" ] >>> tf.nest.pack_sequence_as(structure, flat_sequence, ... expand_composites=False) {'foo': 'two', 'bar': 'one'} With `expand_composites=True`, we expect that the flattened input contains the tensors making up the ragged tensor i.e. the values and row_splits tensors. >>> structure = { "foo": tf.ragged.constant([[1., 2.], [3.]]), ... "bar": tf.constant([[5.]]) } >>> tensors = tf.nest.flatten(structure, expand_composites=True) >>> print(tensors) [, , ] >>> verified_tensors = [tf.debugging.check_numerics(t, 'invalid tensor: ') ... if t.dtype==tf.float32 else t ... for t in tensors] >>> tf.nest.pack_sequence_as(structure, verified_tensors, ... expand_composites=True) {'foo': , 'bar': } - For Modality.DATA: If `structure` is a scalar, `flat_sequence` must be a single-element list; in this case the return value is `flat_sequence[0]`. Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] structure: - For Modality.CORE: Nested structure, whose structure is given by nested lists, tuples, and dicts. Note: numpy arrays and strings are considered scalars. - For Modality.DATA: tuple or list constructed of scalars and/or other tuples/lists, or a scalar. Note: numpy arrays are considered scalars. flat_sequence: flat sequence to pack. expand_composites: Arg valid for Modality.CORE only. If true, then composite tensors such as `tf.sparse.SparseTensor` and `tf.RaggedTensor` are expanded into their component tensors. sequence_fn: Arg valid for Modality.CORE only. Returns: packed: `flat_sequence` converted to have the same recursive structure as `structure`. Raises: ValueError: If `flat_sequence` and `structure` have different atom counts. TypeError: For Modality.CORE only. `structure` is or contains a dict with non-sortable keys. r )r r _tf_core_pack_sequence_asr _tf_data_pack_sequence_asr#r$)r%r& flat_sequencerrs rpack_sequence_asrs^z $="3[ 8== $Y >> 7>>xH rc r|rtnt}|xst}d}||s+tdj ||dt |||sZt |dk7rGtdj t |||dt |t |||d|dS t||d||\}}|t |krt ||S#t$rKt||}t |t |k7r$tdt |t |||fzY\wxYw) zDImplements sequence packing, with the option to alter the structure.c6t|}|d|||dxrdzS)Nz...)r)rlength value_strs rtruncatez+_tf_core_pack_sequence_as..truncateks)E I Wf 67!3!= >>rzYAttempted to pack value: {} into a structure, but found incompatible type `{}` instead.dr/zThe target structure is of type `{}` {} However the input is a sequence ({}) of length {}. {} nest cannot guarantee that it is safe to map one to the other.rrzsCould not pack sequence. Structure had %d atoms, but flat_sequence had %d items. Structure: %s, flat_sequence: %s.) _is_nested_or_compositer!rPrBr$r:rKr#r IndexErrorr) r&rrrrr final_indexrflat_structures rrrbsw "38J,}+? m $  **0& ]C ($}*=+   i  =Q  ??Ev9oy#&=!- }c* @     ;=!\;KS'' ( Y ''  %%6N >c-00  K  #m"4i O P 1 s2&C""AD65D6c t|s2t|ts"tdt |j dt|s+t |dk7rtdt |d|dSt|}t |t |k7r*tdt |dt |d |d |d t||d\}}t||S) a<Returns a given flattened sequence packed into a nest. If `structure` is a scalar, `flat_sequence` must be a single-element list; in this case the return value is `flat_sequence[0]`. Args: structure: tuple or list constructed of scalars and/or other tuples/lists, or a scalar. Note: numpy arrays are considered scalars. flat_sequence: flat sequence to pack. Returns: packed: `flat_sequence` converted to have the same recursive structure as `structure`. Raises: ValueError: If nest and structure have different element counts. z2Argument `flat_sequence` must be a sequence. Got ''.r/z:Argument `structure` is a scalar but `len(flat_sequence)`=z > 1rz2Could not pack sequence. Argument `structure` had z, elements, but argument `flat_sequence` had z elements. Received structure: z, flat_sequence: r) r"rFrDrBr:r rKr#rrrP)r&rrrvrs rrrs$ ] +z-/N    ( ( ) -  I & =Q  ""%m"4!5T ;   #I..C .. < ~  K }  = +&}oQ 8 0 =!L)!V y& ))rc|tjk(rt|g|i|S|tjk(rt |g|i|St dj |)aCreates a new structure by applying `func` to each atom in `structure`. - For Modality.CORE: Refer to [tf.nest](https://www.tensorflow.org/api_docs/python/tf/nest) for the definition of a structure. Applies `func(x[0], x[1], ...)` where x[i] enumerates all atoms in `structure[i]`. All items in `structure` must have the same arity, and the return value will contain results with the same structure layout. Examples: * A single Python dict: >>> a = {"hello": 24, "world": 76} >>> tf.nest.map_structure(lambda p: p * 2, a) {'hello': 48, 'world': 152} * Multiple Python dictionaries: >>> d1 = {"hello": 24, "world": 76} >>> d2 = {"hello": 36, "world": 14} >>> tf.nest.map_structure(lambda p1, p2: p1 + p2, d1, d2) {'hello': 60, 'world': 90} * A single Python list: >>> a = [24, 76, "ab"] >>> tf.nest.map_structure(lambda p: p * 2, a) [48, 152, 'abab'] * Scalars: >>> tf.nest.map_structure(lambda x, y: x + y, 3, 4) 7 * Empty structures: >>> tf.nest.map_structure(lambda x: x + 1, ()) () * Check the types of iterables: >>> s1 = (((1, 2), 3), 4, (5, 6)) >>> s1_list = [[[1, 2], 3], 4, [5, 6]] >>> tf.nest.map_structure(lambda x, y: None, s1, s1_list) Traceback (most recent call last): ... TypeError: The two structures don't have the same nested structure * Type check is set to False: >>> s1 = (((1, 2), 3), 4, (5, 6)) >>> s1_list = [[[1, 2], 3], 4, [5, 6]] >>> tf.nest.map_structure(lambda x, y: None, s1, s1_list, check_types=False) (((None, None), None), None, (None, None)) - For Modality.DATA: Applies `func(x[0], x[1], ...)` where x[i] is an entry in `structure[i]`. All structures in `structure` must have the same arity, and the return value will contain the results in the same structure. Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] func: A callable that accepts as many arguments as there are structures. *structure: - For Modality.CORE: atom or nested structure. - For Modality.DATA: scalar, or tuple or list of constructed scalars and/or other tuples/lists, or scalars. Note: numpy arrays are considered scalars. **kwargs: Valid keyword args are: * `check_types`: - For Modality.CORE: If set to `True` (default) the types of iterables within the structures have to be same (e.g. `map_structure(func, [1], (1,))` raises a `TypeError` exception). To allow this set this argument to `False`. Note that namedtuples with identical name and fields are always considered to have the same shallow structure. - For Modality.DATA: only valid keyword argument is `check_types`. If set to `True` (default) the types of iterables within the structures have to be same (e.g. `map_structure(func, [1], (1,))` raises a `TypeError` exception). To allow this set this argument to `False`. * `expand_composites`: Valid for Modality.CORE only. If set to `True`, then composite tensors such as `tf.sparse.SparseTensor` and `tf.RaggedTensor` are expanded into their component tensors. If `False` (the default), then composite tensors are not expanded. Returns: A new structure with the same arity as `structure[0]`, whose atoms correspond to `func(x[0], x[1], ...)` where `x[i]` is the atom in the corresponding location in `structure[i]`. If there are different structure types and `check_types` is `False` the structure types of the first structure will be used. Raises: TypeError: If `func` is not callable or if the structures do not match each other by depth tree. ValueError: If no structure is provided or if the structures do not match each other by type. ValueError: If wrong keyword arguments are provided. r )r r rr _tf_data_map_structurer#r$)r%funcr&kwargss r map_structurersbB !$ = =f ==8== !$ = =f == 7>>xH rc t|std|z|s td|jdd}|jdd|r+tddj |j z|d dD]}t |d || fd |D}t|}t|d |Dcgc]}|| c} Scc}w)Nzfunc must be callable, got: %s#Must provide at least one structurerTrFzQOnly valid keyword arguments are `check_types` and `expand_composites`, not: `%s`z`, `r/rrrc36K|]}t|ywr1)r)r2rrs rr5z)_tf_core_map_structure..LsNq$Q(9:Nr) callablerBr#popjoinrhrr8r) rr&rrotherrentriesxrs @rrr3s $ 4t; <<  : ;; =$/+jj!4e<  ) ++fkkm $ %  }e"!  + OIN.  ' "l !AtQx!) !s5 C c Ft|std||s td|r'd|vst|dkDrtd|d|d}nd}|ddD]}t |d|| d |D}t |}t |d|Dcgc]}|| c}Scc}w) Nz'Argument `func` must be callable, got: rrr/zIOnly valid keyword argument for `check_types_dict` is 'check_types'. Got rTrrc32K|]}t|ywr1)r)r2rs rr5z)_tf_data_map_structure..ks;A$Q';s)rrBr#rKrr8r)rr&check_types_dictrrrrrs rrrWs $ =dVD EE  : ;;,,4D0E0I  01 4 #=1KK}Qe"9Q<KPQ<;.  ' "9Q>xH  Cs!%A3A/'A3A1 A31A3c#K||s||fytt|}t|D]-\}}||fz}||}t||||D] \}} || f /yw)aYields (path, value) pairs of input_tree flattened up to shallow_tree. Args: shallow_tree: Nested structure. Traverse no further than its leaf nodes. input_tree: Nested structure. Return the paths and values from this tree. Must have the same upper structure as shallow_tree. is_nested_fn: Function used to test if a value should be treated as a nested structure. path: Tuple. Optional argument, only used when recursing. The path from the root of the original shallow_tree, down to the root of the shallow_tree arg of this recursive call. Yields: Pairs of (path, value), where path the tuple path of a leaf node in shallow_tree, and value is the value of the corresponding node in input_tree. )rN)r7rur) rrrr shallow_keyshallow_subtreesubpath input_subtree leaf_path leaf_values rrrs$ l #  1*=>J %\ 2 &  ~%g -m#< =,W$& )Z*%%& &sA A"c#Kt|r>xH rc |rtnt}||r||stdt|zt |t j rt|j}n t|}|rt ||st|d}t|d}|r@|r>t||sttjt|t|t |trt |trnt|s t|rt|s t|rnft |tj rt |tj s2ttjt|t|t|s t|rt|s t|rt|s=t|s2ttjt|t|t|r|n |j"j%}t|r|n |j"j%} t'|dr2t'| dr&|j)| j)k(xsd} n|j+| g} | t-d|d| t|r"t|stdt|zt/|t/|k7r2t-t0jt/|t/|t/|t/|kr2t-t2jt/|t/| t |tj r@t5|t5|z } | r't-t6jt9| t;t=|t=|D]\} } t?| | || yy) NzVIf shallow structure is a sequence, input must also be a sequence. Input has type: %s.F) input_type shallow_type_get_structurez(Incompatible CompositeTensor TypeSpecs: z vs. zWIf shallow structure is a TypeSpec, input must also be a TypeSpec. Input has type: %s.) input_lengthshallow_length) input_size shallow_sizer) rr!rBr:rFrGrHrIr,same_namedtuples!STRUCTURES_HAVE_MISMATCHING_TYPESr$rDrJrNr|r}rL_without_tensor_nameshasattrrmost_specific_common_supertyper#rK#STRUCTURES_HAVE_MISMATCHING_LENGTHS$INPUT_TREE_SMALLER_THAN_SHALLOW_TREEsetSHALLOW_TREE_HAS_INVALID_KEYSrgr8ror)rrrrrrshallow_is_namedtupleinput_is_namedtuple type_spec_1 type_spec_2r4 absent_keysrrs rrrs"38J,  #      , 2 23,223l,'l:j,?,L%@)*e< #6 j9/66!*-D - 4 4 +$|:L 5   L)-A*-M  +}Z/H"<0M,4O - 4 4 +$|:L 5   < ( &&  &j1*z7L7L . /G '5  & & (K,F,F,H H PD ;;[MJ K )  | $ : & 2:   ZC - - / 6 6 _S=N 7   z?S. . 0 7 7z?\9J 8   , 0 8 89 %J7k  ) 0 0 1D E  ),\*Z() $ (  !-  a rct|rct|s"tdt|jd|rMt |t|s8tdt|jdt|jdt |t |k7r$t dt |dt |d|rt |tjrmt|t|k7r$t dt|d t|dt|j}t|j}t||D]\}}t||| yy) NzTIf shallow structure is a sequence, input must also be a sequence. Input has type: 'rzPThe two structures don't have the same sequence type. Input structure has type 'z%', while shallow structure has type 'zSThe two structures don't have the same sequence length. Input structure has length z%, while shallow structure has length rzFThe two structures don't have the same keys. Input structure has keys z#, while shallow structure has keys r)r"rBr:r rFrKr#r|r}rrDrgitemsr8r)rrrrrs rrrs % j )  ":.778 <  :j$|2DE  !!%j!1!:!: ;>xH rc|rtnt}t||||t|||Dcgc]\}}| c}}Scc}}w)Nr)rr!rr)rrrrrrvrws rr r  sZ"38J$) ,  L  !Q  s >cDt||tt||Sr1)rrDr)rrs rr r !s#L*= ' jA BBrc|tjk(rt||g|i|S|tjk(r t ||g|St dj |)a Applies a function or op to a number of partially flattened inputs. The `inputs` are flattened up to `shallow_tree` before being mapped. Use Case: Sometimes we wish to apply a function to a partially flattened structure (for example when the function itself takes structure inputs). We achieve this by specifying a shallow structure, `shallow_tree` we wish to flatten up to. The `inputs`, can be thought of as having the same structure layout as `shallow_tree`, but with leaf nodes that are themselves tree structures. This function therefore will return something with the same base structure as `shallow_tree`. Examples: ```python shallow_tree = [None, None] inp_val = [1, 2, 3] out = map_structure_up_to(shallow_tree, lambda x: 2 * x, inp_val) # Output is: [2, 4] ``` ```python ab_tuple = collections.namedtuple("ab_tuple", "a, b") op_tuple = collections.namedtuple("op_tuple", "add, mul") inp_val = ab_tuple(a=2, b=3) inp_ops = ab_tuple(a=op_tuple(add=1, mul=2), b=op_tuple(add=2, mul=3)) out = map_structure_up_to(inp_val, lambda val, ops: (val + ops.add) * ops.mul, inp_val, inp_ops) # Output is: ab_tuple(a=6, b=15) ``` ```python data_list = [[2, 4, 6, 8], [[1, 3, 5, 7, 9], [3, 5, 7]]] name_list = ['evens', ['odds', 'primes']] out = map_structure_up_to( name_list, lambda name, sec: "first_{}_{}".format(len(sec), name), name_list, data_list) # Output is: ['first_4_evens', ['first_5_odds', 'first_3_primes']] ``` Args: modality: enum value of supported modality [Modality.CORE or Modality.DATA] shallow_tree: a shallow structure, common to all the inputs. func: callable which will be applied to each input individually. *inputs: structures that are compatible with shallow_tree. The function `func` is applied to corresponding structures due to partial flattening of each input, so the function must support arity of `len(inputs)`. **kwargs: Arg valid for Modality.CORE only. kwargs to feed to func(). Special kwarg `check_types` is not passed to func, but instead determines whether the types of iterables within the structures have to be same (e.g. `map_structure(func, [1], (1,))` raises a `TypeError` exception). To allow this set this argument to `False`. Raises: TypeError: If `shallow_tree` is a nested structure but `input_tree` is not. TypeError: If the structure types of `shallow_tree` are different from `input_tree`. ValueError: If the structure lengths of `shallow_tree` are different from `input_tree`. Returns: result of repeatedly applying `func`, with the same structure layout as `shallow_tree`. r )r r -_tf_core_map_structure_with_tuple_paths_up_tor _tf_data_map_structure_up_tor#r$)r%rrinputsrs rmap_structure_up_tor&snT 8d # '- 8== ' d DV DD 7>>xH rc^ |s td|jdd |jdd rtnt}|D]}t |  fd|D}dt |d |D}t |g|Dcgc] }||i| } }t| Scc}w) zZSee comments for map_structure_with_tuple_paths_up_to() in tensorflow/python/util/nest.py.zCannot map over no sequencesrTrFrc3<K|]}t|yw)rN)r )r2rrrrs rr5z@_tf_core_map_structure_with_tuple_paths_up_to..s1     - sc3&K|] \}}| ywr1r)r2rrvs rr5z@_tf_core_map_structure_with_tuple_paths_up_to..s $ sr)r&rr)r#rrr!rrr8r) rrrrrrflat_value_gen flat_path_genrTresultsrrs ` @@rrr|s  3 44 =$/+jj!4e<!28Jj%+ .. q <-),M(KN(K  $dDF '  #)  s B*c|s td|D]}t|fd|D}t|Dcgc]}|| }}t|Scc}w)Nz9Argument `inputs` is empty. Cannot map over no sequences.c36K|]}t|ywr1)r )r2rrs rr5z/_tf_data_map_structure_up_to..s;E\:6r)r&r)r#rr8r)rrrrall_flattened_up_totensorsrs` rrrsz  C @j%lJ?@ IO,/0C+D ET7^ E' E "G  Fs A)F)TF)Tr1)r)Tr collectionsr;enumwraptrGtensorflow.pythonrtensorflow.python.platformrtensorflow.python.utilrtensorflow.python.util.compatrr|+tensorflow.python.util.custom_nest_protocolr IsMappingViewrCIsAttrsrEIsCompositeTensorrJ IsTypeSpecrNIsMutableMappingr6 IsMappingr>IsNestedForDatar"FlattenForDatarIsNestedr!IsNestedOrCompositerSameNamedtuplesrrrrrEnumr objectrrr'r,rPrdr9rmrsroryrurprrrrrrrrrrrrrrrrrrrr r r rrrrrrr4s  # /10MJ!..  ! ! $66(( #44%% "22 //"++';; 00J" $%N *tyy*Z|2 6K \L$D( 2j!JAFhZ7<(F 7;$N>gT=HLFT>B5(p**ZhX HN4D&D   9|CHH+/%X `HCH*C Sl-br