BVhU-dZddlZddlZddlZddlmZmZmZddlm Z ddlm Z ddlm Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZeej0e j2fZeeefZej.d ej:dGddZej.dGddej>Z deedeededdfdZ!y)z6Data structures and utilities for checkpoint sharding.N)HashableMutableMappingSequence)device)dtypes)tensor) tensor_shape) tensor_spec) variables)base)saveable_object) tf_exportz"train.experimental.ShardableTensorT)frozenc$eZdZUdZej ed<ejed<e jed<e jed<e ed<ejed<ej"j$ed<e ed <ej(ed <d efd Zd e fd Zy)ShardableTensoraTensor wrapper containing data necessary for sharding. The tensor representation used as inputs to pre-made and custom `tf.train.experiemental.ShardingCallback`s, which can be specified using the `experimental_sharding_callback` option in `tf.train.CheckpointOptions`. _tensor_save_specrdtypernameshape slice_speccheckpoint_key trackablereturnct|j|jt|j|j fSN)hashrrstrrrselfs c/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/checkpoint/sharding/sharding_util.py__hash__zShardableTensor.__hash__<s- DJJDKK(8$:M:MN OOcd|jjd|jd|jd|jd|j d|j d|jd|jd |jd |jS) N z: _tensor_save_spec=z tensor=z dtype=z device=z name=z shape=z slice_spec=z checkpoint_key=z trackable=) __class____name__rrrrrrrrrrs r __repr__zShardableTensor.__repr__?s(()*##'#9#9"<= 'zzn% 'ii]#zzn% OO./ $ 3 367>>, . /r"N)r& __module__ __qualname____doc__r SaveSpec__annotations__ tensor_libTensorrDType device_lib DeviceSpecrr TensorShaper Variable SaveSliceInfor Trackableintr!r'r"r rr(s%---        )  ! !!  ... ^^PP / /r"rz#train.experimental.ShardingCallbackceZdZdZeej defdZej de e de e fdZ de fdZy)ShardingCallbackaCheckpoint sharding callback function, along with a text description. A callback function wrapper that will be executed to determine how tensors will be split into shards when the saver writes the checkpoint shards to disk. The callback takes a list of `tf.train.experimental.ShardableTensor`s as input (as well as any kwargs defined by the `tf.train.experimental.ShardingCallback` subclass), and organizes the input tensors into different shards. Tensors are first organized by device task (see `tf.DeviceSpec`), then the callback will be called for each collection of tensors. There are a few restrictions to keep in mind when creating a custom callback: - Tensors must not be removed from the checkpoint. - Tensors must not be reshaped. - Tensor dtypes must not change. - Tensors within a shard must belong to the same task. Validation checks will be performed after the callback function is executed to ensure these restrictions aren't violated. Here's an example of a simple custom callback: ``` # Place all tensors in a single shard. class AllInOnePolicy(tf.train.experimental.ShardingCallback): @property def description(self): return "Place all tensors in a single shard." def __call__(self, shardable_tensors): tensors = {} for shardable_tensor in shardable_tensors: tensor = shardable_tensor.tensor_save_spec.tensor checkpoint_key = shardable_tensor.checkpoint_key slice_spec = shardable_tensor.slice_spec tensors.set_default(checkpoint_key, {})[slice_spec] = tensor return [tensors] ckpt.save( "path", options=tf.train.CheckpointOptions( experimental_sharding_callback=AllInOnePolicy())) ``` The `description` attribute is used to identify the callback and to aid debugging during saving and restoration. To take in kwargs, simply define the constructor and pass them in: ``` class ParameterPolicy(tf.train.experimental.ShardingCallback): def __init__(self, custom_param): self.custom_param = custom_param ... ckpt.save( "path", options=tf.train.CheckpointOptions( experimental_sharding_callback=ParameterPolicy(custom_param=...))) ``` rcy)z2Returns a text description of the sharding policy.Nr7rs r descriptionzShardingCallback.descriptions r"shardable_tensorscy)z9Returns a list of shards for the given shardable tensors.Nr7)rr<s r __call__zShardingCallback.__call__s  r"c$t|j}t|jD]^\}}t j |rt j |r2|t|z}t|tsQ|t|z}`|Sr) rr;varsitemsinspectismethod isfunction isinstancer)rhash_val attr_nameattr_vals r r!zShardingCallback.__hash__s{D$$%H#Dz//1% 8x(G,>,>x,HDO# h ) d8n $( % Or"N)r&r(r)r*propertyabcabstractmethodrr;rrShardr>r6r!r7r"r r9r9Lsp=~  3    '8   r"r9shardsr<callback_descriptionrci}|D]5}|j|j|ji|j<7t }|D]{}d}|j D]b\}} | j D]H\} } | j } || f|vrtd|d|d| d||vrtd|d|d| d||| j} | j| k7r%td|d|d| d| d | jd ||| j} | j| k7r%td |d|d| d | d | jd tjj||| jj}tjj| jj}||k7rtd |d|d| d|d|d |tddd| jdd| |d }ntjj|jj}tjj| jj}|<|:||k7r5td|d|jd|jd|d|d| d|d||| =||s||=|j!|| fKe~|rBd}|j D]\}} |dz }|d|dz }|d| dz }td|d|y)z4Validates shards generated by the sharding_callback.NzAfter executing the checkpoint sharding callback, multiple tensors with the same checkpoint key and slice spec were found: callback_description: z checkpoint_key: z slice_spec: r$zAfter executing the checkpoint sharding callback, a tensor not originally in the object graph was found in the checkpoint shards: callback_description: ztAfter executing the checkpoint sharding callback, a tensor was found with an altered shape: callback_description: z original tensor_shape: z new tensor_shape: ztAfter executing the checkpoint sharding callback, a tensor was found with an altered dtype: callback_description: z original tensor_dtype: z new tensor_dtype: zsAfter executing the checkpoint sharding callback, a tensor was found with an altered task: callback_description: z original tensor_task: z new tensor_task: ) rrrrrrrrrzAfter executing the checkpoint sharding callback, tensors with different tasks were found in the same shard: callback_description: z! tensor #1: checkpoint_key: z slice_spec: z task: z! tensor #2: checkpoint_key: z tensor: z checkpoint_key: z slice_spec: zAfter executing the checkpoint sharding callback, tensors in the object graph were not found in the checkpoint shards: callback_description: )r setdefaultrrsetrAstrip RuntimeErrorrrr0r1 from_stringrtaskradd)rMr<rNunseen_tensor_dictshardable_tensorseen_tensor_set shard_tensors task_tensorrtensor_slice_dictr shard_tensor target_shape target_dtype target_taskshard_tensor_tasktask1task2 tensors_infockpt_keys r validate_shardsrgs +A*:)@)@!!''  % %'AE/a:mK-:-@-@-B_:))&7&=&=&?^: "*l%%'  J '? :)*>(>?##1"23)l" ./ / !3 3)*>(>?##1"23)l" ./ /*.9*EKK    -))=(>?##1"23)l+**68%%1%7%7$8 <= =*.9*EKK    -))=(>?##1"23)l+**68%%1%7%7$8 <= =!++77 ~ .z : A ACCG4 &11==   !!%   +))=(>?##1"23)l+))4 6$$5#6b :; ;  ' $!((#+ +''33K4F4FGLL%''33L4G4GHMM%  5#4%++?*@A''2'A'A&BC##.#9#9":;"G$''5&67##-,/"G2 ' ( ( ~ .z :!.1 0^Z89}^:_:a:HL 2 8 8 :8*m#l,XJb99l( B77l8  ##7"8 .   r")"r*rJ dataclassesrBtypingrrrtensorflow.python.frameworkrr0rrr-r r tensorflow.python.opsr tensorflow.python.trackabler !tensorflow.python.training.savingr tensorflow.python.utilr TensorSpecr. TensorSlicesrrL dataclassrABCr9rgr7r"r rss= 55<.<43+,=,k44j6G6GGH  sL()9:d#//$;/D:;UswwU<Up{ UO{0{{ {r"