AVh#dZddlZddlmZmZmZddlmZddlmZddlm Z ddlm Z ddlm Z dd l mZdd lmZdd lmZdd lmZdd lmZddlmZddlmZedgde j4deeej8fdeeeej8fdeeeej8fdeeej8ej:ff dZedgdeej:fdZedgde j4dedeeeej8ej:fffdZ edgde j4deeej8fdeeeej8ej:fffd Z!y)!z=Contains functionaility for Checkpoint/SavedModel in DTensor.N)DictListUnion)api) d_variable)gen_dtensor_ops)layout) mesh_util)context) errors_impl)ops)tensor)io_ops) variables) tf_exportz!experimental.dtensor.sharded_save)v1mesh file_prefix tensor_namesshape_and_slicestensorsctjtj5t j ||||dddt j|jdtj|j5t j|g|d}dddt j|jdS#1swYxYw#1swY;xYw)aSaves given named tensor slices in a sharded, multi-client safe fashion. The method makes sure the checkpoint directory state is correct in a sharded mutli-client saving. Namely, we place a barrier after SaveV2 to make sure every client has done writing the files. And another one after MergeV2Checkpoints to make sure all Metadata is properly merged. Upon existing, the checkpoint is completed and the all directory operations are done. Args: mesh: The Mesh that contains the Tensors to save. file_prefix: The prefix of checkpoint. tensor_names: a list of tensor names used in save op. shape_and_slices: a list of shape and slice specification used in save op. The only supported value is "" as we don't support distributed saving with slices yet. tensors: a list of tensors used in save op. The order should match tensor_names. Returns: A MergeV2Checkpoints op that merged all Metadata. NSaveV2T)checkpoint_prefixesdestination_prefixdelete_old_dirsMergeV2Checkpoints) r devicer device_namersave_v2r barrier host_mesh default_meshr)rrrrrmerge_ops V/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/dtensor/python/save_restore.py sharded_saver&"s> zz#//#$I NN; .>HI DNN$h/ ()(((M&H DNN$&:; /II sCC$C!$C-z(experimental.dtensor.enable_save_as_bf16rcV|D]$}t|tjsd|_&y)aAllows float32 DVariables to be checkpointed and restored as bfloat16. The method only affects the DVariable part inside the model and leaves non-DTensor Variables/Tensors untouched. Args: variables: A list of tf.Variable to be enabled with bfloat16 save/restore. Only has effect on DTensor Variables as they go through d_variables with DTensor Specific logis. TN) isinstancer DVariable save_as_bf16)rvs r%enable_save_as_bf16r,Ss* a!Z))*anz'experimental.dtensor.name_based_restorecheckpoint_prefixname_tensor_dictc Vtjs td|}t|tj st j |}|j D]_\}} tj|jjjdk7rtdj|atj|g|j!zt"j$j'|j)d}tjt+|j-g|j!zt"j$j'|j)d }tjd gt/|zg|j!zt"j$j'|j)d }|j1Dcgc]}|j2} }|j1Dcgc]%}tj|j5'} }t7j8tj:5t=j>|||| | |j1Dcgc]}|j@c} } dddt j tC|j- S#tj$r}td|d}~wwxYwcc}wcc}wcc}w#1swYlxYw) amRestores from checkpoint_prefix to name based DTensors. It is required to have already-initialized DTensor variables that have same shape/dtype for the tensors being restored. Also, we currently only support a named based restore on a single mesh. Args: mesh: The single mesh that all Tensors would be restored to. checkpoint_prefix : The prefix of checkpoint to be restored. name_tensor_dict: A ordered dictionary of tensor_names to a DTensor. The DTensor shape/dtype must match the tensors being saved/restored for now. Returns: A dictionary of name to its restored DTensor value. z$name based restore must run eagerly.CPUz\Restoring a non CPU Tensor is not supported currently. Offending tensor name : {tensor_name}) tensor_namez)Saving/Restoring tensor must be a DTensorNrrank)prefixrr input_shapes input_layoutsdtypes)"r executing_eagerly ValueErrorr( collections OrderedDictitemsr fetch_layoutr device_typeupperformatr OpErrorpacknum_local_devices layout_libLayout replicatedr"listkeyslenvaluesshape to_stringr rrrd_tensor_restore_v2dtypezip) rr.r/ordered_name_tensor_dictnamerop_errorrrr8r9restored_cpu_tensorss r%name_based_restorerWds.  " " $ ; <<- $k&=&= >*667GH/446EldFE  & ! & & 2 2 4 : : < E **0&T&*BD D FEhhD2244""4>>#3!"<>  $ ) ) +,-0F0F0HH""4>>#3!"<>,XX tc*++,t/E/E/GG""4>>#3!"<>.F-L-L-NO6&,,O,O-335  v((*-  zz#//#$*>> !)!#+C+J+J+LM M    " ' ' )+?@ C   E  5 7rrErFrGrHrIr"rJrKr&rLrM)rr.r/rSrs r%name_based_saverYs(  " " $ 8 99- $k&=&= >*667GHhh 12T5K5K5MM)00;;#'>>#3! < => $ ) ) +,-0F0F0HH""4>>#3!"<>, #tc":;;+2245 7r-)"__doc__r=typingrrrtensorflow.dtensor.pythonrrrr rGr tensorflow.python.eagerr tensorflow.python.frameworkr r r tensor_libtensorflow.python.opsrr tf_variables tensorflow.python.util.tf_exportrMeshstrTensorVariabler&r,rWrYr-r%rhsD$$)05:/+3+<(;6 .26- //-sJ---.-S :#4#445-DIz'8'889 - % ))<+@+@@A B -7-` 5"= 4 (=(=#> >   4<H //HH U:$$l&;&;; <<>H=HV 1b9-7 //-7S*"3"334-7 U:$$l&;&;; <<>-7:-7r-