BVhd% :dZddlZddlZddlZddlmZmZmZmZddl m Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZdZ de!dedee!deeejDgejDfde#dejDf dZ$ d'de!dedee!deeejDgejDfdejDf dZ%de!deejLfdZ'de!dejLdeejDgejDfdejDfdZ(de!defdZ)dee*e!fdefdZ+Gd d!ejXZ-Gd"d#ejXZ.Gd$d%ejXZ/de!dejLdede!ddf d&Z0y)(z(Implementation of LoadDataset in Python.N)AnyCallableOptionalUnion)logging)message) text_format) snapshot_pb2)_pywrap_snapshot_utils) dataset_ops)structured_function)dtypes)errors) tensor_spec)gen_experimental_dataset_ops)gfile)nested_structure_coderpath element_spec compression reader_funcwaitreturnc|rt||||S|d}t|}|rt||||t|||S| t |}t ||||S)z$Loads dataset from tf.data snapshot.cl|jdtjtjS)Nc|SN)xs R/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/data/ops/load_op.pyz)_load....8s!) cycle_lengthnum_parallel_calls) interleavemultiprocessing cpu_countr AUTOTUNE)datasetss r!r"z_load..7s.8#6#6$..0&//$7$1r#)_load_with_retry#_load_distributed_snapshot_metadata_validate_snapshot_load_distributed_snapshot_load_element_spec _LoadDataset)rrrrrdistributed_snapshot_metadatas r!_loadr2*s D, [ II1K #Fd"K" +\;H % +[ ::%d+L dL+{ CCr#c tjj||||d}tjd||S#t j tf$r2tjd|tjtYnwxYw)z1Tries loading the snapshot. Retries if not found.F)rrrrrzLoad tf.data snapshot at %s.z;Could not find tf.data snapshot at %s. Will wait and retry.) r Datasetloadrinfor NotFoundErrorFileNotFoundErrortimesleep_RETRY_INTERVAL_SEC)rrrrdatasets r!r+r+Hs  &##((#!! )g ll148 n  "3 4& ll GO jj$%& s;?AB  B c|tj|}tj|sy tj|d5}t j |tjcdddS#1swYyxYw#tjt jtjtf$rYywxYw)zReads the distributed snapshot metadata. Args: path: Base path of the snapshot. Returns: DistributedSnapshotMetadata if the snapshot is a distributed snapshot. Returns None if it is a non-distributed snapshot. Nr)r TF_DATA_SnapshotMetadataFilePathrExistsGFiler ParseLinesr DistributedSnapshotMetadatarr7 ParseErrorr DecodeErrorUnicodeDecodeError)r metadata_filefs r!r,r,as)II$O- m $   ]C (9A  # # \ 5 5 79999      s/B(A5+ B5A>:B>B7B;:B;metadatacRt|}|jfd}||S)zLoads a distributed snapshot.cZt|tjjS)N)rr)_SnapshotChunkDataset_parse_element_specrr) chunk_filerIs r!r"z,_load_distributed_snapshot..s'. *8+@+@A**,r#)_ListSnapshotChunksDatasetmap)rrIrr<s ` r!r.r.}s/ 't ,' KK, -' W r#ctjj|tj}t j |stjddd|dt j|d5}|j}ddd tS#1swYxYw#tj$r"}tjddd|d|d}~wwxYw)zLoads the dataset element spec. Args: path: Base path of the snapshot. Returns: Dataset element_spec. Raises: NotFoundError if the element spec file does not exist or cannot be decoded. Nz.tf.data snapshot element_spec file not found: .)node_defoprrbz9tf.data snapshot element_spec file not found or invalid: )osrjoinr DATASET_SPEC_FILENAMErr@rr7rAreadrMrNotEncodableError)rdataset_spec_filenamerH encoded_speces r!r/r/s'',,t[-N-NO + ,   $@(), --  {{($/1668L4 | ,, 1 14   $K(), -3444s$6B B'B$'C:CCencoded_element_specctjj}|j|tj|Sr)r struct_pb2StructuredValueParseFromString decode_proto)r^ struct_pbs r!rMrMs6$//??A) 01  , ,Y 77r#c ~eZdZdZdedededeejgejfffd Z e defdZ xZ S) r0z.A dataset that loads previously saved dataset.rrrrc ||_||_||_tj|dt j t j |j|_tj|f|jjj||jjd|j}t|9|y)Nzload())input_structure)reader_func_other_argsrr)_path _element_spec _compressionr StructuredFunctionWrapperr DatasetSpec _reader_funcged_ops load_datasetfunctioncaptured_inputs_flat_structuresuper__init__)selfrrrrvariant_tensor __class__s r!ruz_LoadDataset.__init__s DJ%D#D+EE#//  # #D$6$6 79 :D))  #0099II%%..      N  G^$r#rc|jSrrjrvs r!rz_LoadDataset.element_spec   r#) __name__ __module__ __qualname____doc__strrrr r4rupropertyr __classcell__rxs@r!r0r0se6% %% % [001;3F3FFG %0 C r#r0cFeZdZdZdededeffd ZedefdZxZ S)rLzAA dataset for one chunk file from a tf.data distributed snapshot.rNrrc||_||_tj|fd|i|j}t ||y)Nr) _chunk_filerjrosnapshot_chunk_datasetrsrtru)rvrNrrrwrxs r!ruz_SnapshotChunkDataset.__init__sM!D%D33      N G^$r#rc|jSrrzr{s r!rz"_SnapshotChunkDataset.element_specr|r#) r}r~rrrrrurrrrs@r!rLrLs;I%%C%c% C r#rLcReZdZdZdeffd ZedejfdZ xZ S)rOzA dataset for listing snapshot chunk files. It supports listing partially written snapshots. When a snapshot is being written, it returns the currently available chunk files. snapshot_pathcr||_tj|fi|j}t||yr)_snapshot_pathrolist_snapshot_chunks_datasetrsrtru)rvrrwrxs r!ruz#_ListSnapshotChunksDataset.__init__s9'D99/--/N G^$r#rcJtjgtjSr)r TensorSpecrstringr{s r!rz'_ListSnapshotChunksDataset.element_specs  ! !"fmm 44r#) r}r~rrrrurrrrrrs@r!rOrOs5 %C%  5K225 5r#rOc tj|}tj|r6tj|d5}t d|d|j t|j}|r||k7rt d|d|d|d|r/||jk7rt d|d|d |jdyy#1swYlxYw) a8Validates a tf.data distributed snapshot. Args: path: Root path of the distributed snapshot. metadata: The DistributedSnapshotMetadata of the snapshot. element_spec: Dataset element_spec. compression: Compression method used for saving. Raises: ValueError if the snapshot is invalid. r>z#Failed to load tf.data snapshot at z+. The save job failed to write it. Status: Nz. User specified element_spec z!, but the actual element_spec is rRz. User specified compression z , but the actual compression is ) r TF_DATA_SnapshotErrorFilePathrr@rA ValueErrorrYrMrr)rrIrr error_filerHsnapshot_element_specs r!r-r-s"&CCDI* \\* Z %+  /v6 vvxj * ++.h.C.CDl&;;  -dV4$~%F ! $ %% [H$8$88  -dV4"m#C     # $$9[++s  CC )NNN)1rr'rVr9typingrrrrabslrgoogle.protobufrr tensorflow.core.protobufr +tensorflow.python.data.experimental.servicer tensorflow.python.data.opsr r tensorflow.python.frameworkrrrtensorflow.python.opsrrotensorflow.python.platformrtensorflow.python.saved_modelrr;rr4boolr2r+rCr,r.r/bytesrM DatasetSourcer0rLrOr-rr#r!rs+/ 11#'1N2:..3I,@D DD#D(K$7$7#8+:M:M#MNO D  D  D@!%@D & &&#&+%%& (;(;;<> &  &2  l6678   66 ;../1D1DDE   4S4S4<8eE3J.?8C8 ;,,@K55"5!:!:5$#$ #$66#$#$ #$ #$r#