2Vh"nddlZddlZddlZddlZddlmZddlZddlm Z ddl m Z ddl m Z ddlmZe d ddZ dd Zd Zd Zd Zd ZdZdZdZdZ ddZdZdZdZdZdZ y)N) ThreadPool)tree) keras_export)io_utils) tensorflowzkeras.utils.split_datasetct|}|td|dt|| | tdt ||}|rK|t j dtd}t j|t j|t|}t|||\}}t|d|}t|| d} t|||}t| ||} tjj j#|}tjj j#| } |tjj ur:t%|r/t'|} | "|j)| }| j)| } |j+tjj,}| j+tjj,} || fS)a0Splits a dataset into a left half and a right half (e.g. train / test). Args: dataset: A `tf.data.Dataset`, a `torch.utils.data.Dataset` object, or a list/tuple of arrays with the same length. left_size: If float (in the range `[0, 1]`), it signifies the fraction of the data to pack in the left dataset. If integer, it signifies the number of samples to pack in the left dataset. If `None`, defaults to the complement to `right_size`. Defaults to `None`. right_size: If float (in the range `[0, 1]`), it signifies the fraction of the data to pack in the right dataset. If integer, it signifies the number of samples to pack in the right dataset. If `None`, defaults to the complement to `left_size`. Defaults to `None`. shuffle: Boolean, whether to shuffle the data before splitting it. seed: A random seed for shuffling. Returns: A tuple of two `tf.data.Dataset` objects: the left and right splits. Example: >>> data = np.random.random(size=(1000, 4)) >>> left_ds, right_ds = keras.utils.split_dataset(data, left_size=0.8) >>> int(left_ds.cardinality()) 800 >>> int(right_ds.cardinality()) 200 NzThe `dataset` argument must be eithera `tf.data.Dataset`, a `torch.utils.data.Dataset`object, or a list/tuple of arrays. Received: dataset=z of type zoAt least one of the `left_size` or `right_size` must be specified. Received: left_size=None and right_size=Noner.A)_get_type_spec TypeErrortype ValueError_convert_dataset_to_listrandomrandintintseedshufflelen_rescale_dataset_split_sizeslist_restore_dataset_from_listtfdataDatasetfrom_tensor_slices is_batchedget_batch_sizebatchprefetchAUTOTUNE) dataset left_size right_sizerrdataset_type_specdataset_as_list total_length left_split right_split batch_sizes M/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/utils/dataset_utils.py split_datasetr+sJ'w/  !") 4=/ C  i/   /w8IJO <>>!SX.D D''L8:|Izojy12J {|45K+%wJ-&K33J?J''//44[AKBGGOO+ 70C#G,  !#))*5J%++J7K$$RWW%5%56J&&rww'7'78K { ""ct||}g}tj}t||||D]}|j||S)azConvert `dataset` object to a list of samples. Args: dataset: A `tf.data.Dataset`, a `torch.utils.data.Dataset` object, or a list/tuple of arrays. dataset_type_spec: the type of the dataset. data_size_warning_flag: If set to `True`, a warning will be issued if the dataset takes longer than 10 seconds to iterate. Defaults to `True`. ensure_shape_similarity: If set to `True`, the shape of the first sample will be used to validate the shape of rest of the samples. Defaults to `True`. Returns: List: A list of samples. )_get_data_iterator_from_datasettime_get_next_sampleappend)r!r$data_size_warning_flagensure_shape_similaritydataset_iteratorr% start_timesamples r*rrks_,7"OJ" ' v& ' r,c P|turt|dk(r tdd}t|D]\}}t |t j stdt|d|d| |j}J|jd|dk7s`td|d|d t j|jd tt|S|turt|dk(r td d}t|D]\}}t |t j std t|d|d| |j}J|jd|dk7s`td |d|d t j|jdtt|S|tjjur&t!|r|j#}t|St%|r t|S|t j ur t|Std|)a Get the iterator from a dataset. Args: dataset: A `tf.data.Dataset`, a `torch.utils.data.Dataset` object, or a list/tuple of arrays. dataset_type_spec: The type of the dataset. Returns: iterator: An `iterator` object. rzJReceived an empty list dataset. Please provide a non-empty list of arrays.Nz5Expected a list of `numpy.ndarray` objects,Received: at index .zOReceived a list of NumPy arrays with different lengths.Mismatch found at index z, Expected shape=z Received shape=z:.Please provide a list of NumPy arrays of the same length.zJReceived an empty list dataset.Please provide a non-empty tuple of arrays.z6Expected a tuple of `numpy.ndarray` objects,Received: zPReceived a tuple of NumPy arrays with different lengths.Mismatch found at index z;.Please provide a tuple of NumPy arrays of the same length.zInvalid dataset_type_spec: )rrr enumerate isinstancenpndarrayr shapearrayiterziptuplerrrrunbatchis_torch_dataset)r!r$expected_shapeielements r*r.r.seD w<1 =  #G, JAwgrzz2 !!%gz!A?%!(q!^A%66 //0c2&&4%56&&(hhw&7&=&=%>?PP "CM"" e # w<1 >  #G, JAwgrzz2 !!%gz!A?%!(q!^A%66 //0c2&&4%56&&(hhw&7&=&=%>?QQ "CM"" bggoo - g oo'GG} ' "G} bjj (G} 23D2EF GGr,c #Kddlm} t|}t|}t |t j tjfs||r tj|j}nd}d}|t|D]\}}|rq|tj|jk7rOtdtj|jdtj|jd|d |rM|d zdk(rEtj} t| |z d kDr |rt!j"d t$d d}|y#t$r tdwxYww)aYield data samples from the `dataset_iterator`. Args: dataset_iterator: An `iterator` object. ensure_shape_similarity: If set to `True`, the shape of the first sample will be used to validate the shape of rest of the samples. Defaults to `True`. data_size_warning_flag: If set to `True`, a warning will be issued if the dataset takes longer than 10 seconds to iterate. Defaults to `True`. start_time (float): the start time of the dataset iteration. this is used only if `data_size_warning_flag` is set to true. Yields: data_sample: The next sample. r)is_torch_tensorNFzReceived an empty dataset. Argument `dataset` must be a non-empty list/tuple of `numpy.ndarray` objects or `tf.data.Dataset` objects.z StopIterationr r:r/rwarningswarnResourceWarning) r4r3r2r5rI first_samplefirst_sample_shaperFr6cur_times r*r0r0so,  01,- lRYY $; < A "$,!7!=!= !% &+ #/0 6 "!RXXf%5%;%;; '')xx '='C'C&DE'')xx'7'='=&>jc "2v{99;x*,-27MMM3 "1..3* 1   ,   sE,A0E:CE,E))E,ct|drQ|jjD]8}|jdk(st |j j ds8yy)N __class__rztorch.utils.dataTF)hasattrrX__mro____name__str __module__ startswith)r!parents r*rDrD$s]w $''// F)+!!1j+,1-   r,c t|}t|}|/|ttfvr!||ttfvrtd|d|||ttfvrtd|d||ttfvrtd|d|dk(r|dk(r t d|tur |dks||k\s|tur|dks|d k\rt d |d ||tur |dks||k\s|tur|dks|d k\rt d |d |||cxurturnn||zd kDr t d|turt ||z}n|tur t|}|turt ||z}n|tur t|}|||z }n|||z }||z|kDrt d|d||zd||df|dffD]"\}}|dk(s t d|d|d|d|d t|t|}}||fS)aRescale the dataset split sizes. We want to ensure that the sum of the split sizes is equal to the total length of the dataset. Args: left_size: The size of the left dataset split. right_size: The size of the right dataset split. total_length: The total length of the dataset. Returns: tuple: A tuple of rescaled `left_size` and `right_size` integers. zjInvalid `left_size` and `right_size` Types. Expected: integer or float or None, Received: type(left_size)=z and type(right_size)=zTInvalid `left_size` Type. Expected: int or float or None, Received: type(left_size)=z. zUInvalid `right_size` Type. Expected: int or float or None,Received: type(right_size)=r9rz]Both `left_size` and `right_size` are zero. At least one of the split sizes must be non-zero.z=`left_size` should be either a positive integer smaller than z<, or a float within the range `[0, 1]`. Received: left_size=zB`right_size` should be either a positive integer and smaller than z< or a float within the range `[0, 1]`. Received: right_size=z^The sum of `left_size` and `right_size` is greater than 1. It must be less than or equal to 1.zCThe sum of `left_size` and `right_size` should be smaller than the z%. Received: left_size + right_size = zand total_length = leftrightzWith `dataset` of length=z, `left_size`=z and `right_size`=z .Resulting zN side dataset split will be empty. Adjust any of the aforementioned parameters)r rfloatr r round)r"r#r&left_size_typeright_size_typesplitsides r*rr.s)_N:&O .e "D?3,#F C4_4E G  U|!C ))7(8 =  /#u"E **9):! =  A~*/ @   # !^yL8 U " !^yA~ (>*>k    3 1_ l : e # 1_ a ,~.?l    >2U2  "Q & :  )l23 3 )$ %: 45 C :&  :-  !I- : , ##/.122;j2H1I!, 1  #F+j'-BC t A:+L>+/ |<!F#>>  NC OzI j  r,cz|tttjjfvs t |r~|d}|Dcgc]}t j|}}t|Dcgc]}tj|}}t j||}t jd|dS|Scc}wcc}w)z,Restore the dataset from the list of arrays.rc<t|tr t|S|SN)r;rrBxs r*z,_restore_dataset_from_list..s*Q"5eAh1r,F)top_down) rBrrrrrDrflattenrAr<r?pack_sequence_astraverse)r%r$original_dataset element_specr6s r*rrsUD"''//::>N?'q) >MNF4<</NN:=:OP288F+PP// oN}} <   OPs B3$B8ct|dS)z*Check if the `tf.data.Dataset` is batched. _batch_size)rYr!s r*rrs 7M **r,c2t|r |jSy)z"Get the batch size of the dataset.N)rrwrxs r*rrs'"""r,cRt|trtSt|trtSt|tjrtjSt|t j jrt j jSt|rddl m}|Sy)z!Get the type spec of the dataset.r)rN) r;rBrr<r=rrrrDtorch.utils.data)r! TorchDatasets r*r r sl'5! GT " GRZZ (zz GRWW__ -ww ' "<r,c ~|dk(rg}ttjjj D]} tjjj tjjj | sU| jdrg| jdr| dd} |j| |Tt|jt|std|d||}ndg}|| td td |}tt|tt!|} t#} g} g} fd |DD]+}| j| j%t&|| ||f-g}| D]+}|j)\}}|j|| |z } -|dk(rJd }t+j,t!| fd }|D]!}||||t!|z|t!|z }#nm|d}nht!|t!| k7r'tdt!|dt!| ddtt|Dcgc] }t/|}}|rS|#t1j2dt!| dn.t1j2dt!| dt!|d| j5| j | Dcgc],}tjjj |.}}|r|t*j6j9d}t*j6j;|}|j=||0t*j6j;|}|j=||||fScc}wcc}w)aList all files in `directory`, with their labels. Args: directory: Directory where the data is located. If `labels` is `"inferred"`, it should contain subdirectories, each containing files for a class. Otherwise, the directory structure is ignored. labels: Either `"inferred"` (labels are generated from the directory structure), `None` (no labels), or a list/tuple of integer labels of the same size as the number of valid files found in the directory. Labels should be sorted according to the alphanumeric order of the image file paths (obtained via `os.walk(directory)` in Python). formats: Allowlist of file extensions to index (e.g. `".jpg"`, `".txt"`). class_names: Only valid if `labels="inferred"`. This is the explicit list of class names (must match names of subdirectories). Used to control the order of the classes (otherwise alphanumerical order is used). shuffle: Whether to shuffle the data. Defaults to `True`. If set to `False`, sorts the data in alphanumeric order. seed: Optional random seed for shuffling. follow_links: Whether to visits subdirectories pointed to by symlinks. verbose: Whether the function prints number of files found and classes. Defaults to `True`. Returns: tuple (file_paths, labels, class_names). - file_paths: list of file paths (strings). - labels: list of matching integer labels (same length as file_paths) - class_names: names of the classes corresponding to these labels, in order. inferredr9/NzjThe `class_names` passed did not match the names of the subdirectories of the target directory. Expected: z0 (or a subset of it), but received: class_names=zKWhen `labels=None` (no labels), argument `class_names` cannot be specified.zWhen argument `labels` is specified, argument `class_names` cannot be specified (the `class_names` will be the sorted list of labels).c3rK|].}tjjj|0ywrl)riogfilejoin).0subdir directorys r* z"index_directory..9s&NFBEEKK$$Y7Ns47rint32)dtypezfExpected the lengths of `labels` to match the number of files in the target directory. len(labels) is z while we found z files in directory zFound z files.z files belonging to z classes.r )sortedrrrlistdirisdirrr^endswithr1setissubsetr dictrArangerr apply_asyncindex_subdirectorygetr<zerosr\r print_msgcloserr RandomStater)rlabelsformats class_namesrr follow_linksverbosesubdirsr class_indicespoolresults filenamesdirpath labels_listrespartial_filenamespartial_labelsrFlabelfname file_pathsrngs` r*index_directoryrsZRUU[[00;< +Fuu{{  !1!1)V!DE((-s+!'NN6*  +  "{#,,S\: !!( *11< ? "G$  "~ + !: K[%K0@*ABCM >*&& '  3y>+7;) %N2@F1q3~.. / ^$ $A %   v;#i. (Dv;-/I/?@ ){!-  06c&k/BCes5zC C >   I'7w? @   Y()+&'y2  JJLIIKBKL"%%++""9e4LJL <99$$S)Dii##D) J  ))''-C KK  v{ **-DMs +N581N:c#,K|s*tjjj|}nt j||}t |dD]<\}}}t |D](}|j j|s#||f*>yw)N) followlinksc |dS)Nrrms r*roz"iter_valid_files..vs QqTr,)key)rrrwalkosrlowerr)rrrrroot_filesrs r*iter_valid_filesrqs~ uu{{ *wwyl; >:"aE] "E{{}%%g.Ek! ""s BB Bctjj|}t|||}g}g}|D]\}} |j ||t j jj|| } t j jj|tjj| |} |j | ||fS)adRecursively walks directory and list image paths and their class index. Args: directory: string, target directory. class_indices: dict mapping class names to their index. follow_links: boolean, whether to recursively follow subdirectories (if False, we only list top-level images in `directory`). formats: Allowlist of file extensions to index (e.g. ".jpg", ".txt"). Returns: tuple `(filenames, labels)`. `filenames` is a list of relative file paths, and `labels` is a list of integer labels corresponding to these files. ) rpathbasenamerr1rrrrrelpath) rrrrdirname valid_filesrrrr absolute_path relative_paths r*rr|sggy)G"9lGDK FI"( e mG,- ((u5  (( RWW__]I>  ' ( f r,c0|s||fSt|t|z}|dk(r7tjdt||z d|d| }||d| }||fS|dk(r+tjd|d|| d}||| d}||fSt d|)aPotentially restrict samples & labels to a training or validation split. Args: samples: List of elements. labels: List of corresponding labels. validation_split: Float, fraction of data to reserve for validation. subset: Subset of the data to return. Either `"training"`, `"validation"`, or `None`. If `None`, we return all of the data. Returns: tuple (samples, labels), potentially restricted to the specified subset. trainingzUsing z files for training.N validationz files for validation.z>`subset` must be either "training" or "validation", received: )rrrrr )samplesrvalidation_splitsubsetnum_val_sampless r* get_training_or_validation_splitrs *S\9:O S\O344H I +O+,  -o-.F F? < VO#44JKL?*+,  _,-.F F?  **0 3  r,c&tjjj|}|dk(r.|j dtjj }|S|dk(r.|j fdtjj }|S)aCreate a `tf.data.Dataset` from the list/tuple of labels. Args: labels: list/tuple of labels to be converted into a `tf.data.Dataset`. label_mode: String describing the encoding of `labels`. Options are: - `"binary"` indicates that the labels (there can be only 2) are encoded as `float32` scalars with values 0 or 1 (e.g. for `binary_crossentropy`). - `"categorical"` means that the labels are mapped into a categorical vector. (e.g. for `categorical_crossentropy` loss). num_classes: number of classes of labels. Returns: A `tf.data.Dataset` instance. binarycXtjtj|ddS)Nfloat32r)axis)r expand_dimscastrms r*roz#labels_to_dataset..sbnnRWWQ %:Dr,)num_parallel_calls categoricalc0tj|Srl)rone_hot)rn num_classess r*roz#labels_to_dataset..sbjjK0r,)rrrrmapr )r label_moderlabel_dss ` r*labels_to_datasetrs ww11&9HX<< D!ww//   O } $<< 0!ww//   Or,c|rd|cxkrdksntd||s|r|r|s td|dvrtd||r|r| tdyyy) aRaise errors in case of invalid argument values. Args: validation_split: float between 0 and 1, fraction of data to reserve for validation. subset: One of `"training"`, `"validation"`, or `"both"`. Only used if `validation_split` is set. shuffle: Whether to shuffle the data. Either `True` or `False`. seed: random seed for shuffling and transformations. rraz6`validation_split` must be between 0 and 1, received: zBIf `subset` is set, `validation_split` must be set, and inversely.)rrbothNzF`subset` must be either "training", "validation" or "both", received: NzIf using `validation_split` and shuffling the data, you must provide a `seed` argument, to make sure that there is no overlap between the training and validation subset.)r )rrrrs r*check_validation_split_argrs$4 8q 8 )* ,   F-=& P  == 117 :  G  B  )5Gr,)NNFN)TT)NTNFT)!rrr/rQmultiprocessing.poolrnumpyr< keras.srcrkeras.src.api_exportrkeras.src.utilsrkeras.src.utils.module_utilsrrr+rr.r0rDrrrrr rrrrrrrr,r*rs +-$9)*BFX#+X#|  $NHHVDNz!z2+ ,   K+\":#L< r,