BVhQjjdZddlmZddlmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlZddlZddlZddlZddlZddlZddlZddlmZddlmZddlmZddlmZdd lm Z e jBdd k(rd(d Z"ndd l#m"Z"d Z$d)dZ% d*dZ&dZ'd+dZ(d,dZ)d-dZ*Gdde+Z,dZ-Gdde+Z.dZ/ia0da1ejdZ3da4e5Z6da7ejpZ9dZ:dZ;dZGd d!e+Z?Gd"d#e?Z@d(d$ZAd%ZBGd&d'e?ZCy).z(Utilities for file download and caching.)abstractmethod)closingN)tensor)urlopen) tf_inspect)Progbar)path_to_stringcdd}t||}t|d5}|||D]}|j| dddy#1swYyxYw)aReplacement for `urlretrieve` for Python 2. Under Python 2, `urlretrieve` relies on `FancyURLopener` from legacy `urllib` module, known to have issues with proxy management. Args: url: url to retrieve. filename: where to store the retrieved data locally. reporthook: a hook function that will be called once on establishment of the network connection and once after each block read thereafter. The hook will be passed three arguments; a count of blocks transferred so far, a block size in bytes, and the total size of the file. data: `data` argument passed to `urlopen`. Nc3K|jjd}d}|t|j}d} |j |}|dz }| |||||r|ny+w)NzContent-Lengthr)infogetintstripread)response chunk_size reporthook content_type total_sizecountchunks X/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/keras/utils/data_utils.py chunk_readzurlretrieve..chunk_read@s|]]_(()9:lj  !++-. e  j)    ! UJ 3 +  sA+A-wb)r)i N)ropenwrite)urlfilenamerdatarrfdrs r urlretriever$0sW  sD!H h h:>% s !AA)r$c$tttttt f}t |tjtjf|zrytj|xs,t |txst |tjS)z'Check if `x` is a Keras generator type.F)strlisttupledictset frozenset isinstancerTensornpndarrayr isgeneratorSequencetypingIterator)xbuiltin_iteratorss ris_generator_or_sequencer6XsjD%sI>FMM2::.1BBC    # ) Q ! ) Q (*c|y|dk(rddg}t|tr|g}t|}t|}|D]z}|dk(r tj}tj }|dk(r t j}t j}|sV|5} |j| dddyy#tjttf$rktjj|rJtjj!|rtj"|t%j&|wxYw#1swYyxYw)a6Extracts an archive if it matches tar, tar.gz, tar.bz, or zip formats. Args: file_path: path to the archive file path: path to extract the archive file archive_format: Archive format to try for extracting the file. Options are 'auto', 'tar', 'zip', and None. 'tar' includes tar, tar.gz, and tar.bz files. The default 'auto' is ['tar', 'zip']. None or an empty list will return no matches found. Returns: True if a match was found and an archive extraction was completed, False otherwise. NFautotarzipT)r,r&r tarfiler is_tarfilezipfileZipFile is_zipfile extractallTarError RuntimeErrorKeyboardInterruptospathexistsisfileremoveshutilrmtree) file_pathrFarchive_format archive_typeopen_fn is_match_fnarchives r_extract_archiverRbs/  vU^N$$%NY')  $$lu g&&kug&&k9 9      T " '(   ,0AB  WW^^D !ww~~d#iio mmD!    sEB66B D??EE Fc | =tjjtjjdd} |||}d}tjj| } tj| tj s tjjdd} tjj| |} t | t|}|r&tjj| |} | dz} n tjj| |} d}tjj| r*|*t| ||std |zd z|zd zd }nd }|r9td |Gddtfd}d} t|| | d_|r/tjj st5| | d| S|r t5| | || S#tjj$r5}t!|j#||j$|j&d}~wtjj($r5}t!|j#||j*|j,d}~wwxYw#t t.f$r:}tjj| rtj0| d}~wwxYw)aDownloads a file from a URL if it not already in the cache. By default the file at the url `origin` is downloaded to the cache_dir `~/.keras`, placed in the cache_subdir `datasets`, and given the filename `fname`. The final location of a file `example.txt` would therefore be `~/.keras/datasets/example.txt`. Files in tar, tar.gz, tar.bz, and zip formats can also be extracted. Passing a hash will verify the file after download. The command line programs `shasum` and `sha256sum` can compute the hash. Example: ```python path_to_downloaded_file = tf.keras.utils.get_file( "flower_photos", "https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz", untar=True) ``` Args: fname: Name of the file. If an absolute path `/path/to/file.txt` is specified the file will be saved at that location. origin: Original URL of the file. untar: Deprecated in favor of `extract` argument. boolean, whether the file should be decompressed md5_hash: Deprecated in favor of `file_hash` argument. md5 hash of the file for verification file_hash: The expected hash string of the file after download. The sha256 and md5 hash algorithms are both supported. cache_subdir: Subdirectory under the Keras cache dir where the file is saved. If an absolute path `/path/to/folder` is specified the file will be saved at that location. hash_algorithm: Select the hash algorithm to verify the file. options are `'md5'`, `'sha256'`, and `'auto'`. The default 'auto' detects the hash algorithm in use. extract: True tries extracting the file as an Archive, like tar or zip. archive_format: Archive format to try for extracting the file. Options are `'auto'`, `'tar'`, `'zip'`, and `None`. `'tar'` includes tar, tar.gz, and tar.bz files. The default `'auto'` corresponds to `['tar', 'zip']`. None or an empty list will return no matches found. cache_dir: Location to store cached files, when None it defaults to the default directory `~/.keras/`. Returns: Path to the downloaded file N~z.kerasmd5z/tmpz.tar.gzF) algorithmzNA local file was found, but it seems to be incomplete or outdated because the z0 file hash does not match the original value of z! so we will re-download the data.TzDownloading data fromceZdZdZy)!get_file..ProgressTrackerN)__name__ __module__ __qualname__progbarr7rProgressTrackerrXs gr7r^cj|dk(rd}t|_yjj||zy)Nr )r\rupdate)r block_sizerr^s r dl_progresszget_file..dl_progresssB  (  *")*"5&&uz'9:r7z!URL fetch failure on {}: {} -- {}r:)rM)rErFjoin expanduseraccessW_OK_makedirs_exist_okr rG validate_fileprintobjectr$urlliberror HTTPError ExceptionformatcodemsgURLErrorerrnoreasonrDrIr\rR)fnameoriginuntarmd5_hash file_hash cache_subdirhash_algorithmextractrM cache_dir datadir_basedatadir untar_fpathfpathdownloadrb error_msger^s @rget_filersht RWW//4h?I i/IN##I., < )77<<1L GGLL| 4'W  % '',,w.K ) #E GGLL% (E (WW^^E 5)~ F 46DE@ACLM22 3H  !6*& ;4I  EFE;/#O 77>>+ &uge<  UG^4 ,%\\ # #A ((?@@ \\ " "E ((!((CDDE ( )   %  s< G44J0I J!0JJJK"(5KK"c2tj|dy)NT)exist_ok)rEmakedirs)rs rrgrgs++g%r7c|dk(rtjS|dk(r$|"t|dk(rtjStjS)z+Returns hash algorithm as hashlib function.sha256r9@)hashlibrlenrU)rVrys r_resolve_hasherrsJ( >> &Y2s9~7K >>  r7ct|tr t|}n|}t|d5t fddD]}|j | ddd|j S#1swY|j SxYw)aCalculates a file sha256 or md5 hash. Example: ```python _hash_file('/path/to/file.zip') 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' ``` Args: fpath: path to the file being validated algorithm: hash algorithm, one of `'auto'`, `'sha256'`, or `'md5'`. The default `'auto'` detects the hash algorithm in use. chunk_size: Bytes to read at a time, important for large files. Returns: The file hash rbc&jSN)r)r fpath_filesrz_hash_file..?sjooj9r7r7N)r,r&rriterr` hexdigest)rrVrhasherrrs ` @r _hash_filer&s& 3 Y 'F F E4J93? mmE         s 'A--Bcbt||}tt|||t|k(ryy)aValidates a file against a sha256 or md5 hash. Args: fpath: path to the file being validated file_hash: The expected hash string of the file. The sha256 and md5 hash algorithms are both supported. algorithm: Hash algorithm, one of 'auto', 'sha256', or 'md5'. The default 'auto' detects the hash algorithm in use. chunk_size: Bytes to read at a time, important for large files. Returns: Whether the file is valid TF)rr&r)rryrVrrs rrhrhEs0 9i 0&E6: ./3y>A  r7c(eZdZdZdZdZdZdZy)ThreadsafeIterzEWrap an iterator with a lock and propagate exceptions to all threads.cR||_tj|_d|_yr)it threadingLocklock _exception)selfrs r__init__zThreadsafeIter.__init__^s DG DIDOr7c|Srr]rs r__iter__zThreadsafeIter.__iter__ks Kr7c"|jSr)__next__rs rnextzThreadsafeIter.nextns ==?r7c|j5|jr |j t|jcdddS#t$r }||_d}~wwxYw#1swYyxYwr)rrrrrn)rrs rrzThreadsafeIter.__next__qs`  ooDGG}    s(AA AAAAA'N)rYrZr[__doc__rrrrr]r7rrr[sM  r7rcBtjfd}|S)Nc&t|i|Sr)r)akwfs rgzthreadsafe_generator..gs !Q+"+ &&r7 functoolswraps)rrs` rthreadsafe_generatorr}s% ??1'' (r7c<eZdZdZedZedZdZdZy)r1a1Base object for fitting to a sequence of data, such as a dataset. Every `Sequence` must implement the `__getitem__` and the `__len__` methods. If you want to modify your dataset between epochs you may implement `on_epoch_end`. The method `__getitem__` should return a complete batch. Notes: `Sequence` are a safer way to do multiprocessing. This structure guarantees that the network will only train once on each sample per epoch which is not the case with generators. Examples: ```python from skimage.io import imread from skimage.transform import resize import numpy as np import math # Here, `x_set` is list of path to the images # and `y_set` are the associated classes. class CIFAR10Sequence(Sequence): def __init__(self, x_set, y_set, batch_size): self.x, self.y = x_set, y_set self.batch_size = batch_size def __len__(self): return math.ceil(len(self.x) / self.batch_size) def __getitem__(self, idx): batch_x = self.x[idx * self.batch_size:(idx + 1) * self.batch_size] batch_y = self.y[idx * self.batch_size:(idx + 1) * self.batch_size] return np.array([ resize(imread(file_name), (200, 200)) for file_name in batch_x]), np.array(batch_y) ``` ct)zGets batch at position `index`. Args: index: position of the batch in the Sequence. Returns: A batch NotImplementedError)rindexs r __getitem__zSequence.__getitem__  r7ct)zbNumber of batch in the Sequence. Returns: The number of batches in the Sequence. rrs r__len__zSequence.__len__s  r7cy)z-Method called at the end of every epoch. Nr]rs r on_epoch_endzSequence.on_epoch_ends r7c#ZKfdttDD]}|yw)z2Create a generator that iterate over the Sequence.c3(K|] }| ywrr]).0irs r z$Sequence.__iter__..s3Qa3sN)ranger)ritems` rrzSequence.__iter__s)3%D "23 js(+N) rYrZr[rrrrrrr]r7rr1r1s:+Z   r7r1c#$K |D]}| w)zIterates indefinitely over a Sequence. Args: seq: `Sequence` instance. Yields: Batches of data from the `Sequence`. r])seqrs riter_sequence_infiniters$  j scBtjfd}|S)Ncft5tdc}a|i|}|a|cdddS#1swYyxYw)NT)_FORCE_THREADPOOL_LOCK_FORCE_THREADPOOL)argskwargsold_force_threadpooloutrs rwrappedz.dont_use_multiprocessing_pool..wrappedsA 0A4-- t v c.  s'0r)rrs` rdont_use_multiprocessing_poolrs% ??1 .r7cf|rtrtjjStjSr)rmultiprocessingdummyPool)use_multiprocessings rget_pool_classrs&  1  % %%   r7cBttjatS)z,Lazily create the queue to track worker ids.)_WORKER_ID_QUEUErQueuer]r7rget_worker_id_queuers&,,. r7c|ayr_SHARED_SEQUENCES)seqss r init_poolr sr7ct||S)aQGet the value from the Sequence `uid` at index `i`. To allow multiple Sequences to be used at the same time, we use `uid` to get a specific one. A single Sequence would cause the validation to overwrite the training Sequence. Args: uid: int, Sequence identifier i: index Returns: The value at index `i`. r)uidrs r get_indexrs 3  ""r7cleZdZdZ d dZdZd dZdZddZdZ e d Z e d Z e d Z y)SequenceEnqueueraBase class to enqueue inputs. The task of an Enqueuer is to use parallelism to speed up preprocessing. This is done with processes or threads. Example: ```python enqueuer = SequenceEnqueuer(...) enqueuer.start() datas = enqueuer.get() for data in datas: # Use the inputs; training, evaluating, predicting. # ... stop sometime. enqueuer.stop() ``` The `enqueuer.get()` should be an infinite stream of datas. c||_||_t tjddat ttrt|_tdz anKtj5tj|_txjdz c_ dddd|_ d|_ d|_ d|_d|_y#t $rdaYwxYw#1swY>xYw)Nrrr)sequencer_SEQUENCE_COUNTERrValueOSErrorr,rrget_lockvalueworkers executor_fnqueue run_thread stop_signal)rrrs rrzSequenceEnqueuer.__init__8sDM2D +11#q9#S)"dh1  % % '%$**1$%DLDDJDOD' %%sC)/C CCCcX|jduxr|jj Sr)ris_setrs r is_runningzSequenceEnqueuer.is_runningVs*   4 ' I0@0@0G0G0I,IIr7ct|jr|j|_n fd|_|_t j ||_t j|_t j|j|_ d|j_ |jjy)zStarts the handler's workers. Args: workers: Number of workers. max_queue_size: queue size (when full, workers could block on `put()`) c&tdS)NF)r)_rs rrz(SequenceEnqueuer.start..es#8>%#8#Ar7)targetTN)r_get_executor_initrrrrrEventrThread_runrdaemonstart)rrmax_queue_sizes ` rrzSequenceEnqueuer.startYs 009dBdDL^,DJ (D&&dii8DO!DOOOOr7c>|jt|j<y)z&Sends current Iterable to all workers.N)rrrrs r_send_sequencezSequenceEnqueuer._send_sequencems#'--dhhr7Nc|jj|jj5|jjj d|j_|jj jddd|jj|dt|j<y#1swY8xYw)zStops running threads and wait for them to exit, if necessary. Should be called by the same thread which called `start()`. Args: timeout: maximum time to wait on `thread.join()` rN) rr*rmutexclearunfinished_tasksnot_fullnotifyrrcrr)rtimeouts rstopzSequenceEnqueuer.stoprs    # jj$%djj! jj  "# OO!"&dhh ##s ACC cF|jr|jyyr)rr rs r__del__zSequenceEnqueuer.__del__s  iikr7ct)?Submits request to the executor and queue the `Future` objects.rrs rrzSequenceEnqueuer._runs  r7ct)Gets the Pool initializer for multiprocessing. Args: workers: Number of workers. Returns: Function, a Function to initialize the pool r)rrs rrz#SequenceEnqueuer._get_executor_initrr7ct)zCreates a generator to extract data from the queue. Skip the data if it is `None`. # Returns Generator yielding tuples `(inputs, targets)` or `(inputs, targets, sample_weights)`. rrs rrzSequenceEnqueuer.gets  r7)F)r r)rYrZr[rrrrrr rrrrrr]r7rrr#sg*$)<J(0 '   r7rc<eZdZdZdfd ZdZdZdZdZxZ S)OrderedEnqueueraBuilds a Enqueuer from a Sequence. Args: sequence: A `tf.keras.utils.data_utils.Sequence` object. use_multiprocessing: use multiprocessing if True, otherwise threading shuffle: whether to shuffle the data at the beginning of each epoch c<tt| ||||_yr)superrrshuffle)rrrr __class__s rrzOrderedEnqueuer.__init__s /4)(4GHDLr7cfd}|S)rcxtdt|dtf}tj ||SNT) initializerinitargs)rinit_pool_generatorr _DATA_POOLSadd)rpoolrs rpool_fnz3OrderedEnqueuer._get_executor_init..pool_fns; !^D ! 2$ 3 568dood kr7r]rrr$s ` rrz"OrderedEnqueuer._get_executor_inits Nr7c tjd|jjdk(s|jj ryJ)zWait for the queue to be empty.皙?rN)timesleeprrrrrs r _wait_queuezOrderedEnqueuer._wait_queues;  jjo  $ $ )T-=-=-D-D-F r7c ttt|j}|j  |j rt j |t|jt5}|D]c}|jjr dddy|jj|jt|j |fde|j#|jjr dddy ddd|jj%|j #1swY5xYwrTNblock)r'rrrrrrandomrrrrrrput apply_asyncrrr*r)rrexecutorrs rrzOrderedEnqueuer._runsE#dmm,-.H  x 4##$56 7 8 JA    $ $ &    **..""9txxm.>?+IINN;!2!223  LL""$L<r7c&tt|S)aKGets the next value from the generator `uid`. To allow multiple generators to be used at the same time, we use `uid` to get a specific one. A single generator would cause the validation to overwrite the training generator. Args: uid: int, generator identifier Returns: The next value of generator `uid`. )rr)rs r next_samplerFs $ %%r7c:eZdZdZ dfd ZdZdZdZxZS)GeneratorEnqueueraBuilds a queue out of a data generator. The provided generator can be finite in which case the class will throw a `StopIteration` exception. Args: generator: a generator function which yields data use_multiprocessing: use multiprocessing if True, otherwise threading random_seed: Initial seed for workers, will be incremented by one for each worker. c<tt| ||||_yr)rrHrrB)r generatorrrBrs rrzGeneratorEnqueuer.__init__4s  T+I7JK"Dr7cfd}|S)zGets the Pool initializer for multiprocessing. Args: workers: Number of works. Returns: A Function to initialize the pool ctdt|jtf}tj ||Sr)rr rBrr!r")rr#rrs rr$z5GeneratorEnqueuer._get_executor_init..pool_fnCsB !^D ! 2$**,?,ABDdood kr7r]r%s`` rrz$GeneratorEnqueuer._get_executor_init:s Nr7c<|jt|jt5} |jj r dddy|j j|jt|jfda#1swYyxYwr,) rrrrrrrr0r1rFr)rr2s rrzGeneratorEnqueuer._runKs !!"34 5H     " " $ HH   txxk :$  H HHsB=BBc#K |jr\|jjdj}|jj|||jr[yy#t$rg}|jj dkDrI|j |jjd|jj dkDrI|D]}|j|Dcgc]#}|js|j%ncc}w}}|D] }|| Yyt$r/}|jdt|vr td|d}~wwxYww)r4Tr-Nrzgenerator already executingzyYour generator is NOT thread-safe. Keras requires a thread-safe generator when `use_multiprocessing=False, workers > 1`. ) rrrr7 StopIterationqsizeappendwait successfulrnr r&rC)rr9 last_onesrfuturers rrzGeneratorEnqueuer.getVsM OO d+//1   , OO   i JJ    "d34 JJ    "! /8OF6;L;L;N6::<OOiO&  ,  iik &#a& 0 9: : gsTE7A*A1/E71A0E4"E4=D&D&%E45E4;E7=E4*E//E44E7)FN) rYrZr[rrrrrr:r;s@rrHrH's% $)# " H$r7rH)NN).r9)FNNdatasetsr9Fr9Nr)r)r9rX)Drabcr contextlibrrrrmultiprocessing.dummyrErr/rJsysr<rr(r2rkweakrefr>numpyr.tensorflow.python.frameworkrsix.moves.urllib.requestrtensorflow.python.keras.utilsr+tensorflow.python.keras.utils.generic_utilsr&tensorflow.python.keras.utils.io_utilsr version_infor$urllib.requestr6rRrrgrrrhrjrrr1rrrWeakSetr!rr* _WORKER_IDSrRLockrrrrrrrrr rFrHr]r7rrisi /  .,4?AA!#J)*.f$""@F& >,VD KvK\  goo e (*  #"}v}@V&Vr=6 & S(Sr7