1Vh(dZddlZddlmZddlmZddlmZddlmZejZ GddZ Gd d e Z y) z;Contains the implementation for the DirectoryWatcher class.N) io_wrapper)tf)io_util) tb_loggingcJeZdZdZdfdZdZdZdZdZdZ d Z d Z d Z y ) DirectoryWatcheraA DirectoryWatcher wraps a loader to load from a sequence of paths. A loader reads a path and produces some kind of values as an iterator. A DirectoryWatcher takes a directory, a factory for loaders, and optionally a path filter and watches all the paths inside that directory. This class is only valid under the assumption that only one path will be written to by the data source at a time and that once the source stops writing to a path, it will start writing to a new path that's lexicographically greater and never come back. It uses some heuristics to check whether this is true based on tracking changes to the files' sizes, but the check can have false negatives. However, it should have no false positives. cy)NT)xs f/home/dcms/DCMS/lib/python3.12/site-packages/tensorboard/backend/event_processing/directory_watcher.pyzDirectoryWatcher.,sc| td| td||_d|_||_d|_||_d|_i|_y)aConstructs a new DirectoryWatcher. Args: directory: The directory to load files from. loader_factory: A factory for creating loaders. The factory should take a path and return an object that has a Load method returning an iterator that will yield all events that have not been yielded yet. path_filter: If specified, only paths matching this filter are loaded. Raises: ValueError: If path_provider or loader_factory are None. NzA directory is requiredzA loader factory is requiredF) ValueError _directory_path_loader_factory_loader _path_filter_ooo_writes_detected_finalized_sizes)self directoryloader_factory path_filters r __init__zDirectoryWatcher.__init__,s]  67 7  !;< <# - '$)! "rc#K |jD]}|y#tjj$rNtjj j |jstd|jzYywxYww)aLoads new values. The watcher will load from one path at a time; as soon as that path stops yielding events, it will move on to the next path. We assume that old paths are never modified after a newer path has been written. As a result, Load() can be called multiple times in a row without losing events that have not been yielded yet. In other words, we guarantee that every event will be yielded exactly once. Yields: All values that have not been yielded yet. Raises: DirectoryDeletedError: If the directory has been permanently deleted (as opposed to being temporarily unavailable). z)Directory %s has been permanently deletedN) _LoadInternalrerrorsOpErroriogfileexistsrDirectoryDeletedError)revents r LoadzDirectoryWatcher.LoadFs{" ++-   yy   55;;%%doo6+?oo&7 s&B B A(B B B  B c#K|js|j|jsy |jjD]}||j}|s!tj d|j y|jjD]}|tj d|j ||j|w)a6Internal implementation of Load(). The only difference between this and Load() is that the latter will throw DirectoryDeletedError on I/O errors if it thinks that the directory has been permanently deleted. Yields: All values that have not been yielded yet. NzNo path found after %sz)Directory watcher advancing from %s to %s)r_InitializeLoaderr& _GetNextPathloggerinfor_SetPath)rr% next_paths r rzDirectoryWatcher._LoadInternalas||  " " $|| **,   ))+I 4djjA$**,    KK;   MM) $KsCCc|jS)aReturns whether any out-of-order writes have been detected. Out-of-order writes are only checked as part of the Load() iterator. Once an out-of-order write is detected, this function will always return true. Note that out-of-order write detection is not performed on GCS paths, so this function will always return false. Returns: Whether any out-of-order write has ever been detected by this watcher. )r)rs r OutOfOrderWritesDetectedz)DirectoryWatcher.OutOfOrderWritesDetecteds(((rcL|j}|r|j|yyN)r)r,)rpaths r r(z"DirectoryWatcher._InitializeLoaders$  "  MM$  rc|j}|rotj|sZ tjj j |j}tjd||||j|<||_|j||_y#tjj$r!}tjd||Yd}~Wd}~wwxYw)zSets the current path to watch for new events. This also records the size of the old path, if any. If the size can't be found, an error is logged. Args: path: The full path of the file to watch. zSetting latest size of %s to %dzUnable to get size of %s: %sN)rr IsCloudPathrr!r"statlengthr*debugrrr errorrr)rr3old_pathsizees r r,zDirectoryWatcher._SetPaths:: G//9 Juu{{''188 >$O26%%h/ ++D1 99$$ J ;XqII JsABC9CCctfdtjjD}|syj|dSt j |dsijs]tj|j}td|jz }|||D]}j|sd_ntfd|D}|r t|Sy)zGets the next path to load from. This function also does the checking for out-of-order writes as it iterates through the paths. Returns: The next path to load events from, or None if there are no more paths. c3DK|]}j|r|ywr2)r.0r3rs r z0DirectoryWatcher._GetNextPath..s'   &  s NrTc3ZK|]"}j|jkDs|$ywr2)rr?s r rAz0DirectoryWatcher._GetNextPath..s* djj&8D4::r]sE B ;!$'    hhV  I  r