1Vh dZddlZddlZddlZddlmZmZmZddlm Z ddlm Z ddlm Z ddlm Z ddlm Z dd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZej4Zej8dGddZej8dGddZej8dGddZej8dGddZ ej8dGddZ!ej8dGddZ"ej8dGd d!Z#d"d#d$d%d&d'Z$ejJZ%ejLZ&ejNZ'ejPZ(ejRZ)ejTZ*ejVZ+ejXZ,ejZZ-d(Z.e%d)e'd*e(d*e)d+e&d,e*d-iZ/e%de'de(de)de&de*diZ0Gd.d/Z1d0Z2d1Z3y)2zATakes a generator of values, and accumulates them for a frontend.N)OptionalSequenceTuple)directory_watcher)event_file_loader) event_util) io_wrapper)plugin_asset_util) reservoir) tag_types) config_pb2) event_pb2) graph_pb2)meta_graph_pb2) tensor_pb2) compressor) tb_loggingT)frozenc0eZdZUdZeed<eed<eed<y) ScalarEventzContains information of a scalar event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. value: A float or int value of the scalar. wall_timestepvalueN)__name__ __module__ __qualname____doc__float__annotations__intf/home/dcms/DCMS/lib/python3.12/site-packages/tensorboard/backend/event_processing/event_accumulator.pyrr*s I Lr"rc@eZdZUdZeed<eed<eeeefed<y)CompressedHistogramEventa&Contains information of a compressed histogram event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. compressed_histogram_values: A sequence of tuples of basis points and associated values in a compressed histogram. rrcompressed_histogram_valuesN) rrrrrrr rrr!r"r#r%r%9s( I!)%u *=!>>r"r%cdeZdZUdZeed<eed<eed<eed<eed<eeed<eeed<y ) HistogramValueaZHolds the information of the histogram values. Attributes: min: A float or int min value. max: A float or int max value. num: Total number of values. sum: Sum of all values. sum_squares: Sum of squares for all values. bucket_limit: Upper values per bucket. bucket: Numbers of values per bucket. minmaxnumsum sum_squares bucket_limitbucketN)rrrrrrr rr!r"r#r(r(Is7  J J H J5/! SMr"r(c0eZdZUdZeed<eed<eed<y)HistogramEventzContains information of a histogram event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. histogram_value: Information of the histogram values. rrhistogram_valueN)rrrrrrr r(r!r"r#r1r1`s I##r"r1cDeZdZUdZeed<eed<eed<eed<eed<y) ImageEventaContains information of an image event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. encoded_image_string: Image content encoded in bytes. width: Width of the image. height: Height of the image. rrencoded_image_stringwidthheightN)rrrrrrr bytesr!r"r#r4r4os$ I J Kr"r4cNeZdZUdZeed<eed<eed<eed<eed<eed<y) AudioEventaContains information of an audio event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. encoded_audio_string: Audio content encoded in bytes. content_type: A string describes the type of the audio content. sample_rate: Sample rate of the audio in Hz. Must be positive. length_frames: Length of the audio in frames (samples per channel). rrencoded_audio_string content_type sample_rate length_framesN) rrrrrrr r8strr!r"r#r:r:s,  Ir"r:cDeZdZUdZeed<eed<ejed<y) TensorEventzA tensor event. Attributes: wall_time: Timestamp of the event in seconds. step: Global step of the event. tensor_proto: A `TensorProto`. rr tensor_protoN) rrrrrrr r TensorProtor!r"r#rArAs  I(((r"rA_ProcessScalar_ProcessHistogram _ProcessImage _ProcessAudio_ProcessTensor) simple_valuehistoimageaudiotensor) rii3i iii it$'irN ceZdZdZdedfdZdZdZdZdZ d e e fd Z d Z d Zd ZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZdZ dZ!d Z"d!Z#d"Z$y)#EventAccumulatora8An `EventAccumulator` takes an event generator, and accumulates the values. The `EventAccumulator` is intended to provide a convenient Python interface for loading Event data written during a TensorFlow run. TensorFlow writes out `Event` protobuf objects, which have a timestamp and step number, and often contain a `Summary`. Summaries can have different kinds of data like an image, a scalar value, or a histogram. The Summaries also have a tag, which we use to organize logically related data. The `EventAccumulator` supports retrieving the `Event` and `Summary` data by its tag. Calling `Tags()` gets a map from `tagType` (e.g. `'images'`, `'compressedHistograms'`, `'scalars'`, etc) to the associated tags for those data types. Then, various functional endpoints (eg `Accumulator.Scalars(tag)`) allow for the retrieval of all data associated with that tag. The `Reload()` method synchronously loads all of the data written so far. Histograms, audio, and images are very large, so storing all of them is not recommended. Fields: audios: A reservoir.Reservoir of audio summaries. compressed_histograms: A reservoir.Reservoir of compressed histogram summaries. histograms: A reservoir.Reservoir of histogram summaries. images: A reservoir.Reservoir of image summaries. most_recent_step: Step of last Event proto added. This should only be accessed from the thread that calls Reload. This is -1 if nothing has been loaded yet. most_recent_wall_time: Timestamp of last Event proto added. This is a float containing seconds from the UNIX epoch, or -1 if nothing has been loaded yet. This should only be accessed from the thread that calls Reload. path: A file path to a directory containing tf events files, or a single tf events file. The accumulator will load events from this path. scalars: A reservoir.Reservoir of scalar summaries. tensors: A reservoir.Reservoir of tensor summaries. @@Tensors NTc^|xst}i}tD]}||vr ||||<t|||<d|_tj|t|_d|_d|_d|_i|_ i|_ tj|t|_ tj|td|_tj|t|_tj|t"|_tj|t&|_t+j,t.|_t3j4|_||_t;||_||_||_ d|_!d|_"d|_#d|_$d|_%i|_&y)aConstruct the `EventAccumulator`. Args: path: A file path to a directory containing tf events files, or a single tf events file. The accumulator will load events from this path. size_guidance: Information on how much data the EventAccumulator should store in memory. The DEFAULT_SIZE_GUIDANCE tries not to store too much so as to avoid OOMing the client. The size_guidance should be a map from a `tagType` string to an integer representing the number of items to keep per tag for items of that `tagType`. If the size is 0, all events are stored. compression_bps: Information on how the `EventAccumulator` should compress histogram data for the `CompressedHistograms` tag (for details see `ProcessCompressedHistogram`). purge_orphaned_data: Whether to discard any events that were "orphaned" by a TensorFlow restart. N)sizeF)rUalways_keep_last)scalars histogramscompressed_histogramsimagesaudios)'DEFAULT_SIZE_GUIDANCE_first_event_timestampr ReservoirSCALARSrX_graph_graph_from_metagraph _meta_graph_tagged_metadatasummary_metadata HISTOGRAMSrYCOMPRESSED_HISTOGRAMSrZIMAGESr[AUDIOr\TENSORStensors collections defaultdictdict_plugin_to_tag_to_content threadingLock_generator_mutexpath_GeneratorFromPath _generator_compression_bpspurge_orphaned_datamost_recent_stepmost_recent_wall_time file_version_source_writeraccumulated_attrs_tensor_summaries)selfrs size_guidancecompression_bpsrwsizeskeys r#__init__zEventAccumulator.__init__s0&>)> ( 8Cm#*3/c 237c  8 '+# **g?  %*" " "#--53DE%.%8%8,-& " ))uV}= ))uU|<  **g? *5)@)@)F& ) 0 ,T2 /#6 "%'" #" "$r"c|j5|jjD]}|j| ddd|S#1swY|SxYw)zLoads all events added since the last call to `Reload`. If `Reload` was never called, loads all events in the file. Returns: The `EventAccumulator`. N)rrruLoad _ProcessEventr~events r#ReloadzEventAccumulator.ReloadNsU " " *--/ *""5) * *  * s 1A  AcBtj|j|S)aHReturn a list of all plugin assets for the given plugin. Args: plugin_name: The string name of a plugin to retrieve assets for. Returns: A list of string plugin asset names, or empty list if none are available. If the plugin was not registered, an empty list is returned. )r ListAssetsrsr~ plugin_names r# PluginAssetszEventAccumulator.PluginAssets[s!++DII{CCr"cDtj|j||S)a4Return the contents of a given plugin asset. Args: plugin_name: The string name of a plugin. asset_name: The string name of an asset. Returns: The string contents of the plugin asset. Raises: KeyError: If the asset is not available. )r RetrieveAssetrs)r~r asset_names r#RetrievePluginAssetz$EventAccumulator.RetrievePluginAssetgs"!.. II{J  r"c(|j |jS|j5 t|jj }|j ||jcdddS#t $r tdwxYw#1swYyxYw)aReturns the timestamp in seconds of the first event. If the first event has been loaded (either by this method or by `Reload`, this returns immediately. Otherwise, it will load in the first event. Note that this means that calling `Reload` will cause this to block until `Reload` has finished. Returns: The timestamp in seconds of the first event that was loaded. Raises: ValueError: If no events have been loaded and there were no events found on disk. Nz!No event timestamp could be found)r^rrnextrurr StopIteration ValueErrorrs r#FirstEventTimestampz$EventAccumulator.FirstEventTimestampxs  & & 2.. .  " " F FT__1134""5)22  F F ! F !DEE F  F FsB?A00BBBreturncj|j |jS|j5 t|jj }|j ||jcdddS#t $r#tjd|jYnwxYw dddy#1swYyxYw)z%Returns the name of the event writer.Nz.End of file in %s, no source writer was found.) r{rrrrurrrloggerinforsrs r#GetSourceWriterz EventAccumulator.GetSourceWriters    *&& &  " "  T__1134""5)**    !  Ddii     s(B)?A00)BB)BB))B2cX||jvrtd|z|j|S)aReturns a dict mapping tags to content specific to that plugin. Args: plugin_name: The name of the plugin for which to fetch plugin-specific content. Raises: KeyError: if the plugin name is not found. Returns: A dict mapping tag names to bytestrings of plugin-specific content-- by convention, in the form of binary serialized protos. zPlugin %r could not be found.)roKeyErrorrs r#PluginTagToContentz#EventAccumulator.PluginTagToContents4 d<< <:[HI I--k::r"c |j|S)zGiven a summary tag name, return the associated metadata object. Args: tag: The name of a tag, as a string. Raises: KeyError: If the tag is not found. Returns: A `SummaryMetadata` protobuf. )rer~tags r#SummaryMetadataz EventAccumulator.SummaryMetadatas$$S))r"cL|j|j|_|jdrptj|j }|j r>|j |k7r/tjdj|j |||_|jdrptj|j}|jr>|j|k7r/tjdj|j|||_ |j||jdr:|jtjd|j|_d|_y|jd r|j"tjd |j$|_|j |j rt'j(}|j+|j"|jrH|jtjd d |_|jj-|_yyy|jd rc|j.j0}||j2vrtjd|zdz|j.j4|j2|<y|jdrC|j6j8D](}|jdr|j0}||j:vrx|j<|j:|<|j<j>}|j@r'|jB|jD|j@|<ntjd|tFjID]k\}} |j|stK||} |j0}|dk(r|s |jL}tK|| ||j|jN| m+yy)z#Called whenever an event is loaded.Nsource_metadataz;Found new source writer for event.proto. Old: {0}, New: {1}rzzqFound new file_version for event.proto. This will affect purging logic for TensorFlow restarts. Old: {0} New: {1} graph_defzFound more than one graph event per run, or there was a metagraph containing a graph_def, as well as one or more graph events. Overwriting the graph with the newest event.Fmeta_graph_defz]Found more than one metagraph event per run. Overwriting the metagraph with the newest event.zFound multiple metagraphs containing graph_defs,but did not find any graph events. Overwriting the graph with the newest metagraph version.Ttagged_run_metadataz2Found more than one "run metadata" event with tag z'. Overwriting it with the newest event.summarymetadataz?This summary with tag %r is oddly not associated with a plugin.rM)(r^rHasFieldrrrr{rrformatParseFileVersionrzwarning_MaybePurgeOrphanedDatararrbrcrr MetaGraphDefParseFromStringSerializeToStringrrrd run_metadatarrrer plugin_datarcontentro SUMMARY_TYPESitemsgetattr node_namer) r~rnew_source_writernew_file_version meta_graphrrr summary_type summary_funcdatums r#rzEventAccumulator._ProcessEvents  & & .*///D ' >>+ , * : :%%! ""t':':>O'O -fT002CD  #4D  >>. ))::5;M;MN   T%6%6:J%J,fT..0@A !1D  $$U+ >>+ &{{&( //DK).D & ^^, -+K %33D {{"d&@&@,88: **4+;+;<''{{.!K26D.","6"6"H"H"JDK( 'A ^^1 2++//Cd+++H?@ */)B)B)O)OD ! !# & ^^I &,,# >>*-))C$"7"775:^^--c2&+nn&@&@ &22&1%8%8!:: + 7 7!##NN%.!$ 3@2E2E2G .L,~~l3 '| <#ii'83C#(//C3l3%**e 3# 'r"ct|jjt|jjt |j jt|jjt|jjt|jjt|jdut|j dut"t%|j&j)i S)zReturn all tags found in the value stream. Returns: A `{tagType: ['list', 'of', 'tags']}` dictionary. N)rhr[Keysrir\rfrYr`rXrgrZrjrkGRAPHra META_GRAPHrc RUN_METADATAlistrdkeys)r~s r#TagszEventAccumulator.Tags;s DKK$$& 4;;##% ,,. T\\&&( !4#=#=#B#B#D T\\&&( 4;;d* ((4 $t4499;<  r"c8|jj|S)aGiven a summary tag, return all associated `ScalarEvent`s. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `ScalarEvent`s. )rXItemsrs r#ScalarszEventAccumulator.ScalarsO||!!#&&r"ctj}|j|j|j|St d)aMReturn the graph definition, if there is one. If the graph is stored directly, return that. If no graph is stored directly but a metagraph is stored containing a graph, return that. Raises: ValueError: If there is no graph for this run. Returns: The `graph_def` proto. z*There is no graph in this EventAccumulator)rGraphDefrarr)r~graphs r#GraphzEventAccumulator.Graph]s?""$ ;; "  ! !$++ .LEFFr"c|j tdtj}|j |j|S)zReturn the metagraph definition, if there is one. Raises: ValueError: If there is no metagraph for this run. Returns: The `meta_graph_def` proto. z.There is no metagraph in this EventAccumulator)rcrrrr)r~rs r# MetaGraphzEventAccumulator.MetaGraphosD    #MN N#002 ""4#3#34r"c||jvr tdtj}|j |j||S)aGiven a tag, return the associated session.run() metadata. Args: tag: A string tag associated with the event. Raises: ValueError: If the tag is not found. Returns: The metadata in form of `RunMetadata` proto. z+There is no run metadata with this tag name)rdrr RunMetadatar)r~rrs r#rzEventAccumulator.RunMetadata~sK d++ +JK K!--/ $$T%:%:3%?@r"c8|jj|S)zGiven a summary tag, return all associated histograms. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `HistogramEvent`s. )rYrrs r# HistogramszEventAccumulator.Histogramss$$S))r"c8|jj|S)aGiven a summary tag, return all associated compressed histograms. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `CompressedHistogramEvent`s. )rZrrs r#CompressedHistogramsz%EventAccumulator.CompressedHistogramss))//44r"c8|jj|S)zGiven a summary tag, return all associated images. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `ImageEvent`s. )r[rrs r#ImageszEventAccumulator.Images{{  %%r"c8|jj|S)zGiven a summary tag, return all associated audio. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `AudioEvent`s. )r\rrs r#AudiozEventAccumulator.Audiorr"c8|jj|S)zGiven a summary tag, return all associated tensors. Args: tag: A string tag associated with the events. Raises: KeyError: If the tag is not found. Returns: An array of `TensorEvent`s. )rkrrs r#TensorszEventAccumulator.Tensorsrr"c|jsy|jr!|jdk\r|j|y|j|y)aMaybe purge orphaned data due to a TensorFlow crash. When TensorFlow crashes at step T+O and restarts at step T, any events written after step T are now "orphaned" and will be at best misleading if they are included in TensorBoard. This logic attempts to determine if there is orphaned data, and purge it if it is found. Args: event: The event to use as a reference, to determine if a purge is needed. N)rwrz_CheckForRestartAndMaybePurge$_CheckForOutOfOrderStepAndMaybePurgers r#rz(EventAccumulator._MaybePurgeOrphanedDatasF''    !2!2a!7  . .u 5  5 5e Processes a proto histogram by adding it to accumulated state.N)rr1rYAddItemrZ_CompressHistogram)r~rrrrJhisto_evs r#rEz"EventAccumulator._ProcessHistogram#sU11%8!)T59 X. ""** 422 r"ct|j|jtj|j |j S)zCallback for _ProcessHistogram.)r%rrrcompress_histogram_protor2rv)r~rs r#rz#EventAccumulator._CompressHistogram,s>'    MM  / /(($*?*?   r"ct|||j|j|j}|jj ||y)z5Processes an image by adding it to accumulated state.)rrr5r6r7N)r4r5r6r7r[r)r~rrrrKrs r#rFzEventAccumulator._ProcessImage6s?!&!;!;++<<   C'r"ct|||j|j|j|j}|j j ||y)z4Processes a audio by adding it to accumulated state.)rrr;r<r=r>N)r:r;r<r=r>r\r)r~rrrrLrs r#rGzEventAccumulator._ProcessAudioAsL!&!;!;++))--   C'r"cXt|||}|jj||y)z;Processes a simple value by adding it to accumulated state.)rrrN)rrXr)r~rrrscalarsvs r#rDzEventAccumulator._ProcessScalarMs$ 94v F S"%r"cXt|||}|jj||y)N)rrrB)rArkr)r~rrrrMtvs r#rHzEventAccumulator._ProcessTensorRs$ 94f M S"%r"c fd |rN fd}jjDcgc] }|| }}t|Dcgc] }t|}}n2jDcgc]}t |j }}t|dkDrMtjjjjg|}tj|yycc}wcc}wcc}w)aPurge all events that have occurred after the given event.step. If by_tags is True, purge all events that occurred after the given event.step, but only for the tags that the event has. Non-sequential event.steps suggest that a TensorFlow restart occurred, and we discard the out-of-order events to display a consistent view in TensorBoard. Discarding by tags is the safer method, when we are unsure whether a restart has occurred, given that threading in supervisor can cause events of different tags to arrive with unsynchronized step values. If by_tags is False, then purge all events with event.step greater than the given event.step. This can be used when we are certain that a TensorFlow restart has occurred and these events can be discarded. Args: event: The event to use as reference for the purge. All events with the same tags, but with a greater event.step will be purged. by_tags: Bool to dictate whether to discard all out-of-order events or only those that are associated with the given reference event. c6|jjkSN)r)xrs r#z)EventAccumulator._Purge..ms 3r"cjDcgc](}t|j|j*c}Scc}wr)r|r FilterItemsr)rr _NotExpiredr~s r#_ExpiredPerTagz/EventAccumulator._Purge.._ExpiredPerTagqsC"33D!$00eiiHs-ArN)rrzipr,r|rr_GetPurgeMessagerxryrrrr) r~rrrrexpired_per_tagsrexpired_per_type purge_msgrs `` @r#rzEventAccumulator._PurgeVs.4  49==3F3F */u%   145E0FG1AG G// a ,,[9     1 $(%%**   " I NN9 % %  H sC'C,'"C1)%rrrrNORMAL_HISTOGRAM_BPSrrrrrrr?rrrrrrrrrrrrrrrrrrrErrFrGrDrHrr!r"r#rSrSs)\, N$`  D "F4 # ;$ *yv ( 'G$ & * 5 & & '=2.,9"     ( (& &3&r"rSc 4dj||||||||| S)z=Return the string message associated with TensorBoard purges.a"Detected out of order event.step likely caused by a TensorFlow restart. Purging expired events from Tensorboard display between the previous step: {} (timestamp: {}) and current step: {} (timestamp: {}). Removing {} scalars, {} histograms, {} compressed histograms, {} images, and {} audio.)r) rxry event_stepevent_wall_timenum_expired_scalarsnum_expired_histosnum_expired_comp_histosnum_expired_imagesnum_expired_audios r#rrs3   f r"c|s tdtj|rtj|St j |tjtjS)zECreate an event generator for file or directory at given path string.zpath must be a valid string)rr IsSummaryEventsFilerLegacyEventFileLoaderrDirectoryWatcher)rss r#rtrtsX 677%%d+ 66t<< 11   3 3  * *  r")4rrl dataclassesrptypingrrr$tensorboard.backend.event_processingrrrr r r r tensorboard.compat.protor rrrr tensorboard.plugins.distributionrtensorboard.utilr get_loggerr dataclassrr%r(r1r4r:rArrgrfrhrir`rjrrrr r]STORE_EVERYTHING_SIZE_GUIDANCErSrrtr!r"r#rsKH,,BB;;B::/..3/7'    d#  $ d# ? ?$ ?d#$,d# $ $$ $d#$$d#$(d# ) )$ ) %    "77  ! !           ! ! %% K3 A 1 U R 1 A 1 Q Q "w &w &t@  r"