BVhdZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl mZddl mZddl mZddl mZdd l mZdd l mZdd l mZdd l mZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddlm&Z&ddlm'Z'ddlm(Z)ddlm*Z*ddl+m,Z,ddl+m-Z-dd l+m.Z.dd!l+m/Z0dd"l1m2Z2dd#l3m4Z4dd$l3m5Z5dd%l3m6Z6dd&l7m8Z8dd'l9m:Z:d(Z;d)Zd,Z?d-Z@d.ZAd/ZBd0ZCd1ZDd2ZEd3ZFd4ZGd5ZHd6ZId7ZJd8ZKd9ZLd:ZMd;ZNdZQd?ZRd@ZSdAZTdBZUdCZVdDZWdEZXe4jZZe4jZ\e4jZ^e4jZ`e4jZbe4jZde4jZfe4jZhdFZidGZjdHZkdIZldJZme jdKdLdMZodNZpdVdOZqdPZrdQZsdVdRZtdSZuGdTdUZvy)Wz(A utility to trace tensor values on TPU.N) summary_pb2) monitoring) constant_op)dtypes) func_graph)function)graph_io)opstensor) tensor_util)file_io) array_ops)array_ops_stack)cond)control_flow_case)control_flow_ops)control_flow_util) gen_math_ops)init_ops) linalg_ops) logging_ops)math_ops)nn_impl) state_ops) string_ops)summary_ops_v2)variable_scope) analytics)gfile) remote_utils) tf_logging)summary_iterator)tensor_tracer_flags)tensor_tracer_report)tpu_replication)tpu_ops) training_utiltpucpuznot-traced-outside-op-rangeznot-traced-unsafe-opznot-traced-special-whileloop-opznot-traced-control-flow-opznot-traced-in-control-flowznot-traced-unsafe-scalarznot-traced-scalarznot-traced-less-interesting-opznot-traced-device-mismatchznot-traced-dynamic-shapez traced-scalarz traced-tensorztraced-user-includedznot-traced-user-excludedznot-traced-not-in-exec-pathznot-traced-non-numeric-tensorz%not-traced-feeds-special-whileloop-opzfile://tensor_tracer_variablestensor_tracer_summary_writersz trace.allzcompact_trace.gtensor_tracer_storagetensor_tracer_snapshotz #replica-id: Nonetensor_tracer_summary tensor_tracertensor_tracer_host_callz.tensor_tracer z /tensorflow/api/tensor_tracer/v1ztensor tracer usagemethodc| tdtj}|jt |j d|j S)z=Generates and returns a summary tag name for the given graph.z graph is Nonezutf-8) RuntimeErrorhashlibmd5updatereprencode hexdigest)graphhash_ids S/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/tpu/tensor_tracer.py_graph_summary_tagrAusI ]  '' KKM' ..e##G,-    cdtjz}|r#|jD]\}}|d|d|z }|tjtj <y)a]Enables tensor tracer and sets its parameters. Example usage: tensor_tracer_parameters = {'trace_dir': '/usr/tmp/trace_dir', 'trace_mode': 'norm', 'report_file': '/usr/tmp/trace_dir/report.all'} tensor_tracer.set_parameters(tensor_tracer_parameters) This sets up the parameters for tensor tracer. A call to tensor tracer as below is necessary to enable debugging on CPUs and GPUs. On TPUs below can be skipped as this call is hooked into tpu.rewrite. tt = tensor_tracer.TensorTracer() loss = tt.trace_cpu(tf.get_default_graph(), tensor_fetches=loss) Args: tensor_tracer_params: Tensor tracer parameter dictionary. Below gives examples of these parameters: See tensor_tracer_report.py for all parameters. - enable: If set, tensor tracer will be enabled. Calling enable_tensor_tracer automatically adds this parameters. - trace_mode: The trace_mode to be used by tensor tracer. These include: - summary: Collects multiple statistics for traced tensors, and writes them a summary file that can be visualized using tensorboard. This mode currently only works for TPUEstimator. It can be also be used for other models, but outfeed must be handled by the user. - norm: Collects norm of each traced tensor and writes them into a text file pointed by 'trace_dir' flag. (Default mode). - nan-inf: Checks the existince of NaNs and Infs in the tensor, and writes a boolean value to a text file pointed by 'trace_dir' flag. Note that 'norm' mode can also capture this information with more numerical info. - max-abs: Collects the absolute max for each traced tensors and writes it into a text file pointed by 'trace_dir' flag. - full-tensor: Writes the full tensor content of the traced tensors into a text file pointed by 'trace_dir' flag. - part-tensor: Writes a part of the tensor content of the traced tensors into a text file pointed by 'trace_dir' flag. - full_tensor_summary: Writes the full tensors as binary event files. The outputs can be read using: trace = tensor_tracer.read_tensor_tracer_event_file(event_file_path) - report_file: Path to the metadata file that is written during graph construction. If not set, metadata will be printed to stdout during graph construction. - trace_dir: Path where the execution traces will be written during the graph execution. If not set, trace will be printed to stderr. - trace_level: Tensor tracer aims to trace everything it can. This introduces some overhead on graph execution and graph compilation times. Using trace_level parameter, it is possible to trace operation based on their priorities. For example, - trace_level=7 is the highest trace_level, in which every op is traced. - trace_level=6 will skip constant operations such as tf.constant. - trace_level=5 will skip less important ops such as tf.identities. - The default trace_level=3, that will skip concat ops, or random number generators. - To reduce the graph compile time overhead, trace_level can be set to 0, that will skip additions, and substractions, and multiplications as well. - excluded_opnames: If set, any matching op name will not be traced. excluded_opnames can be set as a regular expression. E.g, excluded_opnames=.* will exclude everything. - excluded_optypes: If set, any matching op type will not be traced. excluded_optypes can be set as a regular expression. E.g, excluded_optypes=.* will exclude everything. excluded_optypes=MatMul will exclude all MatMul ops from tracing. - included_opnames: If set, any matching op name will be forced to be traced. included_opnames can be set as a regular expression. E.g, '--included_opnames=some_op --excluded_opname=*.' will only trace some_op. - included_optypes: If set, any matching op type will be forced to be traced. included_optypes can be set as a regular expression. E.g, '--included_optypes=some_op_type --excluded_optypes=*.' will trace only the ops with type 'some_op_type' - flush_summaries: If summary mode is used, flush_summaries=1 will flush summaries using outside compilation. Note that, if used with low level APIs, flush_summaries=1 is necessary to obtain results. Advanced Flags: - trace_scalar: Scalar values are not traced by default. If this flag is set, scalar values will also be traced. - op_range: In the form of '%d:%d' that limits the tracing to the ops within this limit. --op_range='5:10' will trace only the ops that have topological order between 5-10. - submode: 'brief' or 'detailed'. If the trace mode is not compact, brief mode will print only the id of each traced tensor to save some space. 'detailed' mode prints the full tensor name. - use_fingerprint_subdirectory: The trace directory will be chosen as using the fingerprint of the trace metadata under the provided trace_dir. z--%s=1z --=N)r$FLAG_NAME_ENABLEitemsosenviron FLAGS_ENV_VAR)tensor_tracer_params enable_flagskeyvalues r@set_parametersrNsYp/@@@,*0020 UC//l02>"** . ./rBcJ|dvry|dvry|dvry|dvry|d vry |d vry |d vryy )zReturns the priority of the op. If the priority of the op is k, it will be traced if trace_level>=k. Args: op_type: String name of the operation type. Returns: Integer value corresponding the priority of the op. )ConstShapeBroadcastGradientArgsRange VariableShapeFillOneHotShapeN) IdentityCastReshape ExpandDims StopGradientPreventGradientSqueezeGatherGatherNd) ConcatV2Concat StridedSliceSlicePackTileCollectivePermuteSplitVDynamicPartition)PadRandomUniformInt GreaterEqual)SumAddV2AddAddNBiasAddCrossReplicaSumr+)NegSub) MulSquareMatMul RandomUniformSelectMaximumMeanVarianceExpRsqrt)op_types r@ op_priorityrs >>  EE  DD  ;;  MM    @@  rBctjt}g}tj|D]}|j dst |j jdk7r+tdt |j jz|j}||xxdz cc<||dz }t |}||k(r+tjt}|j|n||kDrtd||fz|j jd}|j} |jjjD cgc]} | j } } t#j$|jj&t)j*|jj,j/j1| } | |||| <|Scc} w)aReads the event file written by tensor tracer. This can be used to read the full tensors written into binary event files by by TensorTracer with trace_mode=full_tensor_summary. Example usage: result_dict_list = tensor_tracer.read_tensor_tracer_event_file( event_file_path) for result_dict in result_dict_list: for step, tensor_dict in result_dict.items(): for tensor_name, full_tensor_content in tensor_dict.items(): logging.info(tensor_name, full_tensor_content) Args: event_file: Path to the event file that contains only tensor tracer events. Returns: A list of event dictionaries, each of which with the form: {step_number: {tensor_name: tensor_content}}. This is a list instead of a single event dictionary because it is possible that an event file may have multiple event traces, each of them covering the same step ranges. Raises: ValueError: If an unexpected trace is found. summaryrz3Single step contains %d summary values, expected 1.z6Unexpected: occurrence_idx (%d) > occurrence_size (%d)r) collections defaultdictintr#HasFieldlenrrM ValueErrorstepdictappendtagr tensor_shapedimsizenp frombuffertensor_contentrDTypedtypeas_numpy_dtypereshape) event_filestep_occurrence_countstep_occurrence_list trace_eventroccurrence_idxoccurrence_sizenew_occurrence tensor_value tensor_named real_shapers r@read_tensor_tracer_event_filer s4&11#6%66zB!Mk    * ;   $ $%* &(+K,?,?,E,E(FG HH   D$1$*4014N./O(#..t4n!!.1 / )03A3B3DDE E&&,,Q/L""K"."5"5"B"B"F"FGQ!&&GJG]]** \((../>>@  '* ?M(.{;C!MD  Hs Gc| |j}|jjt|jj t||f|S)ayProgrammatic interface to trace a tensor with Tensor Tracer. Tensor Tracer, by default, traces all tensors in the execution. This function can be used to limit traced tensors. If this function is called for a subset of the tensors, only those will be traced. For example, Tensor Traacer will only trace c below. c = tf.MatMul(a, b) tensor_tracer.trace_tensor(c) d = tf.add(c, 1) Args: tensor: the tensor object for which the tracing is requested. tracepoint_name: an optional tensor tracepoint name string. A tracepoint name is an Tensor Tracer internal name for the tensor. It is useful when comparing equivalent traces from different models that have different tensor namings. Equivalent tensors (with different names) can be mapped to each other by assigning a common tracepoint_name. Returns: The provided tensor. )namer>get_collection_TENSOR_TRACER_COLLECTIONadd_to_collection)r tracepoint_names r@ trace_tensorrQsH,kkO,,78,,  !:"(/!:< -rBc  |j}tj|rt|d|z|Sd}|D]-}tj|rt|d||fz|dz }/ |S#t$rY|St $rY|SwxYw)aAn interface for adding the tensor outputs of a keras layer. Encapsulates trace_tensor. Args: layer: A keras layer. checkpoint_name: a string name for the checkpoint. This name has to be a unique name if used within model comparison. The tensors that have the same checkpoint identifier is compared in model comparison. Returns: The provided layer. z%srz%s_%dr)outputr is_tf_typerAttributeErrorr7)layercheckpoint_nameoutputsidx output_tensors r@keras_layer_tracepointros  llGg&7DO45 , c"-  ! !' * }g#0F&F G q ,   ,   , s0A+4A++ B7BBc6eZdZdZeZedZedZedZ edZ edZ edZ edZ ed Zed Zd Zed Zd ZdZdZdZdZdej0fdZdej0fdZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$dZ%d Z&d!Z'd"Z(d#Z)d$Z*d%Z+d&Z,d'Z-d(Z.d)Z/d*Z0d+Z1d,Z2d-Z3d.Z4d/Z5d0Z6d1Z7d2Z8d3Z9d4Z:d5Z;d6Zd9Z?d:Z@d;ZAd<ZB d@d=ZC dAd>ZDdBd?ZEy)C TensorTraceranA software construct for tracing tensor values in a TF graph. This utility is disabled by default. It is hooked into tpu.rewrite, so it can easily be enabled on TPUs by setting the TENSOR_TRACER_FLAGS env variable as below without a code change. export TENSOR_TRACER_FLAGS="--enable=1" Below is the use example to enable it on CPUs or GPUs, or for more advance use cases on TPUs. a = x + 1 b = a * 2 rs = tf.reduce_sum(b) tensor_tracer.set_parameters({'trace_dir': 'path/to/trace_dir', 'report_file: 'path/to/report/file'}) tt = tensor_tracer.TensorTracer() if on_tpu: rs = tt.trace_tpu(tf.get_default_graph(), tensor_fetches=rs) else: rs = tt.trace_cpu(tf.get_default_graph(), tensor_fetches=rs) session.run(rs) If it is enabled, it will trace the output tensor values of selected Ops in the graph. It has two outputs: (1) the traces and (2) a report. The traces are dumped to a specified directory during the graph execution, while the report is dumped during the graph construction. By passing options via the env variable, users can change: (1) the trace mode (e.g., detecting NaN/Inf, printing partial or full tensor values) (2) which Ops to be traced (via op.name or op.type) (3) output trace file path. c tjj}|r$tj dj d|S#t tf$r }tjd|Yd}~yd}~wwxYw)z(Returns True if TensorTracer is enabled. is_enabledTzKTensor Tracer V1 flags processing error encountered in is_enabled check. %sN) r$ TTParametersrtt_gaugeget_cellsetrr7loggingwarning)enablees r@rzTensorTracer.is_enabledsl "//1<<>f ""<044T: m  % oo   sA A A;A66A;c<|ttfvrtd|zy)z)Checks if the given device type is valid.zInvalid device_type "%s"N)_DEVICE_TYPE_TPU_DEVICE_TYPE_CPUr) device_types r@check_device_typezTensorTracer.check_device_types)+-=>> 1+= >>?rBcb|tjk(r|tk7rtd|d|dyy)zChecks if the given trace mode work on the given device type. Args: device_type: Device type, TPU, GPU, CPU. trace_mode: Tensor tracer trace mode. Raises: ValueError: If the given trace mode is not supported for the device. z Device_type "z'" is not yet supported for trace mode ""N)r$TRACE_MODE_FULL_TENSOR_SUMMARYrr)r trace_modes r@check_trace_modezTensorTracer.check_trace_modesA(GGG ( (.9:GH H )HrBc|jdvS)N)LoopCond RefLoopCondtypeops r@ loop_cond_opzTensorTracer.loop_cond_ops 771 11rBctj|xsjtj|xsStj|xs<tj|xs%t j |xs|jdvS)Returns true if op is one of the special ops of in a while loop. Args: op: A tf.Operation. Returns: True if the given op is one of [Switch, Merge, Enter, Exit, NextIteration, LoopCond], which are all building blocks for TF while loops. )RefNextIteration NextIteration)r IsLoopSwitch IsLoopMerge IsLoopEnter IsLoopExitrrrrs r@ while_loop_opzTensorTracer.while_loop_ops{ + +B />  * *2 .>  * *2 .> ) )" -> & &r * > WW= = ?rBcZtj|xstj|S)r)rIsSwitchIsMergers r@control_flow_opzTensorTracer.control_flow_ops* ' ' ++  & &r *,rBc$|jdk(ryy)z1Returns True if this op is not safe to be traced.AssignTFrrs r@ unsafe_opzTensorTracer.unsafe_ops  ww(  rBcb|tk(r&tj|jjvSy)NF)rr&_TPU_REPLICATE_ATTRnode_defattr)rrs r@device_mismatchzTensorTracer.device_mismatch s)&&  0 0 8H8H HH rBc@|jdvry|jdvryy)zEReturn true if scalar output tensor from Op is not safe to be traced.)rEnterMergerPSwitchLessReadVariableOpT) VarHandleOpIteratorToStringHandleIteratorGetNextOneShotIterator IteratorV2 MakeIteratorBatchDatasetV2 MapDatasetFixedLengthRecordDataset TakeDataset ZipDataset PlaceholderPlaceholderWithDefaultreFrrs r@unsafe_scalar_tracez TensorTracer.unsafe_scalar_traces5  ww77  wwLL  rBcZt|j|jjkS)zDReturns True if the given op is not an interesting one to be traced.)rr _parameters trace_levelselfrs r@_is_interesting_opzTensorTracer._is_interesting_op&s" rww 4#3#3#?#? ??rBcd||fzS)z5Returns reason why the Op at op_idx is traced or not.z%d %sr)op_idxdetailss r@reasonzTensorTracer.reason*s FG$ $$rBcd|_tj|_t j |_i|_i|_i|_ t|_ d|_ i|_ d|_d|_y)zoInitializes a TensorTracer. Sets the various member fields from the flags (if given) or the defaults. N) _replica_idr%TensorTracerConfig _tt_configr$rr _host_call_fn_cache_variables_history_value_cacher_traced_op_names _report_proto_temp_cache_var_report_proto_path_outmost_contextrs r@__init__zTensorTracer.__init__0sq D*==?DO*779DDD "DEDDD D DrBcH|jr |jStd)aGetter for tensor_tracer.proto object for summary and full_tensor_summary modes. Returns: A tensor_tracer.proto object. Raises: ValueError if called before tracing happens, or when trace mode is not summary or full_tensor_summary. zuCall to report_proto must be done after tracing.Report proto only exists for trace_mode=[summary|full_tensor_summary])rrrs r@ report_protozTensorTracer.report_protoEs.     B CCrBc|jS)zkGetter for path where tensor_tracer.proto object should be written. Returns: A string path. )rrs r@report_proto_pathzTensorTracer.report_proto_pathUs  " ""rBcF|jddjddS)N/_:)replace)r variable_names r@_escape_namescopeszTensorTracer._escape_namescopes]s"  c * 2 23 <s r@_cache_variable_for_graphz&TensorTracer._cache_variable_for_graph`s1 D)))%'dE"   ''rBNc n| td||jvri|j|<||j|vr| td|jrtt}nt}|j 5}|j d5tjd|j|z||tj|ddttjjg|j||<dddddd|j||S#1swY#xYw#1swY'xYw)aCreates a variable as the cache to store historic intermediate tensor values. Args: cache_name: Name to be given to the cache (an instance of tf.variable). graph: Tensorflow graph. shape: A list of dimensions. dtype: Data type of created cache. Returns: A ref to newly created or existing cache with the given dimensions. Raises: ValueError: (1) If graph is None, or (2) shape is None when a new cache needs to be created. NInvalid graph.)shape must be provided at cache creation. tt_history_FTshaper initializer trainable use_resourcer)rr is_integerr_COMPACT_TRACE_ENTRY_INIT_VALUE as_default name_scoper get_variabler%rconstant_initializer_TENSOR_TRACER_STORAGEr GraphKeysLOCAL_VARIABLES)r cache_namer>r.rinit_valgs r@*_create_or_get_tensor_history_values_cachez7TensorTracer._create_or_get_tensor_history_values_cachees4& } ' (( D---)+d&22599 DEE   672     ALL$6 (55"T%<%r.rgraph_cache_varr<r=s r@"_create_or_get_tensor_values_cachez/TensorTracer._create_or_get_tensor_values_caches  } ' ((44U;O( DEE   672    QALL$6Q&4&A&A 3 !8!8!D Du 55h?/1N1NO 'Q #QQ : &&?: &&QQQ : &&s%"D4A*C8D8D =DDc|jjratjd5t j t t|jjd|_dddyd|_y#1swYyxYw)z5Adds nodes for computing the replica ID to the graph.N tt_replica_idrunknown) r num_replicasr control_dependenciesr'tpu_replicated_inputlistrangerrs r@_add_replica_id_to_graphz%TensorTracer._add_replica_id_to_graphsp ##  # #D )""77 t334 5 """ #d ""s ABB c||jjdkry|jjddkxs||jjdkS)z > ?@rBc^g}|jdkDra|jt|jfdD]\}}|j |t j |dd}|S|jdk(r|j\\}}|}|Std)aAReturns a tensor that merges the given signatures. Args: signatures: A dictionary of the signature updates from signature name to a tensor of dimension [1]. Returns: A tensor that concats the signature values in a predefined order. Raises: ValueError: Unable to merge signatures. rc|dS)Nrr)itemsignature_indicess r@z7TensorTracer._merge_tensor_signatures..)s,=d1g,FrBrLrmerge_single_op_signaturesaxisrzHCannot merge 0 signatures. Check the value passed for flag --signatures.)rercsortedrFrrstackr)r signatures sorted_updater!valupdatesrqs @r@_merge_tensor_signaturesz%TensorTracer._merge_tensor_signaturessM %%'!+//1:++-FH"&!SS!" %% a&BDg N  ' ' )Q .""$ihq#g N , --rBcH|j|}tj||jg}||jvr t d|t |j|k\r&t d|t |j|fz||j||<y)aReturns an op that will save the given updates to an entry in the cache. Args: cache_idx: The cache index of the tensor within the cache. updates: A dictionary of the signature updates from signature name to a tensor of dimension [1]. graph: A TensorFlow graph. Raises: RuntimeError: (1) graph is not already in self._temp_cache_var, or (2) cache_idx is out of range. $graph is not in self._temp_cache_varz#cache_idx (%d) is out of range (%d)N)r}rrrerr7r)r cache_idxr|r>s r@_save_tensor_value_to_tmp_cachez,TensorTracer._save_tensor_value_to_tmp_cache6s++G4G!%!?!?!A BDG D((( ? @@C,,U344 > S--e45B77 88-4D *rBc |j|}tj|d|jg}t j |g}|j t|}tj|||jS)aReturns an op that will save the given updates to an entry in the cache. Args: cache_idx: The cache index of the tensor within the cache. updates: A dictionary of the signature updates. graph: A TensorFlow graph. Returns: Cache update operation. r) r}rrrerconstantrB_TT_SUMMARY_TAGrscatter_updater)rrr|r>indicescaches r@_save_tensor_value_to_cache_opz+TensorTracer._save_tensor_value_to_cache_opMsw++G4G!"D$B$B$D EGG""I;/G  3 3OU KE  # #E7G < ? ??rBc|j|j|jj|jj |j }tj||jS)zCreates a new tf.Variable and a new tf.Operation that assigns the value of the tensor to this variable. Args: tensor: tensor whose values will be stored in a new tf.Variable. Returns: An assignment operation. ) rBrrr>r.as_listrrassign)rr snapshot_variables r@_snapshot_tensorzTensorTracer._snapshot_tensorasX?? VYY__  .   -v 6 9 99rBcd}ddfd}dfd }dfd }dfd }dfd }dd}dfd } |jjtjk(r|jj||iS|jjtjk(r|jj|iS|jjtj tj fvr|jj|iS|jjtjk(r3|jjtj||d giS|jjtjk(r3|jjtj||d giS|jjtjk(r|jj| |iS|jjtjk(r2tj|tj }i} t"|j%vst&|j%vr ||d \} } t)|j%j+d D]\} }| t,k(r ||d }n| t.k(r ||d }nr| t0k(r | |d }n^| t2k(r ||d }nJ| t4k(r ||}n8| t6k(r ||}n&| t"k(r }n| t&k(r }nt9d| z|| | <| St;d|jjz)zComputes NAN/Norm/Max on TPUs before sending to CPU. Args: tensor: The tensor to be traced. Returns: A tensor that should be input to the trace_function. Raises: RuntimeError: If the signature is invalid. c2|jjrjtjt j t j |t j|}tj|dd}|Stjdg}|S)z7Trace function for detecting any NaN/Inf in the tensor.c.tjdgS)Ng?rrrrBr@rrzQTensorTracer._preprocess_traced_tensor.._detect_nan_inf..K((#/rBc.tjdgS)NrrrBr@rrzQTensorTracer._preprocess_traced_tensor.._detect_nan_inf..rrBr) r is_floatingr reduce_anyr logical_oris_nanis_infrrr)r maskrs r@_detect_nan_infz?TensorTracer._preprocess_traced_tensor.._detect_nan_infzs  ! !""  # ###F+\-@-@-H JK  / /1 $,,cU3 rBc|r$tj|tj}||}|j j st j|g}|Sr')rcastrfloat32 get_shapeis_fully_definedrr)r tf_op cast_to_f32rs r@_compute_signaturezBTensorTracer._preprocess_traced_tensor.._compute_signaturesQ vv~~6Fmm $ $ & 7 7 9!))-< rBc~|tjd}tj|tj S)NFr)rrrrrr)r tsizers r@ _show_sizez:TensorTracer._preprocess_traced_tensor.._show_sizes. !UKe]]5&.. 11rBc4|tj|Sr')r reduce_maxr rrs r@ _show_maxz9TensorTracer._preprocess_traced_tensor.._show_max (;(;[ IIrBc4|tj|Sr')r reduce_minrs r@ _show_minz9TensorTracer._preprocess_traced_tensor.._show_minrrBc4|tj|Sr')rnormrs r@ _show_normz:TensorTracer._preprocess_traced_tensor.._show_norms   EErBc$fd}|||S)Nctjtj|}tj|}t j tj ||Sr')r greater_equalabsrr zero_fractionr)r non_zerosnans tolerances r@ sparsity_fnzSTensorTracer._preprocess_traced_tensor.._show_sparsity..sparsity_fnsJ**8<<+?K v&$$X%8%8D%IJJrBr)r rrrrs ` r@_show_sparsityz>TensorTracer._preprocess_traced_tensor.._show_sparsitysK   [ AArBc|r$tj|tj}t j t j|dgdg\}}|jjst j|g}|jjst j|g}||fS)z2Returns the mean and variance of the given tensor.r)axes) rrrrrmomentsrrrr)r rmeanvars r@_show_mean_and_variancezGTensorTracer._preprocess_traced_tensor.._show_mean_and_variances vv~~6//)"3"3FRD"ALidC ^^  . . 0  r* ]]_ - - /R( 3YrBc|d|S)NcRtjtj|Sr')rrr)ts r@rrzOTensorTracer._preprocess_traced_tensor.._show_max_abs..sH// Q@rBrrs r@ _show_max_absz=TensorTracer._preprocess_traced_tensor.._show_max_abss  @+OOrBrFrc |dSNrr)xs r@rrz8TensorTracer._preprocess_traced_tensor..s AaDrBrszUnknown signature type :%s.z(Unsupported signature for trace mode %s.)T)Tgư>)rrr$r]TRACE_MODE_PART_TENSORTRACE_MODE_FULL_TENSORrr^rrr_r`rarrrr_TT_SUMMARY_MEANrc_TT_SUMMARY_VARrwrF_TT_SUMMARY_NORM_TT_SUMMARY_MAX_TT_SUMMARY_MAX_ABS_TT_SUMMARY_MIN_TT_SUMMARY_SPARSITY_TT_SUMMARY_SIZErr7)rr rrrrrrrr result_dictrvariancesignature_namer!signature_result_tensorrs @r@_preprocess_traced_tensorz&TensorTracer._preprocess_traced_tensoros1 2JJFB O ""&9&L&LL))?6+B CC ##22 3))6 22 ##22::(< <))6 22 ""&9&I&II))9+<+< V qc,#$$ ""&9&L&LL))9+<+< V qc,#$$ ""&9&L&LL))=+@ AA ""&9&L&LL}}VV^^4fk d335 5 T224 40UKh%d&;&;&=&C&C&E*8 :> .! - -$.v5$I !  .$-f%$H ! 2 2$1&e$L !  .$-f%$H ! 3 3$26$: ! / /$.v$6 ! / /$( !  .$, !8>IJ J&= N#+>, 2    % % & ''rBcfdfd}fd}jjtjk(r|Sjjtjtj tj tjtjtjfvr|Stdjjz)alMakes the tensor tracing function called by outside compilation. Args: tensor_name: name of the tensor being traced. tensor_trace_order: TensorTraceOrder object holding tensorname to id map. Returns: A function to be passed as the first argument to outside compilation. Raises: RuntimeError: If the trace mode is invalid. c jjr3|jvrtd|d|ddj|z}nd|z}jjrSt j jjjtjz}t|z}ntj}tj|tj |dj"d|d|| S) aPrints a tensor value to a file. Args: tensor_name: name of the tensor being traced. num_elements: number of elements to print (-1 means print all). tensor: the tensor needs to be returned. output_tensor: the tensor needs to be printed. Returns: The same tensor passed via the "tensor" argument. Raises: ValueError: If tensor_name is not already in tensor_trace_order.tensorname_to_cache_idx. zTensor z with name z& is not in the tensorname_to_cache_idx%dz"%s"@  summarize output_stream)r is_brief_modetensorname_to_cache_idxr trace_dirrGpathjoin_TRACE_FILE_NAME_get_outfile_suffix_OUTPUT_STREAM_ESCAPEsysstderrrprint_v2rr.r) r num_elementsr rmsg output_pathrrtensor_trace_orders r@ _print_tensorz:TensorTracer._make_tensor_trace_fun.._print_tensors"    ' ' ) 0HH H{$% %'?? LL{"   # #ggll    & & t779 9; . ;   ! !#y}'E"%t'7'7"& t,80= ??rBc"t||S)z/Trace function for printing part of the tensor.)_TRACE_MODE_PART_TENSOR_SIZEr rrs r@_show_part_tensorz>TensorTracer._make_tensor_trace_fun.._show_part_tensor.s;(D!6++rBcd||S)z.Trace function for printing the entire tensor.rrrs r@_show_full_tensorz>TensorTracer._make_tensor_trace_fun.._show_full_tensor4s;FF ;;rBz7Full tensor support is not available with trace mode %s) rrr$rr]r^rr`rar_r7)rrrrrrs``` @r@_make_tensor_trace_funz#TensorTracer._make_tensor_trace_funs%?N+ < ##22 3   ""..++22...... '   P((334 55rBc,tj|S)zReturns true if the given op is inside a tf.cond or in tf.while_loop. Args: op: A tensorflow op that should be checked whether in control flow or not. Returns: A boolean value whether the op is in control flow or not. )rIsInCondrs r@_is_in_control_flowz TensorTracer._is_in_control_flowMs  % %b ))rBc|j|}tj|}|tj|jk(S)zReturns true if the op is at the same level with the training loop. Returns false if the op is in an inner while loop or if it is outside of the training loop. Args: op: tf.Operation Returns: A boolean. )_get_op_control_flow_contextrGetContainingWhileContextr)rrctxtouter_while_contexts r@_is_in_outmost_while_loopz&TensorTracer._is_in_outmost_while_loopWsI  , ,R 0D+EEdK "3"M"M # rBc|jry|jjtk(r"|j xs|j Sy)zMReturns false incase it is not safe to trace ops in tf.cond or tf.while_loop.FT)rjrrrrhrgrs r@_should_trace_in_control_flowz*TensorTracer._should_trace_in_control_flowgsH     $ $(8 8 * * , I0G0G0II rBctj|r+|j|tj|tytj |r+|j|tj|t ytj|r+|j|tj|tytj|jj|r+|j|tj|ty||vr+|j|tj|ty|j|s|j|s;|j!s+|j|tj|t"y|j%|re|j|tj|t&t(j*j,r t/j0d|j2y|j5|s+|j|tj|t6y|j9|s+|j|tj|t:y|j=|re|j|tj|t>t(j*j,r t/j0d|j2yy)aUReturns True if we should not trace Op. Args: op_id: Topological index of the op. op: tf.Operation ops_in_exec_path: Set of operations that are in the execution path. report_handler: An instance of tensor_tracer_report.TTReportHandle. Returns: True if the op should not be traced, false otherwise. TzUSER_INCLUDED op %sFzUSER_EXCLUDED op %s) rr instrument_opr _REASON_WHILELOOP_OPr_REASON_CONTROLFLOW_OPr_REASON_UNSAFE_OPrrr_REASON_DEVICE_MISMATCH_REASON_NOT_EXECUTEDrrr_REASON_IN_CONTROL_FLOWrW_REASON_USER_INCLUDEDr$TT_CHECK_FILTERrMrinforrO_REASON_OUTSIDE_OP_RANGEr_REASON_LESS_INTERESTING_OPr[_REASON_USER_EXCLUDED)rop_idrops_in_exec_pathreport_handlers r@_skip_opzTensorTracer._skip_opusj!!"%"" l!!%)=>@ ##B'"" l!!%)?@B b!"" l!!%):;= ##DOO$?$?D"" l!!%)@AC  !!"" l!!%)=>@  #4+I+I"+M  / / 1$$  ##E+BC E   $"" l!!%)>?A  , , 2 2 *BGG4    '"" l!!%)ABD   " "2 &"" l!!%)DEG    $"" l!!%)>?A  , , 2 2 *BGG4  rBcNttjtjtjg}|j |vr+|j |tj|ty|jDcgc]}tj|r|c}r+|j |tj|ty|j|jre|j |tj|tt j"j$r t'j(d|j*y|j-|jre|j |tj|t.t j"j$r t'j(d|j*y|j1j3s|j4j6t j8t j:t j<t j>t j@fvr+|j |tj|tBy|j |tj|tDytG|jH}|dkr|j4jJrutjM|jr+|j |tj|tNy|j |tj|tPy|j |tj|tRy|j |tj|tBycc}w)a-Returns True if we should not trace out_tensor. Args: op_id: Topological index of the op producing tensor. out_tensor: tf.Tensor report_handler: An instance of tensor_tracer_report.TTReportHandle. Returns: True if the tensor should not be traced, false otherwise. TzUSER_INCLUDED tensor %sFzUSER_EXCLUDED tensor %sr)*rrvariantresourcestringrinstrument_tensorrr _REASON_NON_NUMERIC_TENSOR consumersr_REASON_FEEDS_WHILELOOP_OPrWrr r$r rMrrrr[rrrrrr]r^r_r`ra_REASON_TENSOR_GET_TRACED_REASON_DYNAMIC_SHAPErr.trace_scalar_opsr_REASON_UNSAFE_SCALAR_REASON_SCALAR_GET_TRACED_REASON_SKIP_SCALAR)rr out_tensorrnon_numeric_tensor_typesconsumerranks r@ _skip_tensorzTensorTracer._skip_tensors #FNNFOO$*MM$3 433&& l))%1KLN !+!5!5!7.X""8, .&& l))%1KLN    /&& l))%1FGI  , , 2 2 . @    /&& l))%1FGI  , , 2 2 . @     ! 2 2 4    $ $  0 0  - -  0 0  0 0  0 0 )   ((  ++E3LM O((  ++E3HI K z D ax   * *  + +JMM :  * *,--e5JK M  * *,--e5NO Q((  ++E3FG I&& l))%1JKM m.sN"c| t|St|ttfs|g}g}|D]q}t|tj r|j |/t|tjr|j |jetd|zt|}t|} |s |S|j}|jDcgc]}|j} }| j|j| D]?} | |vstj!| r|j#| |j | Acc}w)zFReturns the set of ops in the execution path to compute given fetches.z-Given fetch:%s is neither a tensor nor an op.)r isinstancerJtupler Operationr tensor_libTensorrr7popinputsextendcontrol_inputsrradd) r operationsfetches op_fetchesfetchexecution_path_operationstraverse_stackhead_op tensor_input input_opsinput_ops r@!_filter_execution_path_operationsz.TensorTracer._filter_execution_path_operations sJ _ ge} - gJ# E3== )% eZ.. /%((#J!"# # #!$J*%N    %$""$g7>~~F| \\$**44 5C++5567 76 4rBctjttj|g}t |Dcgc]}|c}|j |<ycc}w)a}Creates a temporary cache with the given dimensions. Fills the self._temp_cache_var with num_traced_tensors tf.constant() ops that have shape of [num_signatures]. Args: num_traced_tensors: Int, denoting total number of traced tensors. num_signatures: Int, denoting the number of statistics collected per tensors. graph: TensorFlow graph. rr.N)rrr3rrrKr)rnum_traced_tensorsnum_signaturesr> init_valuer!s r@_create_temp_cachezTensorTracer._create_temp_cachebsQ%%&E,2NN-;,<>J""45#7 #7D#7s Ac|jtj|}|jt}tj }|j ||||}tjdt||r%tjjr tdtj||}|j} | r|j!r|j#r|j%t|| |nj|j't(|t|| g|j*j,tj.vr"|j1t(|t|| g|j*j,tj2tj4fvrN|j7|j8|j*|||j;|_|j*j>rt@jBjE|j*jF|j<jH|j*_#tjd|j*jF|jK|j*jF||_&|j*jNtPk7r1|jS|jL|j<|j*|S|j*j,tj.vr(|jU|j8|j*|||S)aWork needs to be done prior to TPU or CPU tracing. Args: graph: tf.graph ops_in_exec_path: Set of operations in the execution path. graph_summary_tag: the summary tag name for the given graph. Returns: An instance of tensor_tracer_report.TensorTraceOrder, containing list of tensors to be traced with their topological order information. Raises: RuntimeError: If opname filtering is incorrectly set. z#TensorTracer is tracing %d tensors.z)Verify ops being traced by tensor tracer.z%TensorTracer updating trace_dir to %s)+rJr%sort_tensors_and_opsrrTTReportHandlerFrrrr$r rMr7TensorTraceOrderrerhrjrPrBrrrr_r>rarcreate_report_protorrcruse_fingerprint_subdirrGrrr fingerprintrrreport_file_path_SKIP_REPORT_FILEwrite_report_proto create_report) rr>rgraph_summary_tagrArBrrCrrNs r@"_determine_trace_and_create_reportz/TensorTracer._determine_trace_and_create_reportss &;;EBK../HI)88:NBB%':NLN LL6N8KL-==CC D EE->>{?MO335N$779    N 3^UK // US%8.$I K    & &  2 2 4  9 9us>':N&K M ""..::'<<*== //4++-? t4468d    0 0%'WW\\    & &(:(:(F(F&H" <%%// 1 . @ @    $ $&7!9d    * *.? ?))$*A*A*.*<*N>N P     $ $  0 0 2$$T__d6F6F%79L N rBcn|jjtjtjfvSr')rrr$rarrs r@_create_host_callzTensorTracer._create_host_calls2    & &..::+< <._inspect_tensor..rBcy)Nzhas no NaNs or Infs.rrrBr@rrzNTensorTracer._inspect_summary_cache.._inspect_tensor..rcrB)rrr$r]rrgreater)r rs r@_inspect_tensorz._inspect_tensorsN    % %  0 0 1yy   VS ) $ *, ,  rBz2Inspect mode has no tensors in the cache to check.rcy)NzNaNs or Infs in the step!rrrBr@rrz5TensorTracer._inspect_summary_cache..rcrBcy)Nz1No numerical issues have been found for the step.rrrBr@rrz5TensorTracer._inspect_summary_cache..rcrB core:,step:-->zPrinting tensors for mode:%s...c |dSrrrps r@rrz5TensorTracer._inspect_summary_cache.. arBrsrrrr)rCrwarnrno_oprrr$r]rre reduce_sumrrrrrrcollect_summary_per_corerwrrFr1rr) rr replica_idstep_numrrrfstep_has_nan_or_infstep_error_messagestatsrrs ` r@_inspect_summary_cachez#TensorTracer._inspect_summary_caches   , , llGH  # ## ""&9&L&LL$,,X-@-@-GM%//  ! !!!%(,*=*=e*D FG +CE  00w C(E!043C3C3N3NNPew%1C043C3C3N3NNPe#)2288: #" G Y    2 2 ':sGXs il0C DF G  '8S il0C DF G G   ".; ==rBc|js%tjdtjSdd|dd|gg}t |j jdD]{\}}||df}|j|j|jj|jj|j} tj | g5t#j$| d } d d d tj  g5t'j(|t*j,} t'j.t'j0| | } t#j2| | } |j5| d d d tj  g5t#j$| d }d d d j7d d|dd|d|d |  g ~t9j:|}t'j<|}t?j>t'j@|tBjDjFfd dS#1swYcxYw#1swYxYw#1swYxYw)aGenerates a conditional print operation to log differences in tensor values. Args: cache: Tensor storing the trace results for the step. replica_id: Tensor storing the replica id of the running core. step_num: Step number. tensor_trace_order: TensorTraceOrder object holding tensorname to id map. Returns: The Op to flush the cache to file. z5TT history mode has no tensors in the cache to check.rirjrkrlc |dSrrros r@rrz5TensorTracer._inspect_history_cache..rprBrsrrNrrmc.tjddiS)Nrr)rr)rysr@rrz5TensorTracer._inspect_history_cache../s $$e:r:rBc*tjSr')rrrrrBr@rrz5TensorTracer._inspect_history_cache..0s &&(rB)$rCrrqrrrrwrrFr>rrr>r.rrr rHr assign_addrrrrrsubtractrrr1rrxrrrer$DELTA_THRESHOLDrM)rrrurvrdiffsrrtensor_to_writer old_value new_valuedeltaupdatednew_value_from_var diff_stackstep_maxrys @r@_inspect_history_cachez#TensorTracer._inspect_history_caches;  , , llJK  # ## Wj#w AE E"(2288: #"E Yil+oII    2 2 8 8    ' ' )?+@+@B  # #%6$7 8A(():C@ A  # #YK 0MM/6>>B  X..y)DE""#4i@ U   # #WI .J&112CSIJ ll S'8S ui);UDE)E0!&&u-J"":.H 99#6#F#F#L#LM:( **%AA JJs%IA5I #I,I I) ,I5 c~tj|jjrtjSy)Nr )r!is_remote_pathrrget_appendable_file_encodingrs r@rz TensorTracer._get_outfile_suffix2s.""4#3#3#=#=>  6 6 88 rBc >fd}jt|}jr|}n|j}rjj s%j |}j|d}tj||jtjtj}n(tj} ||j| }jr?tj |g5t#j$dj&cdddStj |g5t#j$t(|j*|j,} t/j0|| j&} tj | g5t#j$dj&cdddcdddS#1swYyxYw#1swYnxYw dddy#1swYyxYw)aEGenerates an Op that will flush the cache to file. Args: num_replicas: total number of replicas. on_tpu: if the graph is executed on TPU. tensor_trace_order: TensorTraceOrder object holding tensorname to id map. graph: TensorFlow graph. Returns: The Op to flush the cache to file. c"  fd}fd}i}|d||d<tdD]E}r0 jjstj|||<5|||||<Gt j |dS)z8Flushes the cache to a file corresponding to replica_id.cfd}|S)z2Generates a func that flushes the cache to a file.c dz} jjrVtjj jjt |z j z}t|z}ntj}t|z}g} jjr jdkDr td jjtj vr%|j# j% nu|j# j' | nOt) jD]3}|j#t+j,|ddd|fdd| 5t/j0|5t3j4d j6cdddS#1swYyxYw) zFlushes the cache to a file.rrz.Inspecting multi signatures are not supported.)rrurvr)rrurvrrrNrrr)rrrGrr_COMPACT_TRACE_FILE_PREFIXrrrr_REPLICA_ID_TAG inspect_tracererrr$r_rrrzrKrrr rHrrr) replica_strrr new_step_line print_opsrEr file_indexrurrvrs r@ _print_cachez[TensorTracer._generate_flush_cache_op.._flush_fun.._f.._print_cacheKs *+    ' '77<<(8(8(B(B(BD()+/+C+C+EFK2K?MJJM)K7-)    + +--/!3OPP**#668--!!+'); .=>--!!+'$1); .=>499;< 0{331+t -  /0 0 '' 2 .''*-- . . .s GG r)rrrrurrvrs` r@_fzETensorTracer._generate_flush_cache_op.._flush_fun.._fIs( .( .RrBc0tj|Sr')requal)rrus r@_eqzFTensorTracer._generate_flush_cache_op.._flush_fun.._eqvs~~j*55rBrrT) exclusive)rKrrtrrrrcase) rrurvrrflush_op_casesrErGon_tpurrs ``` r@ _flush_funz9TensorTracer._generate_flush_cache_op.._flush_funFs++Z6n!!unSVQ %)! $**CC$4#9#9.Q #%a5.Q ) # #Nd CCrBrNrL)rBrrjrMrrtmerge_caches_on_tpuaggregate_global_cacher&outside_compilationrridentityr(get_or_create_global_stepr rHrrrr3rr.rr) rrGrrr>rr cache_valflush_op global_step reset_value assign_ops ```` r@_generate_flush_cache_opz%TensorTracer._generate_flush_cache_op8sADF  3 3OU KE i++-i    6 6,,Y7 // :1=  44 i!1!1   ]DDF GIh";;=kIt'7'7Eh   # #XJ /*##A&))**  # #XJ /,!**+J1616> $$UK8;;  % %yk 2 ,%%a(++ , , ,, ** , , , ,,,s1G1A'H?G= H1G:=H HHc^|jstjd|Stj||Dcgc]}|j c}z5|j |jj|||}tj||gcdddScc}w#1swYyxYw)aFlushes the intermediate tensor values in the graph to the cache. Args: tensor_fetches: list of tensor results returned by the model_fn. op_fetches: list of ops that are returned by the model_fn, e.g., train_op. on_tpu: if the graph is executed on TPU. tensor_trace_order: TensorTraceOrder object holding tensorname to id map. graph: TensorFlow graph. Returns: An identical copy of tensor_fetches. z7No tensor values being traced. No flush cache op added.r2N) rCrrqr rHrrrrGrr+)rtensor_fetchesr6rrr>r flush_cache_ops r@_flush_tensor_values_cachez'TensorTracer._flush_tensor_values_caches  , , llLM  ! !*;I"J699"J#K LE44 // & &0BEKn  # #N4B3CE EE"JEEsB AB##B,c| tdt|ttfs|g}n |s tdg}|D];}t|tj r|j |/td|z|S)z>Check that tensor_fetches is not empty and have valid tensors.z8tensor_fetches provided to tensor_tracer cannot be None.z>tensor_fetches provided to tensor_tracer cannot be empty list.z&Given tensor_fetch:%s is not a tensor.)r7r*rJr+r-r.r)rrr5r7s r@_process_tensor_fetchesz$TensorTracer._process_tensor_fetchess ! "" ntUm 4&'n  ' ((GM E:,, -uCeKLL M NrBcF|gSt|ttfs|g}g}|D]|}t|tjr|j |/t|t jr|j |jetjd|z~|S)z%Check that op_fetches have valid ops.z3Ignoring the given op_fetch:%s, which is not an op.) r*rJr+r r,rr-r.rrr)rr6r5r7s r@_process_op_fetchesz TensorTracer._process_op_fetchess i j4- 0r\rs r@ _write_cachez8TensorTracer._prepare_host_call_fn.._write_cache\s(*k  & ,,k;L-M7:< 44    $ $%)+n  *N<7j   b 6<< RYY --&YYv~~6]]:v||<  $ $ &"&66#33>>7?9:!,,. "JC!!:: o%%++*=*=*?*Ba*G11%8e'')<)<)>(?@ "  $ $W]]#%+c14EE 0&! " " " " "& # #$5 66  " """s% CH=9H1 H=1H: 6H==Irrr')rrrr(rrrrrr rHr(rFr1r.r_TT_HOSTCALL_KEY)rprocessed_t_fetchesr6r>r\rrrr host_call_depscaches_to_write all_cachesr;cache_variablenew_cache_shapers` `` r@_prepare_host_call_fnz"TensorTracer._prepare_host_call_fnLs6 !!) 0 11?7B 99;K   [1# .DD;N"O699"OONO ! !. 1,11%8j(2(8(8(:, $*n#~33;;=>!!./B&+ # ,,#OF-9/+JD'()#P,,s$D(A1D--D6c|jSr')rrs r@host_call_deps_and_fnz"TensorTracer.host_call_deps_and_fns   rBc|jS)z#Returns the set of traced op names.)rrs r@get_traced_op_namesz TensorTracer.get_traced_op_namess   rBc '(d}jj}jj}|j _t jd||gtj|tj||j|}j|}||D cgc]} | jc} z} j|j| } t!|} j#|| | 't%|} g}t'| }|j)d|D]V}t+t-|j.D]1}|j.|}|j0}|'j2vr.j4j7|j0|j9}|Dcgc] }|| vs| }}|| v}|s|sj;|}|r|j=|j?|}r#|jAD]}|||||<jCrXjEr%'j2|}jG|||d}nʼn'j2|}jI|||}njKrGt-|dk7r tMdt'|jOd}jQ|}nK'fd }t-|dk7r tMdtStU|jO}|||}|r|j=j |s|r|jW||D]}|jY|4Y|j=j |rt[j\|| }jCsjKrɉjErVj_|}|j`vr tMd tcjdj`|dd |tf<jir<jk|||| sHjltn\(} (di| }!t[j\||!g }jltn=njjprމjltn\(} tf| vrd| vr| d}"| tf}#js|#d}$jjtsjw|$}$(fd}%tyjz|%|"j||$}&t[j\||&g }jltn=n!tdj||'|}j||Scc} wcc}w)ayCommong tracing function for both CPU and TPUs. The caller function should set device_type, num_replicas, num_replicas_per_host, num_hosts and replica_id before calling _trace_execution. Args: graph: the graph of Ops executed on the TPU. tensor_fetches: a (list,tuple,or a single object) of tensor fetches returned by model_fn given to session.run. Function must be provided with as least one tensor to fetch. op_fetches: A list of op fetches returned by model_fn given to session.run. op_fetches and tensor_fetches are used to determine the nodes that will be executed. Can be None. on_tpu: True if executing on TPU. Returns: tensor_fetches: an exact copy of tensor_fetches that has additional dependencies. Raises: RuntimeError: If tensor_fetches is None or empty. c|jjtjr$t j |tj S|jjtjs)|jjtjr$t j |tjS|S)z!Casts tensor to a supported type.) r__eq__rrrrint32bfloat16float16rr s r@_cast_unsupported_dtypesz?TensorTracer._trace_execution.._cast_unsupported_dtypessy   V\\ *}}VV\\22   V__ -1D1D ..2}}VV^^44 mrBr2c|jSr'rErs r@rrz/TensorTracer._trace_execution..s BGGrBrsNrz0Multiple stats are only allowed in compact mode.rchj|}rtj||S||S)z7Wraps the trace_fn with outside compilation if on TPUs.)rr&r)r out_tensor_nametensor_trace_fnrrrs r@tpu_wrap_trace_fnz8TensorTracer._trace_execution..tpu_wrap_trace_fn<sA"99/:LNO$88!6++%V,,rBrrstack_all_op_signaturesrurctjtj|dfdtjS)NrcdS)N)rrr1r)r tt_summary write_cachesr@rrzHTensorTracer._trace_execution..write_if_core_0..s+44.write_if_core_0s1YY..Q/H"(( **rB)rrurz9Outside compiled flush in only supported for summary mode)rrr>r)Brrrr_get_control_flow_contextrr track_usagerrrrrrr>get_operationsrAr]rrJsortrKrrrrrr3rr_set_control_flow_contextrkeysrhrjrrrgr7valuesrnextiterr_add_control_inputrr+r(rrrxrr_rrr$flush_summaries_with_outside_compilerrtrr&rrrrr))rr>rr6rr rrrr all_fetches exec_op_setr\tensor_fetch_set tracing_opssorted_exec_op_listrrEr$rrcopis_a_fetched_tensorrprocessed_tensorsrrtrace_opprocessed_out_tensorr consumer_oprArcache_write_oprr1tt_core_summaryrwrite_oprrs)` ` @@r@_trace_executionzTensorTracer._trace_executions6 !!,,J//--K!;;=D/J +DE"";/!!+z:66~F))*5J8KLf LLK889M9M9O9DFK*51@@ {-/./K{+!34!X5S_%W5!ZZ]  oo 0HH H  !!"''*((* %.DS 1CSD D),<<$7 "&"C"CB"G "  ) )*A B!:::F ,113 .i+C!),,. i ( .  ( ( * ! ! #*BB;OI  0 01B16 8H*BB;OI::9;L;@BH $ $ & " #q ( '( (!%&7&>&>&@!A!!D **+?@( -" #q ( '( ( "&d+<+C+C+E&F!G &';[I( "  ) )$*?*? @    x ( ' 5k  * *8 4 5kW5X5x ##D$9$9:-223FBMO $$&$*A*A*C   88? ,, ,CD D+:+@+@   'a6O,Q(   ! ""#6 E#4 6)-););2B!D   !12    B B)-););  0 01D FFW M>Es3W? W Wct|tjst|tjrt j d|S|tjvrt j d|Stjj|tj|_ t|j_||j_||j_||j_|jj |jj"d|j_|jj$E||jj"z||jj"zdkDz|j_|jj&r+t)j*||jj&d|j-5|j/|j1|||d}ddd|jj&r+t)j*||jj&d |S#1swYLxYw) aTraces the tensors generated by TPU Ops in a TF graph. Args: graph: the graph of Ops executed on the TPU. tensor_fetches: a (list,tuple,or a single object) of tensor fetches returned by model_fn given to session.run. Function must be provided with as least one tensor to fetch. op_fetches: A list of op fetches returned by model_fn given to session.run. op_fetches and tensor_fetches are used to determine the nodes that will be executed. Can be None. num_replicas: number of replicas used on the TPU. num_replicas_per_host: number of replicas per TPU host. num_hosts: total number of TPU hosts. Returns: tensor_fetches: an exact copy of tensor_fetches that has additional dependencies. HTensor Tracer is not supported for tracing FuncGraphs. Ignoring tracing.GGraph is already rewritten with tensor tracer, ignoring multiple calls.Nrgraph_before_tt.pbtxtTrgraph_after_tt.pbtxt)r*r FuncGraphr _FuncGraphrrr_traced_graphsr3r$rrrrrrGnum_replicas_per_host num_hostsgraph_dump_pathr write_graphr4rLr1)rr>rr6rGr<r=s r@ trace_tpuzTensorTracer.trace_tpus0%--.* x""3$ oo*+  +++ oo() !!%%e,*779D"2DOO#/DOO ,ADOO) )DOO ##/  . . 601-  " " * DOOAA A DOOAA AA E G ! ''5$"2"2"B"B24    : ##%,,UNJ48-:n: ''5$"2"2"B"B13 ::s &IIctt|tjst|tjrt j d|S|tjvrt j d|Stjj|tj|_ t|j_d|j_d|j_d|j_d|_|jj(r+t+j,||jj(d|j/5|j1|||d}ddd|jj(r+t+j,||jj(d |S#1swYLxYw) a{Traces the tensors generated by CPU Ops in a TF graph. Args: graph: the graph of Ops executed on the CPU. tensor_fetches: a (list,tuple,or a single object) of tensor fetches returned by model_fn given to session.run. Function must be provided with as least one tensor to fetch. op_fetches: A list of op fetches returned by model_fn given to session.run. op_fetches and tensor_fetches are used to determine the nodes that will be executed. Can be None. Returns: tensor_fetches: an exact copy of tensor_fetches that has additional dependencies. r3r4rrr6Fr7Nr8)r*rr9rr:rrrr;r3r$rrrrrrGr<r=rr>r r?r4r1)rr>rr6s r@ trace_cpuzTensorTracer.trace_cpusf %--.* x""3$ oo*+  +++ oo() !!%%e,*779D"2DOO#$DOO ,-DOO) !DOOD ''5$"2"2"B"B24    ;,,UNJ49-;n; ''5$"2"2"B"B13  ;;s F..F7)NT)NNNNr')F__name__ __module__ __qualname____doc__rr; staticmethodrrrrrrrrrrr rrrr%r(rrr>rBrLrOrWr[rcrerjrhrgr}rrrrrrrrrr(r>rFrJrPr]r_rzrrrrrrrrrrrrrr1r@rBrrBr@rrs&"H5.  ??  H H22??$ , ,$@%% !*C #=(8<7=~~ /8d046>>&'P #1, (".@ 85.@( :I'VQ5f* DLM^%%NB7 7"?B< C=J2*h p,dE4&& #"F,+@ZiKV! #'"_FF !&* <|-rBrr')wrFrr8rrGos.pathrnumpyrtensorflow.core.frameworkrtensorflow.python.eagerrtensorflow.python.frameworkrrrrr r r r-r tensorflow.python.lib.iortensorflow.python.opsrrrrrrrrrrrrrrrrrtensorflow.python.platformrr r!r"rtensorflow.python.summaryr#tensorflow.python.tpur$r%r&tensorflow.python.tpu.opsr'tensorflow.python.trainingr(rrrrrrrr r!r#rr rr"rr rr rrrrrrrr3r8r@rrYTT_SUMMARY_NORMrTT_SUMMARY_MAXrTT_SUMMARY_MAX_ABSrTT_SUMMARY_MINrTT_SUMMARY_MEANrTT_SUMMARY_VARrTT_SUMMARY_SIZErTT_SUMMARY_SPARSITYrrrrrr BoolGaugerrArNrrrrrrrBr@r]s/  1.3.200+<3,+1&323.*,-*)+,;00,3<6561-4 8*8562)>62++.24<D!5#B -"&0' !&66%44)<<%44&66%44&66*>>)-,( :   B 5x A \?~* ZAH<>||rB