AVhDdZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZdd lmZdd lmZeZd ZdZdZ d!dZ Gdde!Z"edgGdde"Z#Gdde"Z$eddZ%edGdde$Z&dZ'd"d Z(y)#zUtilities to run benchmarks.N)app) config_pb2)rewriter_config_pb2) test_log_pb2)timeline)ops)gfile) tf_logging) tf_inspect) tf_exportTEST_REPORT_FILE_PREFIXTF_OVERRIDE_GLOBAL_THREADPOOLc|j}|j||}tj||j||j |j S)z>MM ++c  tjd|||nd||nd||nd||nd|r t|nd|r t|ndtj}|j j }||_|||_|||_ |||_ |||_ |t|ts td|jD]Q\} } t| t j"r| |j$| _5t| |j$| _S|t|t*s td|D]} d| vr tdd | vr td |j,j } | d| _| d | _d | vr| d | j0_d | vsl| d | j2_t4j6j9t:d} | t=t|y|j?}|jAd d}| |}tCjD|rtGd|ztCjH|d5}|jK|dddy#1swYyxYw)aMethod for recording a benchmark directly. Args: name: The BenchmarkEntry name. iters: (optional) How many iterations were run cpu_time: (optional) Total cpu time in seconds wall_time: (optional) Total wall time in seconds throughput: (optional) Throughput (in MB/s) extras: (optional) Dict mapping string keys to additional benchmark info. metrics: (optional) A list of dict representing metrics generated by the benchmark. Each dict should contain keys 'name' and'value'. A dict can optionally contain keys 'min_value' and 'max_value'. Raises: TypeError: if extras is not a dict. IOError: if the benchmark output file already exists. z]Benchmark [%s] iters: %d, wall_time: %g, cpu_time: %g,throughput: %g, extras: %s, metrics: %sNNonezextras must be a dictzmetrics must be a listrzmetric must has a 'name' fieldvaluezmetric must has a 'value' field min_value max_value/__zFile already exists: %swb)&logginginfostrrBenchmarkEntriesentryaddriterscpu_time wall_time throughput isinstancedict TypeErroritemsnumbersNumberextras double_value string_valuelistmetricsr$r%r&osenvirongetTEST_REPORTER_TEST_ENVprintSerializeToStringrr ExistsIOErrorGFilewrite)rr0r1r2r3r:r>entriesr.kvmetric metric_entrytest_envserialized_entry mangled_name output_pathouts r_global_report_benchmarkrS;st( ,,9:>)ur%1yr#/xR'3z$s6{&&s7|F4  ) ) +' --   %%* EK ENEO!E  fd # - ..,,..A Aw~~ &'( Q$'*1v Q$ .   gt $ . // ; v 899  9::]]&&(l .l!'?l  '-k': $  '-k': $ ;ZZ^^2D 9(  #g, ..0c4(,"L1+ \\+ +k9 :: {{;% II   s I99JceZdZdZdZy)_BenchmarkRegistrarzAThe Benchmark class registrar. Used by abstract Benchmark class.ctj||||}|jstj ||SN)type__new__ is_abstractGLOBAL_BENCHMARK_REGISTRYr/)mcsclsnamebaseattrsnewclasss rrYz_BenchmarkRegistrar.__new__s5||C$6H    !##H- Or N)__name__ __module__ __qualname____doc__rYr rrUrUs Ir rUz(__internal__.test.ParameterizedBenchmark)v1c"eZdZdZfdZxZS)ParameterizedBenchmarkaDMetaclass to generate parameterized benchmarks. Use this class as a metaclass and override the `_benchmark_parameters` to generate multiple benchmark test cases. For example: class FooBenchmark(metaclass=tf.test.ParameterizedBenchmark, tf.test.Benchmark): # The `_benchmark_parameters` is expected to be a list with test cases. # Each of the test case is a tuple, with the first time to be test case # name, followed by any number of the parameters needed for the test case. _benchmark_parameters = [ ('case_1', Foo, 1, 'one'), ('case_2', Bar, 2, 'two'), ] def benchmark_test(self, target_class, int_param, string_param): # benchmark test body The example above will generate two benchmark test cases: "benchmark_test__case_1" and "benchmark_test__case_2". cR|d}d}|jjD]l}|jds||}||=|D]K}|d} |dd} |dz| z} | |vrtdj | ||| } t | d| || <Mnt |||||S)N_benchmark_parameterscfdS)Nc|gSrWre)selforiginal_benchmarkparamss rzSParameterizedBenchmark.__new__..create_benchmark_function..s,T;F;r re)rnros``rcreate_benchmark_functionzAParameterizedBenchmark.__new__..create_benchmark_functions ;;r benchmarkrr(z#Benchmark named {} already defined.)copykeys startswith ExceptionformatrsuperrY)r\r]r^r_param_config_listrqrrn param_configtest_name_suffixrobenchmark_namerr __class__s rrYzParameterizedBenchmark.__new__s56< !!#O __[ ) ; ++ O,'?ab!'77 U "3::>JL L..@&I !1A~ Nn OO( 7?3u 55r )rarbrcrdrY __classcell__r~s@rrhrhs,66r rhc>eZdZdZedZddZ ddZy) BenchmarkaAbstract class that provides helper functions for running benchmarks. Any class subclassing this one is immediately registered in the global benchmark registry. Only methods whose names start with the word "benchmark" will be run during benchmarking. c:t|jdkS)Nlenmroclss rrZzBenchmark.is_abstracts swwy>Q r Nc"tj}d}d}|dddD]<}|dj}|jdd}t |t s5|}|d}n| t d|xs|}t|j}|d|}|S)z?Returns full name of class and method calling report_benchmark.Nr"rrmz,Unable to determine calling Benchmark class..) r stackf_localsrAr4r ValueErrorrXra) rmoverwrite_namer calling_classrframerf_self class_names r _get_namezBenchmark._get_names    EM Dttq""h||FD)f FI & Qx  E FF  !TDm$--J $ 'D Kr c L|j|}t|||||||y)adReport a benchmark. Args: iters: (optional) How many iterations were run cpu_time: (optional) Median or mean cpu time in seconds. wall_time: (optional) Median or mean wall time in seconds. throughput: (optional) Throughput (in MB/s) extras: (optional) Dict mapping string keys to additional benchmark info. Values may be either floats or values that are convertible to strings. name: (optional) Override the BenchmarkEntry name with `name`. Otherwise it is inferred from the top-level method name. metrics: (optional) A list of dict, where each dict has the keys below name (required), string, metric name value (required), double, metric value min_value (optional), double, minimum acceptable metric value max_value (optional), double, maximum acceptable metric value )r)rr0r1r2r3r:r>N)rrS)rmr0r1r2r3r:rr>s rreport_benchmarkzBenchmark.report_benchmarks,4 >>> .D Yfg?r rW)NNNNNNN)rarbrcrd classmethodrZrrrer rrrs<4   ?r r) metaclassztest.benchmark_configctj}tjj|j j _|S)zReturns a tf.compat.v1.ConfigProto for disabling the dependency optimizer. Returns: A TensorFlow ConfigProto object. )r ConfigProtorRewriterConfigOFF graph_optionsrewrite_optionsdependency_optimization)configs rbenchmark_configrs:  ! ! #&((,, &&> -r ztest.BenchmarkcPeZdZdZfdZedZ ddZdZxZ S)TensorFlowBenchmarkz?Abstract class that provides helpers for TensorFlow benchmarks.cPdtjt<t|y)N1)r?r@OVERRIDE_GLOBAL_THREADPOOLry__init__)rmr~s rrzTensorFlowBenchmark.__init__#s.1BJJ)* Gr c:t|jdkS)Nrrrs rrZzTensorFlowBenchmark.is_abstract)s swwy>Q r c lt|D]} |j||dg|z} t|D]G} tj}|j||tj}||z }|| | <I| | ni} i}|s|rtjtjj }tj }|j||||tj|j}|r|j|d<|rH|jd}|j}|jD]\}}|j| d|z<d }d }|| }||| || |z d }|jdi||| \}}||d <||d <|dj!||S)aARun an op or tensor in the given session. Report the results. Args: sess: `Session` object to use for timing. op_or_tensor: `Operation` or `Tensor` to benchmark. feed_dict: A `dict` of values to feed for each op iteration (see the `feed_dict` parameter of `Session.run`). burn_iters: Number of burn-in iterations to run. min_iters: Minimum number of iterations to use for timing. store_trace: Boolean, whether to run an extra untimed iteration and store the trace of iteration in returned extras. The trace will be stored as a string in Google Chrome trace format in the extras field "full_trace_chrome_format". Note that trace will not be stored in test_log_pb2.TestResults proto. store_memory_usage: Boolean, whether to run an extra untimed iteration, calculate memory usage, and store that in extras fields. name: (optional) Override the BenchmarkEntry name with `name`. Otherwise it is inferred from the top-level method name. extras: (optional) Dict mapping string keys to additional benchmark info. Values may be either floats or values that are convertible to strings. mbs: (optional) The number of megabytes moved by this op, used to calculate the ops throughput. Returns: A `dict` containing the key-value pairs that were passed to `report_benchmark`. If `store_trace` option is used, then `full_chrome_trace_format` will be included in return dictionary even though it is not passed to `report_benchmark` with `extras`. ) feed_dictN) trace_level)roptions run_metadatafull_trace_chrome_formatT) show_memoryzallocator_maximum_num_bytes_%scf|syt|}t|}|dz }||dz||dzzdz S)Nr"rsrg@)sortedr)xsllm1s r_medianz5TensorFlowBenchmark.run_op_benchmark.._medianwsC  )a a&a Ec1g#q& !S ((r c|syt|}t||z }|dk(r|dfSt|Dcgc] }||z ||z zc}|dz z }|tj|fScc}w)N)r"r"rsr")rsummathsqrt)rrmeanevariances r_mean_and_stdevz=TensorFlowBenchmark.run_op_benchmark.._mean_and_stdevsr  a&a VaZd aRxq9!q4xAH-9:a!eDh 499X& &&:sA%)r0r2r:rr3wall_time_meanwall_time_stdevr:re)rangeruntimer RunOptions FULL_TRACE RunMetadatarTimeline step_statsgenerate_chrome_trace_formatanalyze_step_statsallocator_maximumsr7 num_bytesrupdate)rmsess op_or_tensorr burn_iters min_iters store_tracestore_memory_usagerr:mbs_deltasi start_timeend_timedeltaunreported_extras run_optionsrtlstep_stats_analysisrrJrKrr median_deltabenchmark_values mean_delta stdev_deltas rrun_op_benchmarkz$TensorFlowBenchmark.run_op_benchmark/sP: 2 hh|yh12Vi F 9 99;j hh|yh1h#efQi )VrF()) ++668k++-l hh|y"?   \44 5b   + + - 45  333E0CC&,,. EDAq9:&1A5 6 E)'6?L!L( D-,--f5J *4&'+6'(X%%&78 r cptjxs|j}|j|S)zEvaluates tensors and returns numpy values. Args: tensors: A Tensor or a nested list/tuple of Tensors. Returns: tensors numpy values. )rget_default_sessioncached_sessionr)rmtensorsrs revaluatezTensorFlowBenchmark.evaluates/  " " $ =(;(;(=D 88G r )Nr FTNNr) rarbrcrdrrrZrrrrs@rrrsEG "&"#!##(*. "iV r rctt}g}|D]}|jd|j}t |}d}|D]~}|j dst ||}t|s-|d|} |dk(stj|| sP|j| |xs|}t ||} | |stdj|y)aRun benchmarks that match regex `regex`. This function goes through the global benchmark registry, and matches benchmark class and method names of the form `module.name.BenchmarkClass.benchmarkMethod` to the given regex. If a method matches, it is run. Args: regex: The string regular expression to match Benchmark classes against. Raises: ValueError: If no benchmarks were selected by the input regex. rNrrallz'No benchmarks matched the pattern: '{}') r=r[rbradirrvgetattrcallableresearchappendrrx) regexregistryselected_benchmarksrrr}r_benchmark_instanceattrcandidate_benchmark_fnfull_benchmark_nameinstance_benchmark_fns r_run_benchmarksrs+ ,( i ) 4 4i6H6HIN  NE  __[ )&y$7 , -'5t< %299U,?@""#67/>9; '(:D A  ,  >EEeL MM r c4|tj}|Dcgc]&}|jds|jdr|(}}|rF|j|d|dj ddt j fd|y|ycc}w) zRun benchmarks as declared in argv. Args: true_main: True main function to run if benchmarks are not requested. argv: the command line arguments (if None, uses sys.argv). Nz--benchmark_filter=z-benchmark_filter=r=rsctSrW)r)rrs rrpz!benchmarks_main..s oe,r )argv)sysrrvremovesplitrr) true_mainrarg found_argrs @rbenchmarks_mainrs \ 88D - . , - ) KK !  aL  s #A &EGG ,48 Ks+B)NNNNNNrW))rdrr8r?rrrrabslrtensorflow.core.protobufrrtensorflow.core.utilrtensorflow.python.clientrtensorflow.python.frameworkrtensorflow.python.platformr r r*tensorflow.python.utilr tensorflow.python.util.tf_exportr setr[rBrrrSrXrUrhrrrrrrer rrs #  /8--+,<-6  E3=+04*.N b$ 5"=16016>16hD?-D?N "# $  D)DDN)NXr