AVh4 |dZddlZddlmZddlZddlZddlZddlZddlZ ddl Z ddl m Z ddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl#m%Z%ddl&m'Z(ddl)m*Z*ddl)m+Z+ddl)m,Z,ddl)m-Z-ddl)m.Z.ddl)m/Z/ddl0m1Z1ddl0m2Z2ddl0m3Z3ddl4m5Z5dd l6m7Z7dd!l6m8Z8dd"l9m:Z:dd#l;me:d&e=d'Z? e>jd(ZBd)ZCGd*d+eDZEe?jr$ejd,d-drejd.z ZInd/ZId0ZJd1ZKd2ZLd3ZMd4ZNeeJfd?ZUe7jdd@ dVdAZWdBZXGdCdDeDZYdEZZdFZ[GdGdHe3jZ]dIZ^dWdJZ_dKZ`dLZadMZbdNZce>" 88G$58 66 (c\t|tjr|jdS|S)z&Decode s if it is a sequence of bytes.r()r*r+ binary_typedecoder.s r0 _to_stringr6[s&4##$ 88G  (r2c(eZdZdZdZdZedZy)TrtPrecisionModeFP32FP16INT8ctjtjtjg}||Dcgc]}|j c}zScc}wN)r8r9r:r;lower) precisionsps r0supported_precision_modesz*TrtPrecisionMode.supported_precision_modesgsG /446F6K6KJ J7q7 777sAN)__name__ __module__ __qualname__r9r:r; staticmethodrAr2r0r8r8bs# $ $ $88r2r8ii@RangeOptimalz Range+OptimalImplicitBatchModeCompatiblec.ttttgSr=)PROFILE_STRATEGY_RANGEPROFILE_STRATEGY_OPTIMALPROFILE_STRATEGY_RANGE_OPTIMAL/PROFILE_STRATEGY_IMPLICIT_BATCH_MODE_COMPATIBLErFr2r0supported_profile_strategiesrQs6$5 r2z&experimental.tensorrt.ConversionParams)v1cDeZdZdZeej ddddffd ZxZS)TrtConversionParamsaParameters that are used for TF-TRT conversion. Fields: max_workspace_size_bytes: the maximum GPU temporary memory that the TRT engine can use at execution time. This corresponds to the 'workspaceSize' parameter of nvinfer1::IBuilder::setMaxWorkspaceSize(). precision_mode: one of the strings in TrtPrecisionMode.supported_precision_modes(). minimum_segment_size: the minimum number of nodes required for a subgraph to be replaced by TRTEngineOp. maximum_cached_engines: max number of cached TRT engines for dynamic TRT ops. Created TRT engines for a dynamic dimension are cached. If the number of cached engines is already at max but none of them supports the input shapes, the TRTEngineOp will fall back to run the original TF subgraph that corresponds to the TRTEngineOp. use_calibration: this argument is ignored if precision_mode is not INT8. If set to True, a calibration graph will be created to calibrate the missing ranges. The calibration graph must be converted to an inference graph by running calibration with calibrate(). If set to False, quantization nodes will be expected for every tensor in the graph (excluding those which will be fused). If a range is missing, an error will occur. Please note that accuracy may be negatively affected if there is a mismatch between which tensors TRT quantizes and which tensors were trained with fake quantization. allow_build_at_runtime: whether to allow building TensorRT engines during runtime if no prebuilt TensorRT engine can be found that can handle the given inputs during runtime, then a new TensorRT engine is built at runtime if allow_build_at_runtime=True, and otherwise native TF is used. Tc 6tt| |||||||Sr=)superrT__new__)clsmax_workspace_size_bytesprecision_modeminimum_segment_sizemaximum_cached_enginesuse_calibrationallow_build_at_runtime __class__s r0rYzTrtConversionParams.__new__s3 $ !#;^24J-/EGGr2) rBrCrD__doc__$DEFAULT_TRT_MAX_WORKSPACE_SIZE_BYTESr8r9rY __classcell__ras@r0rTrTs/ >(L-22#$%&"%) G Gr2rTr[r\r]r^r_r` TRTEngineOpFctj}|j|vr%tdj |j||j dkr|j dk7r tdyy)a8Validate the provided TrtConversionParams. Args: conversion_params: a TrtConversionParams instance. is_v2: whether we're getting a RewriterConfig for TF 2.0. Raises: TypeError: if any of the parameters are of unexpected type. ValueError: if any of the parameters are of unexpected value. z;precision mode '{}' is not supported.It should be one of {}rzQminimum segment size should be positive or -1 (to disable main graph conversion).N)r8rAr\ ValueErrorformatr])conversion_paramsis_v2rAs r0_check_conversion_paramsrns/HHJ%%-FF  "#)6*;*J*J*C$E FF,,1,,2 ; <<32r2cftjs tjdt dt j dk(rtjdtj}tj}tjdt|tjdt|d}tjdd d s |d |tjdd d s |d ||d |d k7stj|sHtjd tj|tj|t d ||k7r>tjdtj|tj|yy)zyCheck compatibility of TensorRT version. Raises: RuntimeError: if the TensorRT library version is incompatible. zVTensorflow needs to be built with TensorRT support enabled to allow TF-TRT to operate.z4Tensorflow has not been built with TensorRT support.WindowszWindows support is provided experimentally. No guarantee is made regarding functionality or engineering support. Use at your own risk.zLinked TensorRT version: %szLoaded TensorRT version: %sc|dvs Jd|ztjdj|tj|t d|z)N)linkedloadedzMIncorrect value received for version_type: %s. Accepted: ['linked', 'loaded']zzThe {version_type} version of TensorRT: `{trt_version}` has now been removed. Please upgrade to TensorRT 7 or more recent. version_type trt_versionz!Incompatible %s TensorRT versions)loggingerrorrk trt_utilsversion_tuple_to_string RuntimeErrorrts r0raise_trt_version_deprecatedzF_check_trt_version_compatibility..raise_trt_version_deprecatedsw  $ #$$   MM EEKV%!99+FFLFHI :\I JJr2rrrrsa6Loaded TensorRT %s but linked TensorFlow against TensorRT %s. A few requirements must be met: -It is required to use the same major version of TensorRT during compilation and runtime. -TensorRT does not support forward compatibility. The loaded version has to be equal or more recent than the linked version.z#Incompatible TensorRT major versionzLoaded TensorRT %s and linked TensorFlow against TensorRT %s. This is supported because TensorRT minor/patch upgrades are backward compatible.N)r&is_tensorrt_enabledrwrxr{platformsystemwarnget_linked_tensorrt_versionget_loaded_tensorrt_versioninfostrry(is_linked_tensorrt_version_greater_equal(is_loaded_tensorrt_version_greater_equalrz)linked_versionloaded_versionr|s r0 _check_trt_version_compatibilityrso  - - / MM  M NN __)# LL PQ$??A.#??A. ,,,c..AB ,,,c..AB K  ; ;Aq! D >:  ; ;Aq! D >:Q>!,,  < -disable_non_trt_optimizers_in_rewriter_config) rlrrrmdisable_non_trt_optimizersrrrewriter_config_with_trt optimizers r0_get_tensorrt_rewriter_configrs2,E: }( C DD =( : ;;}.2L = >>}Zc> M OO0??A(-$ 88AqAB88AqA7 $''..0 ((,,4&88<<>) #> /<)/*,$5$L$L  !2@I,-/4F)./1 4= 5"I./1   ;; " "!r2zBYou shouldn't need a rewriter_config with the current TF-TRT APIs.c t|||||Sr=)r)rlrrrmrs r0get_tensorrt_rewriter_configrus  ''8-'5u'A CCr2c*|jddS)N/ri)splitrs r0_get_canonical_engine_namers C r2c zeZdZdZddddddeej ddddf dZdZd Z d Z d Z d Z d Z ddZdZy)TrtGraphConvertera;A converter for TF-TRT transformation for TF 1.x GraphDef/SavedModels. To run the conversion without quantization calibration (e.g. for FP32/FP16 precision modes): ```python converter = TrtGraphConverter( input_saved_model_dir="my_dir", precision_mode=TrtPrecisionMode.FP16) converted_graph_def = converter.convert() converter.save(output_saved_model_dir) ``` To run the conversion with quantization calibration: ```python converter = TrtGraphConverter( input_saved_model_dir="my_dir", precision_mode=TrtPrecisionMode.INT8) converter.convert() # Run calibration 10 times. converted_graph_def = converter.calibrate( fetch_names=['output:0'], num_runs=10, feed_dict_fn=lambda: {'input:0': my_next_data()}) converter.save(output_saved_model_dir) ``` NrVrUFTc tjr td|r |r td|s |s tdt ||_||_||_d|_d|_ |xstjg|_ |xstj|_d|_d|_|t$j&k(xs!|t$j&j)k(xr| |_|j*r| st-j.dd} | |_| rd|_|8t-j.dn"t5|t6s td ||_t9||| | | d |_t=|j:d|_y) a Initializes the converter. Args: input_saved_model_dir: the directory to load the SavedModel which contains the input graph to transforms. Used only when input_graph_def is None. input_saved_model_tags: list of tags to load the SavedModel. input_saved_model_signature_key: the key of the signature to optimize the graph for. input_graph_def: a GraphDef object containing a model to be transformed. If set to None, the graph will be read from the SavedModel loaded from input_saved_model_dir. nodes_denylist: list of node names to prevent the converter from touching. max_batch_size: max size for the input batch. max_workspace_size_bytes: the maximum GPU temporary memory which the TRT engine can use at execution time. This corresponds to the 'workspaceSize' parameter of nvinfer1::IBuilder::setMaxWorkspaceSize(). precision_mode: one of TrtPrecisionMode.supported_precision_modes(). minimum_segment_size: the minimum number of nodes required for a subgraph to be replaced by TRTEngineOp. is_dynamic_op: whether to generate dynamic TRT ops which will build the TRT network and engine at run time. maximum_cached_engines: max number of cached TRT engines in dynamic TRT ops. If the number of cached engines is already at max but none of them can serve the input, the TRTEngineOp will fall back to run the TF function based on which the TRTEngineOp is created. use_calibration: this argument is ignored if precision_mode is not INT8. If set to True, a calibration graph will be created to calibrate the missing ranges. The calibration graph must be converted to an inference graph by running calibration with calibrate(). If set to False, quantization nodes will be expected for every tensor in the graph (excluding those which will be fused). If a range is missing, an error will occur. Please note that accuracy may be negatively affected if there is a mismatch between which tensors TRT quantizes and which tensors were trained with fake quantization. Raises: ValueError: if the combination of the parameters is invalid. RuntimeError: if this class is used in TF 2.0. z8Please use tf.experimental.tensorrt.Converter in TF 2.0.zACan only specify one of input_graph_def and input_saved_model_dirz=Must specify one of input_graph_def and input_saved_model_dirFNzrINT8 precision mode with calibration is supported with dynamic TRT ops only. Disregarding is_dynamic_op parameter.Tz6When is_dynamic_op==True max_batch_size should be Nonez=When is_dynamic_op==False max_batch_size should be an integerrf) r executing_eagerlyr{rjr_input_graph_def_nodes_denylist_input_saved_model_dir _converted_grappler_meta_graph_defrSERVING_input_saved_model_tagsr!DEFAULT_SERVING_SIGNATURE_DEF_KEY _input_saved_model_signature_key_calibration_graph_calibration_data_collectedr8r;r>_need_calibrationrwr_is_dynamic_op_max_batch_sizer*rrT_conversion_paramsrn%_test_only_disable_non_trt_optimizers) selfinput_saved_model_dirinput_saved_model_tagsinput_saved_model_signature_keyinput_graph_defnodes_denylistrr[r\r]rr^r_s r0__init__zTrtGraphConverter.__init__sh  "  D FF0  M OO #8 / 00$&+D)D"7DDO$(D! 9=#8#8"9   ( >== ) #D',D$ ,11 1 ; ,11779 9 P@O  m ll HIm'D!d  # MN  ,&' '+d1!9%15'# %DT44516D.r2cBtj}t|j|j|j |j d}|jjj|tj||jd|_ d|_y)z9Run Grappler's OptimizeGraph() tool to convert the graph.T)rlrrrrtf_graphgraph_idN)r ConfigProtorrrrr graph_optionsrewrite_optionsCopyFromr OptimizeGraphr_converted_graph_defr)rgrappler_session_configcustom_rewriter_configs r0_run_conversionz!TrtGraphConverter._run_conversions)446:11))++#'#M#M ! ))99BB !- : : %%!DDOr2cP|jr|jjd}|jj}|jD][}t |t jr%|jt|jB|jt|]yy)Ntrain_op) rrcollection_def node_listvaluer*rTensorappendr1r)rrdenylistrs r0_add_nodes_denylistz%TrtGraphConverter._add_nodes_denylist1s 44CCJOn))//h##(! a ' //)AFF+ , //)A, ' (r2cPtj}|j5tj|j ddddt j|jd||_ |j|jy#1swYVxYw)zConvert the input GraphDef.rNT add_shapes graph_defgraph) rGraph as_defaultrimport_graph_defrr export_meta_graph as_graph_defrrr)rrs r0_convert_graph_defz$TrtGraphConverter._convert_graph_def;s IIKE    @ 5 5B?@$)$;$;$$$5U%DD! @@s "BB%ctjjtjjtjjtjj gz}|Dcgc] }||vs| c}Scc}wr=)r GraphKeys_VARIABLE_COLLECTIONSTRAIN_OP WHILE_CONTEXT COND_CONTEXT)rcollection_keyscollections_to_removekeys r0_collections_to_keepz&TrtGraphConverter._collections_to_keepFsd ++ MM " "CMM$?$? MM & &/   + OCc9N.NC OO Os - A>7A>ctj}tj|5}t j ||j |j}|j|j}d}||jj||j}|j|jD]j}|jj!|D]J}t#|tj$s|j'|j(j+ddLlt-j.||jj1dt3|} t5j6|_|j8j:j=| |j|jD]7}|j8j|j=|j|9|j?|j8j@j=|j@|j8j|jj=|ddd|jCy#1swYxYw)zConvert the input SavedModel.)rcj|Dchc]#}||jjdd%c}Scc}w)z%Get the node names from a TensorInfo.:r)rr) tensor_infors r0 _gather_namesz=TrtGraphConverter._convert_saved_model.._gather_names[s1?JK C %%++C03KKKs(0rrTrN)"rrrSessionrrrr signature_defrinputsunionoutputsrrrget_collectionr* Operationrrrr convert_variables_to_constantsrlistr MetaGraphDefrrrr meta_info_defr) rrsessinput_meta_graph_definput_signature_defroutput_node_namescollection_keyopfrozen_graph_defs r0_convert_saved_modelz&TrtGraphConverter._convert_saved_modelRs8 IIKE u %)O#[[t/K/K)-)D)DF0>>  / /1L ((;(B(BCII +33 46!55  - -/9.**++N; 9B CMM *  ! !"''--"4Q"7 8 99.LL  ''4'8  !#'5&A&A&Cd# ##--667GH!55  - -/A. %%44^DMM / / ? AA    ##11::  , ,. ##11  / /119:M1NS)OX Y)O)OsCJ;E/JJ c|jrJ|jr|j|jS|j|jS)zVRun the TF-TRT conversion. Returns: The converted GraphDef for TF 1.x. )rrrrrrs r0convertzTrtGraphConverter.convertsN      $ $$ !  $ $$r2c V|jsJ|jsJ|jrJ|r|s|s |s td|r]|j D]E\}}t |t s tdt |tjr} |jj@D]}|jBtDk(s|jF| vrR|jjG|jF5tHjK| }ddd| |jF<| j7| |jF| tM|jNi }||jPd _) dddd|_ddd|jS#1swYxYw#1swYxYw#1swY=xYw#1swY|jSxYw) a#Run the calibration and return the calibrated GraphDef. Args: fetch_names: a list of output tensor name to fetch during calibration. num_runs: number of runs of the graph during calibration. feed_dict_fn: a function that returns a dictionary mapping input names (as strings) in the GraphDef to be calibrated to values (e.g. Python list, numpy arrays, etc). One and only one of `feed_dict_fn` and `input_map_fn` should be specified. input_map_fn: a function that returns a dictionary mapping input names (as strings) in the GraphDef to be calibrated to Tensor objects. The values of the named input tensors in the GraphDef to be calibrated will be re-mapped to the respective `Tensor` values during calibration. One and only one of `feed_dict_fn` and `input_map_fn` should be specified. Raises: ValueError: if the input combination is invalid. RuntimeError: if this method is called in eager mode. Returns: The GraphDef after the calibration. zAShould specify one and only one of feed_dict_fn and input_map_fn.z(Keys of input_map_fn must be of type strz0Values of input_map_fn must be of type tf.TensorNr) input_mapreturn_elementsrT)r)allow_soft_placementr)rconfig) feed_dictcalibration_data)*rrrrjitemsr*rrrrrrrrrrrrrryrrr GraphOptionsrrrangerunr placeholderr stringnoder_TRT_ENGINE_OP_NAMEdevicer%get_calibration_data_oprrattrr/)r fetch_namesnum_runs feed_dict_fn input_map_fnkvfetchescalibrate_rewriter_cfgcalibrate_configcalibration_sess_device_to_get_resource_op_mapresource_name_inputr"serialized_resources_outputcalibration_results r0 calibratezTrtGraphConverter.calibrates6 ???  ! !! !/// /|/;  M OO.&&(O$!Q!S!EF F!V]]+MN N O "iikD + + -))  # #&2LN% g1??A 11== " "--! --245 %% !&.$4XI! |~4  II')#  " " - - /A'33FMMB--22 AD WW+ +{{"??**11$++>N778KL,N.,DKK8"2!5!5-dkk:')CDII)N"6"  /ADII( ) +% AA.*.d&M&.P  $ $$yXNNAA&.P  $ $$sQ-K/A LAL 3LK< )A"L L/K9<L LL LL(c|jsJ|jr|jsJ|jr t dd}t j |}tjj5tj|jd|tj|j|j|jj t#j$5}|j'||j(|jj*dddddd|j-y#1swY"xYw#1swY&xYw)aSave the converted graph as a SavedModel. Args: output_saved_model_dir: construct a SavedModel using the converted GraphDef and save it to the specified directory. This option only works when the input graph is loaded from a SavedModel, i.e. when input_saved_model_dir is specified and input_graph_def is None in __init__(). Raises: ValueError: if the input to the converter is a GraphDef instead of a SavedModel. z:Not able to save to a SavedModel since input is a GraphDefc dd}|D]}|j|}|jd}|tjd|=t j |}|rk|dk(rft j |}|jjD]7} |} | j|  || |} |j|| 9t||} |dk(rM| jD]<} t j| |} |j| }|j||>|dk(r.| jD]} |j|t#| P| jD](} |j|t j| |*y#Y xYw#ttt f$rYwxYw) z*Restores collections that we need to keep.rkindNz6Cannot identify data type for collection %s. Skipping. bytes_list) import_scoper int64_list)r WhichOneofrwrxrget_from_proto_functionget_collection_proto_typer:rParseFromStringadd_to_collectiongetattrprepend_name_scopeas_graph_element TypeErrorrjKeyErrorr) dest_graphsrc_meta_graph_defrscoperrr9 from_proto proto_typerproto new_valuefieldrcol_ops r0_restore_collectionsz4TrtGraphConverter.save.._restore_collectionsse (Q#+::3?((0 < --F M 005 $,.44S9*%0066 9eLE  ! !% ( $U?i  ( (i 8 9.$/% [  8++E59d#44T:**37 8|# <**3E ; < Q**3+.+A+A%+OQ QM(Q   X6s% F;FFF/.F/rr)signature_def_mapN)rrrrrjrSavedModelBuilderrrrrrrget_default_graphrrrrradd_meta_graph_and_variablesrrr)routput_saved_model_dirrPsaved_model_builderr s r0rzTrtGraphConverter.saves< ???   - -- -   F HH+Q\"334JK    ! K 9 9C   !4#@#@  # #++:: <= ?? K88   ( (";;II 9 KK K KK K Ks%:A?E93E ,E E EE")NN)rBrCrDrbrcr8r9rrrrrrrr6rrFr2r0rrsp@&*&*/3#"(L.33$%"&'#m7^((  P/b % "! h%TQr2rctj|5tj|cdddS#1swYyxYw)N) resource_name)rr$r%create_trt_resource_handle)rr$s r0_get_resource_handlerZMs8 zz&F  1 1 1 EFFFs6?c|jj}d}ttt |j j D]}|j j |}d|jjvs5|dz }tjj|jjrotjj|jjtjj|jjro|j j |=tjd|d|jD]}|jdk(s|j d=!|j j D]0}|j"D]}|jdk(s|j d=!2t%||}|S)a8Remove native segments from the input TF-TRT Converted Function. Args: input_func: provide the concrete function with native segment nodes. The transformed output func will not contain any native segment nodes. All the TRTEngineOp references will be deleted and reset to default empty func. rnative_segmentrVz'Found and deleted native segments from z TRTEngineOp nodes.rg segment_func)rrreversedrlenlibraryfunction signaturerr has_functionremove_functionrwrr"rr&node_def"_construct_function_from_graph_def) input_funcr nodes_deletedfunc_idfr"funcnew_funcs r0_remove_native_segmentsrmRs$$113/-%O$;$;$D$D EFG4g((1A1;;+++qm OO  * *1;;+;+; <))!++*:*:; OO  * *1;;+;+; <  ! ! * *7 3 4 ,,/ *,""$d ww- ))N #$%%..&d & M ! IIn %&& 0 O L( /r2c8eZdZdZ dfd ZdZdZdZxZS)_TRTEngineResourcez/Class to track the serialized engines resource.ctt| |||_|j t j |d|_||_y)N)r$!_serialized_trt_resource_filename) rXror_resource_name_track_trackablerAsset _filename_maximum_cached_engines)rrXfilenamer^r$ras r0rz_TRTEngineResource.__init__}sJ  d,F,;'D** HBDDN#9D r2cBt|j|jSr=)rZrr_resource_devicers r0_create_resourcez#_TRTEngineResource._create_resources  3 3T5J5J KKr2cptj|j|j|jy)N)max_cached_engines_count)r%initialize_trt_resourceresource_handlerurvrs r0 _initializez_TRTEngineResource._initializes.''  !%!=!=(?r2ct|j|j}tj|j5t j |ddddy#1swYyxYw)NT)ignore_lookup_error)rZrrryrr$rdestroy_resource_op)rhandles r0_destroy_resourcez$_TRTEngineResource._destroy_resourcesT !$"5"5t7L7L MF D)) *,33 d,,,,s A!!A*)GPU) rBrCrDrbrrzrrrdres@r0rorozs 7  :L? ,r2rocd}t|D]O\}}t|}||}|dkDr|dz}dj||z|}t||kDsE|d|dz dz}Q||y) z Prints a row.rr z{0:{min_length}}) min_lengthNrHz ...) enumeraterrkr_)fields positionsprint_fnlinerrN end_line_poss r0 _print_rowrs $F#0ha JEQ,>?&&++?!,!4!45vv{{5xA('+&;&; jj##X^^%F%F'H(..# ..(04/N/N(..+ / @5s,G$+G) G. c|jj}|jjD]}d|jvs|jd= t j ||j}dD]}g}|jdjjD]W}tj}|j||jd|j|jY||j|jjddt!j"}|j$|j&zD]1} |j(jj| j*3|jdj-|t/j0} | j2j4} d| _| j8jd t;j<| |} | S) zBApply an inlining optimization to the function's graph definition.api_implementsr) variablesmodel_variablestrainable_variableslocal_variablesrinitializer_nameNrrira)rrr`rar&r rrr:rr VariableDefr@ ClearFieldrSerializeToStringr CollectionDefrrrrrrrrrmin_graph_nodesrrr) rkrra meta_graphrraw_listrawvariablefetch_collectionarrayrr new_graph_defs r0_apply_inliningrsjj%%')##,,*h8==( --( )*&&$**M* CdH((5@@FF4))+hs#,-ooh0023 4 ;CJd#..44Q7 C$113{{T\\)8e$$++EJJ78 J'001AB  ! ! #&((88/$&/!##J/,,VZ@- r2ci}t|jj|jD]\}}|j||j < |j Dcic]}|j |}}|j D]}|jdk(s|jdk(s"|}||jdjdk(r2||jd}||jdjdk(r2|jddz}||vr<||} |jdjj| jtdj|ycc}w) aeAnnotates variable operations with custom `_shape` attribute. This is required for the converters and shape inference. The graph definition is modified in-place. Args: func: Function represented by the graph definition. graph_def: Graph definition to be annotated in-place. Raises: RuntimeError: if some shapes cannot be annotated. ReadVariableOpResourceGatherrIdentityz:0_shapez&Not found in the function captures: {}N)ziprinternal_capturesrshaperr"rinputr&ras_protor{rk) rkr ph_shape_mapphvarr" name_to_nodenode_ph_namers r0_annotate_variable_opsrsK,TZZ114>>B&gb#IIL&/8nn=d$))T/=,=nn Fd ww""dgg1A&Ae Q ( + +z 9U[[^, Q ( + +z 9 A%g L W% (!!**5>>+;< 4 ; ;G DF F F>sE!c, tjt|j}|j |j d_y#tjtjf$r#tjd|jYywxYw)Nrz Warning calibration error for %s) r%r%rrnumpyr&r/r UnknownError NotFoundErrorrwwarning)r"calibration_tables r0_save_calibration_tablersuC#;;"499-/&7&=&=&?DII !#   v33 4C OO6 BCsA A ABBc t|tr9g}|jDcic]\}}|tj|}}}nIi}t|t t frttj|}ntj|g}||fScc}}w#d}tj|dj|zt|xYw)Nz"Failed to convert input to tensor.z inp = `{0}` ) r*dictrrconvert_to_tensorrtuplemaprwrxrkr{)inpargsr+r,kwargs error_msgs r0_convert_to_tensorr!s "#t d8; D13((++DfDf C$ '3((#.%%c*+ vE"4I MM)/66s;;< y !!s%B BA BB6Czexperimental.tensorrt.Converterc eZdZdZdZej ddddddddeejdddddf d Z d Z d Z d Z d ZddZdZ ddZddZy)TrtGraphConverterV2axAn offline converter for TF-TRT transformation for TF 2.0 SavedModels. Windows support is provided experimentally. No guarantee is made regarding functionality or engineering support. Use at your own risk. There are several ways to run the conversion: 1. FP32/FP16 precision ```python params = tf.experimental.tensorrt.ConversionParams( precision_mode='FP16') converter = tf.experimental.tensorrt.Converter( input_saved_model_dir="my_dir", conversion_params=params) converter.convert() converter.save(output_saved_model_dir) ``` In this case, no TRT engines will be built or saved in the converted SavedModel. But if input data is available during conversion, we can still build and save the TRT engines to reduce the cost during inference (see option 2 below). 2. FP32/FP16 precision with pre-built engines ```python params = tf.experimental.tensorrt.ConversionParams( precision_mode='FP16', # Set this to a large enough number so it can cache all the engines. maximum_cached_engines=16) converter = tf.experimental.tensorrt.Converter( input_saved_model_dir="my_dir", conversion_params=params) converter.convert() # Define a generator function that yields input data, and use it to execute # the graph to build TRT engines. def my_input_fn(): for _ in range(num_runs): inp1, inp2 = ... yield inp1, inp2 converter.build(input_fn=my_input_fn) # Generate corresponding TRT engines converter.save(output_saved_model_dir) # Generated engines will be saved. ``` In this way, one engine will be built/saved for each unique input shapes of the TRTEngineOp. This is good for applications that cannot afford building engines during inference but have access to input data that is similar to the one used in production (for example, that has the same input shapes). Also, the generated TRT engines is platform dependent, so we need to run `build()` in an environment that is similar to production (e.g. with same type of GPU). 3. INT8 precision and calibration with pre-built engines ```python params = tf.experimental.tensorrt.ConversionParams( precision_mode='INT8', # Currently only one INT8 engine is supported in this mode. maximum_cached_engines=1, use_calibration=True) converter = tf.experimental.tensorrt.Converter( input_saved_model_dir="my_dir", conversion_params=params) # Define a generator function that yields input data, and run INT8 # calibration with the data. All input data should have the same shape. # At the end of convert(), the calibration stats (e.g. range information) # will be saved and can be used to generate more TRT engines with different # shapes. Also, one TRT engine will be generated (with the same shape as # the calibration data) for save later. def my_calibration_input_fn(): for _ in range(num_runs): inp1, inp2 = ... yield inp1, inp2 converter.convert(calibration_input_fn=my_calibration_input_fn) # (Optional) Generate more TRT engines offline (same as the previous # option), to avoid the cost of generating them during inference. def my_input_fn(): for _ in range(num_runs): inp1, inp2 = ... yield inp1, inp2 converter.build(input_fn=my_input_fn) # Save the TRT engine and the engines. converter.save(output_saved_model_dir) ``` 4. To use dynamic shape, we need to call the build method with an input function to generate profiles. This step is similar to the INT8 calibration step described above. The converter also needs to be created with use_dynamic_shape=True and one of the following profile_strategies for creating profiles based on the inputs produced by the input function: * `Range`: create one profile that works for inputs with dimension values in the range of [min_dims, max_dims] where min_dims and max_dims are derived from the provided inputs. * `Optimal`: create one profile for each input. The profile only works for inputs with the same dimensions as the input it is created for. The GPU engine will be run with optimal performance with such inputs. * `Range+Optimal`: create the profiles for both `Range` and `Optimal`. ctDcgc]}|j}}|j|vr#tdj|t|dk(rt j dyycc}w)Nz>profile_strategy '{}' is not supported. It should be one of {}rKzImplicitBatchModeCompatible strategy is deprecated, and using it may result in errors during engine building. Please consider using a different profile strategy.)rQr>rjrkrwr)rstrategyr/supported_strategiess r0_verify_profile_strategyz,TrtGraphConverterV2._verify_profile_strategysv/K/MN!AGGINN~~33  K F89; < >>00 ll :;1 OsA7Nz+Use individual converter parameters insteadrlrUrVTc TtjsJ| t|||| | | } tt | d| |_||_|xstjg|_ |xstj|_ tjd |_| j t"j$k(xs+| j t"j$j'k(xr | j(|_d|_d|_d|_d|_d|_|d|_n||_|js(|j6st9j:dd|_d|_|j6r-|t>|_d|_!y|jA|||_d|_!y) a Initialize the converter. Args: input_saved_model_dir: the directory to load the SavedModel which contains the input graph to transforms. Required. input_saved_model_tags: list of tags to load the SavedModel. input_saved_model_signature_key: the key of the signature to optimize the graph for. use_dynamic_shape: whether to enable dynamic shape support. None is equivalent to False in the current implementation. dynamic_shape_profile_strategy: one of the strings in supported_profile_strategies(). None is equivalent to Range in the current implementation. max_workspace_size_bytes: the maximum GPU temporary memory that the TRT engine can use at execution time. This corresponds to the 'workspaceSize' parameter of nvinfer1::IBuilder::setMaxWorkspaceSize(). precision_mode: one of the strings in TrtPrecisionMode.supported_precision_modes(). minimum_segment_size: the minimum number of nodes required for a subgraph to be replaced by TRTEngineOp. maximum_cached_engines: max number of cached TRT engines for dynamic TRT ops. Created TRT engines for a dynamic dimension are cached. If the number of cached engines is already at max but none of them supports the input shapes, the TRTEngineOp will fall back to run the original TF subgraph that corresponds to the TRTEngineOp. use_calibration: this argument is ignored if precision_mode is not INT8. If set to True, a calibration graph will be created to calibrate the missing ranges. The calibration graph must be converted to an inference graph by running calibration with calibrate(). If set to False, quantization nodes will be expected for every tensor in the graph (excluding those which will be fused). If a range is missing, an error will occur. Please note that accuracy may be negatively affected if there is a mismatch between which tensors TRT quantizes and which tensors were trained with fake quantization. allow_build_at_runtime: whether to allow building TensorRT engines during runtime if no prebuilt TensorRT engine can be found that can handle the given inputs during runtime, then a new TensorRT engine is built at runtime if allow_build_at_runtime=True, and otherwise native TF is used. conversion_params: a TrtConversionParams instance (deprecated). Raises: ValueError: if the combination of the parameters is invalid. NrfTrdisable_graph_freezingFzZDisabling graph freezing is only possible in dynamic shape mode. The graph will be frozen.Unknown)"r rrTrrnrrrrrrrrry!is_experimental_feature_activatedfreezer\r8r;r>r_r_calibration_input_fnr_device_build_called_once _calibrated_use_dynamic_shaperwr_profile_strategyrMrr) rrrruse_dynamic_shapedynamic_shape_profile_strategyr[r\r]r^r_r`rls r0rzTrtGraphConverterV2.__init__sv  $ $ && & -#;'3!7)!7 9%&.d;/D"7D9=#8#8"9   ( >== ) AA ""DK  ) )-=-B-B B L  ) )-=-B-B-H-H-J J@0??  "&DDODL#DD %d 1d ;;t66 ll '(dk&D  ' /!7 27D. %%&DE!?27D.r2c|jSr=)rrs r0_need_trt_profilesz&TrtGraphConverterV2._need_trt_profiless  " ""r2c8tj}t|jj ddd|j |j |j}|jjj|tj||dS)zRun Grappler's OptimizeGraph() tool to convert the graph. Args: meta_graph_def: the MetaGraphDef instance to run the optimizations on. Returns: The optimized GraphDef. T)r`N)rlrrrrrrr) rrrr_replacerrrrrrrr)rmeta_graph_defrrs r0rz#TrtGraphConverterV2._run_conversions)446:11::#';)#'#M#M#666//1))99BB  % %+ GGr2c|jD]}|jtk(s|| |jjD]/}|j D]}|jtk(s|| 1y)z;Helper method to manipulate all TRTEngineOps in a GraphDef.N)r"rr#r`rare)rrfnr"rks r0_for_each_trt_nodez&TrtGraphConverterV2._for_each_trt_node7sm ' ' 4!!**--$ 77) ) T(r2c|D]"}t|\}}|j|i|$|j|jtt |j|j|_d|_y)z@Run INT8 calibration with the provided input generator function.TN)r_converted_funcrrrrfr)rcalibration_input_fnrrrs r0_execute_calibrationz(TrtGraphConverterV2._execute_calibrationAsx#%,',ldFdD+F+, D557NO> d779DDr2c4|jrJtjgj}d|j vrt d|d|j vr6||_tjd|jd|j |jr |s t d|js |r t dtj|j|j|_ |jj|j}|j rt#j$|}n#t'|}t)||t+||}|j,j/}tjd|j0D] }d |_ |j "t3j4||j, }nt7j8j;5} t7j|j 5t=j>|d t3j4| j/| }d d d d d d tAjB} |jD|jFzD]1} | jHjJjM| jN3jPd jS| |jU||_+t+||jV||_,|jr)|j[s|j]|n||_/d|_t`jbjedd } | H tgjh|jXj,j/d| |jXS|jXS#1swYvxYw#1swY{xYw#tj$r-} tjld| Yd } ~ |jXSd } ~ wwxYw)aConvert the input SavedModel in 2.0 format. Args: calibration_input_fn: a generator function that yields input data as a list or tuple or dict, which will be used to execute the converted signature for calibration. All the returned input data should have the same shape. Example: `def input_fn(): yield input1, input2, input3` If dynamic_shape_mode==False, (or if the graph has static input shapes) then we run calibration and build the calibrated engine during conversion. If dynamic_shape_mode==True (and the graph has any unknown input shape), then the reference to calibration_input_fn is stored, and the calibration is actually performed when we build the engine (see build()). Raises: ValueError: if the input combination is invalid. Returns: The TF-TRT converted Function. gpuzSpecified device is not a GPU: zgpu:0zPlacing imported graph from `z ` on device: zFShould specify calibration_input_fn because INT8 calibration is neededzNShould not specify calibration_input_fn because INT8 calibration is not neededz7Clearing prior device assignments in loaded saved modelrNrrrTTF_TRT_EXPORT_GRAPH_VIZ_PATH)defaultr)graphdefdot_output_filenamezDAn Exception occurred during the export of the graph visualization: )7rrzerosr$r>rjrrwrrrrr _saved_model signaturesrrr !convert_variables_to_constants_v2rrrfrrr"r rrrrrrrrrrrrrrrrrrrrrrosenvirongetrydraw_graphdef_as_graphviz Exceptionrx)rrdevice_requestedrkrinlined_graph_defrr"grappler_meta_graph_defrrr graphviz_pathes r0rzTrtGraphConverterV2.convertPs0 !r*11 $**,, 89I8JK LL&,,..%dl ll223=PQ '; / 00  " "'; 3 44 $"="="&">">@D    ' '(M(M ND {{(JJ4Pk)$/T#456t=NOk"((557 LLJK %%dk || % 7 7$K,=,=!? 99; ! ! #9ucjj.F9!!"2<"'"9"9((*%#999 &335##k&9&99:  &&--ejj9:**:6??!% 4 45L MD= d''6D   $ $ & !!"67%9"DOJJNN#A4NPM  ++))//<<<M - /   4  M9999@   S "      s<6 O=OO4;O!O OO! P*PPc$|jr td|s td|js td|jr#|j s|j tdd}|jrG|j|jt|dt|j|j}n |j}d}|D]}||}t|\}}||i||jr&|j|jt|d|jr/|j s#|j|j d|_yt|\}}|j|i|d|_y) aRun inference with converted graph in order to build TensorRT engines. If the conversion requires INT8 calibration, then a reference to the calibration function was stored during the call to convert(). Calibration will be performed while we build the TensorRT engines. Args: input_fn: a generator function that provides the input data as a single array, OR a list or tuple of the arrays OR a dict, which will be used to execute the converted signature to generate TRT engines. Example 1: `def input_fn(): # Let's assume a network with 1 input tensor. # We generate 2 sets of dummy input data: input_shapes = [(1, 16), # 1st shape (2, 32)] # 2nd shape for shapes in input_shapes: # return an input tensor yield np.zeros(shape).astype(np.float32)' Example 2: `def input_fn(): # Let's assume a network with 2 input tensors. # We generate 3 sets of dummy input data: input_shapes = [[(1, 16), (2, 16)], # 1st input list [(2, 32), (4, 32)], # 2nd list of two tensors [(4, 32), (8, 32)]] # 3rd input list for shapes in input_shapes: # return a list of input tensors yield [np.zeros(x).astype(np.float32) for x in shapes]` Raises: NotImplementedError: build() is already called. RuntimeError: the input_fx is None. zNbuild() is already called. It is not supported to call build() more than once.zbinput_fn is None. Method build() needs input_fn to be specified in order to build TensorRT enginesz%Need to call convert() before build()NzENeed to provide the calibration_input_fn arg while calling convert().c,||jd_y)N_profile_generation_moder&r)rr"s r0_set_profile_generation_modez?TrtGraphConverterV2.build.._set_profile_generation_modes05dii*+-r2TF)rNotImplementedErrorr{rrrrrrrrrfrrr)rinput_fnrrk first_inputrrrs r0buildzTrtGraphConverterV2.buildsH  !L MM  N OO ?? @ AA t'7'7 ""* . //6   d77%&BDIK 00D0D040I0IKd ! !dKz   ',ldF DF     d77%&BEJL  d&6&6  : :;#D( 4ldFdD+F+"Dr2c jsJtjdrXtjdt j _j jj_ jrjs tdtj i fd}jj|j rt#j$n j&} |_j*j,sd}jj|t/j0jj j2Dcgc]}|j4c}j j6Dcgc]}|j4c}}t9j:j jj<|jj<|j_j j>|j_|_j@j i} tCjB||| |ycc}wcc}w)aSave the converted SavedModel. Args: output_saved_model_dir: directory to saved the converted SavedModel. save_gpu_specific_engines: whether to save TRT engines that have been built. When True, all engines are saved and when False, the engines are not saved and will be rebuilt at inference time. By using save_gpu_specific_engines=False after doing INT8 calibration, inference can be done on different GPUs than the GPU that the model was calibrated and saved on. options: `tf.saved_model.SaveOptions` object for configuring save options. Raises: RuntimeError: if the needed calibration hasn't been done. remove_native_segmentsz_'remove_native_segments' experimental feature is enabled during saving of converted SavedModel.zA model that requires INT8 calibration has to be built before saving it. Call build() to build and calibrate the TensorRT engines.cZt|j}|vrytjj d|z} t j ||dt||jj|<y#tj$rtjd|YywxYw)z2Serialize TRT engines in the cache and track them.Nztrt-serialized-engine.T)rXrwdelete_resourcesave_gpu_specific_engineszCould not find %s in TF-TRT cache. This can happen if build() is not called, which means TensorRT engines will be built and cached at runtime.)rrrpathjoinr%serialize_trt_resourcerrrwrrorr^)r"canonical_engine_namerwengine_asset_dir resource_maprrs r0_serialize_and_track_enginez=TrtGraphConverterV2.save.._serialize_and_track_engineDs9C , ..69NNPh **/ &? + A-?   ! ! 8 8-:l() ! !  %'<  =  sA>>)B*)B*c,d|jd_y)NF_allow_build_at_runtimer)r"s r0_reset_allow_build_at_runtimez?TrtGraphConverterV2.save.._reset_allow_build_at_runtimens16 +,.r2)optionsN)"rryrrwrrmrrrrrrr{tempfilemkdtemprrr AutoTrackablertrt_engine_resourcesrr`r rrrrr"rrrrr) rrUrrr trackablerrreset_converted_funcrrrs ` ` @@r0rzTrtGraphConverterV2.saves$ ???223KL ll 45T5I5IJd"&"6"6"<"<"I"I"Kd d&6&6 ; <<  '')L:: D5579 ++++%1I"  " " 9 97 d77;=+BB  # #%)%9%9%@%@ A66;; A%)%9%9%A%A B66;; BD7;6K6K    $ $ 7 7  $ $ 7 779  3    9 9  ;1d779M9MNJIIi/WM B Bs 8I4 %I9 c |js:td|jjd|jjd|dkrt d|d|t }gd}|Dcgc]\}}t ||z}}}tj|j}|Dcgc]\}}| } }}t| || |d |zd } d } d } |jjjd } t}| jD]+}|j dk7r| dz } |||j"<| dz } -t%|j'D]1\}}|j(j+dd}t-j.|d}t-j.|d}t-j0|d}t-j0|d}t-j2|d}t-j2|d}t-j4| |\}}| |z } | dk7r |dddt|||||||||g |||s|t%t|j'D]\}}|d|d|d4|dd |z|d| | | z}| |z dz}|d|dd | d|d!ycc}}wcc}}w)"aThis method describes the results of the conversion by TF-TRT. It includes information such as the name of the engine, the number of nodes per engine, the input and output dtype, along with the input shape of each TRTEngineOp. Args: line_length: Default line length when printing on the console. Minimum 160 characters long. detailed: Whether or not to show the nodes inside each TRTEngineOp. print_fn: Print function to use. Defaults to `print`. It will be called on each line of the summary. You can set it to a custom function in order to capture the string summary. Raises: RuntimeError: if the graph is not converted. zImpossible to call `z.summary()` before calling z .convert()`.z/Invalid `line_length` value has been received: z. Minimum: 160.N) )zTRTEngineOP Nameg?)Device ףp= ?)z# Nodesg?)z# Inputsr%)z # Outputsr%)z Input DTypesQ?)z Output Dtypesr&)z Input Shapesr&)z Output Shapesr&)r=rTrrgrVrri input_shapes_output_shapesInTOutT z(----------------------------------------)rrrz - z: xz&[*] Total number of TensorRT engines: dz[*] % of OPs Converted: z.2fz% [z] )rr{rarBrjprintrnpcumsumtolistrrrrrr"rrsortedrr$rryget_node_io_shapesget_trtengineop_io_dtypesget_trtengineop_io_nodes_countget_trtengineop_node_op_count)r line_lengthdetailedrcolumnsr1r@rhheaders n_enginesn_ops_convertedn_ops_not_convertedrtrtengineops_dictr"r node_device in_shapes out_shapes in_dtypes out_dtypesin_nodes_countout_nodes_count node_countconverted_ops_dictrr total_opsconversion_ratios r0summaryzTrtGraphConverterV2.summarys2$ ??  !8!8 9:^^,,-\ ; <<S H%o7 88h G3::$!Q[1_%:I: )$++-I$%TQq%G%w H5 S; IO##))66$6GH  M !q '+$))$Q .4467+ dKK%%c*2.k..t^Di//6FGj55dEBi66tVDj ??eLn!@@vNo'0'N'N D($j$#o a2fXR!K^_Y      &8!9!?!?!AB +JC T#bq) * +5+: r#k/" #$ 5i[ AB#o5I&2S8 '(8'=> !9+S23o;%s ,K/ Kr=)TN)r#TN)rBrCrDrbrr!deprecated_argsrcr8r9rrrrrrr rrLrFr2r0rr4sdL ;;tL24&*&*/3!%.2(L.33$%&'#&*!%o74o7b#G0 f P]#B&*jNXf3r2rrVrUc zt|| | ||||||||d } | j} | r| j| | S)a Python wrapper for the TRT transformation. Args: input_graph_def: a GraphDef object containing a model to be transformed. If set to None, the graph will be read from the SavedModel loaded from input_saved_model_dir. outputs: list of tensors or node names for the model outputs. Only used when input_graph_def is not None. max_batch_size: max size for the input batch. max_workspace_size_bytes: the maximum GPU temporary memory which the TRT engine can use at execution time. This corresponds to the 'workspaceSize' parameter of nvinfer1::IBuilder::setMaxWorkspaceSize(). precision_mode: one of TrtPrecisionMode.supported_precision_modes(). minimum_segment_size: the minimum number of nodes required for a subgraph to be replaced by TRTEngineOp. is_dynamic_op: whether to generate dynamic TRT ops which will build the TRT network and engine at run time. maximum_cached_engines: max number of cached TRT engines in dynamic TRT ops. If the number of cached engines is already at max but none of them can serve the input, the TRTEngineOp will fall back to run the TF function based on which the TRTEngineOp is created. input_saved_model_dir: the directory to load the SavedModel which contains the input graph to transforms. Used only when input_graph_def is None. input_saved_model_tags: list of tags to load the SavedModel. input_saved_model_signature_key: the key of the signature to optimize the graph for. output_saved_model_dir: if not None, construct a SavedModel using the returned GraphDef and save it to the specified directory. This option only works when the input graph is loaded from a SavedModel, i.e. when input_saved_model_dir is specified and input_graph_def is None. Returns: A GraphDef transformed from input_graph_def (or the SavedModel graph def loaded from input_saved_model_dir, if input_graph_def is not present), where all TRT compatible subgraphs are replaced with TRTEngineOps, and a TF function is added for each of the subgraphs. If is_dynamic_op is True, each TRTEngineOp will contain a serialized subgraph GraphDef, which will be converted to a TRT engine at execution time and the TRT engine will be cached for future usage. A new TRT engine will be created each time when none of the cached engines match the input shapes. If it fails to execute the TRT engine or the number of cached engines reaches maximum_cached_engines, the op will fall back to call the corresponding TF function. If is_dynamic_op is False, each TRTEngineOp will contain a serialized TRT engine created from the corresponding subgraph. No more engines will be created on the fly, and the op will fall back to call the corresponding TF function when it fails to execute the engine. Raises: ValueError: if the combination of the parameters is invalid. F) rrrrrrr[r\r]rr^r_)rrr)rrrr[r\r]rr^rrrrU trt_converterconverted_graph_defs r0create_inference_graphrQs\D$13&E%#7#/!3 -&--/-. r2)F)NNFFr=)grb collections functoolsrrrsysrrr0sixr+tensorflow.core.frameworkrtensorflow.core.protobufrrrtensorflow.python.clientr#tensorflow.python.compiler.tensorrtr rytensorflow.python.eagerr r tensorflow.python.frameworkr r rrrrtensorflow.python.grapplerrtensorflow.python.opsrrtensorflow.python.platformrrwtensorflow.python.saved_modelrrrrrrtensorflow.python.trackablerrrtensorflow.python.trainingr tensorflow.python.utilr!r""tensorflow.python.util.lazy_loaderr# tensorflow.python.util.tf_exportr$globalsr%r& trt_engine_opAttributeErrorr1r6objectr8r~rmaxsizercrMrNrOrPrQ namedtuplerTDEFAULT_TRT_CONVERSION_PARAMSr#rnrr deprecatedrrrrZrmTrackableResourcerorrfrrrrrr9rQrFr2r0rns:  2/38,B+1<..0+.3+;<1.0.=7-50,.'96 7957  68     8v 8 ((*6I66q!Q?),s):&)0& $!02O/ 3;-GK03-G<-G`!4 5#<.?;F1515(-=B593I \"~ NP0404',  <,^FDC& ,4t 3&t 35t 3vA#(($(R{4s H22H;:H;