BVh ` >dZddlZddlmZddlmZddlmZddlmZddlmZ ddl m Z dd l mZdd l mZdd l mZdd lmZdd lmZedgGddej,dgdZdZddZ ddZdZdZedg ddZy)z,Utilities to warm-start TF.Learn Estimators.N)errors)ops) state_ops)variable_scope) variables) tf_logging)checkpoint_ops)checkpoint_utils)saver)saveable_object_util) tf_exportztrain.VocabInfo)v1c*eZdZdZ dfd ZxZS) VocabInfoa Vocabulary information for warm-starting. See `tf.estimator.WarmStartSettings` for examples of using VocabInfo to warm-start. Args: new_vocab: [Required] A path to the new vocabulary file (used with the model to be trained). new_vocab_size: [Required] An integer indicating how many entries of the new vocabulary will used in training. num_oov_buckets: [Required] An integer indicating how many OOV buckets are associated with the vocabulary. old_vocab: [Required] A path to the old vocabulary file (used with the checkpoint to be warm-started from). old_vocab_size: [Optional] An integer indicating how many entries of the old vocabulary were used in the creation of the checkpoint. If not provided, the entire old vocabulary will be used. backup_initializer: [Optional] A variable initializer used for variables corresponding to new vocabulary entries and OOV. If not provided, these entries will be zero-initialized. axis: [Optional] Denotes what axis the vocabulary corresponds to. The default, 0, corresponds to the most common use case (embeddings or linear weights for binary classification / regression). An axis of 1 could be used for warm-starting output layers with class vocabularies. Returns: A `VocabInfo` which represents the vocabulary information for warm-starting. Raises: ValueError: `axis` is neither 0 or 1. Example Usage: ```python embeddings_vocab_info = tf.VocabInfo( new_vocab='embeddings_vocab', new_vocab_size=100, num_oov_buckets=1, old_vocab='pretrained_embeddings_vocab', old_vocab_size=10000, backup_initializer=tf.compat.v1.truncated_normal_initializer( mean=0.0, stddev=(1 / math.sqrt(embedding_dim))), axis=0) softmax_output_layer_kernel_vocab_info = tf.VocabInfo( new_vocab='class_vocab', new_vocab_size=5, num_oov_buckets=0, # No OOV for classes. old_vocab='old_class_vocab', old_vocab_size=8, backup_initializer=tf.compat.v1.glorot_uniform_initializer(), axis=1) softmax_output_layer_bias_vocab_info = tf.VocabInfo( new_vocab='class_vocab', new_vocab_size=5, num_oov_buckets=0, # No OOV for classes. old_vocab='old_class_vocab', old_vocab_size=8, backup_initializer=tf.compat.v1.zeros_initializer(), axis=0) #Currently, only axis=0 and axis=1 are supported. ``` c |dk7r|dk7rtdj|tt|||||||||S)NrOThe only supported values for the axis argument are 0 and 1. Provided axis: {}) ValueErrorformatsuperr__new__) cls new_vocabnew_vocab_sizenum_oov_buckets old_vocabold_vocab_sizebackup_initializeraxis __class__s ]/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/training/warm_starting_util.pyrzVocabInfo.__new__ls] qyTQY 3396$< AA C (    )Nr)__name__ __module__ __qualname____doc__r __classcell__)r s@r!rr s?L !%r"r)rrrrrrrctj|}t|dkDrtd|d|t |j dS)aXReturns name of the `var`. Args: var: A list. The list can contain either of the following: (i) A single `Variable` (ii) A single `ResourceVariable` (iii) Multiple `Variable` objects which must be slices of the same larger variable. (iv) A single `PartitionedVariable` Returns: Name of the `var` rz`var` = z= passed as arg violates the constraints. name_to_var_dict = r)r op_list_to_dictlen TypeErrorlistkeys)varname_to_var_dicts r!_infer_var_namer1sX*99#> Q /24DF GG ##% &q ))r"cftj|r t|g}nt|trt d|Dr t|}nZt|t jrt|g}|j}n#tdjt||s|}||fS)aHelper method for standarizing Variable and naming. Args: var: Current graph's variable that needs to be warm-started (initialized). Can be either of the following: (i) `Variable` (ii) `ResourceVariable` (iii) list of `Variable`: The list must contain slices of the same larger variable. (iv) `PartitionedVariable` prev_tensor_name: Name of the tensor to lookup in provided `prev_ckpt`. If None, we lookup tensor with same name as given `var`. Returns: A tuple of the Tensor name and var. c3FK|]}tj|ywNr _is_variable.0vs r! z _get_var_info.. :  ) )! , :!`var MUST be one of the following: a Variable, list of Variable or PartitionedVariable, but is {}) r r6r1 isinstancer-all variables_libPartitionedVariable_get_variable_listr,rtype)r/prev_tensor_namecurrent_var_names r! _get_var_inforFs""3'&u-3 :c ::&s+#}889&u- "C  ))/S ): << ' 3 r"c |r|r|r|s tdtj|r|g}nst|trt d|Dr|}nNt|t jr|j}n#tdjt||s t|}td|D} |D]} | jj} | j!} d}| r+t#j$| j&| j(}| dk(r|}| d}|}|}|}d}d}|}d}n2| dk(r| }|}d }d}d}|}|}d}|}ntd j| t+j,tj.|||||||||||| }t1j2|| | }t5j6| || _y) aWarm-starts given variable from `prev_tensor_name` tensor in `prev_ckpt`. Use this method when the `var` is backed by vocabulary. This method stitches the given `var` such that values corresponding to individual features in the vocabulary remain consistent irrespective of changing order of the features between old and new vocabularies. Args: var: Current graph's variable that needs to be warm-started (initialized). Can be either of the following: (i) `Variable` (ii) `ResourceVariable` (iii) list of `Variable`: The list must contain slices of the same larger variable. (iv) `PartitionedVariable` current_vocab_path: Path to the vocab file used for the given `var`. current_vocab_size: An `int` specifying the number of entries in the current vocab. prev_ckpt: A string specifying the directory with checkpoint file(s) or path to checkpoint. The given checkpoint must have tensor with name `prev_tensor_name` (if not None) or tensor with name same as given `var`. prev_vocab_path: Path to the vocab file used for the tensor in `prev_ckpt`. previous_vocab_size: If provided, will constrain previous vocab to the first `previous_vocab_size` entries. -1 means use the entire previous vocab. current_oov_buckets: An `int` specifying the number of out-of-vocabulary buckets used for given `var`. prev_tensor_name: Name of the tensor to lookup in provided `prev_ckpt`. If None, we lookup tensor with same name as given `var`. initializer: Variable initializer to be used for missing entries. If None, missing entries will be zero-initialized. axis: Axis of the variable that the provided vocabulary corresponds to. Raises: ValueError: If required args are not provided. zgInvalid args: Must provide all of [current_vocab_path, current_vocab_size, prev_ckpt, prev_vocab_path}.c3FK|]}tj|ywr4r5r7s r!r:z-_warm_start_var_with_vocab..r;r<r=c3^K|]%}|jjd'yw)rN) get_shapeas_listr7s r!r:z-_warm_start_var_with_vocab..s$C!1;;=00215Cs+-N) full_shape var_offsetrrr#r) ckpt_pathold_tensor_namenew_row_vocab_sizenew_col_vocab_sizeold_row_vocab_sizeold_row_vocab_filenew_row_vocab_fileold_col_vocab_filenew_col_vocab_filenum_row_oov_bucketsnum_col_oov_buckets initializer)shapepartition_info)rr r6r>r-r?r@rArBr,rrCr1sumrJrK_get_save_slice_infor_PartitionInforLrMr "_load_and_remap_matrix_initializer_get_checkpoint_filenamerconvert_to_tensorrassign_initializer_op)r/current_vocab_pathcurrent_vocab_size prev_ckptprev_vocab_pathprevious_vocab_sizecurrent_oov_bucketsrDrYrtotal_v_first_axisr9v_shape slice_infor[rPrQrRrSrTrUrVrWrXinit new_init_vals r!_warm_start_var_with_vocabros#Z !3  H II""3' %C3 :c :: C#}889 "C  ))/S ): << &s+CsCC 2:akkm##%G'')JN%44**z7L7LNn qy-"1:.*-/ .-*-/ 3396$< AA  < <";;IF(-------// !D(( 7>:strr7s r!r:z)_get_grouped_variables..Us :!:a  :sc3FK|]}tj|ywr4r5r7s r!r:z)_get_grouped_variables..Zs J!  * *1 - Jr<z^If `vars_to_warm_start` is a list, it must be all `Variable` or all `str`. Given types are {}z@`vars_to_warm_start must be a `list` or `str`. Given type is {})r>rslogginginforget_collection GraphKeysTRAINABLE_VARIABLESr-r?GLOBAL_VARIABLESrrrCr1 setdefaultappend)vars_to_warm_start list_of_varsr9grouped_variablestvar_names r!_get_grouped_variablesr7sd."C(,>,F LLGH%% ))1CEL$d+ :'9 ::l!5!** MM * *!5 5 5 J7I J J'l FFLf-?@DG@GB CC ""(&.@)A"B DD 9aa&AAq!H  2.55a89 AsE+ cBtj|} tj|}t |t |jz }|rtdj||Dcic]}||| c}S#tj $ricYSwxYwcc}w)a#Returns a dictionary mapping variable names to checkpoint keys. The warm-starting utility expects variable names to match with the variable names in the checkpoint. For object-based checkpoints, the variable names and names in the checkpoint are different. Thus, for object-based checkpoints, this function is used to obtain the map from variable names to checkpoint keys. Args: path: path to checkpoint directory or file. variable_names: list of variable names to load from the checkpoint. Returns: If the checkpoint is object-based, this function returns a map from variable names to their corresponding checkpoint keys. If the checkpoint is name-based, this returns an empty dict. Raises: ValueError: If the object-based checkpoint is missing variables. zAttempting to warm-start from an object-based checkpoint, but found that the checkpoint did not contain values for all variables. The following variables were missing: {}) r r` saver_libobject_graph_key_mappingr NotFoundErrorsetr.rr)pathvariable_namesfname names_to_keys missing_namesnames r!_get_object_checkpoint_renamesrns*  3 3D 9%66u=M n%M,>,>,@(AA-  /     1? ?$ d# # ??    I @sB1 BBBztrain.warm_startc*tjdj|t|}|i}|st ||j }d}t }t }i}|jD]\} } |j| } | r|j| |j| } | r|j| |dz }tjdj| | j| j| j| jdkDr | jnd| j| xsd| j xsdt#| | j| j|| j| j| j| | j | j$ 1|s5|dz }tjd j| | xsdt'| dk(r| d} t)| | \} }| |vrOtjd j| || |t+j,|||j/||| <|rt+j,||t |j |z }t |j |z }tjd ||r)t1d j||j |r)t1dj||j y)a Warm-starts a model using the given settings. If you are using a tf.estimator.Estimator, this will automatically be called during training. Args: ckpt_to_initialize_from: [Required] A string specifying the directory with checkpoint file(s) or path to checkpoint from which to warm-start the model parameters. vars_to_warm_start: [Optional] One of the following: - A regular expression (string) that captures which variables to warm-start (see tf.compat.v1.get_collection). This expression will only consider variables in the TRAINABLE_VARIABLES collection -- if you need to warm-start non_TRAINABLE vars (such as optimizer accumulators or batch norm statistics), please use the below option. - A list of strings, each a regex scope provided to tf.compat.v1.get_collection with GLOBAL_VARIABLES (please see tf.compat.v1.get_collection). For backwards compatibility reasons, this is separate from the single-string argument type. - A list of Variables to warm-start. If you do not have access to the `Variable` objects at the call site, please use the above option. - `None`, in which case only TRAINABLE variables specified in `var_name_to_vocab_info` will be warm-started. Defaults to `'.*'`, which warm-starts all variables in the TRAINABLE_VARIABLES collection. Note that this excludes variables such as accumulators and moving statistics from batch norm. var_name_to_vocab_info: [Optional] Dict of variable names (strings) to `tf.estimator.VocabInfo`. The variable names should be "full" variables, not the names of the partitions. If not explicitly provided, the variable is assumed to have no (changes to) vocabulary. var_name_to_prev_var_name: [Optional] Dict of variable names (strings) to name of the previously-trained variable in `ckpt_to_initialize_from`. If not explicitly provided, the name of the variable is assumed to be same between previous checkpoint and current model. Note that this has no effect on the set of variables that is warm-started, and only controls name mapping (use `vars_to_warm_start` for controlling what variables to warm-start). Raises: ValueError: If the WarmStartSettings contains prev_var_name or VocabInfo configuration for variable names that are not used. This is to ensure a stronger check for variable configuration than relying on users to examine the logs. zWarm-starting from: {}NrrzWarm-starting variable: {}; current_vocab: {} current_vocab_size: {} prev_vocab: {} prev_vocab_size: {} current_oov: {} prev_tensor: {} initializer: {}All Unchangedzzero-initialized) rdrerfrgrhrirDrYrz-Warm-starting variable: {}; prev_var_name: {}zSRequested prev_var_name {} initialize both {} and {}; calling init_from_checkpoint.zWarm-started %d variables.zYou provided the following variables in var_name_to_prev_var_name that were not used: {0}. Perhaps you misspelled them? Here is the list of viable variable names: {1}zYou provided the following variables in var_name_to_vocab_info that were not used: {0}. Perhaps you misspelled them? Here is the list of viable variable names: {1})rurvrrrr.ritemsgetadddebugrrrrrrrorr+rFr init_from_checkpointclearr)ckpt_to_initialize_fromr}var_name_to_vocab_infovar_name_to_prev_var_namerwarmstarted_countprev_var_name_usedvocab_info_usedvocabless_varsrvariable prev_var_name vocab_inforDr/prev_var_name_not_usedvocab_info_not_useds r! warm_startrs/f ,,'../FGH,-?@# " !?!2!7!7!9!;uE/.-3358/h-11(;MX&'++H5J(#1 mm #V ,,j.G.G""%/%>%>%B&0%>%>HM((-*F;++A/A CD! '11'66+$..(77(88( 33   Q ELL m2{4 5 x=A a[( -h F# ~ - --88>,*+;<9! "  / /0G0> @    +.'(q8/t))*A*8:$$&(*<=2779:_L ,,+->?  %f%;%6%;%;%=?  @@   V/1B1G1G1IJ  LLr"r4)r#rNNr)z.*NN)r' collectionstensorflow.python.frameworkrrtensorflow.python.opsrrrr@tensorflow.python.platformrrutensorflow.python.trainingr r r r!tensorflow.python.training.savingr tensorflow.python.util.tf_exportr namedtuplerr1rFrorrrr"r!rs3.++0<<579B6  !"`K;)`#`F**R463404+/$%t:t4n$@N !"#"&&*)-[L$[Lr"