AVhrdZddlZddlZddlmZddlmZddlmZddlm Z ddlm Z ddlm Z dd lm Z dd lm Z dd lmZdd lmZdd lmZddlmZddlmZddlmZGddZeddgej0dGddeZeddgej0ddGddeZeddgej0dGddeZed d!gej0d Gd"d#eZed$d%gej0d$Gd&d'eZed(d)gej0d(d)Gd*d+eZed,d-gej0d-d,Gd.d/eZed0d1gej0d0d1Gd2d3eZ ed4d5gej0d4d5Gd6d7eZ!Gd8d9eZ"Gd:d;eZ#Gd<d=e#Z$Gd>d?e#Z%Gd@dAe#Z&edBgej0dBGdCdDeZ'edEdFgej0dEdFGdGdHe Z(edIdJgej0dIdJGdKdLe Z)eZ*eZ+eZ,eZ-eZ.eZ/eZ0e Z1e(Z2e)Z3e!Z4e'Z5e"Z6e%Z7e$Z8e&Z9edMgdWdNZ:edOgdWdPZ;edQgdWdRZ<edSgdWdTZ=dUZ>dVZ?y)XaOperations often used for initializing tensors. All variable initializers returned by functions in this file should have the following signature: def _initializer(shape, dtype=dtypes.float32, partition_info=None): Args: shape: List of `int` representing the shape of the output `Tensor`. Some initializers may also be able to accept a `Tensor`. dtype: (Optional) Type of the output `Tensor`. partition_info: (Optional) variable_scope._PartitionInfo object holding additional information about how the variable is partitioned. May be `None` if the variable is not partitioned. Returns: A `Tensor` of type `dtype` and `shape`. N) constant_op)dtypes) tensor_shape) array_ops)array_ops_stack)gen_linalg_ops)linalg_ops_impl)math_ops) random_ops) deprecation) deprecated)deprecated_arg_values)deprecated_args) tf_exportc.eZdZdZddZdZedZy) InitializerzAInitializer base class: all initializers inherit from this class.Nct)a4Returns a tensor object initialized as specified by the initializer. Args: shape: Shape of the tensor. dtype: Optional dtype of the tensor. If not provided use the initializer dtype. partition_info: Optional information about the possible partitioning of a tensor. NotImplementedErrorselfshapedtypepartition_infos N/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/init_ops.py__call__zInitializer.__call__7s  ciS)zReturns the configuration of the initializer as a JSON-serializable dict. Returns: A JSON-serializable Python dict. rs r get_configzInitializer.get_configCs Irc|di|S)a{Instantiates an initializer from a configuration dictionary. Example: ```python initializer = RandomUniform(-1, 1) config = initializer.get_config() initializer = RandomUniform.from_config(config) ``` Args: config: A Python dictionary. It will typically be the output of `get_config`. Returns: An Initializer instance. rr)clsconfigs r from_configzInitializer.from_configKs& ==rNN)__name__ __module__ __qualname____doc__rr! classmethodr%rrrrr4s#I rrzinitializers.zeroszeros_initializer)v1cVeZdZdZedddej fdZddZdZ y) ZerosaInitializer that generates tensors initialized to 0. @compatibility(TF2) `tf.compat.v1.zeros_initializer` is compatible with eager execution and `tf.function`. To migrate to TF2, please use `tf.zeros_initializer` instead. The `dtype` argument in `tf.compat.v1.zeros_initializer.__init__()` does not exist in `tf.zeros_initializer.__init__()`. However, you can specify the `dtype` in `__call__()` in both cases. #### Structural Mapping to TF2 Before: ```python initializer = tf.compat.v1.zeros_initializer(dtype=tf.float32) variable = tf.Variable(initializer(shape=[3, 3])) ``` After: ```python initializer = tf.zeros_initializer() variable = tf.Variable(initializer(shape=[3, 3], dtype=tf.float32)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :------------------- | :--------------- | :------------------------- | | `dtype` | `dtype` | In `__call__()` method | | `partition_info` | - | (`__call__` arg in TF1) Not supported | #### Before & After Usage Example Before: >>> initializer = tf.compat.v1.zeros_initializer(dtype=tf.float32) >>> tf.Variable(initializer(shape=[3])).numpy() array([0., 0., 0.], dtype=float32) >>> tf.Variable(initializer(shape=[3, 3])).numpy() array([[0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], dtype=float32) >>> initializer = tf.compat.v1.zeros_initializer() >>> tf.Variable(initializer(shape=[3], dtype=tf.float32)).numpy() array([0., 0., 0.], dtype=float32) >>> tf.Variable(initializer(shape=[3, 3], dtype=tf.float32)).numpy() array([[0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], dtype=float32) After: >>> initializer = tf.zeros_initializer() >>> tf.Variable(initializer(shape=[3], dtype=tf.float32)).numpy() array([0., 0., 0.], dtype=float32) >>> tf.Variable(initializer(shape=[3, 3], dtype=tf.float32)).numpy() array([[0., 0., 0.], [0., 0., 0.], [0., 0., 0.]], dtype=float32) @end_compatibility NZCall initializer instance with the dtype argument instead of passing it to the constructorrc8tj||_yNras_dtyperrrs r__init__zZeros.__init__'DJrcJ| |j}tj||Sr2)rrzerosrs rrzZeros.__call__s! }jje ??5% ((rc2d|jjiSNrrnamer s rr!zZeros.get_config TZZ__ %%rr& r'r(r)r*rrfloat32r6rr!rrrr/r/asAAF467>@">>(@() &rr/zinitializers.onesones_initializercVeZdZdZedddej fdZddZdZ y) Onesa:Initializer that generates tensors initialized to 1. @compatibility(TF2) This API is compatible with TF2 behavior and `tf.function`, and can be migrated immediately with `tf.keras.initializers.ones`. Before: >>> initializer = tf.compat.v1.keras.initializers.ones() >>> initializer((1, 1)) After: >>> initializer = tf.keras.initializers.ones() >>> initializer((1, 1)) @end_compatibility Nr0rc8tj||_yr2r3r5s rr6z Ones.__init__r7rcJ| |j}tj||Sr2)rronesrs rrz Ones.__call__s! }jje >>% ''rc2d|jjiSr;r<r s rr!zOnes.get_configr>rr&r?rrrrCrCs?&467>@">>(@(( &rrCzinitializers.constantconstant_initializercteZdZdZedddeddddej dfd Zd d Zd Z y) ConstantaiInitializer that generates tensors with constant values. The resulting tensor is populated with values of type `dtype`, as specified by arguments `value` following the desired `shape` of the new tensor (see examples below). The argument `value` can be a constant value, or a list of values of type `dtype`. If `value` is a list, then the length of the list must be less than or equal to the number of elements implied by the desired shape of the tensor. In the case where the total number of elements in `value` is less than the number of elements required by the tensor shape, the last element in `value` will be used to fill the remaining entries. If the total number of elements in `value` is greater than the number of elements required by the tensor shape, the initializer will raise a `ValueError`. Args: value: A Python scalar, list or tuple of values, or a N-dimensional numpy array. All elements of the initialized variable will be set to the corresponding value in the `value` argument. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. verify_shape: Boolean that enables verification of the shape of `value`. If `True`, the initializer will throw an error if the shape of `value` is not compatible with the shape of the initialized tensor. Raises: TypeError: If the input `value` is not one of the expected types. Examples: The following example can be rewritten using a numpy.ndarray instead of the `value` list, even reshaped, as shown in the two commented lines below the `value` list initialization. >>> value = [0, 1, 2, 3, 4, 5, 6, 7] >>> init = tf.compat.v1.constant_initializer(value) >>> # fitting shape >>> with tf.compat.v1.Session(): ... x = tf.compat.v1.get_variable('x', shape=[2, 4], initializer=init) ... x.initializer.run() ... print(x.eval()) [[0. 1. 2. 3.] [4. 5. 6. 7.]] >>> # Larger shape >>> with tf.compat.v1.Session(): ... y = tf.compat.v1.get_variable('y', shape=[3, 4], initializer=init) ... y.initializer.run() ... print(y.eval()) [[0. 1. 2. 3.] [4. 5. 6. 7.] [7. 7. 7. 7.]] >>> # Smaller shape >>> with tf.compat.v1.Session(): ... z = tf.compat.v1.get_variable('z', shape=[2, 3], initializer=init) Traceback (most recent call last): ... ValueError: Too many elements provided. Needed at most 6, but received 8 >>> # Shape verification >>> init_verify = tf.compat.v1.constant_initializer(value, verify_shape=True) >>> with tf.compat.v1.Session(): ... u = tf.compat.v1.get_variable('u', shape=[3, 4], ... initializer=init_verify) Traceback (most recent call last): ... TypeError: Expected Tensor's shape: (3, 4), got (8,). @compatibility(TF2) Although it is a legacy API endpoint, `tf.compat.v1.constant_initializer` is compatible with eager execution and `tf.function`. To migrate to a non-legacy TF2 API, please use `tf.constant_initializer` instead. The `dtype` argument in `tf.compat.v1.constant_initializer.__init__()` does not exist in `tf.constant_initializer.__init__()`. However, you can specify the `dtype` in `__call__()` in both cases. In the `compat.v1` symbol, if `verify_shape` is set to `True`, an exception is raised when initializing a variable with a different shape from `value`. If set to `False`, `value` is reshaped to initialize the variable if necessary. An exception would only be raised when the number of elements are different. The `verify_shape` argument is not supported in TF2. Using `tf.constant_initializer` is equivalent to setting `verify_shape` to `False`. #### Structural Mapping to TF2 Before: ```python value = [0, 1, 2, 3, 4, 5, 6, 7] initializer = tf.compat.v1.constant_initializer( value=value, dtype=tf.float32, verify_shape=False) variable = tf.Variable(initializer(shape=[2, 4])) ``` After: ```python value = [0, 1, 2, 3, 4, 5, 6, 7] initializer = tf.constant_initializer(value=value) tf.Variable(initializer(shape=[2, 4], dtype=tf.float32)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :-------------------- | :--------------- | :-------------------------- | | `value` | `value` | In constructor | | `dtype` | `dtype` | In `__call__()` method | | `verify_shape` | Not Supported | Equivalent to set to `False`| | `partition_info` | - | (`__call__` arg in TF1) Not supported | #### Before & After Usage Example Before: >>> value = [1., 2., 3., 4.] >>> initializer = tf.compat.v1.constant_initializer( ... value=value, dtype=tf.float32, verify_shape=True) >>> tf.Variable(initializer(shape=[2, 2])).numpy() Traceback (most recent call last): ... TypeError: Expected Tensor's shape: (2, 2), got (4,). >>> initializer = tf.compat.v1.constant_initializer( ... value=value, dtype=tf.float32, verify_shape=False) >>> tf.Variable(initializer(shape=[2, 2])).numpy() array([[1., 2.], [3., 4.]], dtype=float32) After: >>> value = [1., 2., 3., 4.] >>> initializer = tf.constant_initializer(value=value) >>> tf.Variable(initializer(shape=[2, 2], dtype=tf.float32)).numpy() array([[1., 2.], [3., 4.]], dtype=float32) @end_compatibility Nr0rzCObjects must now be the required shape or no shape can be specified verify_shaperFctj|sJt|tttj fs%t d|dt|jd||_ tj||_ ||_ y)NzInvalid type for initial value=z of type: zD. Expected Python scalar, list or tuple of values, or numpy.ndarray.)npisscalar isinstancelisttuplendarray TypeErrortyper'valuerr4r _verify_shape)rrUrrKs rr6zConstant.__init__msu KK *UT5"**4M"N  +E7* %[ ! ! "#& & '' DJ'DJ%Drc| |j}| |j}tj|j|||S)N)rrrK)rrVr constant_v1rU)rrrrrKs rrzConstant.__call__}sC }jje''l  " " %u< IIrcH|j|jjdS)N)rUr)rUrr=r s rr!zConstant.get_configs ZZ$**// ::r)NNNr?rrrrJrJs]M^467>@4&'57FNN &7@ &I;rrJzinitializers.random_uniformrandom_uniform_initializerc\eZdZdZeddddddej fdZd dZdZ y) RandomUniformaRInitializer that generates tensors with a uniform distribution. Args: minval: A python scalar or a scalar tensor. Lower bound of the range of random values to generate. maxval: A python scalar or a scalar tensor. Upper bound of the range of random values to generate. Defaults to 1 for float types. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. @compatibility(TF2) Although it is a legacy compat.v1 API, this symbol is compatible with eager execution and `tf.function`. To switch to TF2, switch to using either `tf.initializers.RandomUniform` or `tf.keras.initializers.RandomUniform` (neither from `compat.v1`) and pass the dtype when calling the initializer. Keep in mind that the default minval, maxval and the behavior of fixed seeds have changed. #### Structural Mapping to TF2 Before: ```python initializer = tf.compat.v1.random_uniform_initializer( minval=minval, maxval=maxval, seed=seed, dtype=dtype) weight_one = tf.Variable(initializer(shape_one)) weight_two = tf.Variable(initializer(shape_two)) ``` After: ```python initializer = tf.initializers.RandomUniform( minval=minval, maxval=maxval, seed=seed) weight_one = tf.Variable(initializer(shape_one, dtype=dtype)) weight_two = tf.Variable(initializer(shape_two, dtype=dtype)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :-------------------- | :-------------- | :------------------------- | | `minval` | `minval` | Default changes from 0 to -0.05 | | `maxval` | `maxval` | Default changes from 1.0 to 0.05 | | `seed` | `seed` | | | `dtype` | `dtype` | The TF2 native api only takes it | : : : as a `__call__` arg, not a constructor arg. : | `partition_info` | - | (`__call__` arg in TF1) Not supported | @end_compatibility Nr0rcb||_||_||_tj||_yr2)minvalmaxvalseedrr4r)rr_r`rars rr6zRandomUniform.__init__s)DKDKDI'DJrc| |j}tj||j|j||j SNra)rr random_uniformr_r`rars rrzRandomUniform.__call__s< }jje  $ $ t{{DKKTYY @@rct|j|j|j|jjdS)N)r_r`rar)r_r`rarr=r s rr!zRandomUniform.get_configs,++++   rr&r?rrrr\r\sG=~467>@t$fnn(@( @ rr\zinitializers.random_normalrandom_normal_initializerc\eZdZdZeddddddej fdZd dZd Z y) RandomNormalaYInitializer that generates tensors with a normal distribution. Args: mean: a python scalar or a scalar tensor. Mean of the random values to generate. stddev: a python scalar or a scalar tensor. Standard deviation of the random values to generate. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. @compatibility(TF2) Although it is a legacy `compat.v1` API, this symbol is compatible with eager execution and `tf.function`. To switch to TF2, switch to using either `tf.initializers.RandomNormal` or `tf.keras.initializers.RandomNormal` (neither from `compat.v1`) and pass the dtype when calling the initializer. Keep in mind that the default stddev and the behavior of fixed seeds have changed. #### Structural Mapping to TF2 Before: ```python initializer = tf.compat.v1.random_normal_initializer( mean=mean, stddev=stddev, seed=seed, dtype=dtype) weight_one = tf.Variable(initializer(shape_one)) weight_two = tf.Variable(initializer(shape_two)) ``` After: ```python initializer = tf.initializers.RandomNormal( mean=mean, seed=seed, stddev=stddev) weight_one = tf.Variable(initializer(shape_one, dtype=dtype)) weight_two = tf.Variable(initializer(shape_two, dtype=dtype)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :----------------- | :-------------- | :------------------------- | | `mean` | `mean` | No change to defaults | | `stddev` | `stddev` | Default changes from 1.0 to 0.05 | | `seed` | `seed` | | | `dtype` | `dtype` | The TF2 native api only takes it as a | : : : `__call__` arg, not a constructor arg. : | `partition_info` | - | (`__call__` arg in TF1) Not supported. | @end_compatibility Nr0rr]?ct||_||_||_tt j ||_yr2meanstddevra_assert_float_dtyperr4rrrmrnrars rr6zRandomNormal.__init__).DIDKDI$V__U%;>rct|j|j|j|jjdSN)rmrnrarrmrnrarr=r s rr!zRandomNormal.get_config8, ++   rr&r?rrrririsF=~467>@cFNN=@= > rrizinitializers.truncated_normaltruncated_normal_initializerc\eZdZdZeddddddej fdZd dZd Z y) TruncatedNormala. Initializer that generates a truncated normal distribution. These values are similar to values from a `random_normal_initializer` except that values more than two standard deviations from the mean are discarded and re-drawn. This is the recommended initializer for neural network weights and filters. Args: mean: a python scalar or a scalar tensor. Mean of the random values to generate. stddev: a python scalar or a scalar tensor. Standard deviation of the random values to generate. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. @compatibility(TF2) Although it is a legacy `compat.v1` API, this symbol is compatible with eager execution and `tf.function`. To switch to TF2, switch to using either `tf.initializers.truncated_normal` or `tf.keras.initializers.TruncatedNormal` (neither from `compat.v1`) and pass the dtype when calling the initializer. Keep in mind that the default stddev and the behavior of fixed seeds have changed. #### Structural Mapping to TF2 Before: ```python initializer = tf.compat.v1.truncated_normal_initializer( mean=mean, stddev=stddev, seed=seed, dtype=dtype) weight_one = tf.Variable(initializer(shape_one)) weight_two = tf.Variable(initializer(shape_two)) ``` After: ```python initializer = tf.initializers.truncated_normal( mean=mean, seed=seed, stddev=stddev) weight_one = tf.Variable(initializer(shape_one, dtype=dtype)) weight_two = tf.Variable(initializer(shape_two, dtype=dtype)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :-------------------- | :-------------- | :------------------------- | | `mean` | `mean` | No change to defaults | | `stddev` | `stddev` | Default changes from 1.0 to 0.05 | | `seed` | `seed` | | | `dtype` | `dtype` | The TF2 native api only takes it | : : : as a `__call__` arg, not a constructor arg. : | `partition_info` | - | (`__call__` arg in TF1) Not supported | @end_compatibility Nr0rr]rjct||_||_||_tt j ||_yr2rlrps rr6zTruncatedNormal.__init__rqrc| |j}tj||j|j||j Src)rr truncated_normalrmrnrars rrzTruncatedNormal.__call__s: }jje  & & tyy$++u499 >>rct|j|j|j|jjdSrurvr s rr!zTruncatedNormal.get_configrwrr&r?rrrrzrzAsGBH467>@cFNN=@= > rrzz!initializers.uniform_unit_scaling uniform_unit_scaling_initializercreZdZdZedddeddddejfdZd dZ d Z y) UniformUnitScalingaInitializer that generates tensors without scaling variance. When initializing a deep network, it is in principle advantageous to keep the scale of the input variance constant, so it does not explode or diminish by reaching the final layer. If the input is `x` and the operation `x * W`, and we want to initialize `W` uniformly at random, we need to pick `W` from [-sqrt(3) / sqrt(dim), sqrt(3) / sqrt(dim)] to keep the scale intact, where `dim = W.shape[0]` (the size of the input). A similar calculation for convolutional networks gives an analogous result with `dim` equal to the product of the first 3 dimensions. When nonlinearities are present, we need to multiply this by a constant `factor`. See (Sussillo et al., 2014) for deeper motivation, experiments and the calculation of constants. In section 2.3 there, the constants were numerically computed: for a linear layer it's 1.0, relu: ~1.43, tanh: ~1.15. Args: factor: Float. A multiplicative factor by which the values will be scaled. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Sussillo et al., 2014](https://arxiv.org/abs/1412.6558) ([pdf](http://arxiv.org/pdf/1412.6558.pdf)) Nr0rzbUse tf.initializers.variance_scaling instead with distribution=uniform to get equivalent behavior.rjcf||_||_ttj||_yr2)factorrarorr4r)rrrars rr6zUniformUnitScaling.__init__s'DKDI$V__U%;@d45 d&..=5@ = 9&Prrzinitializers.variance_scalingvariance_scaling_initializerczeZdZdZedddeddddd d dejfd Zdd Z d Z y)VarianceScalingah Initializer capable of adapting its scale to the shape of weights tensors. @compatibility(TF2) Although it is a legacy `compat.v1` API, this symbol is compatible with eager execution and `tf.function`. To switch to TF2 APIs, move to using either `tf.initializers.variance_scaling` or `tf.keras.initializers.VarianceScaling` (neither from `compat.v1`) and pass the dtype when calling the initializer. #### Structural Mapping to TF2 Before: ```python initializer = tf.compat.v1.variance_scaling_initializer( scale=scale, mode=mode, distribution=distribution seed=seed, dtype=dtype) weight_one = tf.Variable(initializer(shape_one)) weight_two = tf.Variable(initializer(shape_two)) ``` After: ```python initializer = tf.keras.initializers.VarianceScaling( scale=scale, mode=mode, distribution=distribution seed=seed) weight_one = tf.Variable(initializer(shape_one, dtype=dtype)) weight_two = tf.Variable(initializer(shape_two, dtype=dtype)) ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :----------------- | :-------------- | :------------------------- | | `scale` | `scale` | No change to defaults | | `mode` | `mode` | No change to defaults | | `distribution` | `distribution` | No change to defaults. | : : : 'normal' maps to 'truncated_normal' : | `seed` | `seed` | | | `dtype` | `dtype` | The TF2 api only takes it | : : : as a `__call__` arg, not a constructor arg. : | `partition_info` | - | (`__call__` arg in TF1) Not supported | @end_compatibility With `distribution="truncated_normal" or "untruncated_normal"`, samples are drawn from a truncated/untruncated normal distribution with a mean of zero and a standard deviation (after truncation, if used) `stddev = sqrt(scale / n)` where n is: - number of input units in the weight tensor, if mode = "fan_in" - number of output units, if mode = "fan_out" - average of the numbers of input and output units, if mode = "fan_avg" With `distribution="uniform"`, samples are drawn from a uniform distribution within [-limit, limit], with `limit = sqrt(3 * scale / n)`. Args: scale: Scaling factor (positive float). mode: One of "fan_in", "fan_out", "fan_avg". distribution: Random distribution to use. One of "normal", "uniform". seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. Raises: ValueError: In case of an invalid value for the "scale", mode" or "distribution" arguments. Nr0rz5`normal` is a deprecated alias for `truncated_normal`normal) distributionrjfan_inr}c|dkrtd||dvrtd||j}|dvrtd|||_||_||_||_t tj||_ y)Nr]z5Argument `scale` must be a positive float. Received: >rfan_avgfan_outzMArgument `mode` should be one of ('fan_in', 'fan_out', 'fan_avg'). Received: >runiformr}untruncated_normalzsArgument `distribution` should be one of ('normal', uniform', 'truncated_normal', 'untruncated_normal'). Received: ) ValueErrorlowerscalemoderrarorr4r)rrrrrars rr6zVarianceScaling.__init__:s { N" ## 33 004v7 88%%'L $$0>3 44DJDI$DDI$V__U%;@ = .^^ = @=276rrzinitializers.orthogonalorthogonal_initializercZeZdZdZedddddej fdZd dZdZ y) OrthogonalaInitializer that generates an orthogonal matrix. If the shape of the tensor to initialize is two-dimensional, it is initialized with an orthogonal matrix obtained from the QR decomposition of a matrix of random numbers drawn from a normal distribution. If the matrix has fewer rows than columns then the output will have orthogonal rows. Otherwise, the output will have orthogonal columns. If the shape of the tensor to initialize is more than two-dimensional, a matrix of shape `(shape[0] * ... * shape[n - 2], shape[n - 1])` is initialized, where `n` is the length of the shape vector. The matrix is subsequently reshaped to give a tensor of the desired shape. Args: gain: multiplicative factor to apply to the orthogonal matrix seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Saxe et al., 2014](https://openreview.net/forum?id=_wzZwKpTDF_9C) ([pdf](https://arxiv.org/pdf/1312.6120.pdf)) Nr0rrjcf||_ttj||_||_yr2gainrorr4rrarrrars rr6zOrthogonal.__init__s'DI$V__U%;@D@ 3>Lrrc@eZdZdZddej fdZddZdZy)ConvolutionDeltaOrthogonalaInitializer that generates a delta orthogonal kernel for ConvNets. The shape of the tensor must have length 3, 4 or 5. The number of input filters must not exceed the number of output filters. The center pixels of the tensor form an orthogonal matrix. Other pixels are set to be zero. See algorithm 2 in (Xiao et al., 2018). Args: gain: Multiplicative factor to apply to the orthogonal matrix. Default is 1. The 2-norm of an input is multiplied by a factor of `gain` after applying this convolution. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Xiao et al., 2018](http://proceedings.mlr.press/v80/xiao18a.html) ([pdf](http://proceedings.mlr.press/v80/xiao18a/xiao18a.pdf)) rjNcf||_ttj||_||_yr2rrs rr6z#ConvolutionDeltaOrthogonal.__init__%DI$V__U%;> $ $U1XuQxq BF hmmDIIU33F Mrctjt|Dcgc]3}tjt|Dcgc] }|||f c}5c}}Scc}wcc}}w)zConvert a dictionary to a tensor. Args: x: A k1 * k2 dictionary. k1: First dimension of x. k2: Second dimension of x. Returns: A k1 * k2 tensor. rstackrange)rxk1k2ijs r_dict_to_tensorz'ConvolutionOrthogonal2D._dict_to_tensorys]  EJ2Y"P@Ab :1qAw:;"P QQ:"Ps!A# A A# A# c|jj|jjk7r&td|jd|jd|jjd}i}tj||j }t j|||d<t j|||z |d<t j||z ||d<t j||z ||z |d <|S) a_Construct a 2 x 2 kernel. Used to construct orthgonal kernel. Args: p1: A symmetric projection matrix. p2: A symmetric projection matrix. Returns: A 2 x 2 kernel [[p1p2, p1(1-p2)], [(1-p1)p2, (1-p1)(1-p2)]]. Raises: ValueError: If the dimensions of p1 and p2 are different. BThe dimension of the matrices must be the same. Received p1.shape=z and p2.shape=rrrrrrr)rr)rr)ras_listrr eyerr r)rp1p2r kernel2x2rs r _block_orthz#ConvolutionOrthogonal2D._block_orths xxRXX--// ,,.HH:^BHH:QP QQ 1AI   atzz 2Coob"-IdOoob385IdOoosRx"5IdOoosRx38=IdO rc |djjd}||djjdk7r,td|djd|djdtt j t |}tt j t |}i}||zdz }t|D]}t|D]} tj||g|j||| f<tt||dzD]d} tt|| dzD]G} || z |ks | | z |ks||| fxxtj|| | f||| z | | z fz cc<If|S)aOMatrix convolution. Args: m1: A k x k dictionary, each element is a n x n matrix. m2: A l x l dictionary, each element is a n x n matrix. Returns: (k + l - 1) * (k + l - 1) dictionary each element is a n x n matrix. Raises: ValueError: if the entries of m1 and m2 are of different dimensions. rrzYThe entries in matrices m1 and m2 must have the same dimensions. Received m1[0, 0].shape=z and m2[0, 0].shape=rr)rrrrrMrrrrr9rminr r) rm1m2rklresultsizerrindex1index2s r _matrix_convz$ConvolutionOrthogonal2D._matrix_convs D  "1%ARX   $ $ &q )) >>@hnn=MN--/X^^, cout. "The number of input channels (cin=4) cannot exceed the number of output channels (cout=).rNrr rrrrrrrr r rr) rksizecincoutorthp_temprrs rrz*ConvolutionOrthogonal2D._orthogonal_kernelsY Tz ;C5A??CfBH II  " "4 (3 2D z  " "9#8#8q#A1 EE  ""4($*D*DT*J LA 519 %     $ $T *D,F,Ft,LNd   At $a%5\1U|1!//$!Q$0!Q$11   5% 00rr& r'r(r)r*rrrr rrrrrrPs"(( Q8B1rrc0eZdZdZddZdZdZdZdZy) ConvolutionOrthogonal1DaInitializer that generates a 1D orthogonal kernel for ConvNets. The shape of the tensor must have length 3. The number of input filters must not exceed the number of output filters. The orthogonality(==isometry) is exact when the inputs are circular padded. There are finite-width effects with non-circular padding (e.g. zero padding). See algorithm 1 in (Xiao et al., 2018). Args: gain: Multiplicative factor to apply to the orthogonal matrix. Default is 1. The 2-norm of an input is multiplied by a factor of `gain` after applying this convolution. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Xiao et al., 2018](http://proceedings.mlr.press/v80/xiao18a.html) ([pdf](http://proceedings.mlr.press/v80/xiao18a/xiao18a.pdf)) Nc | |j}t|dk7rtd||d|dkDrtd|dd|dd|j|d|d|d}|t j |j | z}|S) NrzbThe tensor to initialize, specified by argument `shape` must be three-dimensional. Received shape=rrrrrrrrrs rrz ConvolutionOrthogonal1D.__call__s }jje 5zQ EEJGM NN Ry59  ?d   At $a%5\) __T1Q4 (ad)   5 ))rr&rrrrrrs * ;$<*rrc0eZdZdZddZdZdZdZdZy) ConvolutionOrthogonal3DaInitializer that generates a 3D orthogonal kernel for ConvNets. The shape of the tensor must have length 5. The number of input filters must not exceed the number of output filters. The orthogonality(==isometry) is exact when the inputs are circular padded. There are finite-width effects with non-circular padding (e.g. zero padding). See algorithm 1 (Xiao et al., 2018). Args: gain: Multiplicative factor to apply to the orthogonal matrix. Default is 1. The 2-norm of an input is multiplied by a factor of `gain` after applying this convolution. seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Xiao et al., 2018](http://proceedings.mlr.press/v80/xiao18a.html) ([pdf](http://proceedings.mlr.press/v80/xiao18a/xiao18a.pdf)) Nc | |j}t|dk7rtd||d|dkDrtd|dd|dd|d|d k7s |d|d k7rtd |dd |d d |d d|j|d|d|d}|t j |j |z}|S)NrzaThe tensor to initialize, specified by argument `shape` must be five-dimensional. Received shape=rrrrrrrrrz , shape[1]=z and shape[2]=rrrrs rrz ConvolutionOrthogonal3D.__call__}s }jje 5zQ DDI7L MM Ry59 tjt|Dcgc]`}tjt|Dcgc]4}tjt|Dcgc] }||||f c}6c}}bc}}}Scc}wcc}}wcc}}}w)zConvert a dictionary to a tensor. Args: x: A k1 * k2 dictionary. k1: First dimension of x. k2: Second dimension of x. k3: Third dimension of x. Returns: A k1 * k2 * k3 tensor. r)rrrrk3rrrs rrz'ConvolutionOrthogonal3D._dict_to_tensors  ',Ry"2"2"##2"7"7)     E"I>q!Q' > ? #"2 33> "2s)"B !B"B "1 B: B BB c  |jj}||jjk7s||jjk7r3td|jd|jd|jd|d}tj||j  i}d} fd}d D]5} d D].} d D]'} ||| ||| ||| ||| | | f<)07|S) aEConstruct a 3 x 3 kernel. Used to construct orthgonal kernel. Args: p1: A symmetric projection matrix. p2: A symmetric projection matrix. p3: A symmetric projection matrix. Returns: A 2 x 2 x 2 kernel. Raises: ValueError: If the dimensions of p1, p2 and p3 are different. rz , p2.shape=z and p3.shape=rrrcVtjtj|||Sr2)r r)rrp3s rrz3ConvolutionOrthogonal3D._block_orth..matmuls __X__R4b 99rc&||zd|z |z zzS)zReturn p or (1-p).rr)rrrs rrz1ConvolutionOrthogonal3D._block_orth..casts Ua!ea( ((rr)rrrr rr) rrrr*p1_shaper kernel2x2x2rrrrrrs @rrz#ConvolutionOrthogonal3D._block_orthsxx!H288##%%RXX5E5E5G)G ,,.HH:[ K$$&HH:Q0 11  A   atzz 2CK:)OO! OA!'Q T!R[$q"+!N+aAg  OOO rc|djjd}||djjdk7r,td|djd|djdtt j t |}tt j t |}i}||zdz }t|D]}t|D]} t|D]} tj||g|j||| | f<tt||dzD]} tt|| dzD]s} tt|| dzD]V} || z |ks | | z |ks| | z |ks||| | fxxtj|| | | f||| z | | z | | z fz cc<Xu|S)arMatrix convolution. Args: m1: is a k x k x k dictionary, each element is a n x n matrix. m2: is a l x l x l dictionary, each element is a n x n matrix. Returns: (k + l - 1) x (k + l - 1) x (k + l - 1) dictionary each element is a n x n matrix. Raises: ValueError: if the entries of m1 and m2 are of different dimensions. )rrrrz\The entries in matrices m1 and m2 must have the same dimensions. Received m1[0, 0, 0].shape=z and m2[0, 0, 0].shape=rr)rrrrrMcbrtrrrr9rrr r)rrrrrrrrrrrr r index3s rr z$ConvolutionOrthogonal3D._matrix_convs G##%a(AR[   ' ' )! ,, AW+++,,CW+++,A/ 00 BGGCG A BGGCG A F q519D 4[ >T{ >!t >A%OOQFDJJ?&Aq/c!QUm, >fAq1u . >!#aQ-0>&J!#Vq(8a&jA=MAq/X__/0VQZV;<&>>/> > > > > > Mrc ||kDrtd|d|d|j|d|ddf}|dk(r>tjtjtj|dddS|j |j ||j ||j |}t |dz D]T}|j |j ||j ||j |}|j||}Vt |D]D}t |D]4} t |D]$} tj|||| | f||| | f<&6F|j||||S)aConstruct orthogonal kernel for convolution. Args: ksize: Kernel size. cin: Number of input channels. cout: Number of output channels. Returns: An [ksize, ksize, ksize, cin, cout] orthogonal kernel. Raises: ValueError: If cin > cout. rrrrNrrr) rrrrrrrrrrrs rrz*ConvolutionOrthogonal3D._orthogonal_kernels Tz ;C5A??CfBH II  " "4 (3 2D z  " "    5 5dA > BAGG  ""4($*D*DT*J ""4( *A519 %     $ $T *D,F,Ft,L  $ $T *,d   At $a % 5\9U|9!u 9AtQq!QwZ8!Aq!G* 999   5% 77rr&rrrrr$r$gs"**3"#J$L"8rr$zinitializers.identitycXeZdZdZeddddej fdZd dZdZ y) Identitya,Initializer that generates the identity matrix. Only use for 2D matrices. Args: gain: Multiplicative factor to apply to the identity matrix. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. Nr0rrjcX||_ttj||_yr2)rrorr4r)rrrs rr6zIdentity.__init__ s DI$V__U%;@V^^=@=# 9rr3glorot_uniform_initializerzinitializers.glorot_uniformc\eZdZdZeddddej ffd ZdZxZ S) GlorotUniformaThe Glorot uniform initializer, also called Xavier uniform initializer. It draws samples from a uniform distribution within [-limit, limit] where `limit` is `sqrt(6 / (fan_in + fan_out))` where `fan_in` is the number of input units in the weight tensor and `fan_out` is the number of output units in the weight tensor. Args: seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Glorot et al., 2010](http://proceedings.mlr.press/v9/glorot10a.html) ([pdf](http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf)) Nr0rc4tt| ddd|y)Nrjrrrrrra)superr>r6rrar __class__s rr6zGlorotUniform.__init__Ps# -' (FrcH|j|jjdSN)rarrarr=r s rr!zGlorotUniform.get_configWr;r r'r(r)r*rrr@r6r! __classcell__rCs@rr>r>;s?"467>@fnnF@F9rr>glorot_normal_initializerzinitializers.glorot_normalc\eZdZdZeddddej ffd ZdZxZ S) GlorotNormala)The Glorot normal initializer, also called Xavier normal initializer. It draws samples from a truncated normal distribution centered on 0 with standard deviation (after truncation) given by `stddev = sqrt(2 / (fan_in + fan_out))` where `fan_in` is the number of input units in the weight tensor and `fan_out` is the number of output units in the weight tensor. Args: seed: A Python integer. Used to create random seeds. See `tf.compat.v1.set_random_seed` for behavior. dtype: Default data type, used if no `dtype` argument is provided when calling the initializer. Only floating point types are supported. References: [Glorot et al., 2010](http://proceedings.mlr.press/v9/glorot10a.html) ([pdf](http://jmlr.org/proceedings/papers/v9/glorot10a/glorot10a.pdf)) Nr0rc4tt| ddd|y)Nrjrr}r@)rArLr6rBs rr6zGlorotNormal.__init__qs$ ,& 0B'OrcH|j|jjdSrErFr s rr!zGlorotNormal.get_configxr;rrGrIs@rrLrL[s?$467>@fnnO@O9rrLzinitializers.lecun_normalc tddd|S)aLeCun normal initializer. It draws samples from a truncated normal distribution centered on 0 with standard deviation (after truncation) given by `stddev = sqrt(1 / fan_in)` where `fan_in` is the number of input units in the weight tensor. Args: seed: A Python integer. Used to seed the random generator. Returns: An initializer. References: - Self-Normalizing Neural Networks, [Klambauer et al., 2017](https://papers.nips.cc/paper/6698-self-normalizing-neural-networks) # pylint: disable=line-too-long ([pdf](https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf)) - Efficient Backprop, [Lecun et al., 1998](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) rjrr}r@rrds r lecun_normalrQs0  X,>T KKrzinitializers.lecun_uniformc tddd|S)aLeCun uniform initializer. It draws samples from a uniform distribution within [-limit, limit] where `limit` is `sqrt(3 / fan_in)` where `fan_in` is the number of input units in the weight tensor. Args: seed: A Python integer. Used to seed the random generator. Returns: An initializer. References: - Self-Normalizing Neural Networks, [Klambauer et al., 2017](https://papers.nips.cc/paper/6698-self-normalizing-neural-networks) # pylint: disable=line-too-long ([pdf](https://papers.nips.cc/paper/6698-self-normalizing-neural-networks.pdf)) - Efficient Backprop, [Lecun et al., 1998](http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf) rjrrr@rPrds r lecun_uniformrSs.  XID BBrzinitializers.he_normalc tddd|S)aHe normal initializer. It draws samples from a truncated normal distribution centered on 0 with standard deviation (after truncation) given by `stddev = sqrt(2 / fan_in)` where `fan_in` is the number of input units in the weight tensor. Args: seed: A Python integer. Used to seed the random generator. Returns: An initializer. References: [He et al., 2015] (https://www.cv-foundation.org/openaccess/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html) # pylint: disable=line-too-long ([pdf](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf)) rrr}r@rPrds r he_normalrUs*  X,>T KKrzinitializers.he_uniformc tddd|S)aHe uniform variance scaling initializer. It draws samples from a uniform distribution within [-limit, limit] where `limit` is `sqrt(6 / fan_in)` where `fan_in` is the number of input units in the weight tensor. Args: seed: A Python integer. Used to seed the random generator. Returns: An initializer. References: [He et al., 2015] (https://www.cv-foundation.org/openaccess/content_iccv_2015/html/He_Delving_Deep_into_ICCV_2015_paper.html) # pylint: disable=line-too-long ([pdf](https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/He_Delving_Deep_into_ICCV_2015_paper.pdf)) rrrr@rPrds r he_uniformrWs(  XID BBrct|dkrdx}}nPt|dk(r|dx}}n:t|dk(r |d}|d}n!d}|ddD]}||z} |d|z}|d|z}t|t|fS)zComputes the number of input and output units for a weight shape. Args: shape: Integer shape tuple or TF tensor shape. Returns: A tuple of integer scalars (fan_in, fan_out). rrrNrr)rr)rrrreceptive_field_sizers rrrs Z!^FW 5zQQxFW 5zQ 1XFAhGSbz"c!" 2Y- -FBi..G Vc'l ""rc<|jstd|d|S)zValidate and return floating point type based on `dtype`. `dtype` must be a floating point type. Args: dtype: The data type to validate. Returns: Validated type. Raises: ValueError: if `dtype` is not a floating point type. z=Argument `dtype` is expected to be floating point. Received: r) is_floatingrrs rroros0    ""'+ ,, ,rr2)@r*rnumpyrMtensorflow.python.frameworkrrrtensorflow.python.opsrrrr r r tensorflow.python.utilr "tensorflow.python.util.deprecationr rr tensorflow.python.util.tf_exportrrdeprecated_endpointsr/rCrJr\rirzrrrrrrrr$r3r>rLr,rArHrZrgrxrrr<rJridentity_initializerconvolutional_delta_orthogonalconvolutional_orthogonal_1dconvolutional_orthogonal_2dconvolutional_orthogonal_3drQrSrUrWrrorrrrhs" 3.4+101*,.9D>6**Z #%89:!!!"67P&KP&8;P&f "$678!!!"57IJ &; &K9 &F &(>?@!!!"89m;{m;:Am;` ,.JKL!!!"?@UKUAMUp +-HIJ!!!">?U;U@KUp .0NOP!!!"A"@BZkZBQZz ')K"!!"D"EG<P<PG <P~ .0NOP!!!"A"@BTkTBQTn (*BCD!!!";":<ALAL<EALLBLBLJ@=K@=FS13S1l~*3~*Bi83i8X &'(!!!"9:#9{#9;)#9L +-JKL!!!">"?A9O9AM9: *,HIJ!!!"=">@9?9@K9B*(.#5 .*(#!;555 *+,K-K6 +,-B.B4 '()K*K0 ()*B+B4#6r