2Vh(ddlmZddlmZddlmZddlmZddlmZddlmZddlm Z edGd d e Z y ) )backend) constraints) initializers)ops) regularizers) keras_export)Layerzkeras.layers.LayerNormalizationcVeZdZdZ dfd ZdZdZdZfdZxZ S)LayerNormalizationaLayer normalization layer (Ba et al., 2016). Normalize the activations of the previous layer for each given example in a batch independently, rather than across a batch like Batch Normalization. i.e. applies a transformation that maintains the mean activation within each example close to 0 and the activation standard deviation close to 1. If `scale` or `center` are enabled, the layer will scale the normalized outputs by broadcasting them with a trainable variable `gamma`, and center the outputs by broadcasting with a trainable variable `beta`. `gamma` will default to a ones tensor and `beta` will default to a zeros tensor, so that centering and scaling are no-ops before training has begun. So, with scaling and centering enabled the normalization equations are as follows: Let the intermediate activations for a mini-batch to be the `inputs`. For each sample `x_i` in `inputs` with `k` features, we compute the mean and variance of the sample: ```python mean_i = sum(x_i[j] for j in range(k)) / k var_i = sum((x_i[j] - mean_i) ** 2 for j in range(k)) / k ``` and then compute a normalized `x_i_normalized`, including a small factor `epsilon` for numerical stability. ```python x_i_normalized = (x_i - mean_i) / sqrt(var_i + epsilon) ``` And finally `x_i_normalized ` is linearly transformed by `gamma` and `beta`, which are learned parameters: ```python output_i = x_i_normalized * gamma + beta ``` `gamma` and `beta` will span the axes of `inputs` specified in `axis`, and this part of the inputs' shape must be fully defined. For example: >>> layer = keras.layers.LayerNormalization(axis=[1, 2, 3]) >>> layer.build([5, 20, 30, 40]) >>> print(layer.beta.shape) (20, 30, 40) >>> print(layer.gamma.shape) (20, 30, 40) Note that other implementations of layer normalization may choose to define `gamma` and `beta` over a separate set of axes from the axes being normalized across. For example, Group Normalization ([Wu et al. 2018](https://arxiv.org/abs/1803.08494)) with group size of 1 corresponds to a Layer Normalization that normalizes across height, width, and channel and has `gamma` and `beta` span only the channel dimension. So, this Layer Normalization implementation will not match a Group Normalization layer with group size set to 1. Args: axis: Integer or List/Tuple. The axis or axes to normalize across. Typically, this is the features axis/axes. The left-out axes are typically the batch axis/axes. `-1` is the last dimension in the input. Defaults to `-1`. epsilon: Small float added to variance to avoid dividing by zero. Defaults to 1e-3. center: If True, add offset of `beta` to normalized tensor. If False, `beta` is ignored. Defaults to `True`. scale: If True, multiply by `gamma`. If False, `gamma` is not used. When the next layer is linear (also e.g. `nn.relu`), this can be disabled since the scaling will be done by the next layer. Defaults to `True`. rms_scaling: If True, `center` and `scale` are ignored, and the inputs are scaled by `gamma` and the inverse square root of the square of all inputs. This is an approximate and faster approach that avoids ever computing the mean of the input. Note that this *isn't* equivalent to the computation that the `keras.layers.RMSNormalization` layer performs. beta_initializer: Initializer for the beta weight. Defaults to zeros. gamma_initializer: Initializer for the gamma weight. Defaults to ones. beta_regularizer: Optional regularizer for the beta weight. None by default. gamma_regularizer: Optional regularizer for the gamma weight. None by default. beta_constraint: Optional constraint for the beta weight. None by default. gamma_constraint: Optional constraint for the gamma weight. None by default. **kwargs: Base layer keyword arguments (e.g. `name` and `dtype`). Reference: - [Lei Ba et al., 2016](https://arxiv.org/abs/1607.06450). c  Jt |di| t|ttfrt||_n&t|t r||_ntd|z||_||_ ||_ ||_ tj||_tj||_t!j||_t!j| |_t'j| |_t'j| |_d|_d|_y)NzQExpected an int or a list/tuple of ints for the argument 'axis', but received: %rTF)super__init__ isinstancelisttupleaxisint TypeErrorepsiloncenterscale rms_scalingrgetbeta_initializergamma_initializerrbeta_regularizergamma_regularizerrbeta_constraintgamma_constraintsupports_maskingautocast)selfrrrrrrrrrrr kwargs __class__s b/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/layers/normalization/layer_normalization.pyrzLayerNormalization.__init__ns "6" dT5M *T DI c "DI46:;     & , 0 01A B!-!1!12C!D , 0 01A B!-!1!12C!D*? +0@ A $ c @t|jtr't|jDcgc]}|| c}}n"||jf}|jg|_|js |j r<|j d||j|j|jdd|_ nd|_ |jrH|j s<|j d||j|j|jdd|_yd|_ycc}w)NgammaTF)nameshape initializer regularizer constraint trainabler"beta)rrrrrr add_weightrrr r)rrrrr0)r# input_shapedimr+s r&buildzLayerNormalization.builds dii &tyyA;s+ABE +-E DI ::)) 22 2200)DJDJ ;;t// 11 11//(DIDI9Bs Dc |j}t|dgz jD] }|| |<  fd}tj|j d}t j||}jr{t j|jd}t j|jz}||zt j|j|j z}nt j|jd\} }|j|j} } t j|jz}| %t j| |j } || z}| |z} | %t j| |j } | | z} ||z| z}t j|jS)Nc|At|jk7r)jdz gk7rtj|S|S)Nr6)lenr+rrreshape)vbroadcast_shapendimsr#s r& _broadcastz+LayerNormalization.call.._broadcasts@ LE)II%!),{{1o66Hr'float32T)rkeepdims)axesr?)r+r8rr result_typedtypercastrvarrsqrtrr)momentsr0 compute_dtype)r#inputsr2r3r=rGvarianceinvoutputsmeanr)r0resr;r<s` @@r&callzLayerNormalization.callsll K #+99 4C#.s#3OC  4  ++FLL)D &-0   wwvDIIEH))Ht||34C sxx 4::(> MM  ![[dii$OND($TZZ0*TYY2G4E))Ht||34C  5Ek%#+Cxxfll3DjslS(Gxx!3!344r'c t|jtr|jg}n |j}|D]=}|t|k\s|t| ks!t d|d|d|j|S)NzAxis z" is out of bounds for input shape z. Received: axis=)rrrr8 ValueError)r#r2r@rs r&compute_output_shapez'LayerNormalization.compute_output_shapes dii %II;D99D Ds;''43{3C2C+C D6"##.-0&&*ii[2 r'c |j|j|j|j|jt j |jt j |jtj |jtj |jtj |jtj |jd }t|A}i||S)N) rrrrrrrrrrr )rrrrrr serializerrrrrrrr r get_config)r#config base_configr%s r&rTzLayerNormalization.get_configsII||kkZZ++ , 6 6t7L7L M!-!7!78N8N!O , 6 6t7L7L M!-!7!78N8N!O*44T5I5IJ + 5 5d6K6K L  g(* (+(((r') gMbP?TTFzerosonesNNNN) __name__ __module__ __qualname____doc__rr4rNrQrT __classcell__)r%s@r&r r sN`H  &P@35j ))r'r N) keras.srcrrrrrkeras.src.api_exportrkeras.src.layers.layerr r r r'r&rbs>!""-(/0~)~)1~)r'