2VhTddlmZddlmZddlmZedGddeZy))ops) keras_export) Attentionzkeras.layers.AdditiveAttentionc>eZdZdZ dfd ZdZdZfdZxZS)AdditiveAttentiona[ Additive attention layer, a.k.a. Bahdanau-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)` as a non-linear sum `scores = reduce_sum(tanh(query + key), axis=-1)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. Call arguments: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. c *t|d||d|y)N) use_scaledropout)super__init__)selfr r kwargs __class__s ]/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/layers/attention/additive_attention.pyr zAdditiveAttention.__init__<s H9gHHc|j||dd}d|_|jr'|jd|gd|jd|_yy)Nrscaleglorot_uniformT)nameshape initializerdtype trainable)_validate_inputsrr add_weightr)r input_shapedims rbuildzAdditiveAttention.buildDs[ k*!nR  >>e,jj )DJ rctj|d}tj|d}|jr |jnd}tj|tj ||zzdS)aCalculates attention scores as a nonlinear sum of query and key. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. )axisg?r)r expand_dimsr rsumtanh)rquerykey q_reshaped k_reshapedrs r_calculate_scoresz#AdditiveAttention._calculate_scoresQsY__U4 __Sr2 "nn #wwusxx Z(?@@rJJrc*t|}|d=|S)N score_mode)r get_config)r base_configrs rr/zAdditiveAttention.get_configcsg(*  %r)Tg) __name__ __module__ __qualname____doc__r r r,r/ __classcell__)rs@rrrs,2lI K$rrN) keras.srcrkeras.src.api_exportr$keras.src.layers.attention.attentionrrr rrr9s2-:./_ _0_r