2Vh4lddlmZddlmZddlmZddlmZddlmZedGddeZ y ) )backend)ops) keras_export) KerasTensor)Layerzkeras.layers.AttentionceZdZdZ d fd ZdZdZddZdZ ddZ ddZ d Z dd Z dd Z fd ZxZS) Attentiona Dot-product attention layer, a.k.a. Luong-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as a `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. seed: A Python integer to use as random seed in case of `dropout`. score_mode: Function to use to compute attention scores, one of `{"dot", "concat"}`. `"dot"` refers to the dot product between the query and key vectors. `"concat"` refers to the hyperbolic tangent of the concatenation of the `query` and `key` vectors. Call arguments: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. c t|di|||_||_||_|jdkDr%t j j||_|jdvrtd|d|_ y)Nrseed)dotconcatz_Invalid value for argument score_mode. Expected one of {'dot', 'concat'}. Received: score_mode=F) super__init__ use_scale score_modedropoutrrandom SeedGeneratorseed_generator ValueError_return_attention_scores)selfrrrr kwargs __class__s T/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/layers/attention/attention.pyrzAttention.__init__Bs "6""$ <">D">"ID  ??"3 3((2|5  ).%c|j|d|_d|_|jr%|j ddd|j d|_|j dk(r&|j ddd|j d|_yy)NscaleronesT)nameshape initializerdtype trainablerconcat_score_weight)_validate_inputsr r'r add_weightr%r)r input_shapes rbuildzAttention.buildZs k* #' >>"jj )DJ ??h &'+*"jj (7(D $ 'rcf|jdk(rJtj|tj|gd}|j||jz}|S|jdk(rtj |d}tj |d}|jI|j tjtj|j||zzdz}|S|j tjtj||zdz}|Std ) aCalculates attention scores as a query-key dot product. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. r )r)axesraxiszscores not computed) rrmatmul transposer expand_dimsr'sumtanhr)rquerykeyscores q_reshaped k_reshapeds r_calculate_scoreszAttention._calculate_scoresos ??e #ZZs}}Sy'IJFzz%$**$$ #__ (R8J26Jzz%11CGGHHTZZ: +BCD25 11CGGHHZ*45B5  23 3rc|Otj|}|jdk(rdnd}||tj||jzz}tj|d}|rE|j dkDr6t jj ||j |j}tj|||fS) aApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `(batch_size, Tq)` and `key` tensor of shape `(batch_size, Tv)` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `(batch_size, Tq, Tv)`. value: Value tensor of shape `(batch_size, Tv, dim)`. scores_mask: A boolean mask tensor of shape `(batch_size, 1, Tv)` or `(batch_size, Tq, Tv)`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `(batch_size, Tq, dim)`. Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. float16g@geAr%r4r1rr ) r logical_notr%castsoftmaxrrrrr5)rr<value scores_masktraining padding_mask max_valueweightss r _apply_scoreszAttention._apply_scoress<  "??;7L$*<<9#<%I i#((JzAI 926I 926I++IyAK!b9v{;; Mrc|j||||_|d}|d}t|dkDr|dn|}|r|dnd} |r|dnd} |j||} |j | | |} |j | || |\} }| ;t j| d} | t j| | j z} |r| |fS| S) Ninputsmaskrr.r-)r:r;)r<rFrGrHr4r1rB) r(rlenr?rZrLrr7rDr%)rr]r^rHreturn_attention_scoresrSqvkq_maskrRr<rGattention_outputattention_scoress rcallzAttention.calls V$7(?% 1I 1IVqF1Ia ad ad''aQ'700 FO .2-?-? h.@. **  __V"5F 7G7M7M N N  "$&67 7# #rch|j||||dytj|dS)Nr\r)r(rconvert_to_tensor)rr]r^s r compute_maskzAttention.compute_masks: V$7 <47?$$T!W--rcp|\}}}||}g|dd|d}|jr|d|d|df}||fS|S)Nr4rr.)r)rr* query_shape value_shape key_shape output_shape scores_shapes rcompute_output_shapezAttention.compute_output_shapese.9+ [)  #I;Sb);;r?;  ( ('NKNIaLIL- -rc|j|||d}|d}t|dkDr|dn|}|j|j|j|jg} t | |j } |j s|rG|jd|jd|jdf} t | |j } | | fS| S)Nrr.r-rB)r(r_rqr#r compute_dtyper) rr]r^r`rHrSr:rFr;ro output_specrpattention_scores_specs rcompute_output_speczAttention.compute_output_specs fd+q q v;?fQi00 [[%++syy 1 ",d6H6HI   ( (,C A A ! L %0D$6$6% ! !67 7rc|jj}t|tst |d|dt |dkst |dkDrt |dt |d|Rt|tst |d|dt |dkst |dkDrt |d|d |dyy) z'Validates arguments of the call method.zj layer must be called on a list of inputs, namely [query, value] or [query, value, key]. Received: inputs=.r-zl layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Received length: NzL layer mask must be a list, namely [query_mask, value_mask]. Received: mask=z< layer accepts mask list of length 2 or 3. Received: inputs=z, mask=)r__name__ isinstancelistrr_)rr]r^ class_names rr(zAttention._validate_inputs's^^,, &$',$$*81.  v;?c&kAo,$$'K=3   dD) !l#GGKfAO4y1}D A  !l#((.xwtfA?!. rctt|}|j|j|jd}i||S)N)rrr)r get_configrrr)r base_configconfigrs rrzAttention.get_configBs>g(* //||  )+(((r)Fr gN)NF)NFFF)N)NFNF)rz __module__ __qualname____doc__rr+r?rLrZrgrjrqrvr(r __classcell__)rs@rr r sr6t  .0*B,3\4 % $>.  % D6))rr N) keras.srcrrkeras.src.api_exportrkeras.src.backendrkeras.src.layers.layerrr rrrrs8-)(&'@)@)(@)r