Vh/dZddlmZmZddlZddlmZmZmZm Z m Z dgZ dee e ejfdee e ejffdZGd dZy) zu This module implements Paged Attention on top of flex_attention. This module is experimental and subject to change. )OptionalUnionN) _identity_mask_mod_signature_score_mod_signature BlockMask noop_maskPagedAttentionxmultiplec||zdz |zS)N)r r s `/home/dcms/DCMS/lib/python3.12/site-packages/torch/nn/attention/experimental/_paged_attention.py_cdivrs L1  ))ceZdZdZ ddedededefdZdejdejd d fd Z dejd d fd Z dejd ejdejdejdejdejd d fdZ dde de ejd e fdZde ed efdZde ed efdZy )r aa PagedAttention supports flex attention inference with a large batch size. With PagedAttention, a batch of key/value tensors with varying kv length is splitted into tensor blocks of fixed length and cached in a compact way. Thus we can avoid redundant memory consumption due to varying kv length and support a larger batch size. n_pages page_sizemax_batch_sizedevicec~||_||_tj||jftj| |_tj |tj||_tt|dz dd|_ tj||ftj| |_ y)Ndtyperr) rrtorchonesint64 page_tablezeroscapacitylistrange empty_pagesphysical_to_logical)selfrrrrs r__init__zPagedAttention.__init__%s #!:: T\\ *%++f    N%++fU  gk2r :;%*JJ W %U[[% $  r batch_idxseq_lenreturnNc||j|kryt||j|z |j}t|j|k\s/Jd|j dt|jd|j||jz}||z}t j|j| d|j}|jd| |_||j|||f<t j|j |j |j|j||f<|j|xx||jzz cc<y)a' Requests the capacity of a given batch to be at least enough to hold `seq_len` elements. Args: batch_idx (Tensor): batch index to be reserved; shape :math:`(1)`. seq_len (Tensor): minimum capacity for the given batch; shape :math:`(1)`. Nz requested z pages but there are only z empty pagesr) r!rrlenr$itemrtensorrraranger%)r&r(r)num_pages_to_allocatestart_page_idx end_page_idxallocated_pagess rreservezPagedAttention.reserveBs dmmI. .  % dmmI. .! 4##$(== .33567""%d&6&6"7!8  F = y1T^^C%(==  ,,   334 5(//  ++,C.C-CD    < ' ) @E||    !    (//@   O!;< i $9DNN$JJ rc|j|dk7}|j||}d|j|<|xj|jz c_d|j|dd|f<d|j|<y)z Removes a single batch from paged attention. Args: batch_idx (Tensor): batch index to be removed; shape :math:`(1)`. rrN)rr!r$tolistr%)r&r(allocated_page_idxr4s rerasezPagedAttention.eraseps~"__Y72=//)45GH$% i  O2244BD  +A,>?%' "r input_posk_valv_valk_cachev_cachec|jr td|j\}}} } |jd} ||jdk7rtd|d|jdd||jdk7rtd|d |jdd| |jdk7rtd | d |jdd| |jdk7rtd | d |jdd| |jdk7rtd| d |jdd||jz} ||jz} t j |j |d| jtjjtj}||jz| zjd}|jddddjjd||| z| }|jddddjjd||| z| }||dddd|ddf<||dddd|ddf<y)a Assigns new contents `val` to the storage `cache` at the location `batch_idx` and `input_pos`. Args: batch_idx (Tensor): batch index; shape :math:`(B)`. input_pos (Tensor): input positions to be assigned for the given batch; shape :math:`(B, S)`. val (Tensor): value to be assigned; shape :math:`(B, H, S, D)` cache (Tensor): the cache to store the values; shape:`(1, H, MAX_S, D)` zval must not require gradientrz