2VhBddlZddlmZddlmZddlmZddlmZddlmZddlmZddlmZdd l m Z dd l m Z dd l mZe d Gd deZy)N)backend) constraints)dtype_policies) initializers)ops) quantizers) regularizers) keras_export) KerasTensor)Layerzkeras.layers.EmbeddingceZdZdZ dfd ZddZedZdZddZ dZ dZ dd Z d Z d Zfd Zd Z dZdZdZfdZddZddZdZxZS) Embeddingau Turns nonnegative integers (indexes) into dense vectors of fixed size. e.g. `[[4], [20]] -> [[0.25, 0.1], [0.6, -0.2]]` This layer can only be used on nonnegative integer inputs of a fixed range. Example: >>> model = keras.Sequential() >>> model.add(keras.layers.Embedding(1000, 64)) >>> # The model will take as input an integer matrix of size (batch, >>> # input_length), and the largest integer (i.e. word index) in the input >>> # should be no larger than 999 (vocabulary size). >>> # Now model.output_shape is (None, 10, 64), where `None` is the batch >>> # dimension. >>> input_array = np.random.randint(1000, size=(32, 10)) >>> model.compile('rmsprop', 'mse') >>> output_array = model.predict(input_array) >>> print(output_array.shape) (32, 10, 64) Args: input_dim: Integer. Size of the vocabulary, i.e. maximum integer index + 1. output_dim: Integer. Dimension of the dense embedding. embeddings_initializer: Initializer for the `embeddings` matrix (see `keras.initializers`). embeddings_regularizer: Regularizer function applied to the `embeddings` matrix (see `keras.regularizers`). embeddings_constraint: Constraint function applied to the `embeddings` matrix (see `keras.constraints`). mask_zero: Boolean, whether or not the input value 0 is a special "padding" value that should be masked out. This is useful when using recurrent layers which may take variable length input. If this is `True`, then all subsequent layers in the model need to support masking or an exception will be raised. If `mask_zero` is set to `True`, as a consequence, index 0 cannot be used in the vocabulary (`input_dim` should equal size of vocabulary + 1). weights: Optional floating-point matrix of size `(input_dim, output_dim)`. The initial embeddings values to use. lora_rank: Optional integer. If set, the layer's forward pass will implement LoRA (Low-Rank Adaptation) with the provided rank. LoRA sets the layer's embeddings matrix to non-trainable and replaces it with a delta over the original matrix, obtained via multiplying two lower-rank trainable matrices. This can be useful to reduce the computation cost of fine-tuning large embedding layers. You can also enable LoRA on an existing `Embedding` layer by calling `layer.enable_lora(rank)`. lora_alpha: Optional integer. If set, this parameter scales the low-rank adaptation delta (computed as the product of two lower-rank trainable matrices) during the forward pass. The delta is scaled by `lora_alpha / lora_rank`, allowing you to fine-tune the strength of the LoRA adjustment independently of `lora_rank`. Input shape: 2D tensor with shape: `(batch_size, input_length)`. Output shape: 3D tensor with shape: `(batch_size, input_length, output_dim)`. c  | jdd} | tjdt |di| ||_||_tj||_ tj||_ tj||_ ||_||_d|_||_| | n||_d|_|C|j)t+|t,rt/|dk(s|g}|j1|yy)N input_lengthz6Argument `input_length` is deprecated. Just remove it.F)popwarningswarnsuper__init__ input_dim output_dimrgetembeddings_initializerr embeddings_regularizerrembeddings_constraint mask_zerosupports_maskingautocast lora_rank lora_alpha lora_enabledbuild isinstancelistlen set_weights) selfrrrrrrweightsr!r"kwargsr __class__s O/home/dcms/DCMS/lib/python3.12/site-packages/keras/src/layers/core/embedding.pyrzEmbedding.__init__Rszz.$7  # MMH  "6""$&2&6&67M&N#&2&6&67M&N#%0__5J%K"" ) "(2(>*I!   JJLw-#g,!2C")   W % c|jry|j|jf}|j|j ||j|jdk7r:|j ||j d|j|jd|_ d|_|jr|j|jyy)N)modeint8 embeddingsT)shape initializername regularizer constraint trainable) builtrrquantization_modequantized_build add_weightrrr _embeddingsr! enable_lora)r) input_shapeembeddings_shapes r-r$zEmbedding.buildws ::  NNDOO<  ! ! -  !18N8N O  ! !V +#& 77! 7755 / D  >>   T^^ , r.c|jrQ|j|j|jz t j |j |jzzS|jSN)r#r=r"r!rmatmullora_embeddings_alora_embeddings_b)r)s r-r2zEmbedding.embeddingss^   ##$..0 41143I3IJ'KK Kr.c|jdk7r%|jdk7rtj|d}tj|j|d}tj||j S)Nint32int64raxisdtype)rLrcasttaker2 compute_dtype)r)inputsoutputss r-callzEmbedding.callsV <<7 "v||w'>XXfg.F((4??F;xxt'9'9::r.cH|jsytj|dS)Nr)rr not_equal)r)rPmasks r- compute_maskzEmbedding.compute_masks~~}}VQ''r.c$g||jSrB)r)r)r?s r-compute_output_shapezEmbedding.compute_output_shapes..doo..r.c|j|j}t|dd}t||j|S)NraggedF)rLrZ)rXr3getattrr rO)r)rP output_shaperZs r-compute_output_speczEmbedding.compute_output_specs>00> 51  2 26  r.c|jr td|js td|jr td|jj |j d|jjd|ftj||j|_ |j d||jjdftj||j|_ d |j_|jjd |_||_|||_y||_y) NzLora is incompatible with embedding constraints. In order to enable lora on this layer, remove the `embeddings_constraint` argument.z3Cannot enable lora on a layer that isn't yet built.z>lora is already enabled. This can only be done once per layer.rDr)r5r3r4r6rErFT)r ValueErrorr9r#_trackerunlockr<r2r3rrrrDrEr8lockr!r")r)rankr" a_initializer b_initializers r-r>zEmbedding.enable_loras4  % %4  zzE    P  !%$??((+T2$((733 "1"  "&$..q12$((733 "1"  %*!  (2(>*Dr.c|jsy|j\}}|g}|j<|jdk(r|j|n|j |jt |D]\}}||t |<yNr1)r9 _get_embeddings_with_merged_lorar:append_quantization_mode_error enumeratestr)r)storeembeddings_valueembeddings_scaletarget_variablesivariables r-save_own_variableszEmbedding.save_own_variablesszz   1 1 3 +*--  ! ! -%%/ ''(8933D4J4JKK$%56 %KAx$E#a&M %r.c|js|j||jsy|jg}|jF|jdk(r|j |j n|j|jt|D]"\}}|j|t|$|jr|jjtj|jj|jjtj|jjyyrg)r#_check_load_own_variablesr9r=r:rirorjrkassignrlrDrzerosr3rE)r)rmrprqrrs r-load_own_variableszEmbedding.load_own_variabless   * *5 1zz !,,-  ! ! -%%/ ''(=(=>33D4J4JKK$%56 +KAx OOE#a&M * +     " " ) ) $00667   " " ) ) $00667  r.ct|}|j|jt j |j tj |jtj |jtj |j|jd}|jr|j|d<|j|d<i||S)N)rrrractivity_regularizerrrr!r")r get_configrrr serializerr rrzrrrr!r")r) base_configconfigr,s r-r{zEmbedding.get_configsg(* //&2&<&<++''3&<&<++'%1$:$:))%&1%:%:**& " >>"&..F; #'??F< (+(((r.c  |j|jz}t|jt|k7rt|dk(rY|jsMt d|j dt|jd|j d|j d t d|j dt|dt|jd |Dcgc]}|j c}ycc}w) NrzLayer 'zY' was never built and thus it doesn't have any variables. However the weights file lists z variables for this layer. In most cases, this error indicates that either: 1. The layer is owned by a parent layer that implements a `build()` method, but calling the parent's `build()` method did NOT create the state of the child layer 'z'. A `build()` method must create ALL state for the layer, including the state of any children layers. 2. You need to implement the `def build_from_config(self, config)` method on layer 'a-', to specify how to rebuild it during loading. In this case, you might also want to implement the method that generates the build config at saving time, `def get_build_config(self)`. The method `build_from_config()` is meant to create the state of the layer (i.e. its variables) upon deserialization.z ' expected z variables, but received z% variables during loading. Expected: )_trainable_variables_non_trainable_variablesr'keysr9r_r5)r)rmall_varsvs r-ruz#Embedding._check_load_own_variabless,,t/L/LL uzz| H -8}!$** dii[)669%**,6G5HI( )- {3! "& ,NN.$))KH ? uzz|$%&.67aff78: 3 .:8s#Dctd|S)NzHInvalid quantization mode. Expected 'int8'. Received: quantization_mode=)NotImplementedError)r)r0s r-rjz"Embedding._quantization_mode_error6s" ++/& 2  r.c`|dk(r|j|d|_y|j|)Nr1T) _int8_buildrj _is_quantized)r)r@r0s r-r;zEmbedding.quantized_build<s5 6>   - ."//5 5r.c|jd|ddd|_|jd|jfdd|_y) Nr2rwr1F)r5r3r4rLr8roones)r5r3r4r8)r<r=rro)r)r@s r-rzEmbedding._int8_buildCsR??" + !%#>># !0! r.cv|jdk7r|j|jt| |i|Srg)r:rjrquantized_call)r)argsr+r,s r-rzEmbedding.quantized_callUs=  ! !V +//0F0FG Gw%t6v66r.ctj|jdvrtj|d}tj |j |d}tj |j|d}tjtj||jtj|d}|jrrtj |j|d}tj||j}tj||j |j"z |z}|S)N)rGrHrGrrIrK)rstandardize_dtyperLrrMrNror=dividerO expand_dimsr#rDrCrEaddr"r!)r)rPtrainingrorQ lora_outputss r- _int8_callzEmbedding._int8_callZs  $ $V\\ 2:L LXXfg.F88D$9$96J((4++V!<** HHWD$6$6 7 OO,2 6    88D$:$:FKL::lD4J4JKLgg$//DNN:lJGr.cH|r,t|tur|j|j|j|j f}|dk(r>t j|jdd\}}tj|d}|`|j|||dk(r6|jj|jj|jj6t!j"|d|jj$}||_yy)Nr1rTrJto_numpyrI_from_)typer_not_implemented_errorquantizerrrabs_max_quantizer=rsqueezer;rvro dtype_policyr:rrr5)r)r0 type_checkr@rnropolicys r-rzEmbedding.quantizens  4:Y6--dmm< < NNDOO< 6>2<1L1L  rD2 . . #{{+;"E   -t4 6>    # #$4 5  ! ! ( ()9 :    . . 6#''4&t7H7H7M7M6N(OPF &D  7r.c|jj|j}|j}|jrt j |t j|d}t j|t j|j|j}tj|dd\}}t j|d}||fS|jdfS)NrrITr)rr:r=ror#rrrrrCrDrErrrr2)r)rnros r-rhz*Embedding._get_embeddings_with_merged_loras    . . :#// #44   $'::$coo6FR&P$ $'77$JJt55t7M7MN$ //(rD3 "2 $';;/?b#I #%55 5$$r.)uniformNNFNNNrB)N he_uniformrw)T)__name__ __module__ __qualname____doc__rr$propertyr2rRrVrXr]r>rsrxr{rurjr;rrrrrh __classcell__)r,s@r-rrs?J )#"#&J-&  ; ( / " &IP%$0)0 D. " $7 ('0%r.r)r keras.srcrrrrrrr keras.src.api_exportr keras.src.backendr keras.src.layers.layerr rrr.r-rsJ!$" "-)(&'K%K%(K%r.