AVh+2dZddlZddlmZddlmZddlmZddlm Z ddl m Z ddlmZdd l mZdd l mZdd l mZdd lmZdd lmZddlmZddlmZm Z m!Z!ddl"m#Z#ejHdgdZ%edddddZ&dpde#e!e&fde#e!e&fde#e!e&fde#e!e&fde'de'de'de(d e)d!e)d"e*fd#Z+ed$ejXe+Z-de#e!e&fde#e!e&fde#e!e&fde#e!e&fde'de'de'de(d e)d!e)d"e*fd%Z.ejHd&gd'Z/ed(ddddZ0dqde#e!e0fde#e!e0fde#e!e0fde#e!e0fd)e#e!e0fd*e#e!e0fd+e#e!e0fd,e#e!e0fd-e#e!e0fd.e#e!e0fd/e#e!e0fde'de'de'de(d e)d!e)f"d0Z1ed1ejXe1Z2de#e!e0fde#e!e0fde#e!e0fde#e!e0fd)e#e!e0fd*e#e!e0fd+e#e!e0fd,e#e!e0fd-e#e!e0fd.e#e!e0fd/e#e!e0fde'de'de'de(d e)d!e)f"d2Z3ejHd3gd'Z4ed4ddddZ5dqde#e!e5fde#e!e5fde#e!e5fde#e!e5fd)e#e!e5fd*e#e!e5fd+e#e!e5fd,e#e!e5fd-e#e!e5fd.e#e!e5fd/e#e!e5fd5e#e!ejlfde'de'de'de(d e)d!e)f$d6Z7ed7ejXe7Z8de#e!e5fde#e!e5fde#e!e5fde#e!e5fd)e#e!e5fd*e#e!e5fd+e#e!e5fd,e#e!e5fd-e#e!e5fd.e#e!e5fd/e#e!e5fd5e#e!ejlfde'de'de'de(d e)d!e)f$d8Z9ejHd9gd'Z:ed:ddddZ;drde#e!e;fde#e!e;fde#e!e;fde#e!e;fd;e#e!ejxfd)e#e!e;fd*e#e!e;fd+e#e!e;fd,e#e!e;fd-e#e!e;fd.e#e!e;fd/e#e!e;fd5e#e!ejlfde'de'de'de(d e)d!e)dZ=ed?ejXe=Z>de#e!e;fde#e!e;fde#e!e;fde#e!e;fd;e#e!ejxfd)e#e!e;fd*e#e!e;fd+e#e!e;fd,e#e!e;fd-e#e!e;fd.e#e!e;fd/e#e!e;fd5e#e!ejlfde'de'de'de(d e)d!e)d+'88%'!H't[t E!QX QK' ""$3%%c*Jll:& ll<(+ll;'CLL4Kc''/(-  / 1F::L L&'3  ! !' *' .i  & &- ##At,,  # #    % '6H9g5k   # #   0?H;;JI))JJJJ54J5zraw_ops.CudnnRNNc |d}tj|d}|d}tj|d}|d}tj|d}|d}tj|d}|d}tj|d }| d} tj| d } | d } tj| d } tj ||||g| t jt jt jt jg\} }|\}}}}||||g}d | d|d|d|d|d |d | d | f}tjdd||| | }tjrtjd|||tj|}|S)Nr(rr)rr*rrr r!r"Tr#r,sCudnnRNNrFattrsr'r&r)r:r;r<r=r>args_to_matching_eager_dtypesbfloat16halffloat32float64rrArGr1r2rrrrrrrr r!r"r#r&r'_attr_T _inputs_TrQrPrKs rRr8r8s H   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +%K""; >+66wQW7XZ]`g`p`pryr~r~AHAPAPRYRaRa`de'9&/#5'7F'62, *h j y)WfdGU &   [!L#)s ?' ""$ L&'3  ! !' *' .CudnnRNNBackprop)input_backpropinput_h_backpropinput_c_backpropparams_backpropTV_CudnnRNNBackprop_Trrroutput_backpropoutput_h_backpropoutput_c_backproprctjxstj}|j}|jrF t j |d||||||||||| | d| d| d| d|d|d|}t j|}|S| d } tj | d} | d } tj | d} | d } tj | d} |d }tj"|d}|d }tj$|d}|d }tj$|d}t'j( did|d|d|d|d|d|d|d|d|d| d| d| d| d| d|d|d|d|\}}}}|dd}tj*rd|j-dd|j/dd|j/dd|j/dd|j/dd|j1dd|j1df}|j2}tj4d|||t j|}|S#tj$r }tj||Yd}~nd}~wtj$rYnwxYw t|||||||||| | | | | ||||| S#tj$rY7wxYw)ahBackprop step of CudnnRNN. Compute the backprop of both data and weights in a RNN. rnn_mode: Indicates the type of the RNN model. input_mode: Indicate whether there is a linear projection between the input and the actual computation before the first layer. 'skip_input' is only allowed when input_size == num_units; 'auto_select' implies 'skip_input' when input_size == num_units; otherwise, it implies 'linear_input'. direction: Indicates whether a bidirectional model will be used. Should be "unidirectional" or "bidirectional". dropout: Dropout probability. When set to 0., dropout is disabled. seed: The 1st part of a seed to initialize dropout. seed2: The 2nd part of a seed to initialize dropout. input: A 3-D tensor with the shape of [seq_length, batch_size, input_size]. input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size, num_units]. input_c: For LSTM, a 3-D tensor with the shape of [num_layer * dir, batch, num_units]. For other models, it is ignored. params: A 1-D tensor that contains the weights and biases in an opaque layout. The size must be created through CudnnRNNParamsSize, and initialized separately. Note that they might not be compatible across different generations. So it is a good idea to save and restore output: A 3-D tensor with the shape of [seq_length, batch_size, dir * num_units]. output_h: The same shape has input_h. output_c: The same shape as input_c for LSTM. An empty tensor for other models. output_backprop: A 3-D tensor with the same shape as output in the forward pass. output_h_backprop: A 3-D tensor with the same shape as output_h in the forward pass. output_c_backprop: A 3-D tensor with the same shape as output_c in the forward pass. reserve_space: The same reserve_space produced in for forward operation. input_backprop: The backprop to input in the forward pass. Has the same shape as input. input_h_backprop: The backprop to input_h in the forward pass. Has the same shape as input_h. input_c_backprop: The backprop to input_c in the forward pass. Has the same shape as input_c. params_backprop: The backprop to the params buffer in the forward pass. Has the same shape as params. Args: input: A `Tensor`. Must be one of the following types: `bfloat16`, `half`, `float32`, `float64`. input_h: A `Tensor`. Must have the same type as `input`. input_c: A `Tensor`. Must have the same type as `input`. params: A `Tensor`. Must have the same type as `input`. output: A `Tensor`. Must have the same type as `input`. output_h: A `Tensor`. Must have the same type as `input`. output_c: A `Tensor`. Must have the same type as `input`. output_backprop: A `Tensor`. Must have the same type as `input`. output_h_backprop: A `Tensor`. Must have the same type as `input`. output_c_backprop: A `Tensor`. Must have the same type as `input`. reserve_space: A `Tensor`. Must have the same type as `input`. rnn_mode: An optional `string` from: `"rnn_relu", "rnn_tanh", "lstm", "gru"`. Defaults to `"lstm"`. input_mode: An optional `string` from: `"linear_input", "skip_input", "auto_select"`. Defaults to `"linear_input"`. direction: An optional `string` from: `"unidirectional", "bidirectional"`. Defaults to `"unidirectional"`. dropout: An optional `float`. Defaults to `0`. seed: An optional `int`. Defaults to `0`. seed2: An optional `int`. Defaults to `0`. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (input_backprop, input_h_backprop, input_c_backprop, params_backprop). input_backprop: A `Tensor`. Has the same type as `input`. input_h_backprop: A `Tensor`. Has the same type as `input`. input_c_backprop: A `Tensor`. Has the same type as `input`. params_backprop: A `Tensor`. Has the same type as `input`. rcrrrr r!r"Nrrrr r!r"r&r'r(r)r*rrrrrrrrrirjrkrr&r,)rc)r-rr.r/rr0_CudnnRNNBackpropOutputr2r3r4r5r6r7!cudnn_rnn_backprop_eager_fallbackr9r:r;r<r=r?r@rArBrCrDrFrG)rrrrrrrrirjrkrrrrr r!r"r&rIrJrKrLrMrNrOrPrQs rRcudnn_rnn_backproprps`N    0h..0$ #\\  11  $w(O5F=*h KIw g (--g6g nH   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +%'88 '"' '18 'BI '#) '28 'CK '&. '-< ' /@ ' /@ '+8 'CK '(2 '>G '%, '37 '?D '"& '!QX QK' ""$3%%c*Jll:& ll<(+ll;'CLL4Kc''/( *F ::L L&'; # ) )' 2' .o  & &- ##At,,  # #    . '668X ,.? (zwT   # #   s1AH==JI++JJJ&&J=<J=zraw_ops.CudnnRNNBackpropc| d} tj| d} | d} tj| d} | d} tj| d} |d}tj|d}|d}tj|d }|d}tj|d }tj|||||||||| | g |t j t jt jt jg\}}|\ }}}}}}}}}} } |||||||||| | g }d |d| d| d| d|d |d |f}tjd d ||||}tjrtjd|||tj|}|S)Nr(rr)rr*rrr r!r"r,sCudnnRNNBackproprVrWrc)r:r;r<r=rYrZr[r\r]r^rrArGrnr2)rrrrrrrrirjrkrrrrr r!r"r&r'r`rarQrPrKs rRroroTs) H   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +%66wQWY_aiksvEGXZkmz8{}@CJCSCSU\UaUacjcrcrt{tCtCBFG'9IRF5'7FFHhQbduxE'668X`qtEGTU, *h j y)WfdGU L&   0!L#)s ?' ""$ L&'; # ) )' 2' .rbCudnnRNNBackpropV2TV_CudnnRNNBackpropV2_T host_reservedctjxstj}|j}|jrG t j |d||||||||||| | | d| d| d|d|d|d|}t j|}|S| d } tj | d} | d } tj | d} |d }tj |d}|d }tj"|d}|d }tj$|d}|d }tj$|d}t'j( did|d|d|d|d|d|d|d|d|d| d| d| d| d| d|d|d|d|d|\}}}}|dd}tj*rd|j-dd|j/dd|j/dd|j/dd|j/dd|j1dd|j1df}|j2}tj4d|||t j|}|S#tj$r }tj||Yd}~nd}~wtj$rYnwxYw t|||||||||| | | | | |||||| S#tj$rY;wxYw)amBackprop step of CudnnRNN. Compute the backprop of both data and weights in a RNN. Takes an extra "host_reserved" inupt than CudnnRNNBackprop, which is used to determine RNN cudnnRNNAlgo_t and cudnnMathType_t. rnn_mode: Indicates the type of the RNN model. input_mode: Indicates whether there is a linear projection between the input and the actual computation before the first layer. 'skip_input' is only allowed when input_size == num_units; 'auto_select' implies 'skip_input' when input_size == num_units; otherwise, it implies 'linear_input'. direction: Indicates whether a bidirectional model will be used. Should be "unidirectional" or "bidirectional". dropout: Dropout probability. When set to 0., dropout is disabled. seed: The 1st part of a seed to initialize dropout. seed2: The 2nd part of a seed to initialize dropout. input: A 3-D tensor with the shape of [seq_length, batch_size, input_size]. input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size, num_units]. input_c: For LSTM, a 3-D tensor with the shape of [num_layer * dir, batch, num_units]. For other models, it is ignored. params: A 1-D tensor that contains the weights and biases in an opaque layout. The size must be created through CudnnRNNParamsSize, and initialized separately. Note that they might not be compatible across different generations. So it is a good idea to save and restore output: A 3-D tensor with the shape of [seq_length, batch_size, dir * num_units]. output_h: The same shape has input_h. output_c: The same shape as input_c for LSTM. An empty tensor for other models. output_backprop: A 3-D tensor with the same shape as output in the forward pass. output_h_backprop: A 3-D tensor with the same shape as output_h in the forward pass. output_c_backprop: A 3-D tensor with the same shape as output_c in the forward pass. reserve_space: The same reserve_space produced in the forward operation. host_reserved: The same host_reserved produced in the forward operation. input_backprop: The backprop to input in the forward pass. Has the same shape as input. input_h_backprop: The backprop to input_h in the forward pass. Has the same shape as input_h. input_c_backprop: The backprop to input_c in the forward pass. Has the same shape as input_c. params_backprop: The backprop to the params buffer in the forward pass. Has the same shape as params. Args: input: A `Tensor`. Must be one of the following types: `bfloat16`, `half`, `float32`, `float64`. input_h: A `Tensor`. Must have the same type as `input`. input_c: A `Tensor`. Must have the same type as `input`. params: A `Tensor`. Must have the same type as `input`. output: A `Tensor`. Must have the same type as `input`. output_h: A `Tensor`. Must have the same type as `input`. output_c: A `Tensor`. Must have the same type as `input`. output_backprop: A `Tensor`. Must have the same type as `input`. output_h_backprop: A `Tensor`. Must have the same type as `input`. output_c_backprop: A `Tensor`. Must have the same type as `input`. reserve_space: A `Tensor`. Must have the same type as `input`. host_reserved: A `Tensor` of type `int8`. rnn_mode: An optional `string` from: `"rnn_relu", "rnn_tanh", "lstm", "gru"`. Defaults to `"lstm"`. input_mode: An optional `string` from: `"linear_input", "skip_input", "auto_select"`. Defaults to `"linear_input"`. direction: An optional `string` from: `"unidirectional", "bidirectional"`. Defaults to `"unidirectional"`. dropout: An optional `float`. Defaults to `0`. seed: An optional `int`. Defaults to `0`. seed2: An optional `int`. Defaults to `0`. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (input_backprop, input_h_backprop, input_c_backprop, params_backprop). input_backprop: A `Tensor`. Has the same type as `input`. input_h_backprop: A `Tensor`. Has the same type as `input`. input_c_backprop: A `Tensor`. Has the same type as `input`. params_backprop: A `Tensor`. Has the same type as `input`. rrrrrr r!r"Nrmr(r)r*rrrrrrrrrirjrkrrtr&r,)rr)r-rr.r/rr0_CudnnRNNBackpropV2Outputr2r3r4r5r6r7$cudnn_rnn_backprop_v2_eager_fallbackr9r:r;r<r=r?r@rArBrCrDrFrG)rrrrrrrrirjrkrrtrrrr r!r"r&rIrJrKrLrMrNrOrPrQs rRcudnn_rnn_backprop_v2rx{spV    0h..0$ #\\  11 "D%'6(O5F=-Xj+y)Wgu &g *//8g nH   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +%'88 )$) )3: )DK )%+ )4: )EM )(0 )/> ) 1B ) 1B )-: )-: )EM )*4 )@I )'. )59 )AF )$( )!QX QK' ""$3%%c*Jll:& ll<(+ll;'CLL4Kc''/( *F ::L lFG= % + +G 4' .q  & &- ##At,,  # #    1 '668X ,.? 9g5t 77  # #   s1AIJI//JJ J++KKzraw_ops.CudnnRNNBackpropV2cX| d} tj| d} | d} tj| d} |d}tj|d}|d}tj|d}|d}tj|d }|d}tj|d }tj|||||||||| | g |t j t jt jt jg\}}|\ }}}}}}}}}} } tj| t j} |||||||||| | | g }d |d| d| d|d|d |d |f}tjd d ||||}tjrtjd|||t j#|}|S)Nr(rr)rr*rrr r!r"r,sCudnnRNNBackpropV2rVrWrr)r:r;r<r=rYrZr[r\r]r^r5convert_to_tensorint8rrArGrvr2)rrrrrrrrirjrkrrtrrrr r!r"r&r'r`rarQrPrKs rRrwrwsF H   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +%66wQWY_aiksvEGXZkmz8{}@CJCSCSU\UaUacjcrcrt{tCtCBFG'9IRF5'7FFHhQbduxE(( E-'668X`qtEGTVcd, *h j y)WfdGU L&   2Al#)s ?' ""$ lFG= % + +G 4' .rbCudnnRNNBackpropV3TV_CudnnRNNBackpropV3_Tsequence_lengthsnum_proj time_majorctjxstj}|j}|jrk t j g|d||||||||||| | | | d| d|d|d|d|d|d|d |}t j|}|S| d } tj | d} |d }tj |d}|d}tj |d}|d}tj"|d}|d}tj$|d}|d}tj$|d}|d}tj$|d}|d}tj&|d }t)j* d id|d|d|d|d|d|d|d|d|d| d| d| d| d| d|d|d|d|d|d|d |d|\}}}}|d d }tj,rd|j/dd|j1dd|j1dd|j1dd|j1dd|j3dd|j3dd|j3dd |j5d f}|j6}tj8d|||t j|}|S#tj$r }tj||Yd }~nd }~wtj$rYnwxYw t|||||||||| | | | f | |||||||||d S#tj$rYwxYw)!auBackprop step of CudnnRNNV3. Compute the backprop of both data and weights in a RNN. Takes an extra "sequence_lengths" input than CudnnRNNBackprop. rnn_mode: Indicates the type of the RNN model. input_mode: Indicates whether there is a linear projection between the input and the actual computation before the first layer. 'skip_input' is only allowed when input_size == num_units; 'auto_select' implies 'skip_input' when input_size == num_units; otherwise, it implies 'linear_input'. direction: Indicates whether a bidirectional model will be used. Should be "unidirectional" or "bidirectional". dropout: Dropout probability. When set to 0., dropout is disabled. seed: The 1st part of a seed to initialize dropout. seed2: The 2nd part of a seed to initialize dropout. input: If time_major is true, this is a 3-D tensor with the shape of [seq_length, batch_size, input_size]. If time_major is false, the shape is [batch_size, seq_length, input_size]. input_h: If time_major is true, this is a 3-D tensor with the shape of [num_layer * dir, batch_size, num_units]. If time_major is false, the shape is [batch_size, num_layer * dir, num_units]. input_c: For LSTM, a 3-D tensor with the shape of [num_layer * dir, batch, num_units]. For other models, it is ignored. params: A 1-D tensor that contains the weights and biases in an opaque layout. The size must be created through CudnnRNNParamsSize, and initialized separately. Note that they might not be compatible across different generations. So it is a good idea to save and restore sequence_lengths: a vector of lengths of each input sequence. output: If time_major is true, this is a 3-D tensor with the shape of [seq_length, batch_size, dir * num_units]. If time_major is false, the shape is [batch_size, seq_length, dir * num_units]. output_h: The same shape has input_h. output_c: The same shape as input_c for LSTM. An empty tensor for other models. output_backprop: A 3-D tensor with the same shape as output in the forward pass. output_h_backprop: A 3-D tensor with the same shape as output_h in the forward pass. output_c_backprop: A 3-D tensor with the same shape as output_c in the forward pass. time_major: Indicates whether the input/output format is time major or batch major. reserve_space: The same reserve_space produced in the forward operation. input_backprop: The backprop to input in the forward pass. Has the same shape as input. input_h_backprop: The backprop to input_h in the forward pass. Has the same shape as input_h. input_c_backprop: The backprop to input_c in the forward pass. Has the same shape as input_c. params_backprop: The backprop to the params buffer in the forward pass. Has the same shape as params. Args: input: A `Tensor`. Must be one of the following types: `bfloat16`, `half`, `float32`, `float64`. input_h: A `Tensor`. Must have the same type as `input`. input_c: A `Tensor`. Must have the same type as `input`. params: A `Tensor`. Must have the same type as `input`. sequence_lengths: A `Tensor` of type `int32`. output: A `Tensor`. Must have the same type as `input`. output_h: A `Tensor`. Must have the same type as `input`. output_c: A `Tensor`. Must have the same type as `input`. output_backprop: A `Tensor`. Must have the same type as `input`. output_h_backprop: A `Tensor`. Must have the same type as `input`. output_c_backprop: A `Tensor`. Must have the same type as `input`. reserve_space: A `Tensor`. Must have the same type as `input`. host_reserved: A `Tensor` of type `int8`. rnn_mode: An optional `string` from: `"rnn_relu", "rnn_tanh", "lstm", "gru"`. Defaults to `"lstm"`. input_mode: An optional `string` from: `"linear_input", "skip_input", "auto_select"`. Defaults to `"linear_input"`. direction: An optional `string` from: `"unidirectional", "bidirectional"`. Defaults to `"unidirectional"`. dropout: An optional `float`. Defaults to `0`. seed: An optional `int`. Defaults to `0`. seed2: An optional `int`. Defaults to `0`. num_proj: An optional `int`. Defaults to `0`. time_major: An optional `bool`. Defaults to `True`. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (input_backprop, input_h_backprop, input_c_backprop, params_backprop). input_backprop: A `Tensor`. Has the same type as `input`. input_h_backprop: A `Tensor`. Has the same type as `input`. input_c_backprop: A `Tensor`. Has the same type as `input`. params_backprop: A `Tensor`. Has the same type as `input`. r|rrrr r!r"rrN) rrrr r!r"rrr&r'r(r)r*rTrrrrr~rrrrirjrkrrtr&r,)r|)r-rr.r/rr0_CudnnRNNBackpropV3Outputr2r3r4r5r6r7$cudnn_rnn_backprop_v3_eager_fallbackr9r:r;r<r=r>r?r@rArBrCrDrErFrG)rrrrr~rrrrirjrkrrtrrrr r!r"rrr&rIrJrKrLrMrNrOrPrQs rRcudnn_rnn_backprop_v3r7sf    0h..0$ #\\  118 8"8$(8*/8188:A8CI88 8"*8,486E8 8-8/<8>K8  8 8+ 8-7 89D 8  8  8 & 8 (. 8 04 8 6= 8 ?D 8  8  8 + 8 -7 8g*//8g n H   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +% H   x 4(J!!*l;*'88)$))3:)DK)%+)0@)&, )7? ) (0 ) /> )1B)1B)-:)-:)EM)*4)@I)'.)59)AF)(0)=G)$()!QX QK' ""$3%%c*Jll:& ll<(+ll;'CLL4Kc''/(* +\  .0F::L lFG= % + +G 4' .I  & &- ##At,,  # #     1 '6+;VX O%68I 2:9g58    # #   s1A)KL K22L  L  L00MMzraw_ops.CudnnRNNBackpropV3c| d} tj| d} |d}tj|d}|d}tj|d}|d}tj|d}|d}tj|d }|d}tj|d }|d}tj|d }|d }tj|d }tj ||||||||| | | g |t jt jt jt jg\}}|\ }}}}}}}}} } } tj|t j}tj| t j} |||||||||| | | | g }d|d| d|d|d|d |d |d |d |f}tjdd||||}tj rtj"d|||t$j'|}|S)Nr(rr)rr*rrr r!r"rTrr,sCudnnRNNBackpropV3rVrWr|)r:r;r<r=r>rYrZr[r\r]r^r5rzint32r{rrArGrr2)rrrrr~rrrrirjrkrrtrrrr r!r"rrr&r'r`rarQrPrKs rRrrs H   x 4(J  \:* I ;7) _G    3' \ D   4 ($ ] E   E7 +% H   x 4(J!!*l;*66wQWY_aiksvEGXZkmz8{}@CJCSCSU\UaUacjcrcrt{tCtCBFG'9IRF5'7FFHhQbduxE++,