AVh1 dZddlZddlZddlZddlmZddlmZddlm Z ddlm Z ddlm Z ddlm Z dd lm Z dd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlmZddlddl m!Z!ddl"m#Z#ddl"m$Z$ddl%m&Z&ddl'm(Z(ddl'm)Z)ddl*m+Z+ejXZ-e.hd Z/d!Z0 dd"Z1Gd#d$Z2dd%Z3e+d&g'e$jh dd(Z5e+d&g'e$jh dd)Z6ejnje6_e+d*e$jh dd+Z8Gd,d-Z9d.Z:d/Z;d0Z<e+d1g'e$jh dd2Z=e+d1g'e$jh dd3Z>e#j~e#j~e=jd4d5d6d7e>_ dd9Z@Gd:d;ZAe+dZCe+d?e$jhdd@ZDddAZEe+dBg'e$jhe#jddCd8dDEe#jddFd8dGE ddHZGe+dBg'e$jh ddIZHe+dJe$jh ddKZIe+dLg'e$jh ddMZJe+dLg'e$jhdddd8dGgdNddfdOZKe+dPg'e$jhd8dGgdNdfdQZLe+dRg'e$jhddddd8dGgdNddf dSZMe+dTg'e$jh ddUZNe+dTg'e$jh ddVZOdWZPe+dXe$jh ddYZQe+dZg'e$jhe#jdZdGgdNdfd[ZSe+d\d]d\g'e$jhe#jd]dGgdNdfd^ZTe+d_d`d_g'e$jhe#jd`dGgdNdfdaZU ddbZVe+dcg'e$jh ddeZWe+dcg'e$jhdddddgdfddfdgZXe#j~ejjd6d7eW_ejjeX_e+dhg'e$jh ddiZZe+dhg'e$jh ddjZ[eIeOe[fZ\e+dke$jh ddlZ]e+dme$jhddnZ^ddoZ_e+dpg'e$jhddqZ`e+dpg'e$jhddrZae`jea_e+dse$je$jhddtZce+due$je$jhddvZde+dwg'e$je$jhddyZedzZfdd{Zge+d|d}g'e$jhdd~Zhe+d|d}g'e$jhe#jPdddddZiehjei_e+ddg'e$je$jhe#jPdddddZje+ddg'e$jhddZkdZle+dg'e$jhddZme+dg'e$jhe(ddd ddZndZoe+dg'e$jhe#jdeo ddZqdZre+dg'e$jh ddZse+dg'e$jhddZte+ddg'e$jhddZue+ddg'e$jh ddZve+dg'e$jhddZwe+de$jhddZxe+de$jhddZye+ddg'e$jhddZze+dg'e$jh ddZ{e+de$jhddZ|e+de$jhddZ}e+de$jhddZ~e+dg'e$jhdGe jdxdfdZe+dg'e$jh ddZejje_e jdddZe jdddZe jdddZe jdddZe+dg'e$jhddZddZdZe+dg'e$jhe#jPddd ddZe+dg'e$jhddZe+de$jhddZe+de$jhddZdZe+dde$jhdd8e j dfdÄZe+ddūe$jh ddƄZe+ddȫe$jh ddɄZddʄZe+dg'e$jhe#jdd̬ dd̈́Ze+dg'e$jh dd΄Ze+dg'e$jhe#jddЬ ddфZe+dg'e$jh dd҄Ze jdddԄZe+dg'e$jhddքZe+dg'e$jh ddׄZe+ddg'e$jhddڄZe+ddg'e$jhddۄZe+dg'e$jhej<Ze+dg'e$jhej>Ze+dg'e$jhej@Ze+dg'e$jhejBZe+dg'e$jhddZe$jejFe$jejHe$jejJe$jejLy(a9Primitive Neural Net (NN) Operations. ## Notes on padding Several neural network operations, such as `tf.nn.conv2d` and `tf.nn.max_pool2d`, take a `padding` parameter, which controls how the input is padded before running the operation. The input is padded by inserting values (typically zeros) before and after the tensor in each spatial dimension. The `padding` parameter can either be the string `'VALID'`, which means use no padding, or `'SAME'` which adds padding according to a formula which is described below. Certain ops also allow the amount of padding per dimension to be explicitly specified by passing a list to `padding`. In the case of convolutions, the input is padded with zeros. In case of pools, the padded input values are ignored. For example, in a max pool, the sliding window ignores padded values, which is equivalent to the padded values being `-infinity`. ### `'VALID'` padding Passing `padding='VALID'` to an op causes no padding to be used. This causes the output size to typically be smaller than the input size, even when the stride is one. In the 2D case, the output size is computed as: ```python out_height = ceil((in_height - filter_height + 1) / stride_height) out_width = ceil((in_width - filter_width + 1) / stride_width) ``` The 1D and 3D cases are similar. Note `filter_height` and `filter_width` refer to the filter size after dilations (if any) for convolutions, and refer to the window size for pools. ### `'SAME'` padding With `'SAME'` padding, padding is applied to each spatial dimension. When the strides are 1, the input is padded such that the output size is the same as the input size. In the 2D case, the output size is computed as: ```python out_height = ceil(in_height / stride_height) out_width = ceil(in_width / stride_width) ``` The amount of padding used is the smallest amount that results in the output size. The formula for the total amount of padding per dimension is: ```python if (in_height % strides[1] == 0): pad_along_height = max(filter_height - stride_height, 0) else: pad_along_height = max(filter_height - (in_height % stride_height), 0) if (in_width % strides[2] == 0): pad_along_width = max(filter_width - stride_width, 0) else: pad_along_width = max(filter_width - (in_width % stride_width), 0) ``` Finally, the padding on the top, bottom, left and right are: ```python pad_top = pad_along_height // 2 pad_bottom = pad_along_height - pad_top pad_left = pad_along_width // 2 pad_right = pad_along_width - pad_left ``` Note that the division by 2 means that there might be cases when the padding on both sides (top vs bottom, right vs left) are off by one. In this case, the bottom and right sides always get the one additional padded pixel. For example, when pad_along_height is 5, we pad 2 pixels at the top and 3 pixels at the bottom. Note that this is different from existing libraries such as PyTorch and Caffe, which explicitly specify the number of padded pixels and always pad the same number of pixels on both sides. Here is an example of `'SAME'` padding: >>> in_height = 5 >>> filter_height = 3 >>> stride_height = 2 >>> >>> in_width = 2 >>> filter_width = 2 >>> stride_width = 1 >>> >>> inp = tf.ones((2, in_height, in_width, 2)) >>> filter = tf.ones((filter_height, filter_width, 2, 2)) >>> strides = [stride_height, stride_width] >>> output = tf.nn.conv2d(inp, filter, strides, padding='SAME') >>> output.shape[1] # output_height: ceil(5 / 2) 3 >>> output.shape[2] # output_width: ceil(2 / 1) 2 ### Explicit padding Certain ops, like `tf.nn.conv2d`, also allow a list of explicit padding amounts to be passed to the `padding` parameter. This list is in the same format as what is passed to `tf.pad`, except the padding must be a nested list, not a tensor. For example, in the 2D case, the list is in the format `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]` when `data_format` is its default value of `'NHWC'`. The two `[0, 0]` pairs indicate the batch and channel dimensions have no padding, which is required, as only spatial dimensions can have padding. For example: >>> inp = tf.ones((1, 3, 3, 1)) >>> filter = tf.ones((2, 2, 1, 1)) >>> strides = [1, 1] >>> padding = [[0, 0], [1, 2], [0, 1], [0, 0]] >>> output = tf.nn.conv2d(inp, filter, strides, padding=padding) >>> tuple(output.shape) (1, 5, 3, 1) >>> # Equivalently, tf.pad can be used, since convolutions pad with zeros. >>> inp = tf.pad(inp, padding) >>> # 'VALID' means to use no padding in conv2d (we already padded inp) >>> output2 = tf.nn.conv2d(inp, filter, strides, padding='VALID') >>> tf.debugging.assert_equal(output, output2) ### Difference between convolution and pooling layers How padding is used in convolution layers and pooling layers is different. For convolution layers, padding is filled with values of zero, and padding is multiplied with kernels. For pooling layers, padding is excluded from the computation. For example when applying average pooling to a 4x4 grid, how much padding is added will not impact the output. Here is an example that demonstrates the difference. >>> x_in = np.array([[ ... [[2], [2]], ... [[1], [1]], ... [[1], [1]]]]) >>> kernel_in = np.array([ # simulate the avg_pool with conv2d ... [ [[0.25]], [[0.25]] ], ... [ [[0.25]], [[0.25]] ]]) >>> x = tf.constant(x_in, dtype=tf.float32) >>> kernel = tf.constant(kernel_in, dtype=tf.float32) >>> conv_out = tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='SAME') >>> pool_out = tf.nn.avg_pool(x, [2, 2], strides=[1, 1, 1, 1], padding='SAME') >>> print(conv_out.shape, pool_out.shape) (1, 3, 2, 1) (1, 3, 2, 1) >>> tf.reshape(conv_out, [3, 2]).numpy() # conv2d takes account of padding array([[1.5 , 0.75], [1. , 0.5 ], [0.5 , 0.25]], dtype=float32) >>> tf.reshape(pool_out, [3, 2]).numpy() # avg_pool excludes padding array([[1.5, 1.5], [1. , 1. ], [1. , 1. ]], dtype=float32) API docstring: tensorflow.nn N)context)config) constant_op)dtypes) errors_impl) graph_util)ops) random_seed)tensor) tensor_shape) tensor_util) array_ops)array_ops_stack) check_ops) gen_math_ops) gen_nn_ops)math_ops)nn_grad) random_ops)stateless_random_ops) variables)*)device_context) deprecation)dispatch)collections_abc)deprecated_args)deprecated_argument_lookup) tf_export> NHCNWCNHWCNWHCNDHWCNDWHCNHDWCNHWDCNWDHCNWHDCc | dg|dzzSt|trnYt|tr t|}n=t|tr|g}n)t|tj s|g}n t|}t |}||dzk(r|S|dk(r||z}n$||k7rt|d|d|dzd|d|d| |dk(rddg|zSdg|zdgzS)z%Formats a value input for gen_nn_ops.z should be of length 1, z or z . Received: = of length ) isinstancelisttupleintrSizedlen ValueError)valuen channel_indexname len_values L/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/nn_ops.py _get_sequencer<s  ] 3!a%=t% KE% GE e_22 3 GE KE%j)!a% L!^ AIEA~ v5aSQUGD""&q{9+G HHa q6E> 3;! c  tj|d||g5}tj|d}|j}tj|d}|j}t ||||||} | ||cdddS#1swYyxYw)axComputes sums of N-D convolutions (actually cross correlation). It is required that 1 <= N <= 3. This is used to implement the more generic `convolution` function, which extends the interface of this function with a `dilation_rate` parameter. Args: input: Rank N+2 tensor of type T of shape `[batch_size] + input_spatial_shape + [in_channels]` if `data_format` does not start with `"NC"`, or `[batch_size, in_channels] + input_spatial_shape` if `data_format` starts with `"NC"`. filter: Rank N+2 tensor of type T of shape `filter_spatial_shape + [in_channels, out_channels]`. Rank of either `input` or `filter` must be known. padding: Padding method to use, must be either "VALID" or "SAME". data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". strides: Sequence of N positive integers, defaults to `[1] * N`. name: Name prefix to use. Returns: Rank N+2 tensor of type T of shape `[batch_size] + output_spatial_shape + [out_channels]`, where if padding == "SAME": output_spatial_shape = input_spatial_shape if padding == "VALID": output_spatial_shape = input_spatial_shape - filter_spatial_shape + 1. Raises: ValueError: if ranks are incompatible. non_atrous_convolutioninputr9filter) filter_shapepadding data_formatstridesr9N)r name_scopeconvert_to_tensorshape_NonAtrousConvolution) r@rBrDrErFr9scope input_shaperCops r;_non_atrous_convolutionrNs\ ~~d4ufoF %  ! !%g 6E++K  " "6 9F<NCWr!NCHWr"<`data_format` must be 'NWC' or 'NCW'. Received: data_format=rr,r"rWz>`data_format` must be 'NHWC' or 'NCHW'. Received: data_format=r$NCDHWz@`data_format` must be 'NDHWC' or 'NCDHW'. Received: data_format=)ndims with_rankrDr9r5rankr4rFrE_conv1dconv_opr0conv2d_conv3d_expanded_batch) selfrLrCrDrErFr9num_batch_dims conv_dimss r;__init__z_NonAtrousConvolution.__init__Ss$!++   n ,q 02lDLDI%))   ~ - 13k  ##.-y9I9I8J L MM1 1 1N BQ F J  00;0@0@/AB*+ - ..!!N2Q6Iig W "  %i[1&i{3w<. B CCA~   < <22=@A AQZdl$d\\dl a   v 5 #W %+ & a&4=(22=@A Adl$ddl a   w 6#W %+ ' !a&4=(22=@A Adl$d+dl r=c$t||||||S)N)r6filtersstriderDrEr9conv1d)rar@rBrFrDrEr9s r;r]z_NonAtrousConvolution._conv1ds!    r=c|j|||j|j|j|jS)N)r@rBrFrDrEr9)r^rFrDrEr9rainprBs r;__call__z_NonAtrousConvolution.__call__s< <<  $$ YY  r=NNNr+)__name__ __module__ __qualname____doc__rdr]rmr=r;rJrJAs&, E,Tr=rJctj|d|g5tj|d}|j}|| d}|j st j|| d}|d| }|j st j|d| }t |tjr)t j|dg|jz}n.t j|t jdg|fd}||}|j| d} | j st j|| d} t j|t j|| fd} | j|jd| | j| dz| cdddS#1swYyxYw)a*Returns `unsqueeze_batch(op(squeeze_batch(inp)))`. Where `squeeze_batch` reshapes `inp` to shape `[prod(inp.shape[:-inner_rank])] + inp.shape[-inner_rank:]` and `unsqueeze_batch` does the reverse reshape but on the output. Args: inp: A tensor with dims `batch_shape + inner_shape` where `inner_shape` is length `inner_rank`. op: A callable that takes a single input tensor and returns a single. output tensor. inner_rank: A python integer. name: A string. Returns: `unsqueeze_batch_op(squeeze_batch(inp))`. squeeze_batch_dimsr@rANaxis) r rGrHrIis_fully_definedrr/r TensorShapereshapeas_listconcat set_shape) rlrM inner_rankr9rI inner_shape batch_shape inp_reshaped out_reshapedout_inner_shapeouts r;rurus$ ~~d03%8  ' 2C IIE %K  ' ' )OOC(*6k *%K  ' ' )OOC(:+6k+|778&&sRD;3F3F3H,HIl&& y"{ 3"=?ll#L"((*6O  + + -! 5zklCo   i&& _'EBO QCMM#))Lj[)CIIzkl,CCD 9s FF55F>z nn.dilation2d)v1c^|dk7rtd|tj||||||S)a Computes the grayscale dilation of 4-D `input` and 3-D `filters` tensors. The `input` tensor has shape `[batch, in_height, in_width, depth]` and the `filters` tensor has shape `[filter_height, filter_width, depth]`, i.e., each input channel is processed independently of the others with its own structuring function. The `output` tensor has shape `[batch, out_height, out_width, depth]`. The spatial dimensions of the output tensor depend on the `padding` algorithm. We currently only support the default "NHWC" `data_format`. In detail, the grayscale morphological 2-D dilation is the max-sum correlation (for consistency with `conv2d`, we use unmirrored filters): output[b, y, x, c] = max_{dy, dx} input[b, strides[1] * y + rates[1] * dy, strides[2] * x + rates[2] * dx, c] + filters[dy, dx, c] Max-pooling is a special case when the filter has size equal to the pooling kernel size and contains all zeros. Note on duality: The dilation of `input` by the `filters` is equal to the negation of the erosion of `-input` by the reflected `filters`. Args: input: A `Tensor`. Must be one of the following types: `float32`, `float64`, `int32`, `uint8`, `int16`, `int8`, `int64`, `bfloat16`, `uint16`, `half`, `uint32`, `uint64`. 4-D with shape `[batch, in_height, in_width, depth]`. filters: A `Tensor`. Must have the same type as `input`. 3-D with shape `[filter_height, filter_width, depth]`. strides: A list of `ints` that has length `>= 4`. The stride of the sliding window for each dimension of the input tensor. Must be: `[1, stride_height, stride_width, 1]`. padding: A `string` from: `"SAME", "VALID"`. The type of padding algorithm to use. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A `string`, only `"NHWC"` is currently supported. dilations: A list of `ints` that has length `>= 4`. The input stride for atrous morphological dilation. Must be: `[1, rate_height, rate_width, 1]`. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input`. r"Q`data_format` values other than 'NHWC' are not supported. Received: data_format=r@rBrFratesrDr9)r5r dilation2dr@rfrFrDrE dilationsr9s r; dilation2d_v2rsRvF 99D G HH   U&-'.%.'.$(  **r=cntd|d|}td|d|}tj||||||S)NrfrBrr)rrr)r@rBrFrrDr9rfrs r; dilation2d_v1r)s? &i(F K& $[)We L%   ufgugt LLr=znn.with_space_to_batchc tj|d}|j}fd}t|||||||} | |dS)a|Performs `op` on the space-to-batch representation of `input`. This has the effect of transforming sliding window operations into the corresponding "atrous" operation in which the input is sampled at the specified `dilation_rate`. In the special case that `dilation_rate` is uniformly 1, this simply returns: op(input, num_spatial_dims, padding) Otherwise, it returns: batch_to_space_nd( op(space_to_batch_nd(input, adjusted_dilation_rate, adjusted_paddings), num_spatial_dims, "VALID") adjusted_dilation_rate, adjusted_crops), where: adjusted_dilation_rate is an int64 tensor of shape [max(spatial_dims)], adjusted_{paddings,crops} are int64 tensors of shape [max(spatial_dims), 2] defined as follows: We first define two int64 tensors `paddings` and `crops` of shape `[num_spatial_dims, 2]` based on the value of `padding` and the spatial dimensions of the `input`: If `padding = "VALID"`, then: paddings, crops = required_space_to_batch_paddings( input_shape[spatial_dims], dilation_rate) If `padding = "SAME"`, then: dilated_filter_shape = filter_shape + (filter_shape - 1) * (dilation_rate - 1) paddings, crops = required_space_to_batch_paddings( input_shape[spatial_dims], dilation_rate, [(dilated_filter_shape - 1) // 2, dilated_filter_shape - 1 - (dilated_filter_shape - 1) // 2]) Because `space_to_batch_nd` and `batch_to_space_nd` assume that the spatial dimensions are contiguous starting at the second dimension, but the specified `spatial_dims` may not be, we must adjust `dilation_rate`, `paddings` and `crops` in order to be usable with these operations. For a given dimension, if the block size is 1, and both the starting and ending padding and crop amounts are 0, then space_to_batch_nd effectively leaves that dimension alone, which is what is needed for dimensions not part of `spatial_dims`. Furthermore, `space_to_batch_nd` and `batch_to_space_nd` handle this case efficiently for any number of leading and trailing dimensions. For 0 <= i < len(spatial_dims), we assign: adjusted_dilation_rate[spatial_dims[i] - 1] = dilation_rate[i] adjusted_paddings[spatial_dims[i] - 1, :] = paddings[i, :] adjusted_crops[spatial_dims[i] - 1, :] = crops[i, :] All unassigned values of `adjusted_dilation_rate` default to 1, while all unassigned values of `adjusted_paddings` and `adjusted_crops` default to 0. Note in the case that `dilation_rate` is not uniformly 1, specifying "VALID" padding is equivalent to specifying `padding = "SAME"` with a filter_shape of `[1]*N`. Advanced usage. Note the following optimization: A sequence of `with_space_to_batch` operations with identical (not uniformly 1) `dilation_rate` parameters and "VALID" padding net = with_space_to_batch(net, dilation_rate, "VALID", op_1) ... net = with_space_to_batch(net, dilation_rate, "VALID", op_k) can be combined into a single `with_space_to_batch` operation as follows: def combined_op(converted_input, num_spatial_dims, _): result = op_1(converted_input, num_spatial_dims, "VALID") ... result = op_k(result, num_spatial_dims, "VALID") net = with_space_to_batch(net, dilation_rate, "VALID", combined_op) This eliminates the overhead of `k-1` calls to `space_to_batch_nd` and `batch_to_space_nd`. Similarly, a sequence of `with_space_to_batch` operations with identical (not uniformly 1) `dilation_rate` parameters, "SAME" padding, and odd filter dimensions net = with_space_to_batch(net, dilation_rate, "SAME", op_1, filter_shape_1) ... net = with_space_to_batch(net, dilation_rate, "SAME", op_k, filter_shape_k) can be combined into a single `with_space_to_batch` operation as follows: def combined_op(converted_input, num_spatial_dims, _): result = op_1(converted_input, num_spatial_dims, "SAME") ... result = op_k(result, num_spatial_dims, "SAME") net = with_space_to_batch(net, dilation_rate, "VALID", combined_op) Args: input: Tensor of rank > max(spatial_dims). dilation_rate: int32 Tensor of *known* shape [num_spatial_dims]. padding: str constant equal to "VALID" or "SAME" op: Function that maps (input, num_spatial_dims, padding) -> output filter_shape: If padding = "SAME", specifies the shape of the convolution kernel/pooling window as an integer Tensor of shape [>=num_spatial_dims]. If padding = "VALID", filter_shape is ignored and need not be specified. spatial_dims: Monotonically increasing sequence of `num_spatial_dims` integers (which are >= 1) specifying the spatial dimensions of `input` and output. Defaults to: `range(1, num_spatial_dims+1)`. data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". Returns: The output Tensor as described above, dimensions will vary based on the op provided. Raises: ValueError: if `padding` is invalid or the arguments are incompatible. ValueError: if `spatial_dims` are invalid. r@rAcfdS)Nc|SNrs)rl_num_spatial_dimsrMrDs r;z7with_space_to_batch..build_op..s"S"2G<r=rs)rrDrMs``r;build_opz%with_space_to_batch..build_ops < (function that maps (input, filter) -> output). filter_shape: see `with_space_to_batch`. spatial_dims: `see with_space_to_batch`. data_format: see `with_space_to_batch`. num_batch_dims: (Optional). Number of batch dims in `input_shape`. Nc tj|tjd}|jj dvr%t d|d|jj|jjst d|d|j|jjdj} ||jd r|d z} n|} |t| | | z}t|} ttd | D}|| k7st!d |Drt d| ||jd r|d} n|dd z} |j#| t%j&|} |}| O| }t)j | d krt d| t)j*| d k(r|| ||_y t/|\}}|dk(rl|t d|d|tj|d}t%j&|}||}t1|| ||_n| |_||_d |_n|dk(r,t)j8| dgt(j|_ne|dk(rRt)j:|j=| dzdg}||jd r |dd |_n|d d|_nt d|||_||_ ||_!||_"|| d|_#|jH|_y #t $rt d| d|d|jwxYw)z&Helper class for _with_space_to_batch.rrA)Nr+z>`dilation_rate.shape.rank` must be 1. Received: dilation_rate=rQzE`dilation_rate.shape` must be fully defined. Received: dilation_rate=z with shape rNNCr+c32K|]}t|ywr)r2.0xs r; z-_WithSpaceToBatch.__init__..s@c!f@sc3&K|] }|dk ywr+Nrsrs r;rz-_WithSpaceToBatch.__init__..s/L!A/Lzh`spatial_dims` must be a monotonically increasing sequence of positive integers. Received: spatial_dims=rvz$`input.shape.rank` must be at least z. Received: input.shape=z with rank z:`dilation_rate` must be positive. Received: dilation_rate=SAMEzN`filter_shape` must be specified for `padding='SAME'`. Received: filter_shape=z and padding=rCVALIDr,EXPLICITz>`padding` must be one of 'SAME' or 'VALID'. Received: padding=)%r rHrint32rIrZr5r\rydimsr6 startswithranger0sortedsetanywith_rank_at_leastr constant_valuenpallcallconvert_padding"_with_space_to_batch_base_paddings base_paddingsrrate_or_const_ratezerosarrayr{rLrrrErM_with_space_to_batch_call)rarLrrDrrCrrErbrstarting_spatial_dimorig_spatial_dimsexpected_input_rank const_raterexplicit_paddingsconst_filter_shapers r;rdz_WithSpaceToBatch.__init__s))v||/;M  1  (/=3F3F3K3K2L N OO    / / 1  (/  ! # $$ %**//288;#9#9$#?+a/+/+.BBDl\*#@.?@@AL((C/L|/L,L  77H6I K LL;#9#9$#?(,(,q0O$$%89 ++M:J&%  Q  ''1l 45 5  a -w7 !0!9G &   &&2^=  KL L**M*8Q EH 'q, ?E11}xAOWW_f -F 22-u> #(8(8(C(CD(I  # # ( ( + 1 1f6H'--557  ,,r* Q""<0 MH 8s G17 G6c&|j||Sr)rrks r;rmz_WithSpaceToBatch.__call__s 99S& !!r=rn)rorprqrrrdrrmrsr=r;rrs'.! i/V.`"r=rc|d|}|dz |z}|dz}||z }tjt|Dcgc] }||||gc}}|Scc}w)z)Helper function to compute base_paddings.Nr+r,)rrr) rCrrfilter_spatial_shapepad_extra_shapepad_extra_start pad_extra_endrrs r;rrsy &&7'78)A-1CC/$q(/!O3-!''7<=M7NO!M!,-OQ- PsAc|jjdd}|jj}g}t j |}||n|}d}d} | t |kr| } || } | dkDr1|jtj| dz |z g|z||| dzt |kr9|| dz|| dzk(r(| dz } | dzt |kr|| dz|| dzk(r(|j|| | dz|| }| dz } | t |kr|tj|Stj|dS)aReturns an `adjusted` version of `orig` based on `spatial_dims`. Tensor of the same type as `orig` and with shape `[max(spatial_dims), ...]` where: adjusted[spatial_dims[i] - 1, ...] = orig[i, ...] for 0 <= i < len(spatial_dims), and adjusted[j, ...] = fill_value for j != spatial_dims[i] - 1 for some i. If `orig` is a constant value, then the result will be a constant value. Args: orig: Tensor of rank > max(spatial_dims). fill_value: Numpy scalar (of same data type as `orig) specifying the fill value for non-spatial dimensions. spatial_dims: See with_space_to_batch. Returns: `adjusted` tensor. r+Nrdtype) get_shaper|ras_numpy_dtyper rr4appendrfull concatenaterr}) orig fill_valuer fill_dimsrparts const_orig const_or_origprev_spatial_dimrstart_istart_spatial_dims r;rrs{2nn&&(,) ** # #% %))$/* * 6*D-! C G$Q1 ll '' 1$'7789D  q53|$ $ A ,q/A"5 51fa q53|$ $ A ,q/A"5 5 LLwq1u-.#AFA) C * >>%    E1 %%r=c |dg|z}n+t||k7rtd|d|dt|tj|tj}tj |dkrtd||dg|z}n+t||k7rtd|d|dt|tj|tj}tj |dkrtd |tj |dkDr)tj |dkDrtd |d |||fS) alHelper function for verifying strides and dilation_rate arguments. This is used by `convolution` and `pool`. Args: num_spatial_dims: int strides: Optional. List of N ints >= 1. Defaults to `[1]*N`. If any value of strides is > 1, then all values of dilation_rate must be 1. dilation_rate: Optional. List of N ints >= 1. Defaults to `[1]*N`. If any value of dilation_rate is > 1, then all values of strides must be 1. Returns: Normalized (strides, dilation_rate) as int32 numpy arrays of shape [num_spatial_dims]. Raises: ValueError: if the parameters are invalid. r+z`len(dilation_rate)` should be z. Received: dilation_rate=r.rzHall values of `dilation_rate` must be positive. Received: dilation_rate=rTrUz 1` not supported in conjunction with `dilation_rate > 1`. Received: strides= and dilation_rate=)r4r5rrrr)rrFrs r;_get_strides_and_dilation_raters&C**M =-- 67G6HI00=kM*+- ..((=9-VVMA  00=@ AA _c$$G 7|'' 01A0BC**1+c'l^M NN HHWBHH -'VVGaK **14 55VVGaKRVVMA$56  $I%8 I JJ - r=znn.convolutionc ^td|d|}td|d|}t|||||||S)aComputes sums of N-D convolutions (actually cross-correlation). This also supports either output striding via the optional `strides` parameter or atrous convolution (also known as convolution with holes or dilated convolution, based on the French word "trous" meaning holes in English) via the optional `dilation_rate` parameter. Currently, however, output striding is not supported for atrous convolutions. Specifically, in the case that `data_format` does not start with "NC", given a rank (N+2) `input` Tensor of shape [num_batches, input_spatial_shape[0], ..., input_spatial_shape[N-1], num_input_channels], a rank (N+2) `filter` Tensor of shape [spatial_filter_shape[0], ..., spatial_filter_shape[N-1], num_input_channels, num_output_channels], an optional `dilation_rate` tensor of shape N (defaults to `[1]*N`) specifying the filter upsampling/input downsampling rate, and an optional list of N `strides` (defaults to `[1]*N`), this computes for each N-D spatial output position `(x[0], ..., x[N-1])`: ``` output[b, x[0], ..., x[N-1], k] = sum_{z[0], ..., z[N-1], q} filter[z[0], ..., z[N-1], q, k] * padded_input[b, x[0]*strides[0] + dilation_rate[0]*z[0], ..., x[N-1]*strides[N-1] + dilation_rate[N-1]*z[N-1], q] ``` where b is the index into the batch, k is the output channel number, q is the input channel number, and z is the N-D spatial offset within the filter. Here, `padded_input` is obtained by zero padding the input using an effective spatial filter shape of `(spatial_filter_shape-1) * dilation_rate + 1` and output striding `strides`. In the case that `data_format` does start with `"NC"`, the `input` and output (but not the `filter`) are simply transposed as follows: ```python convolution(input, data_format, **kwargs) = tf.transpose(convolution(tf.transpose(input, [0] + range(2,N+2) + [1]), **kwargs), [0, N+1] + range(1, N+1)) ``` It is required that 1 <= N <= 3. Args: input: An (N+2)-D `Tensor` of type `T`, of shape `[batch_size] + input_spatial_shape + [in_channels]` if data_format does not start with "NC" (default), or `[batch_size, in_channels] + input_spatial_shape` if data_format starts with "NC". filter: An (N+2)-D `Tensor` with the same type as `input` and shape `spatial_filter_shape + [in_channels, out_channels]`. padding: A string, either `"VALID"` or `"SAME"`. The padding algorithm. `"valid"` means no padding. `"same"` results in padding evenly to the left/right or up/down of the input such that output has the same height/width dimension as the input when the strides are 1. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. strides: Optional. Sequence of N ints >= 1. Specifies the output stride. Defaults to `[1]*N`. If any value of strides is > 1, then all values of dilation_rate must be 1. dilation_rate: Optional. Sequence of N ints >= 1. Specifies the filter upsampling/input downsampling rate. In the literature, the same parameter is sometimes called `input stride` or `dilation`. The effective filter size used for the convolution will be `spatial_filter_shape + (spatial_filter_shape - 1) * (rate - 1)`, obtained by inserting (dilation_rate[i]-1) zeros between consecutive elements of the original filter in each spatial dimension i. If any value of dilation_rate is > 1, then all values of strides must be 1. name: Optional name for the returned tensor. data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". Returns: A `Tensor` with the same type as `input` of shape `[batch_size] + output_spatial_shape + [out_channels]` if data_format is None or does not start with "NC", or `[batch_size, out_channels] + output_spatial_shape` if data_format starts with "NC", where `output_spatial_shape` depends on the value of `padding`. If padding == "SAME": output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides[i]) If padding == "VALID": output_spatial_shape[i] = ceil((input_spatial_shape[i] - (spatial_filter_shape[i]-1) * dilation_rate[i]) / strides[i]). Raises: ValueError: If input/output depth does not match `filter` shape, if padding is other than `"VALID"` or `"SAME"`, or if data_format is invalid. rfrBrrrFrDrErr9)rconvolution_internal) r@rBrDrFrr9rErfrs r; convolutionr sJD &i(F K&,9o}>-     r=c &t|||||||S)Nr)rrs r;convolution_v2rs&     r=rrrBrfTc t|tjsNtj|s9t j dd||g5t j|d}dddt|tjsNtj|s9t j dd||g5t j|d}ddd|jj} |jj} |;| r| dz }n[| r| dz }nStd|jd| d |jd| | r&| dz |k7rtd |jd| d || r | |z d z } nd } |d vrtd|d||tvr| |z} n| } |t||| d}d} n t||| d}td|D} t||| d}t!j"du}|rd}n|r|rd}n|dk(rd}n |dk(rd}nd}t j ||||g5}| r|r6|dk(rt$}n|dk(rt&}nt(}||||||||cdddS| d k(r |dd}|dd}n |d d}|d d}t+t-j.|jt-j.|j||||||}|||cdddS#1swYwxYw#1swYxYw#1swYyxYw)aInternal function which performs rank agnostic convolution. Args: input: See `convolution`. filters: See `convolution`. strides: See `convolution`. padding: See `convolution`. data_format: See `convolution`. dilations: See `convolution`. name: See `convolution`. call_from_convolution: See `convolution`. num_spatial_dims: (Optional.). It is a integer describing the rank of the spatial dimensions. For `1-D`, `2-D` and `3-D` convolutions, the value of `num_spatial_dims` is `1`, `2`, and `3`, respectively. This argument is only required to disambiguate the rank of `batch_shape` when `filter_shape.ndims is None` and `len(batch_shape) > 1`. For backwards compatibility, if `num_spatial_dims is None` and `filter_shape.ndims is None`, then `len(batch_shape)` is assumed to be `1` (i.e., the input is expected to be `[batch_size, num_channels] + input_spatial_shape` or `[batch_size] + input_spatial_shape + [num_channels]`. Returns: A tensor of shape and dtype matching that of `input`. Raises: ValueError: If input and filter both have unknown shapes, or if `num_spatial_dims` is provided and incompatible with the value estimated from `filters.shape`. rNrfrAr@r,{When `num_spatial_dims` is not set, one of `input.shape.rank` or `filters.shape.rank` must be known. Received: input.shape=rQ and filters.shape=zR`filters.shape.rank - 2` should equal `num_spatial_dims`. Received: filters.shape= and num_spatial_dims=r+>r+r,rRzB`num_spatial_dims` must be 1, 2, or 3. Received: num_spatial_dims=.rFc3&K|] }|dk7 ywrrs)rrs r;rz'convolution_internal.. s4Q!q&4rrFrConv2DrRConv3DrirDrErr9rv)rFrr9rEr)r/ variables_libVariabler is_tf_typer rGrH tensor_libTensorrIr\r5_CHANNELS_LAST_FORMATSr<rrenclosing_tpu_context_conv2d_expanded_batchr`ri Convolutionr as_shape)r@rfrFrDrErr9call_from_convolutionr filters_rank inputs_rankrbr8is_dilated_convhas_tpu_context default_namerMs r;rrs|P Wm44 5   ) .w6F G?%%gI>g? UJ-- .{7M7M 8 .w6F G9##E8e9##,   +%) $q  ##(;;-y F"==/<. B CC  q(,<<   y?,- / 00  #33a7NNY&  &&6%7q : ;;K+AA"%55M"Mi)9=)+IOi)9=)+I4)44O '#3]I N'"88:$F/ L 3 L1L1LL ~~dL5'*:;$ t o Q  # q #     !$ $ ( ! !"+abM !B-aO     ,    . !!+ -bw I$ $ A??99x$ $ s+ K1K#"1K0A/K0K #K-0K9c.eZdZdZ ddZdZdZy)raHelper class for convolution. Note that this class assumes that shapes of input and filter passed to `__call__` are compatible with `input_shape`, `filter_shape`, and `num_spatial_dims` passed to the constructor. Arguments input_shape: static shape of input. i.e. input.shape. Its length is `batch_shape + input_spatial_shape + [num_channels]` if `data_format` does not start with `NC`, or `batch_shape + [num_channels] + input_spatial_shape` if `data_format` starts with `NC`. filter_shape: static shape of the filter. i.e. filter.shape. padding: The padding algorithm, must be "SAME" or "VALID". strides: see convolution. dilation_rate: see convolution. name: see convolution. data_format: A string or `None`. Specifies whether the channel dimension of the `input` and output is the last dimension (if `data_format` is `None` or does not start with `NC`), or the first post-batch dimension (i.e. if `data_format` starts with `NC`). num_spatial_dims: (Usually optional.) Python integer, the rank of the spatial and channel dimensions. For `1-D`, `2-D` and `3-D` convolutions, the value of `num_spatial_dims` is `1`, `2`, and `3`, respectively. This argument is only required to disambiguate the rank of `batch_shape` when `filter_shape.ndims is None` and `len(batch_shape) > 1`. For backwards compatibility, if `num_spatial_dims is None` and `filter_shape.ndims is None`, then `len(batch_shape)` is assumed to be `1` (i.e., the input is expected to be `[batch_size, num_channels] + input_spatial_shape` or `[batch_size] + input_spatial_shape + [num_channels]`. Nc d} tj|}tj|}|jA|0|j|dzk7rtd|d|jd||jdz }|j||j|z dz } ||jdz }nP|jD|j|dzkrtd|d|jd|| |j|z dz } |+td|d|jd |d |j| d} | dkr1td | d |d|jd |d|jd| ||j ds)tj ||| z} t| || z} n+tj || } t| dz|| zdz} tj ||} | | zjdstd|d| d|d| d t|||\}}||_ ||_ ||_ ||_ ||_||_||_| |_||_t'||||j(|| || |_y)z Helper function for convolution.Nr,zM`filters.shape.rank` must be `num_spatial_dims + 2`. Received: filters.shape=rQrr+zQ`input.shape.rank` must be >= than `num_spatial_dims + 2`. Received: input.shape=rz and `filters.shape=z ` of rank z%Batch dims should be >= 1, but found z. Batch dims was estimated as `input.shape.rank - num_spatial_dims - 1` and `num_spatial_dims` was either provided or estimated as `filters.shape.rank - 2`. Received: input.shape=z, filters.shape=z, and num_spatial_dims=rrzsThe number of input channels is not divisible by the corresponding number of output filters. Received: input.shape=z with z channels and filters.shape=z output filters.)rrDrrCrrErb)r rrZr5r\rdimension_at_indexris_compatible_withrrLrCrErFrDr9rrbrr _build_opr^) rarLrCrDrFrr9rErrbinput_channels_dimr filter_dims r;rdzConvolution.__init__esaN((6L'' 4K%  &    01 4 4 )N)L4E4E3FG 01 34 4 (--1$)9)E"((+;;a?n$**Q.    &   /!3 3''2m9[=M=ML4E4E3F H II n  1.1AB#$/-y9I9I8JK'. ,2C2C1DE./  1 22+"8"8">':: '.8:>+;n+LMl':: ~' 1 .?!CEl00?OPJ  + ? ? B  ==HM  <\NK- / 00 <'=2G]#D$D"DDLDLDI&D(D,D$#!!%'DLr=c t|j|j||j|j|j |j S)N)rCrDrErFr9rb)rJrLrCrErFr9rb)rarrDs r;rzConvolution._build_opsD  &&$$ YY** ,,r=c tjPt|||j|j|j |j |jd|j S|j||S)NF)rFrDrErr9r r) rrrrFrDrErr9rr^rks r;rmzConvolution.__call__sj++-9 ! ,,,,&&&&yy %00 2 2\\#v &&r=NNNNN)rorprqrrrdrrmrsr=r;rrCs*J! $b'H,'r=rznn.poolc  td|d|}tj|d|jz|g5tj|d}t | dks dkDrt d|d t ||jj d zt ||\}}|d k(r)tj|dkDrt d |d |tj||kDrt d|d|ttttttd| fvrt d|d||j!ds6dgt#|zdgz dgt#|zdgz t%d dzn/ddgt#|z ddgt#|z t%d d z dk(rA||dk(r t'd n |dk(r t'd nt d|dg z dg z n t'| fd} t)|||| |cdddS#1swYyxYw)aG Performs an N-D pooling operation. In the case that `data_format` does not start with "NC", computes for 0 <= b < batch_size, 0 <= x[i] < output_spatial_shape[i], 0 <= c < num_channels: ``` output[b, x[0], ..., x[N-1], c] = REDUCE_{z[0], ..., z[N-1]} input[b, x[0] * strides[0] - pad_before[0] + dilation_rate[0]*z[0], ... x[N-1]*strides[N-1] - pad_before[N-1] + dilation_rate[N-1]*z[N-1], c], ``` where the reduction function REDUCE depends on the value of `pooling_type`, and pad_before is defined based on the value of `padding` as described in the "returns" section of `tf.nn.convolution` for details. The reduction never includes out-of-bounds positions. In the case that `data_format` starts with `"NC"`, the `input` and output are simply transposed as follows: ```python pool(input, data_format, **kwargs) = tf.transpose(pool(tf.transpose(input, [0] + range(2,N+2) + [1]), **kwargs), [0, N+1] + range(1, N+1)) ``` Args: input: Tensor of rank N+2, of shape `[batch_size] + input_spatial_shape + [num_channels]` if data_format does not start with "NC" (default), or `[batch_size, num_channels] + input_spatial_shape` if data_format starts with "NC". Pooling happens over the spatial dimensions only. window_shape: Sequence of N ints >= 1. pooling_type: Specifies pooling operation, must be "AVG" or "MAX". padding: The padding algorithm, must be "SAME" or "VALID". See the "returns" section of `tf.nn.convolution` for details. dilation_rate: Optional. Dilation rate. List of N ints >= 1. Defaults to `[1]*N`. If any value of dilation_rate is > 1, then all values of strides must be 1. strides: Optional. Sequence of N ints >= 1. Defaults to `[1]*N`. If any value of strides is > 1, then all values of dilation_rate must be 1. name: Optional. Name of the op. data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". dilations: Alias for dilation_rate Returns: Tensor of rank N+2, of shape [batch_size] + output_spatial_shape + [num_channels] if data_format is None or does not start with "NC", or [batch_size, num_channels] + output_spatial_shape if data_format starts with "NC", where `output_spatial_shape` depends on the value of padding: If padding = "SAME": output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides[i]) If padding = "VALID": output_spatial_shape[i] = ceil((input_spatial_shape[i] - (window_shape[i] - 1) * dilation_rate[i]) / strides[i]). Raises: ValueError: if arguments are invalid. rrz%s_poolr@rAr+rRz?`len(window_shape)` must be 1, 2, or 3. Received: window_shape=r.r,rzZpooling with 'SAME' padding is not implemented for `dilation_rate` > 1. Received: padding=rzu`strides` > `window_shape` not supported due to inconsistency between CPU and GPU implementations. Received: strides=z and window_shape=))MAXr+)rr,)rrR)AVGr+)rr,)rrRz-D z pooling is not supported.Nrr!r")rErVrWzAdata_format must be either 'NWC' or 'NCW'. Received: data_format=cdk(rtj| d} ||fd i}dk(rtj| dg}|S)Nr+rr9)r expand_dimssqueeze) converted_inputrconverted_paddingradjusted_stridesadjusted_window_shapedata_format_kwargsrop_key pooling_opsrKrs r;rMzpool..ops~ Q #//0- ~~dI););)=>gS#"'  ! !%g 6E<(!/!3 ''3nKl+,. // OO 01 45;'=2G]&RVVMA$56  44;9=(/ + ,,  vvg $%  DDK9M*^ - ..  K, -F [   c,/I J LL+"8"8"> cD$66!<tG},s21.23l !ftL'99Q$w-/1.23l1   u 4!f5 % !f522=@A A c$99//K8   # !!  #[S#S#S#s G0H::Ic (t||||||||S)aV Performs an N-D pooling operation. In the case that `data_format` does not start with "NC", computes for 0 <= b < batch_size, 0 <= x[i] < output_spatial_shape[i], 0 <= c < num_channels: ``` output[b, x[0], ..., x[N-1], c] = REDUCE_{z[0], ..., z[N-1]} input[b, x[0] * strides[0] - pad_before[0] + dilation_rate[0]*z[0], ... x[N-1]*strides[N-1] - pad_before[N-1] + dilation_rate[N-1]*z[N-1], c], ``` where the reduction function REDUCE depends on the value of `pooling_type`, and pad_before is defined based on the value of `padding` as described in the "returns" section of `tf.nn.convolution` for details. The reduction never includes out-of-bounds positions. In the case that `data_format` starts with `"NC"`, the `input` and output are simply transposed as follows: ```python pool(input, data_format, **kwargs) = tf.transpose(pool(tf.transpose(input, [0] + range(2,N+2) + [1]), **kwargs), [0, N+1] + range(1, N+1)) ``` Args: input: Tensor of rank N+2, of shape `[batch_size] + input_spatial_shape + [num_channels]` if data_format does not start with "NC" (default), or `[batch_size, num_channels] + input_spatial_shape` if data_format starts with "NC". Pooling happens over the spatial dimensions only. window_shape: Sequence of N ints >= 1. pooling_type: Specifies pooling operation, must be "AVG" or "MAX". strides: Optional. Sequence of N ints >= 1. Defaults to `[1]*N`. If any value of strides is > 1, then all values of dilation_rate must be 1. padding: The padding algorithm, must be "SAME" or "VALID". Defaults to "SAME". See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". dilations: Optional. Dilation rate. List of N ints >= 1. Defaults to `[1]*N`. If any value of dilation_rate is > 1, then all values of strides must be 1. name: Optional. Name of the op. Returns: Tensor of rank N+2, of shape [batch_size] + output_spatial_shape + [num_channels] if data_format is None or does not start with "NC", or [batch_size, num_channels] + output_spatial_shape if data_format starts with "NC", where `output_spatial_shape` depends on the value of padding: If padding = "SAME": output_spatial_shape[i] = ceil(input_spatial_shape[i] / strides[i]) If padding = "VALID": output_spatial_shape[i] = ceil((input_spatial_shape[i] - (window_shape[i] - 1) * dilation_rate[i]) / strides[i]). Raises: ValueError: if arguments are invalid. )r@r,r-rDrrFr9rE)r.)r@r,r-rFrDrErr9s r;pool_v2r0s*t    r=znn.atrous_conv2dc Jt|||tj|d|S)aAtrous convolution (a.k.a. convolution with holes or dilated convolution). This function is a simpler wrapper around the more general `tf.nn.convolution`, and exists only for backwards compatibility. You can use `tf.nn.convolution` to perform 1-D, 2-D, or 3-D atrous convolution. Computes a 2-D atrous convolution, also known as convolution with holes or dilated convolution, given 4-D `value` and `filters` tensors. If the `rate` parameter is equal to one, it performs regular 2-D convolution. If the `rate` parameter is greater than one, it performs convolution with holes, sampling the input values every `rate` pixels in the `height` and `width` dimensions. This is equivalent to convolving the input with a set of upsampled filters, produced by inserting `rate - 1` zeros between two consecutive values of the filters along the `height` and `width` dimensions, hence the name atrous convolution or convolution with holes (the French word trous means holes in English). More specifically: ``` output[batch, height, width, out_channel] = sum_{dheight, dwidth, in_channel} ( filters[dheight, dwidth, in_channel, out_channel] * value[batch, height + rate*dheight, width + rate*dwidth, in_channel] ) ``` Atrous convolution allows us to explicitly control how densely to compute feature responses in fully convolutional networks. Used in conjunction with bilinear interpolation, it offers an alternative to `conv2d_transpose` in dense prediction tasks such as semantic image segmentation, optical flow computation, or depth estimation. It also allows us to effectively enlarge the field of view of filters without increasing the number of parameters or the amount of computation. For a description of atrous convolution and how it can be used for dense feature extraction, please see: (Chen et al., 2015). The same operation is investigated further in (Yu et al., 2016). Previous works that effectively use atrous convolution in different ways are, among others, (Sermanet et al., 2014) and (Giusti et al., 2013). Atrous convolution is also closely related to the so-called noble identities in multi-rate signal processing. There are many different ways to implement atrous convolution (see the refs above). The implementation here reduces ```python atrous_conv2d(value, filters, rate, padding=padding) ``` to the following three operations: ```python paddings = ... net = space_to_batch(value, paddings, block_size=rate) net = conv2d(net, filters, strides=[1, 1, 1, 1], padding="VALID") crops = ... net = batch_to_space(net, crops, block_size=rate) ``` Advanced usage. Note the following optimization: A sequence of `atrous_conv2d` operations with identical `rate` parameters, 'SAME' `padding`, and filters with odd heights/ widths: ```python net = atrous_conv2d(net, filters1, rate, padding="SAME") net = atrous_conv2d(net, filters2, rate, padding="SAME") ... net = atrous_conv2d(net, filtersK, rate, padding="SAME") ``` can be equivalently performed cheaper in terms of computation and memory as: ```python pad = ... # padding so that the input dims are multiples of rate net = space_to_batch(net, paddings=pad, block_size=rate) net = conv2d(net, filters1, strides=[1, 1, 1, 1], padding="SAME") net = conv2d(net, filters2, strides=[1, 1, 1, 1], padding="SAME") ... net = conv2d(net, filtersK, strides=[1, 1, 1, 1], padding="SAME") net = batch_to_space(net, crops=pad, block_size=rate) ``` because a pair of consecutive `space_to_batch` and `batch_to_space` ops with the same `block_size` cancel out when their respective `paddings` and `crops` inputs are identical. Args: value: A 4-D `Tensor` of type `float`. It needs to be in the default "NHWC" format. Its shape is `[batch, in_height, in_width, in_channels]`. filters: A 4-D `Tensor` with the same type as `value` and shape `[filter_height, filter_width, in_channels, out_channels]`. `filters`' `in_channels` dimension must match that of `value`. Atrous convolution is equivalent to standard convolution with upsampled filters with effective height `filter_height + (filter_height - 1) * (rate - 1)` and effective width `filter_width + (filter_width - 1) * (rate - 1)`, produced by inserting `rate - 1` zeros along consecutive elements across the `filters`' spatial dimensions. rate: A positive int32. The stride with which we sample input values across the `height` and `width` dimensions. Equivalently, the rate by which we upsample the filter values by inserting zeros across the `height` and `width` dimensions. In the literature, the same parameter is sometimes called `input stride` or `dilation`. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. name: Optional name for the returned tensor. Returns: A `Tensor` with the same type as `value`. Output shape with `'VALID'` padding is: [batch, height - rate * (filter_width - 1), width - rate * (filter_height - 1), out_channels]. Output shape with `'SAME'` padding is: [batch, height, width, out_channels]. Raises: ValueError: If input/output depth does not match `filters`' shape, or if padding is other than `'VALID'` or `'SAME'`. References: Multi-Scale Context Aggregation by Dilated Convolutions: [Yu et al., 2016](https://arxiv.org/abs/1511.07122) ([pdf](https://arxiv.org/pdf/1511.07122.pdf)) Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs: [Chen et al., 2015](http://arxiv.org/abs/1412.7062) ([pdf](https://arxiv.org/pdf/1412.7062)) OverFeat - Integrated Recognition, Localization and Detection using Convolutional Networks: [Sermanet et al., 2014](https://arxiv.org/abs/1312.6229) ([pdf](https://arxiv.org/pdf/1312.6229.pdf)) Fast Image Scanning with Deep Max-Pooling Convolutional Neural Networks: [Giusti et al., 2013] (https://ieeexplore.ieee.org/abstract/document/6738831) ([pdf](https://arxiv.org/pdf/1302.1700.pdf)) )r,)r@rBrDrr9)rr broadcast_to)r6rfraterDr9s r; atrous_conv2dr4s,^   OOD$/   r=c g}|dk(r tdt|ttfrt |D]t\}}t|ttfstd|d|dt |t |dk7rtd|d|dt ||j|vt ||k7rtd|d |dt |d}||fS) aConverts Python padding to C++ padding for ops which take EXPLICIT padding. Args: padding: the `padding` argument for a Python op which supports EXPLICIT padding. expected_length: Expected number of entries in the padding list when explicit padding is used. Returns: (padding, explicit_paddings) pair, which should be passed as attributes to a C++ op. Raises: ValueError: If padding is invalid. rza'EXPLICIT' is not a valid value for `padding`. To use explicit padding, `padding` must be a list.zfWhen `padding` is a list, each element of `padding` must be a list/tuple of size 2. Received: padding=z with element at index z of type r,z of size z+When padding is a list, it must be of size z. Received: padding=)r5r/r0r1 enumeratetyper4extend)rDexpected_lengthrr dim_paddingss r;rrs-   C DD$'$W- -<  tUm 4$$+9,CA3i ./12 2 \ a $$+9,CA3i -.01 1|, - 7|&  77HI&iyW @ AAG # ##r=z nn.conv1dz7`NCHW` for data_format is deprecated, use `NCW` insteadrW) warn_oncerEz7`NHWC` for data_format is deprecated, use `NWC` insteadr"c tjd|d|}tj|d||g5}| |dk(s|dk(rd}d} d} n|d k(s|d k(rd }d } d } nt d |d gt |d | dz} d gt |d | dz}t j|| }t j|d}|jjdvrtj||| |||||} n7t|tjtj|| ||||d|} t j| | gcdddS#1swYyxYw)aqComputes a 1-D convolution of input with rank `>=3` and a `3-D` filter. Given an input tensor of shape `batch_shape + [in_width, in_channels]` if `data_format` is `"NWC"`, or `batch_shape + [in_channels, in_width]` if `data_format` is `"NCW"`, and a filter / kernel tensor of shape `[filter_width, in_channels, out_channels]`, this op reshapes the arguments to pass them to `conv2d` to perform the equivalent convolution operation. Internally, this op reshapes the input tensors and invokes `tf.nn.conv2d`. For example, if `data_format` does not start with "NC", a tensor of shape `batch_shape + [in_width, in_channels]` is reshaped to `batch_shape + [1, in_width, in_channels]`, and the filter is reshaped to `[1, filter_width, in_channels, out_channels]`. The result is then reshaped back to `batch_shape + [out_width, out_channels]` \(where out_width is a function of the stride and padding as in conv2d\) and returned to the caller. Args: value: A Tensor of rank at least 3. Must be of type `float16`, `float32`, or `float64`. filters: A Tensor of rank at least 3. Must have the same type as `value`. stride: An int or list of `ints` that has length `1` or `3`. The number of entries by which the filter is moved right at each step. padding: 'SAME' or 'VALID' use_cudnn_on_gpu: An optional `bool`. Defaults to `True`. data_format: An optional `string` from `"NWC", "NCW"`. Defaults to `"NWC"`, the data is stored in the order of `batch_shape + [in_width, in_channels]`. The `"NCW"` format stores data as `batch_shape + [in_channels, in_width]`. name: A name for the operation (optional). input: Alias for value. dilations: An int or list of `ints` that has length `1` or `3` which defaults to 1. The dilation factor for each dimension of input. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. Dilations in the batch and depth dimensions must be 1. Returns: A `Tensor`. Has the same type as input. Raises: ValueError: if `data_format` is invalid. r@r6riNr"r!r,rWrVr+rXrgrrrRr,r+rNuse_cudnn_on_gpurErr9)rBrFrDrBrErrRrr9)rrr rGr5r<rrrIrZrr_ru functoolspartialr) r6rfrgrDrBrEr9r@rspatial_start_dimr8rFrs r;ririsN  0 0%% P% ~~dHug&67*:4kV3{e7Kkm  +"6km 00;}> ??cM&!]HEEGmIq-MMI  ! !%): ;E##GQ/G {{11     +!f"    /%!  f   V&7%8 9U*:*:*:s DEE c (t||||d|||S)aComputes a 1-D convolution given 3-D input and filter tensors. Given an input tensor of shape `batch_shape + [in_width, in_channels]` if `data_format` is `"NWC"`, or `batch_shape + [in_channels, in_width]` if `data_format` is `"NCW"`, and a filter / kernel tensor of shape `[filter_width, in_channels, out_channels]`, this op reshapes the arguments to pass them to `conv2d` to perform the equivalent convolution operation. Internally, this op reshapes the input tensors and invokes `tf.nn.conv2d`. For example, if `data_format` does not start with `"NC"`, a tensor of shape `batch_shape + [in_width, in_channels]` is reshaped to `batch_shape + [1, in_width, in_channels]`, and the filter is reshaped to `[1, filter_width, in_channels, out_channels]`. The result is then reshaped back to `batch_shape + [out_width, out_channels]` \(where out_width is a function of the stride and padding as in conv2d\) and returned to the caller. Args: input: A Tensor of rank at least 3. Must be of type `float16`, `float32`, or `float64`. filters: A Tensor of rank at least 3. Must have the same type as `input`. stride: An int or list of `ints` that has length `1` or `3`. The number of entries by which the filter is moved right at each step. padding: 'SAME' or 'VALID'. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: An optional `string` from `"NWC", "NCW"`. Defaults to `"NWC"`, the data is stored in the order of `batch_shape + [in_width, in_channels]`. The `"NCW"` format stores data as `batch_shape + [in_channels, in_width]`. dilations: An int or list of `ints` that has length `1` or `3` which defaults to 1. The dilation factor for each dimension of input. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as input. Raises: ValueError: if `data_format` is invalid. T)rBrEr9rrh)r@rfrgrDrErr9s r; conv1d_v2rH3s*v       r=znn.conv1d_transposec Ftj|d|||g5}||dk(rd}d}d} n|dk(rd}d}d} ntd |dgt|d| d z}dgt|d| d z}t j ||}t j |d }t |tjs t|n|}t j|d|dg||dgd }tj|||||||| } t j| |cdddS#1swYyxYw)aThe transpose of `conv1d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is actually the transpose (gradient) of `conv1d` rather than an actual deconvolution. Args: input: A 3-D `Tensor` of type `float` and shape `[batch, in_width, in_channels]` for `NWC` data format or `[batch, in_channels, in_width]` for `NCW` data format. filters: A 3-D `Tensor` with the same type as `input` and shape `[filter_width, output_channels, in_channels]`. `filter`'s `in_channels` dimension must match that of `input`. output_shape: A 1-D `Tensor`, containing three elements, representing the output shape of the deconvolution op. strides: An int or list of `ints` that has length `1` or `3`. The number of entries by which the filter is moved right at each step. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string. `'NWC'` and `'NCW'` are supported. dilations: An int or list of `ints` that has length `1` or `3` which defaults to 1. The dilation factor for each dimension of input. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. Dilations in the batch and depth dimensions must be 1. name: Optional name for the returned tensor. Returns: A `Tensor` with the same type as `input`. Raises: ValueError: If input/output depth does not match `filter`'s shape, if `output_shape` is not at 3-element vector, if `padding` is other than `'VALID'` or `'SAME'`, or if `data_format` is invalid. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) conv1d_transposeNr!r"r+r,rVrWrXrgrr input_sizesrB out_backproprFrDrErr9)r rGr5r<rrr/rrr0r}rconv2d_backprop_inputr) r@rfrrFrDrErr9rFr8rs r;rJrJyssj ~~d.g|46#89=kU2km  km 00;}> ??cM'1mXFFGmIq-MMI  ! !%): ;E##GQ/G-7j''.)4 %.:##\2E4E%F%12C2D%E%GHIKL - -  F   V%6 7G#8#8#8s C2DD z nn.conv2dc (t||||d|||S)aComputes a 2-D convolution given `input` and 4-D `filters` tensors. The `input` tensor may have rank `4` or higher, where shape dimensions `[:-3]` are considered batch dimensions (`batch_shape`). Given an input tensor of shape `batch_shape + [in_height, in_width, in_channels]` and a filter / kernel tensor of shape `[filter_height, filter_width, in_channels, out_channels]`, this op performs the following: 1. Flattens the filter to a 2-D matrix with shape `[filter_height * filter_width * in_channels, output_channels]`. 2. Extracts image patches from the input tensor to form a *virtual* tensor of shape `[batch, out_height, out_width, filter_height * filter_width * in_channels]`. 3. For each patch, right-multiplies the filter matrix and the image patch vector. In detail, with the default NHWC format, output[b, i, j, k] = sum_{di, dj, q} input[b, strides[1] * i + di, strides[2] * j + dj, q] * filter[di, dj, q, k] Must have `strides[0] = strides[3] = 1`. For the most common case of the same horizontal and vertical strides, `strides = [1, stride, stride, 1]`. Usage Example: >>> x_in = np.array([[ ... [[2], [1], [2], [0], [1]], ... [[1], [3], [2], [2], [3]], ... [[1], [1], [3], [3], [0]], ... [[2], [2], [0], [1], [1]], ... [[0], [0], [3], [1], [2]], ]]) >>> kernel_in = np.array([ ... [ [[2, 0.1]], [[3, 0.2]] ], ... [ [[0, 0.3]], [[1, 0.4]] ], ]) >>> x = tf.constant(x_in, dtype=tf.float32) >>> kernel = tf.constant(kernel_in, dtype=tf.float32) >>> tf.nn.conv2d(x, kernel, strides=[1, 1, 1, 1], padding='VALID') Args: input: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. A Tensor of rank at least 4. The dimension order is interpreted according to the value of `data_format`; with the all-but-inner-3 dimensions acting as batch dimensions. See below for details. filters: A `Tensor`. Must have the same type as `input`. A 4-D tensor of shape `[filter_height, filter_width, in_channels, out_channels]` strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. The dimension order is determined by the value of `data_format`, see below for details. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: `batch_shape + [height, width, channels]`. Alternatively, the format could be "NCHW", the data storage order of: `batch_shape + [channels, height, width]`. dilations: An int or list of `ints` that has length `1`, `2` or `4`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 4-d tensor must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input` and the same outer batch shape. TrA)r_rs r; conv2d_v2rPs)B !%'# r=r+r+r+r+c tjd|d|}t|\}} |d}|jdrdnd} t |d| d}t |d| d }|j } t | d d } | d k(r t| } | d vrtj|||||| ||| St|tjtj||||| ||d|S)a Computes a 2-D convolution given 4-D `input` and `filter` tensors. Given an input tensor of shape `[batch, in_height, in_width, in_channels]` and a filter / kernel tensor of shape `[filter_height, filter_width, in_channels, out_channels]`, this op performs the following: 1. Flattens the filter to a 2-D matrix with shape `[filter_height * filter_width * in_channels, output_channels]`. 2. Extracts image patches from the input tensor to form a *virtual* tensor of shape `[batch, out_height, out_width, filter_height * filter_width * in_channels]`. 3. For each patch, right-multiplies the filter matrix and the image patch vector. In detail, with the default NHWC format, output[b, i, j, k] = sum_{di, dj, q} input[b, strides[1] * i + di, strides[2] * j + dj, q] * filter[di, dj, q, k] Must have `strides[0] = strides[3] = 1`. For the most common case of the same horizontal and vertical strides, `strides = [1, stride, stride, 1]`. Args: input: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. A 4-D tensor. The dimension order is interpreted according to the value of `data_format`, see below for details. filter: A `Tensor`. Must have the same type as `input`. A 4-D tensor of shape `[filter_height, filter_width, in_channels, out_channels]` strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. The dimension order is determined by the value of `data_format`, see below for details. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. use_cudnn_on_gpu: An optional `bool`. Defaults to `True`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, height, width, channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, channels, height, width]. dilations: An int or list of `ints` that has length `1`, `2` or `4`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 4-d tensor must be 1. name: A name for the operation (optional). filters: Alias for filter. Returns: A `Tensor`. Has the same type as `input`. rfrBr"rr+rRr,rFrrZrvr?)rBrFrDrBrrErr9)rBrFrDrBrrErrC) rrrrr<rIgetattrr4rr_rurDrE) r@rBrFrDrBrErr9rfrr8rIrZs r;r_r_? s\  1 1(F ,&.w7' K"--d3!- '1mY ?'Iq-E) ++% %" %% b[ JE ##    )+        +-!   r=znn.conv2d_backprop_filterc Zt|\}} tj||||||| ||| S)a@ Computes the gradients of convolution with respect to the filter. Args: input: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. 4-D with shape `[batch, in_height, in_width, in_channels]`. filter_sizes: A `Tensor` of type `int32`. An integer vector representing the tensor shape of `filter`, where `filter` is a 4-D `[filter_height, filter_width, in_channels, out_channels]` tensor. out_backprop: A `Tensor`. Must have the same type as `input`. 4-D with shape `[batch, out_height, out_width, out_channels]`. Gradients w.r.t. the output of the convolution. strides: A list of `ints`. The stride of the sliding window for each dimension of the input of the convolution. Must be in the same order as the dimension specified with format. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. use_cudnn_on_gpu: An optional `bool`. Defaults to `True`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, in_height, in_width, in_channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, in_channels, in_height, in_width]. dilations: An optional list of `ints`. Defaults to `[1, 1, 1, 1]`. 1-D tensor of length 4. The dilation factor for each dimension of `input`. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input`. )rrconv2d_backprop_filter) r@ filter_sizesrMrFrDrBrErr9rs r;rUrU s?n /w7'   * * \<';Ki 77r=znn.conv2d_backprop_inputc tjd| d|}t|\}} tj||||||| ||| S)aN Computes the gradients of convolution with respect to the input. Args: input_sizes: A `Tensor` of type `int32`. An integer vector representing the shape of `input`, where `input` is a 4-D `[batch, height, width, channels]` tensor. filter: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. 4-D with shape `[filter_height, filter_width, in_channels, out_channels]`. out_backprop: A `Tensor`. Must have the same type as `filter`. 4-D with shape `[batch, out_height, out_width, out_channels]`. Gradients w.r.t. the output of the convolution. strides: A list of `ints`. The stride of the sliding window for each dimension of the input of the convolution. Must be in the same order as the dimension specified with format. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. use_cudnn_on_gpu: An optional `bool`. Defaults to `True`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, in_height, in_width, in_channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, in_channels, in_height, in_width]. dilations: An optional list of `ints`. Defaults to `[1, 1, 1, 1]`. 1-D tensor of length 4. The dilation factor for each dimension of `input`. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). filters: Alias for filter. Returns: A `Tensor`. Has the same type as `filter`. rfrB)rrrrrN) rLrBrMrFrDrBrErr9rfrs r;rNrN sXr  1 1(F ,&.w7'   ) )6<';Ki 77r=znn.conv2d_transposec td|d|}td|d|}tj|d|||g5}t||||||| |cdddS#1swYyxYw)aF The transpose of `conv2d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `conv2d` rather than an actual deconvolution. Args: value: A 4-D `Tensor` of type `float` and shape `[batch, height, width, in_channels]` for `NHWC` data format or `[batch, in_channels, height, width]` for `NCHW` data format. filter: A 4-D `Tensor` with the same type as `value` and shape `[height, width, output_channels, in_channels]`. `filter`'s `in_channels` dimension must match that of `value`. output_shape: A 1-D `Tensor` representing the output shape of the deconvolution op. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 0. The dimension order is determined by the value of `data_format`, see below for details. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See the "returns" section of `tf.nn.convolution` for details. data_format: A string. 'NHWC' and 'NCHW' are supported. name: Optional name for the returned tensor. input: Alias for value. filters: Alias for filter. dilations: An int or list of `ints` that has length `1`, `2` or `4`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 4-d tensor must be 1. Returns: A `Tensor` with the same type as `value`. Raises: ValueError: If input/output depth does not match `filter`'s shape, or if padding is other than `'VALID'` or `'SAME'`. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) r@r6rfrBconv2d_transposerN)rr rGconv2d_transpose_v2 r6rBrrFrDrEr9r@rfrs r;rYrY7 sx| %WeWe D% %i(F K& ~~d.fl35 8<       s AAc ,tj|d|t|g5}|d}|jdrdnd}t |d|d}t |d|d }t |\}} t j|||||| ||| cdddS#1swYyxYw) a The transpose of `conv2d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `atrous_conv2d` rather than an actual deconvolution. Args: input: A 4-D `Tensor` of type `float` and shape `[batch, height, width, in_channels]` for `NHWC` data format or `[batch, in_channels, height, width]` for `NCHW` data format. filters: A 4-D `Tensor` with the same type as `input` and shape `[height, width, output_channels, in_channels]`. `filter`'s `in_channels` dimension must match that of `input`. output_shape: A 1-D `Tensor` representing the output shape of the deconvolution op. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 0. The dimension order is determined by the value of `data_format`, see below for details. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. data_format: A string. 'NHWC' and 'NCHW' are supported. dilations: An int or list of `ints` that has length `1`, `2` or `4`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 4-d tensor must be 1. name: Optional name for the returned tensor. Returns: A `Tensor` with the same type as `input`. Raises: ValueError: If input/output depth does not match `filter`'s shape, or if padding is other than `'VALID'` or `'SAME'`. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) rYNr"rr+rRr,rFr) rLrBrMrFrDrrErr9)r rGrBrr<rrrN) r@rfrrFrDrErr9r8rs r;rZrZ sB ~~d.fl358<k$//5A1MGQ yAGiM;GI!0!9G   + + +  s A!B  Bc |jj}||dkrtj|||||||St |t j tj|||||d|S)zEHelper function for `convolution_internal`; handles expanded batches.rS)rBrFrDrErr9rBrFrDrErrRrC)rIr\rr_rurDrE)r@rfrFrDrErr9 input_ranks r;rr s{{*:>           !    r=znn.atrous_conv2d_transposec *tj|d|||g5}tj|d}tj|d}|jjdj |jdsVt d|jd|jdd|jd |jdd |d krt d ||d k(rt|||gd |dcdddStj|d}|jj tjdgst d|jt|tr t|}t|ttjfri|jjdj |ds:t d|d|dd|jd|jdd |dk(r|jjr|jj!}nt#j$|}|d|d } }||d z |d z zz} | | d z |d z zz} | d z } | d z } | dz}| |z }| dz}| |z }n|dk(r d}d}d}d}nt d||d |z|z}|d|z|z}|||zz |z}|||zz |z}d|gd|gg}t#j&|||}||z|dz||z|z||z|z|dg}t)j*|||gd dd}|||zg|||zgg}t#j,|||cdddS#1swYyxYw)aThe transpose of `atrous_conv2d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `atrous_conv2d` rather than an actual deconvolution. Args: value: A 4-D `Tensor` of type `float`. It needs to be in the default `NHWC` format. Its shape is `[batch, in_height, in_width, in_channels]`. filters: A 4-D `Tensor` with the same type as `value` and shape `[filter_height, filter_width, out_channels, in_channels]`. `filters`' `in_channels` dimension must match that of `value`. Atrous convolution is equivalent to standard convolution with upsampled filters with effective height `filter_height + (filter_height - 1) * (rate - 1)` and effective width `filter_width + (filter_width - 1) * (rate - 1)`, produced by inserting `rate - 1` zeros along consecutive elements across the `filters`' spatial dimensions. output_shape: A 1-D `Tensor` of shape representing the output shape of the deconvolution op, of form `[batch, out_height, out_width, out_channels]`. rate: A positive int32. The stride with which we sample input values across the `height` and `width` dimensions. Equivalently, the rate by which we upsample the filter values by inserting zeros across the `height` and `width` dimensions. In the literature, the same parameter is sometimes called `input stride` or `dilation`. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. name: Optional name for the returned tensor. Returns: A `Tensor` with the same type as `value`. Raises: ValueError: If input/output depth does not match `filters`' shape, or if padding is other than `'VALID'` or `'SAME'`, or if the `rate` is less than one, or if the output_shape is not a tensor with 4 elements. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) atrous_conv2d_transposer6rArfrRzc`value` channel count must be compatible with `filters` input channel count. Received: value.shape=z with channel count rz with input channel count rr+z/`rate` cannot be less than one. Received: rate=rQr")rFrDrENrr@z<`output_shape` must have shape (4,). Received: output_shape=r,zl`output_shape` channel count must be compatible with `filters` output channel count. Received: output_shape=z with output channel count rrrz>`padding` must be either 'VALID' or 'SAME'. Received: padding=)r@r block_size)rLrBrMrFrDrE)r@rrb)r rGrHrrrr5rYr rzr/r1r0rndarrayryr|rrIspace_to_batchrrNbatch_to_space)r6rfrr3rDr9 output_shape_rC filter_height filter_widthfilter_height_upfilter_width_up pad_height pad_widthpad_top pad_bottompad_left pad_right in_heightin_widthpad_bottom_extrapad_right_extraspace_to_batch_padrLbatch_to_space_crops r;rara snh ~~d5g|46nA9=  ! !%g 6E##G)1H1H1J0KM NN,&,'l,rzz 23    % %a ( ; ;LO L <J)!_-.$..011L  "1%&a  )* *&     - - /((*224  w/ $0O\!_\m'-!*;q)II$ q(8TAX'FFo#a'j!A%iag'jahh&i G gjhi ,,396 77Q')J6IA)I5Hy4//47ho-5O./!_1EF  $ $0T CE t l1o% 4D(D'M O #,l1oK  , ,  E$Z2B%BC$i/&ABD  # #.4 A[nAnAnAsC,N I.N  Nznn.depthwise_conv2d_nativec Xt|\}}tj||||||||S)a Computes a 2-D depthwise convolution. Given an input tensor of shape `[batch, in_height, in_width, in_channels]` and a filter / kernel tensor of shape `[filter_height, filter_width, in_channels, channel_multiplier]`, containing `in_channels` convolutional filters of depth 1, `depthwise_conv2d` applies a different filter to each input channel (expanding from 1 channel to `channel_multiplier` channels for each), then concatenates the results together. Thus, the output has `in_channels * channel_multiplier` channels. ``` for k in 0..in_channels-1 for q in 0..channel_multiplier-1 output[b, i, j, k * channel_multiplier + q] = sum_{di, dj} input[b, strides[1] * i + di, strides[2] * j + dj, k] * filter[di, dj, k, q] ``` Must have `strides[0] = strides[3] = 1`. For the most common case of the same horizontal and vertices strides, `strides = [1, stride, stride, 1]`. Args: input: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. filter: A `Tensor`. Must have the same type as `input`. strides: A list of `ints`. 1-D of length 4. The stride of the sliding window for each dimension of `input`. padding: Controls how to pad the image before applying the convolution. Can be the string `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, height, width, channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, channels, height, width]. dilations: An optional list of `ints`. Defaults to `[1, 1, 1, 1]`. 1-D tensor of length 4. The dilation factor for each dimension of `input`. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input`. rrErr9)rrdepthwise_conv2d_native)r@rBrFrDrErr9rs r;ryry s@~ /w7'   + +    )  r=z"nn.depthwise_conv2d_backprop_inputz)nn.depthwise_conv2d_native_backprop_inputc Zt|\}}tj||||||||| S)a Computes the gradients of depthwise convolution with respect to the input. Args: input_sizes: A `Tensor` of type `int32`. An integer vector representing the shape of `input`, based on `data_format`. For example, if `data_format` is 'NHWC' then `input` is a 4-D `[batch, height, width, channels]` tensor. filter: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. 4-D with shape `[filter_height, filter_width, in_channels, depthwise_multiplier]`. out_backprop: A `Tensor`. Must have the same type as `filter`. 4-D with shape based on `data_format`. For example, if `data_format` is 'NHWC' then out_backprop shape is `[batch, out_height, out_width, out_channels]`. Gradients w.r.t. the output of the convolution. strides: A list of `ints`. The stride of the sliding window for each dimension of the input of the convolution. padding: Controls how to pad the image before applying the convolution. Can be the string `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, height, width, channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, channels, height, width]. dilations: An optional list of `ints`. Defaults to `[1, 1, 1, 1]`. 1-D tensor of length 4. The dilation factor for each dimension of `input`. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `filter`. rx)rr&depthwise_conv2d_native_backprop_input) rLrBrMrFrDrErr9rs r;r{r{ sCv /w7'   : :   )   r=z#nn.depthwise_conv2d_backprop_filterz*nn.depthwise_conv2d_native_backprop_filterc Zt|\}}tj||||||||| S)a Computes the gradients of depthwise convolution with respect to the filter. Args: input: A `Tensor`. Must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. 4-D with shape based on `data_format`. For example, if `data_format` is 'NHWC' then `input` is a 4-D `[batch, in_height, in_width, in_channels]` tensor. filter_sizes: A `Tensor` of type `int32`. An integer vector representing the tensor shape of `filter`, where `filter` is a 4-D `[filter_height, filter_width, in_channels, depthwise_multiplier]` tensor. out_backprop: A `Tensor`. Must have the same type as `input`. 4-D with shape based on `data_format`. For example, if `data_format` is 'NHWC' then out_backprop shape is `[batch, out_height, out_width, out_channels]`. Gradients w.r.t. the output of the convolution. strides: A list of `ints`. The stride of the sliding window for each dimension of the input of the convolution. padding: Controls how to pad the image before applying the convolution. Can be the string `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. data_format: An optional `string` from: `"NHWC", "NCHW"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. With the default format "NHWC", the data is stored in the order of: [batch, height, width, channels]. Alternatively, the format could be "NCHW", the data storage order of: [batch, channels, height, width]. dilations: An optional list of `ints`. Defaults to `[1, 1, 1, 1]`. 1-D tensor of length 4. The dilation factor for each dimension of `input`. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions must be 1. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input`. rx)rr'depthwise_conv2d_native_backprop_filter) r@rVrMrFrDrErr9rs r;r}r}6 sCx /w7'   ; ;   )   r=c |j}t|dd}|dk(r t|}|dvrtj|||||||St |t jtj|||||d|S)z7Helper function for `conv3d`; handles expanded batches.rZrv)rSr@rRr,r+rN)rErr9r^r@rC)rIrSr4rconv3drurDrE) r@rBrFrDrErr9rIrZs r;r`r` s ++% %" %% b[ JE &&           #  !   r=z nn.conv3dr$c 0|gd}t|||||||S)Nr+r+r+r+r+)r`rs r; conv3d_v2r s+I w+ )4 11r=rc Ttd|d|}tj|||||||S)NrfrB)rrr)r@rBrFrDrErr9rfs r; conv3d_v1r s8 &i(F K&    VWg{It EEr=znn.conv3d_transposec `td|d|}td|d|}t||||||| |S)aAThe transpose of `conv3d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `conv3d` rather than an actual deconvolution. Args: value: A 5-D `Tensor` of type `float` and shape `[batch, depth, height, width, in_channels]`. filter: A 5-D `Tensor` with the same type as `value` and shape `[depth, height, width, output_channels, in_channels]`. `filter`'s `in_channels` dimension must match that of `value`. output_shape: A 1-D `Tensor` representing the output shape of the deconvolution op. strides: A list of ints. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See the "returns" section of `tf.nn.convolution` for details. data_format: A string, either `'NDHWC'` or `'NCDHW`' specifying the layout of the input and output tensors. Defaults to `'NDHWC'`. name: Optional name for the returned tensor. input: Alias of value. filters: Alias of filter. dilations: An int or list of `ints` that has length `1`, `3` or `5`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `D`, `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 5-d tensor must be 1. Returns: A `Tensor` with the same type as `value`. Raises: ValueError: If input/output depth does not match `filter`'s shape, or if padding is other than `'VALID'` or `'SAME'`. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) rfrBr@r6r)rconv3d_transpose_v2r[s r;conv3d_transposer sJv &i(F K& $WeWe D%      r=c tj|d|t|g5}|d}|jdrdnd}t |d|d}t |d|d }t j |||||||| cdddS#1swYyxYw) aThe transpose of `conv3d`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `conv3d` rather than an actual deconvolution. Args: input: A 5-D `Tensor` of type `float` and shape `[batch, depth, height, width, in_channels]` for `NDHWC` data format or `[batch, in_channels, depth, height, width]` for `NCDHW` data format. filters: A 5-D `Tensor` with the same type as `input` and shape `[depth, height, width, output_channels, in_channels]`. `filter`'s `in_channels` dimension must match that of `input`. output_shape: A 1-D `Tensor` representing the output shape of the deconvolution op. strides: An int or list of `ints` that has length `1`, `3` or `5`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the `D`, `H` and `W` dimension. By default the `N` and `C` dimensions are set to 0. The dimension order is determined by the value of `data_format`, see below for details. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string. 'NDHWC' and 'NCDHW' are supported. dilations: An int or list of `ints` that has length `1`, `3` or `5`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the `D`, `H` and `W` dimension. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. Dilations in the batch and depth dimensions if a 5-d tensor must be 1. name: Optional name for the returned tensor. Returns: A `Tensor` with the same type as `input`. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) rNr$rr+r@rRrFrrK)r rGrBrr<rconv3d_backprop_input_v2) r@rfrrFrDrErr9r8s r;rr sl ~~d.fl358<k$//5A1MGQ yAGiM;GI  . .   s AA;;Bznn.conv_transposec tj|d|t|g5}tj|r|j ddz }n7t |tjrt|dz }ntd|d|cxkrdksntd|d|dzd t|dz } | |||||||| cd d d S#1swYy xYw) a The transpose of `convolution`. This operation is sometimes called "deconvolution" after (Zeiler et al., 2010), but is really the transpose (gradient) of `conv3d` rather than an actual deconvolution. Args: input: An N+2 dimensional `Tensor` of shape `[batch_size] + input_spatial_shape + [in_channels]` if data_format does not start with "NC" (default), or `[batch_size, in_channels] + input_spatial_shape` if data_format starts with "NC". It must be one of the following types: `half`, `bfloat16`, `float32`, `float64`. filters: An N+2 dimensional `Tensor` with the same type as `input` and shape `spatial_filter_shape + [in_channels, out_channels]`. output_shape: A 1-D `Tensor` representing the output shape of the deconvolution op. strides: An int or list of `ints` that has length `1`, `N` or `N+2`. The stride of the sliding window for each dimension of `input`. If a single value is given it is replicated in the spatial dimensions. By default the `N` and `C` dimensions are set to 0. The dimension order is determined by the value of `data_format`, see below for details. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string or None. Specifies whether the channel dimension of the `input` and output is the last dimension (default, or if `data_format` does not start with "NC"), or the second dimension (if `data_format` starts with "NC"). For N=1, the valid values are "NWC" (default) and "NCW". For N=2, the valid values are "NHWC" (default) and "NCHW". For N=3, the valid values are "NDHWC" (default) and "NCDHW". dilations: An int or list of `ints` that has length `1`, `N` or `N+2`, defaults to 1. The dilation factor for each dimension of`input`. If a single value is given it is replicated in the spatial dimensions. By default the `N` and `C` dimensions are set to 1. If set to k > 1, there will be k-1 skipped cells between each filter element on that dimension. The dimension order is determined by the value of `data_format`, see above for details. name: A name for the operation (optional). If not specified "conv_transpose" is used. Returns: A `Tensor` with the same type as `value`. References: Deconvolutional Networks: [Zeiler et al., 2010] (https://ieeexplore.ieee.org/abstract/document/5539957) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.232.4023&rep=rep1&type=pdf)) conv_transposerr,zL`output_shape` must be a tensor or sized collection. Received: output_shape=r+rRzC`output_shape` must be of length 3, 4 or 5. Received: output_shape=r.rrN) r rGrBr rrIr/rr3r4r5CONV_TRANSPOSE_OPS) r@rfrrFrDrErr9r7rMs r;rrd sz ~~d,fl358<l+   Q ! #a L/"7"7 8 l a a 11=@ AA ;Q;  $$0>QUG1 F GG AaC B    !s BCCz nn.bias_addctj|d||g5}|G|jdrd}n3|jdr|jdrd}nt d|t j s9tj|d }tj||jd }tj|||| cdddS#1swYyxYw)aAdds `bias` to `value`. This is (mostly) a special case of `tf.add` where `bias` is restricted to 1-D. Broadcasting is supported, so `value` may have any number of dimensions. Unlike `tf.add`, the type of `bias` is allowed to differ from `value` in the case where both types are quantized. Args: value: A `Tensor` with type `float`, `double`, `int64`, `int32`, `uint8`, `int16`, `int8`, `complex64`, or `complex128`. bias: A 1-D `Tensor` with size matching the channel dimension of `value`. Must be the same type as `value` unless `value` is a quantized type, in which case a different quantized type may be used. data_format: A string. 'N...C' and 'NC...' are supported. If `None` (the default) is specified then 'N..C' is assumed. name: A name for the operation (optional). Returns: A `Tensor` with the same type as `value`. Raises: ValueError if data format is unrecognized, if `value` has less than two dimensions when `data_format` is 'N..C'/`None` or `value` has less then three dimensions when `data_format` is `NC..`, if `bias` does not have exactly one dimension (is a vector), or if the size of `bias` does not match the size of the channel dimension of `value`. BiasAddNrrWNCr"zL`data_format` must be of the form `N...C` or `NC...`. Received: data_format=r@rAbiasrr9)rEr9) r rGrendswithr5rexecuting_eagerlyrHrrbias_add)r6rrEr9s r;rr s< ~~dIt}5P    %  ! !# &;+?+?+D ;;F-IJ J  $ $ &##E8e  " "4u{{ Hd   ud $ OPPPs B/CCctj|d||g5}tj|d}tj||jd}t j |||cdddS#1swYyxYw)a=Adds `bias` to `value`. This is a deprecated version of bias_add and will soon to be removed. This is (mostly) a special case of `tf.add` where `bias` is restricted to 1-D. Broadcasting is supported, so `value` may have any number of dimensions. Unlike `tf.add`, the type of `bias` is allowed to differ from `value` in the case where both types are quantized. Args: value: A `Tensor` with type `float`, `double`, `int64`, `int32`, `uint8`, `int16`, `int8`, `complex64`, or `complex128`. bias: A 1-D `Tensor` with size matching the last dimension of `value`. Must be the same type as `value` unless `value` is a quantized type, in which case a different quantized type may be used. name: A name for the operation (optional). Returns: A `Tensor` with the same type as `value`. BiasAddV1r@rArrN)r rGrHrr bias_add_v1)r6rr9s r;rr sk* ~~dK%7:4  ! !%g 6E  U[[v FD  ! !%D 9:::s AA55A>znn.creluctj|d|g5}tj|d}tj|| g||}t j |cdddS#1swYyxYw)aComputes Concatenated ReLU. Concatenates a ReLU which selects only the positive part of the activation with a ReLU which selects only the *negative* part of the activation. Note that as a result this non-linearity doubles the depth of the activations. Source: [Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units. W. Shang, et al.](https://arxiv.org/abs/1603.05201) Args: features: A `Tensor` with type `float`, `double`, `int32`, `int64`, `uint8`, `int16`, or `int8`. name: A name for the operation (optional). axis: The axis that the output values are concatenated along. Default is -1. Returns: A `Tensor` with the same type as `features`. References: Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units: [Shang et al., 2016](http://proceedings.mlr.press/v48/shang16) ([pdf](http://proceedings.mlr.press/v48/shang16.pdf)) CRelufeaturesrAN)r rGrHrr}rrelu)rr9rxcs r;crelursf6 ~~dGhZ0D$$XJ?H(XI.4@A ??1 s AA**A3ct|||S)N)r9rx)r)rrxr9s r;crelu_v2r's xd ..r=znn.relu6ctj|d|g5}tj|d}tj||cdddS#1swYyxYw)aComputes Rectified Linear 6: `min(max(features, 0), 6)`. In comparison with `tf.nn.relu`, relu6 activation functions have shown to empirically perform better under low-precision conditions (e.g. fixed point inference) by encouraging the model to learn sparse features earlier. Source: [Convolutional Deep Belief Networks on CIFAR-10: Krizhevsky et al., 2010](http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf). For example: >>> x = tf.constant([-3.0, -1.0, 0.0, 6.0, 10.0], dtype=tf.float32) >>> y = tf.nn.relu6(x) >>> y.numpy() array([0., 0., 0., 6., 6.], dtype=float32) Args: features: A `Tensor` with type `float`, `double`, `int32`, `int64`, `uint8`, `int16`, or `int8`. name: A name for the operation (optional). Returns: A `Tensor` with the same type as `features`. References: Convolutional Deep Belief Networks on CIFAR-10: Krizhevsky et al., 2010 ([pdf](http://www.cs.utoronto.ca/~kriz/conv-cifar10-aug2010.pdf)) Relu6rrAN)r rGrHrrelu6)rr9s r;rr.sP@ ~~dGhZ01D$$XJ?H   H4 0111s .AAz nn.leaky_reluctj|d||g5}tj|d}|jjr$t j |tj}t|tjr|j}tj|||cdddS#1swYyxYw)aCCompute the Leaky ReLU activation function. Source: [Rectifier Nonlinearities Improve Neural Network Acoustic Models. AL Maas, AY Hannun, AY Ng - Proc. ICML, 2013] (https://ai.stanford.edu/~amaas/papers/relu_hybrid_icml2013_final.pdf). Args: features: A `Tensor` representing preactivation values. Must be one of the following types: `float16`, `float32`, `float64`, `int32`, `int64`. alpha: Slope of the activation function at x < 0. name: A name for the operation (optional). Returns: The activation value. References: Rectifier Nonlinearities Improve Neural Network Acoustic Models: [Maas et al., 2013] (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.693.1422) ([pdf] (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.693.1422&rep=rep1&type=pdf)) LeakyRelurrA)alphar9N)r rGrHr is_integerrcastrfloat32r/rrcitemr leaky_relu)rrr9s r;rrSs4 ~~dK(E):;Ct$$XJ?H~~  x8h%$jjle  T B CCCs BB77Cznn.geluFc 0tj|d|g5tj|d}|jjst d|j|rdt jd|j}d|zdt jd||t j|d zzzzzcd d d Sd|zt j| t jd |jzz cd d d S#1swYy xYw) aCCompute the Gaussian Error Linear Unit (GELU) activation function. Gaussian error linear unit (GELU) computes `x * P(X <= x)`, where `P(X) ~ N(0, 1)`. The (GELU) nonlinearity weights inputs by their value, rather than gates inputs by their sign as in ReLU. For example: >>> x = tf.constant([-3.0, -1.0, 0.0, 1.0, 3.0], dtype=tf.float32) >>> y = tf.nn.gelu(x) >>> y.numpy() array([-0.00404951, -0.15865529, 0. , 0.8413447 , 2.9959507 ], dtype=float32) >>> y = tf.nn.gelu(x, approximate=True) >>> y.numpy() array([-0.00363752, -0.15880796, 0. , 0.841192 , 2.9963627 ], dtype=float32) Args: features: A `float Tensor` representing preactivation values. approximate: An optional `bool`. Defaults to `False`. Whether to enable approximation. name: A name for the operation (optional). Returns: A `Tensor` with the same type as `features`. Raises: ValueError: if `features` is not a floating point `Tensor`. References: [Gaussian Error Linear Units (GELUs)](https://arxiv.org/abs/1606.08415). GelurrAzI`features.dtype` must be a floating point tensor.Received:features.dtype=gHm?g??gQ63E?rRNg;f?) r rGrHr is_floatingr5rrtanhpowerfc)r approximater9coeffs r;gelurvsL ~~dFXJ/$$XJ?H >> % %  %%-^^$4 6 77mmHhnn5e 8^  0'%(,,x2K*KKMN NO    MMi(--(:HNNKK  sB"D =D  Dctj|}tjtj|t j |dgdg}tj |tjdg|gd}tjsa|j}|O|jC|j}d}d}|ddD] }|d}n||z}|r||dg}|j||S)z>Flattens logits' outer dimensions and keep its last dimension.r+rvrNTF)rr\slicerIrsubtractr{r}rrrrr|r~) logitsr\ last_dim_sizeoutputrIproduct product_validdrs r;_flatten_outer_dimsrs  $//oof 1 1$ :;aSB-   VY%5%5t]6KQ%O P&  " " $    E UZZ3mmoegmSbz! 9-  Q,'  r+ & -r=c  d dtj|}|j |dk(xs| jdz k(}|r ||S|}t |t j rtj|}|d j |cxkr jksDntjddd j d jd jd |tj|}t |t j s |d kr4||z }n.tjtj|d ||z|}tj| | jz | tj d}  fd  ||}t |t"rt# fd |DS |S)aHelper function for ops that accept and return 2d inputs of same shape. It reshapes and transposes the inputs into a 2-D Tensor and then invokes the given function. The output would be transposed and reshaped back. If the given function returns a tuple of tensors, each of them will be transposed and reshaped. Args: inputs: A non-empty `Tensor`. Must be one of the following types: `half`, `float32`, `float64`. compute_op: The function to wrap. Must accept the input tensor as its first arugment, and a second keyword argument `name`. dim: The dimension softmax would be performed on. The default is -1 which indicates the last dimension. name: A name for the operation (optional). Returns: A `Tensor`. Has the same shape as inputs. If compute_op returns multiple tensors, each of them have the same shape as the input. Raises: InvalidArgumentError: if `inputs` is empty or `dim` is beyond the last dimension of `inputs`. Nc tj|tjtj||gtj|dz||ggd|S)z(Swaps logits's dim_index and last_index.r+rrAr transposer}rr) input_tensor dim_index last_indexr9s r; _swap_axisz%_wrap_2d_function.._swap_axiss[    NN9 % | NN9q=* 5 {     r=rvr+rAz`dim` must be in the range [z, z) where z9 is the number of dimensions in the input. Received: dim=rcj|tjd}|j|S)Nr+rA)rrr~)rrdim_axisr_r9rIs r; fix_outputz%_wrap_2d_function..fix_outputs9 (++J:GF U Mr=c3.K|] }|ywrrs)rrrs r;rz$_wrap_2d_function..s:F#:sr)r rHrrZr/rrr rrInvalidArgumentErrorrr\whererlessrr1) inputs compute_opdimr9 is_last_dimdim_valrZoutputsrrrr_rIs ` @@@@@r;_wrap_2d_functionrs2   (&    %7u{{Q 6+ f4 (( 'Z&&'((-G %++!F5;;!F  * * d & |nBu{{m8 ;;-  # $$ .. % C** + Qw Ulc //(--Q/uc BC~~f%* 5;; ( fh(9(9*a(H I& v ' :': :: g r=z nn.softmaxz math.softmaxcB|d}t|tj||S)aComputes softmax activations. Used for multi-class predictions. The sum of all outputs generated by softmax is 1. This function performs the equivalent of ```python softmax = tf.exp(logits) / tf.reduce_sum(tf.exp(logits), axis, keepdims=True) ``` Example usage: >>> softmax = tf.nn.softmax([-1, 0., 1.]) >>> softmax >>> sum(softmax) Args: logits: A non-empty `Tensor`. Must be one of the following types: `half`, `float32`, `float64`. axis: The dimension softmax would be performed on. The default is -1 which indicates the last dimension. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type and shape as `logits`. Raises: InvalidArgumentError: if `logits` is empty or `axis` is beyond the last dimension of `logits`. rv)rrsoftmaxrrxr9s r; softmax_v2r"s&H \ D 6:#5#5tT BBr=z#dim is deprecated, use axis insteadrcrtjd|d|}|d}t|tj||S)Nrxrrv)rrrrrrrxr9rs r;rrKs;  / /eS I$ \ D 6:#5#5tT BBr=znn.log_softmaxzmath.log_softmaxcrtjd|d|}|d}t|tj||S)aComputes log softmax activations. For each batch `i` and class `j` we have logsoftmax = logits - log(reduce_sum(exp(logits), axis)) Args: logits: A non-empty `Tensor`. Must be one of the following types: `half`, `float32`, `float64`. axis: The dimension softmax would be performed on. The default is -1 which indicates the last dimension. name: A name for the operation (optional). dim: Deprecated alias for `axis`. Returns: A `Tensor`. Has the same type as `logits`. Same shape as `logits`. Raises: InvalidArgumentError: if `logits` is empty or `axis` is beyond the last dimension of `logits`. rxrrv)rrrr log_softmaxrs r;rrXs;4  / /eS I$ \ D 6:#9#94 FFr=cB|d}t|tj||S)alComputes log softmax activations. For each batch `i` and class `j` we have logsoftmax = logits - log(reduce_sum(exp(logits), axis)) Args: logits: A non-empty `Tensor`. Must be one of the following types: `half`, `float32`, `float64`. axis: The dimension softmax would be performed on. The default is -1 which indicates the last dimension. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `logits`. Same shape as `logits`. Raises: InvalidArgumentError: if `logits` is empty or `axis` is beyond the last dimension of `logits`. rv)rrrrs r;log_softmax_v2rxs%. \ D 6:#9#94 FFr=c4||td|d|d|y)Nz0Both `labels` and `logits` must be provided for zReceived: labels=z and logits=)r5)r9labelsrs r;_ensure_xent_argsrs9 ^v~ Gv( VHF GG&r=z$nn.softmax_cross_entropy_with_logitsc t||||S)a Computes softmax cross entropy between `logits` and `labels`. Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both. **NOTE:** While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of `labels` is a valid probability distribution. If they are not, the computation of the gradient will be incorrect. If using exclusive `labels` (wherein one and only one class is true at a time), see `sparse_softmax_cross_entropy_with_logits`. Usage: >>> logits = [[4.0, 2.0, 1.0], [0.0, 5.0, 1.0]] >>> labels = [[1.0, 0.0, 0.0], [0.0, 0.8, 0.2]] >>> tf.nn.softmax_cross_entropy_with_logits(labels=labels, logits=logits) **WARNING:** This op expects unscaled logits, since it performs a `softmax` on `logits` internally for efficiency. Do not call this op with the output of `softmax`, as it will produce incorrect results. A common use case is to have logits and labels of shape `[batch_size, num_classes]`, but higher dimensions are supported, with the `axis` argument specifying the class dimension. `logits` and `labels` must have the same dtype (either `float16`, `float32`, or `float64`). Backpropagation will happen into both `logits` and `labels`. To disallow backpropagation into `labels`, pass label tensors through `tf.stop_gradient` before feeding it to this function. **Note that to avoid confusion, it is required to pass only named arguments to this function.** Args: labels: Each vector along the class dimension should hold a valid probability distribution e.g. for the case in which labels are of shape `[batch_size, num_classes]`, each row of `labels[i]` must be a valid probability distribution. logits: Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized log probabilities. axis: The class dimension. Defaulted to -1 which is the last dimension. name: A name for the operation (optional). Returns: A `Tensor` that contains the softmax cross entropy loss. Its type is the same as `logits` and its shape is the same as `labels` except that it does not have the last dimension of `labels`. rrrxr9)+softmax_cross_entropy_with_logits_v2_helperrs r;$softmax_cross_entropy_with_logits_v2rsv 5 FD ::r=z'nn.softmax_cross_entropy_with_logits_v2c td|d|}~|d}tj|d||g5}tj|d}tj|d}|jt j k(xs|jt jk(}|r$tj|t jn|}tj||j}tj|}|j}|dk7rd } | |||}| |||}tj|} t|}t|}t!j"r't%|} tj&|| zd  } nt)j*|||\} } tj,| d gtj.|d g}tj0| |} t3j4s2|0|j6$|j9}||=| j;||r)tj| |jcdddS| cdddS#1swYyxYw) aComputes softmax cross entropy between `logits` and `labels`. Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both. **NOTE:** While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of `labels` is a valid probability distribution. If they are not, the computation of the gradient will be incorrect. If using exclusive `labels` (wherein one and only one class is true at a time), see `sparse_softmax_cross_entropy_with_logits`. **WARNING:** This op expects unscaled logits, since it performs a `softmax` on `logits` internally for efficiency. Do not call this op with the output of `softmax`, as it will produce incorrect results. A common use case is to have logits and labels of shape `[batch_size, num_classes]`, but higher dimensions are supported, with the `axis` argument specifying the class dimension. `logits` and `labels` must have the same dtype (either `float16`, `float32`, or `float64`). Backpropagation will happen into both `logits` and `labels`. To disallow backpropagation into `labels`, pass label tensors through `tf.stop_gradient` before feeding it to this function. **Note that to avoid confusion, it is required to pass only named arguments to this function.** Args: labels: Each vector along the class dimension should hold a valid probability distribution e.g. for the case in which labels are of shape `[batch_size, num_classes]`, each row of `labels[i]` must be a valid probability distribution. logits: Unscaled log probabilities. axis: The class dimension. Defaulted to -1 which is the last dimension. name: A name for the operation (optional). dim: Deprecated alias for axis. Returns: A `Tensor` that contains the softmax cross entropy loss. Its type is the same as `logits` and its shape is the same as `labels` except that it does not have the last dimension of `labels`. rxrNrv!softmax_cross_entropy_with_logitsrrArc tj|tjtj|tj|dz||ggdS)Nr+rr)r rr\s r;_move_dim_to_endzEsoftmax_cross_entropy_with_logits_v2_helper.._move_dim_to_end'sR""    y)y1}d3i[  r=r+rwr)rr rGrHrrfloat16bfloat16rrrrr\rrIrris_op_determinism_enabledr reduce_sumrrrrr{rrrr|r~)rrrxr9rconvert_to_float32precise_logitsr_rIrrL log_probscostunused_backproprs r;rrs6p $FD% =$  \ D ~~d?v&(<+/  " "6 9F  " "6 9F &I&,,&//*I$6]] ;A]]6>#7#7 8F/J    E rz(jInj9f//.1K)8N  (F'') 0i!!&9"41= =d)JJ &t-dO??;$,$5$5j!$D#EGL   T< 0D  $ $    6mmoe + nnU ]]4 .u<<xy<<<sHI!I!!I*z Future major versions of TensorFlow will allow gradients to flow into the labels input on backprop by default. See `tf.nn.softmax_cross_entropy_with_logits_v2`. )date instructionsctd|d|}td||tj|d||g5}t j |d}dddt ||||S#1swYxYw) aComputes softmax cross entropy between `logits` and `labels`. Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both. **NOTE:** While the classes are mutually exclusive, their probabilities need not be. All that is required is that each row of `labels` is a valid probability distribution. If they are not, the computation of the gradient will be incorrect. If using exclusive `labels` (wherein one and only one class is true at a time), see `sparse_softmax_cross_entropy_with_logits`. **WARNING:** This op expects unscaled logits, since it performs a `softmax` on `logits` internally for efficiency. Do not call this op with the output of `softmax`, as it will produce incorrect results. A common use case is to have logits and labels of shape `[batch_size, num_classes]`, but higher dimensions are supported, with the `dim` argument specifying the class dimension. Backpropagation will happen only into `logits`. To calculate a cross entropy loss that allows backpropagation into both `logits` and `labels`, see `tf.nn.softmax_cross_entropy_with_logits_v2`. **Note that to avoid confusion, it is required to pass only named arguments to this function.** Args: labels: Each vector along the class dimension should hold a valid probability distribution e.g. for the case in which labels are of shape `[batch_size, num_classes]`, each row of `labels[i]` must be a valid probability distribution. logits: Per-label activations, typically a linear output. These activation energies are interpreted as unnormalized log probabilities. dim: The class dimension. Defaulted to -1 which is the last dimension. name: A name for the operation (optional). axis: Alias for dim. Returns: A `Tensor` that contains the softmax cross entropy loss. Its type is the same as `logits` and its shape is the same as `labels` except that it does not have the last dimension of `labels`. rxrr$softmax_cross_entropy_with_logits_sglabels_stop_gradientrANr)rrr rGr stop_gradientr)rrrr9rxs r;rr]sn #64<#7H ~~dBv&(J+/  $ $V2H IFJ . F4 99 JJs A$$A-c tjrt|}tjt j ||d}tjtd|j}t j|t j|}tjt j|d|j}t jtjtj |dtj"||||}|St%j&|||\}}|S)Nr+) batch_dimsNanrrvrrA)rrrrnegativergatherrconstantfloatrr2rIrr logical_orr greater_equalr(sparse_softmax_cross_entropy_with_logits) rrr9rr nan_tensor cost_all_nans class_countrs r;0_sparse_softmax_cross_entropy_with_rank_2_logitsrs %%' v&I   Y--iAN OD%%eEl&,,GJ**:yt7LMM--  7 ;V\\JK ?? MM&! $  " "6; 7 9:G OD +AAT#GD! +r=z+nn.sparse_softmax_cross_entropy_with_logitsc .td||tj|d||g5tj|}tj|}t j |j tjk(r$tj|tjn|}|j}tj|}|jxr!|jddj}|jj,|jjdk(rt!d|d|jj}|jq|j|jjdz k7rGt!d |d |j"d |jd |jj"|r5||jddk7rt!d |d |j|jjd k(rct%|||}|j tjk(r-tj|tjcdddS|cdddSg}|sN|j't)j*tj|tj|ddtj,|5tj|tj"|dz } tj.|d| g}tj.|dg}t%|||}tj.||}|j1||j tjk(r6tj|tjcdddcdddS|cdddcdddS#1swYnxYw dddy#1swYyxYw)a? Computes sparse softmax cross entropy between `logits` and `labels`. Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both. **NOTE:** For this operation, the probability of a given label is considered exclusive. That is, soft classes are not allowed, and the `labels` vector must provide a single specific index for the true class for each row of `logits` (each minibatch entry). For soft softmax classification with a probability distribution for each entry, see `softmax_cross_entropy_with_logits_v2`. **WARNING:** This op expects unscaled logits, since it performs a `softmax` on `logits` internally for efficiency. Do not call this op with the output of `softmax`, as it will produce incorrect results. A common use case is to have logits of shape `[batch_size, num_classes]` and have labels of shape `[batch_size]`, but higher dimensions are supported, in which case the `dim`-th dimension is assumed to be of size `num_classes`. `logits` must have the dtype of `float16`, `float32`, or `float64`, and `labels` must have the dtype of `int32` or `int64`. **Note that to avoid confusion, it is required to pass only named arguments to this function.** Args: labels: `Tensor` of shape `[d_0, d_1, ..., d_{r-1}]` (where `r` is rank of `labels` and result) and dtype `int32` or `int64`. Each entry in `labels` must be an index in `[0, num_classes)`. Other values will raise an exception when this op is run on CPU, and return `NaN` for corresponding loss and gradient rows on GPU. logits: Per-label activations (typically a linear output) of shape `[d_0, d_1, ..., d_{r-1}, num_classes]` and dtype `float16`, `float32`, or `float64`. These activation energies are interpreted as unnormalized log probabilities. name: A name for the operation (optional). Returns: A `Tensor` of the same shape as `labels` and of the same type as `logits` with the softmax cross entropy loss. Raises: ValueError: If logits are scalars (need to have rank >= 1) or if the rank of the labels is not equal to the rank of the logits minus one. r#SparseSoftmaxCrossEntropyWithLogitsNrvrz-`logits` cannot be a scalar. Received logits=`r+zO`labels.shape.rank` must equal `logits.shape.rank - 1`. Received: labels.shape=rQz and logits.shape=z_`labels.shape` must equal `logits.shape` except for the last dimension. Received: labels.shape=r,rA)rr rGrHras_dtyperrrrrrrrIryrZr5r\rrr assert_equalcontrol_dependenciesr{r~) rrr9rlabels_static_shape labels_shapestatic_shapes_fully_definedr shape_checks num_classess r;rrsl>O ~~dAv&(;  " "6 *F  " "6 *F?E @@(X]]66>>:.4!**,??6*L,,. 33B002 +0@0@0B0H0HA0M  9& C EE +!!-!!V%5%5%7%=%=%AA  $$7#8 % % &&89I9I9K8LM%%',,- / 00 $v//1#266  88K7LM$..01 3 44 1$ = &t-d  '}}T6>>2G;;JK;;RL &  oof%oof%cr* ,- ! !, / OOF+INN6,BQ,FGk (("k9JKn  ".f = &t-d   t\ 2d nn()  '}}T6>>2  _;;v  _;;^   _;;;s>IP  P A'P ;CO5 P !O5" P 5O> :P  Pct|||S)a Computes sparse softmax cross entropy between `logits` and `labels`. Measures the probability error in discrete classification tasks in which the classes are mutually exclusive (each entry is in exactly one class). For example, each CIFAR-10 image is labeled with one and only one label: an image can be a dog or a truck, but not both. Note: For this operation, the probability of a given label is considered exclusive. That is, soft classes are not allowed, and the `labels` vector must provide a single specific index for the true class for each row of `logits` (each minibatch entry). For soft softmax classification with a probability distribution for each entry, see `softmax_cross_entropy_with_logits_v2`. Warning: This op expects unscaled logits, since it performs a `softmax` on `logits` internally for efficiency. Do not call this op with the output of `softmax`, as it will produce incorrect results. A common use case is to have logits of shape `[batch_size, num_classes]` and have labels of shape `[batch_size]`, but higher dimensions are supported, in which case the `dim`-th dimension is assumed to be of size `num_classes`. `logits` must have the dtype of `float16`, `float32`, or `float64`, and `labels` must have the dtype of `int32` or `int64`. >>> logits = tf.constant([[2., -5., .5, -.1], ... [0., 0., 1.9, 1.4], ... [-100., 100., -100., -100.]]) >>> labels = tf.constant([0, 3, 1]) >>> tf.nn.sparse_softmax_cross_entropy_with_logits( ... labels=labels, logits=logits).numpy() array([0.29750752, 1.1448325 , 0. ], dtype=float32) To avoid confusion, passing only named arguments to this function is recommended. Args: labels: `Tensor` of shape `[d_0, d_1, ..., d_{r-1}]` (where `r` is rank of `labels` and result) and dtype `int32` or `int64`. Each entry in `labels` must be an index in `[0, num_classes)`. Other values will raise an exception when this op is run on CPU, and return `NaN` for corresponding loss and gradient rows on GPU. logits: Unscaled log probabilities of shape `[d_0, d_1, ..., d_{r-1}, num_classes]` and dtype `float16`, `float32`, or `float64`. name: A name for the operation (optional). Returns: A `Tensor` of the same shape as `labels` and of the same type as `logits` with the softmax cross entropy loss. Raises: ValueError: If logits are scalars (need to have rank >= 1) or if the rank of the labels is not equal to the rank of the logits minus one. rrr9)rrs r;+sparse_softmax_cross_entropy_with_logits_v2r7sr 2 F //r=z nn.avg_poolznn.avg_pool_v2c|jt|jdz }n,|t|dz }ntd|jd|d|cxkrdks!ntd|jd|dzd||dz}n|jd rdn|dz}t |||d }t |||d }t t jt jd }||} | |||||| S)aPerforms the avg pooling on the input. Each entry in `output` is the mean of the corresponding size `ksize` window in `value`. Args: input: Tensor of rank N+2, of shape `[batch_size] + input_spatial_shape + [num_channels]` if `data_format` does not start with "NC" (default), or `[batch_size, num_channels] + input_spatial_shape` if data_format starts with "NC". Pooling happens over the spatial dimensions only. ksize: An int or list of `ints` that has length `1`, `N` or `N+2`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `N` or `N+2`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string. Specifies the channel dimension. For N=1 it can be either "NWC" (default) or "NCW", for N=2 it can be either "NHWC" (default) or "NCHW" and for N=3 either "NDHWC" (default) or "NCDHW". name: Optional name for the operation. Returns: A `Tensor` of format specified by `data_format`. The average pooled output tensor. r,zW`input` must have a static shape or `data_format` must be given. Received: input.shape= and data_format=r+rR<`input.shape.rank` must be 3, 4 or 5. Received: input.shape=rQrrksizerFr+r,rRrrFrDrEr9) rIr4r5rr< avg_pool1drr)r*) r@rrFrDrEr9r7r8avg_pooling_opsrMs r; avg_pool_v2rts+: [[ EKK1A K1A  !!& ."m % && a1  {{m9QUG1 6 77EM$//5A1q5M q- 9% '1mY ?'       / q"     r=z nn.avg_pool2dc .tj|d|g5}tjd|d|}|d}|j drdnd}t |d |d }t |d |d }t j|||||| cdddS#1swYyxYw) aPerforms the average pooling on the input. Each entry in `output` is the mean of the corresponding size `ksize` window in `value`. Args: value: A 4-D `Tensor` of shape `[batch, height, width, channels]` and type `float32`, `float64`, `qint8`, `quint8`, or `qint32`. ksize: An int or list of `ints` that has length `1`, `2` or `4`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See the "returns" section of `tf.nn.convolution` for details. data_format: A string. 'NHWC' and 'NCHW' are supported. name: Optional name for the operation. input: Alias for value. Returns: A `Tensor` with the same type as `value`. The average pooled output tensor. AvgPoolr@r6Nr"rr+rRr,rrFr)r rGrrrr<rr))r6rrFrDrEr9r@r8s r;r)r)s2 ~~dIw/4  2 2 (Ek$//5A1M %M7 ;EGQ yAG       s A(B  Bc tj|d|g5}|d}|jdrdnd}t|d|d}t|d|d }t j |||||| cdddS#1swYyxYw) aPerforms the average pooling on the input. Each entry in `output` is the mean of the corresponding size `ksize` window in `value`. Args: input: A 4-D `Tensor` of shape `[batch, height, width, channels]` and type `float32`, `float64`, `qint8`, `quint8`, or `qint32`. ksize: An int or list of `ints` that has length `1`, `2` or `4`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string. 'NHWC' and 'NCHW' are supported. name: Optional name for the operation. Returns: A `Tensor` with the same type as `value`. The average pooled output tensor. AvgPool2DNr"rr+rRr,rrFr)r rGrr<rr)r@rrFrDrEr9r8s r; avg_pool2dr$s0 ~~dK%1Tk$//5A1M %M7 ;EGQ yAG        AA33A<z nn.avg_pool1dc tj|d|g5}|d}|jdrdnd}dgt|d|dz}dgt|d|dz}|dk(rdnd}|dk(rd nd }t j ||}t j|||||| }t j||cdddS#1swYyxYw) aPerforms the average pooling on the input. Each entry in `output` is the mean of the corresponding size `ksize` window in `value`. Note internally this op reshapes and uses the underlying 2d operation. Args: input: A 3-D `Tensor` of the format specified by `data_format`. ksize: An int or list of `ints` that has length `1` or `3`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1` or `3`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: An optional string from: "NWC", "NCW". Defaults to "NWC". name: A name for the operation (optional). Returns: A `Tensor` of format specified by `data_format`. The max pooled output tensor. AvgPool1DNr!rr+r,rrFr"rWr) r rGrr<rexpand_dims_v2rr)r) r@rrFrDrEr9r8 expanding_dimrs r;rr s4 ~~dK%14Tk$//5A1M C-q-A AEcM'1mYGGG$-A1M'50&fK  $ $UM :E     F   V] 3%444s BB99Cz nn.avg_pool3dc tj|d|g5}|d}|jdrdnd}t|d|d}t|d|d}t j |||||| cdddS#1swYyxYw) aPerforms the average pooling on the input. Each entry in `output` is the mean of the corresponding size `ksize` window in `value`. Args: input: A 5-D `Tensor` of shape `[batch, depth, height, width, channels]` and type `float32`, `float64`, `qint8`, `quint8`, or `qint32`. ksize: An int or list of `ints` that has length `1`, `3` or `5`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `3` or `5`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A string. 'NDHWC' and 'NCDHW' are supported. name: Optional name for the operation. Returns: A `Tensor` with the same type as `value`. The average pooled output tensor. AvgPool3DNr$rr+rRrrFr)r rGrr<rr*r#s r;r*r*<s0 ~~dK%1Tk$//5A1M %M7 ;EGQ yAG      r%z nn.max_poolznn.max_pool_v2cv|jt|jdz }n,|t|dz }ntd|jd|d|cxkrdks!ntd|jd|dzd||dz}n|jd rdn|dz}t |t t fr|d k(rtd |t|||d }t|||d }t |t t fr|dk(rtd|tttjd}||} | ||||||S)aPerforms max pooling on the input. For a given window of `ksize`, takes the maximum value within that window. Used for reducing computation and preventing overfitting. Consider an example of pooling with 2x2, non-overlapping windows: >>> matrix = tf.constant([ ... [0, 0, 1, 7], ... [0, 2, 0, 0], ... [5, 2, 0, 0], ... [0, 0, 9, 8], ... ]) >>> reshaped = tf.reshape(matrix, (1, 4, 4, 1)) >>> tf.nn.max_pool(reshaped, ksize=2, strides=2, padding="SAME") We can adjust the window size using the `ksize` parameter. For example, if we were to expand the window to 3: >>> tf.nn.max_pool(reshaped, ksize=3, strides=2, padding="SAME") We've now picked up two additional large numbers (5 and 9) in two of the pooled spots. Note that our windows are now overlapping, since we're still moving by 2 units on each iteration. This is causing us to see the same 9 repeated twice, since it is part of two overlapping windows. We can adjust how far we move our window with each iteration using the `strides` parameter. Updating this to the same value as our window size eliminates the overlap: >>> tf.nn.max_pool(reshaped, ksize=3, strides=3, padding="SAME") Because the window does not neatly fit into our input, padding is added around the edges, giving us the same result as when we used a 2x2 window. We can skip padding altogether and simply drop the windows that do not fully fit into our input by instead passing `"VALID"` to the `padding` argument: >>> tf.nn.max_pool(reshaped, ksize=3, strides=3, padding="VALID") Now we've grabbed the largest value in the 3x3 window starting from the upper- left corner. Since no other windows fit in our input, they are dropped. Args: input: Tensor of rank N+2, of shape `[batch_size] + input_spatial_shape + [num_channels]` if `data_format` does not start with "NC" (default), or `[batch_size, num_channels] + input_spatial_shape` if data_format starts with "NC". Pooling happens over the spatial dimensions only. ksize: An int or list of `ints` that has length `1`, `N` or `N+2`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `N` or `N+2`. The stride of the sliding window for each dimension of the input tensor. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. When using explicit padding, the size of the paddings cannot be greater than the sliding window size. data_format: A string. Specifies the channel dimension. For N=1 it can be either "NWC" (default) or "NCW", for N=2 it can be either "NHWC" (default) or "NCHW" and for N=3 either "NDHWC" (default) or "NCDHW". name: Optional name for the operation. Returns: A `Tensor` of format specified by `data_format`. The max pooled output tensor. Raises: ValueError: If - explicit padding is used with an input tensor of rank 5. - explicit padding is used with data_format='NCHW_VECT_C'. r,zW`input` must have a static shape or a data format must be given. Received: input.shape=rr+rRrrQrr NCHW_VECT_CV`data_format='NCHW_VECT_C'` is not supported with explicit padding. Received: padding=rrFzTExplicit padding is not supported with an input tensor of rank 5. Received: padding=rr) rIr4r5rr/r0r1r< max_pool1d max_pool2drr() r@rrFrDrEr9r7r8max_pooling_opsrMs r; max_pool_v2r2fsD [[ EKK1A K1A  !!& ."m % && a1  {{m9QUG1 6 77EM$//5A1q5M$'K=,H <.&s6M!qAv6Mrz(`ksize` cannot be zero. Received: ksize=rrFrDrrEr9)rrr rGrr<r/r0r1r5rrisscalarrcrrr') r6rrFrDrEr9r@r8rs r;r'r's5D  0 0%% P% ~~dIw/4k$//5A1M %M7 ;EGQ yAG'D%=)k].J >>EYH II!0!9G  U  E5"**- /366Mu6M3M A%I JJ    +  !s C'D""D+z nn.max_pool1dc tj|d|g5}t|ttfr|dk(rt d||d}|j drdnd}dgt|d|d z}dgt|d|d z}t|d \}}|d k(rd d g|z}|dk(rdnd}|dk(rdnd}tj||}tj|||||||} tj| |cdddS#1swYyxYw)amPerforms the max pooling on the input. Note internally this op reshapes and uses the underlying 2d operation. Args: input: A 3-D `Tensor` of the format specified by `data_format`. ksize: An int or list of `ints` that has length `1` or `3`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1` or `3`. The stride of the sliding window for each dimension of the input tensor. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NWC"`, this should be in the form `[[0, 0], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCW"`, this should be in the form `[[0, 0], [0, 0], [pad_left, pad_right]]`. When using explicit padding, the size of the paddings cannot be greater than the sliding window size. data_format: An optional string from: "NWC", "NCW". Defaults to "NWC". name: A name for the operation (optional). Returns: A `Tensor` of format specified by `data_format`. The max pooled output tensor. MaxPool1dr-r.Nr!rr+r,rrFrRrrr"rWr7)r rGr/r0r1r5rr<rrr(rr'r) r@rrFrDrEr9r8rr)rs r;r/r/4s=< ~~dK%14T'D%=)k].J >>EYH IIk$//5A1M C-q-A AEcM'1mYGGG!0!!<G *a4$-A1M'50&fK  $ $UM :E   + F   V] 33444s CC>>Dz nn.max_pool2dc ntj|d|g5}|d}|jdrdnd}t|d|d}t|d|d }t |t t fr|d k(rtd |t|\}}tj||||||| cdddS#1swYyxYw) a=Performs max pooling on 2D spatial data such as images. This is a more specific version of `tf.nn.max_pool` where the input tensor is 4D, representing 2D spatial data such as images. Using these APIs are equivalent Downsamples the input images along theirs spatial dimensions (height and width) by taking its maximum over an input window defined by `ksize`. The window is shifted by `strides` along each dimension. For example, for `strides=(2, 2)` and `padding=VALID` windows that extend outside of the input are not included in the output: >>> x = tf.constant([[1., 2., 3., 4.], ... [5., 6., 7., 8.], ... [9., 10., 11., 12.]]) >>> # Add the `batch` and `channels` dimensions. >>> x = x[tf.newaxis, :, :, tf.newaxis] >>> result = tf.nn.max_pool2d(x, ksize=(2, 2), strides=(2, 2), ... padding="VALID") >>> result[0, :, :, 0] With `padding=SAME`, we get: >>> x = tf.constant([[1., 2., 3., 4.], ... [5., 6., 7., 8.], ... [9., 10., 11., 12.]]) >>> x = x[tf.newaxis, :, :, tf.newaxis] >>> result = tf.nn.max_pool2d(x, ksize=(2, 2), strides=(2, 2), ... padding='SAME') >>> result[0, :, :, 0] We can also specify padding explicitly. The following example adds width-1 padding on all sides (top, bottom, left, right): >>> x = tf.constant([[1., 2., 3., 4.], ... [5., 6., 7., 8.], ... [9., 10., 11., 12.]]) >>> x = x[tf.newaxis, :, :, tf.newaxis] >>> result = tf.nn.max_pool2d(x, ksize=(2, 2), strides=(2, 2), ... padding=[[0, 0], [1, 1], [1, 1], [0, 0]]) >>> result[0, :, :, 0] For more examples and detail, see `tf.nn.max_pool`. Args: input: A 4-D `Tensor` of the format specified by `data_format`. ksize: An int or list of `ints` that has length `1`, `2` or `4`. The size of the window for each dimension of the input tensor. If only one integer is specified, then we apply the same window for all 4 dims. If two are provided then we use those for H, W dimensions and keep N, C dimension window size = 1. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of the input tensor. If only one integer is specified, we apply the same stride to all 4 dims. If two are provided we use those for the H, W dimensions and keep N, C of stride = 1. padding: Either the `string` `"SAME"` or `"VALID"` indicating the type of padding algorithm to use, or a list indicating the explicit paddings at the start and end of each dimension. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. When explicit padding is used and data_format is `"NHWC"`, this should be in the form `[[0, 0], [pad_top, pad_bottom], [pad_left, pad_right], [0, 0]]`. When explicit padding used and data_format is `"NCHW"`, this should be in the form `[[0, 0], [0, 0], [pad_top, pad_bottom], [pad_left, pad_right]]`. When using explicit padding, the size of the paddings cannot be greater than the sliding window size. data_format: A string. 'NHWC', 'NCHW' and 'NCHW_VECT_C' are supported. name: Optional name for the operation. Returns: A `Tensor` of format specified by `data_format`. The max pooled output tensor. Raises: ValueError: If explicit padding is used with data_format='NCHW_VECT_C'. MaxPool2dNr"rr+rRr,rrFr-r.r7) r rGrr<r/r0r1r5rrr')r@rrFrDrEr9r8rs r;r0r0psr ~~dK%1Tk$//5A1M %M7 ;EGQ yAG'D%=)k].J >>EYH II!0!9G     +  s BB++B4z nn.max_pool3dc tj|d|g5}|d}|jdrdnd}t|d|d}t|d|d }t j |||||| cdddS#1swYyxYw) a}Performs the max pooling on the input. Args: input: A 5-D `Tensor` of the format specified by `data_format`. ksize: An int or list of `ints` that has length `1`, `3` or `5`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `3` or `5`. The stride of the sliding window for each dimension of the input tensor. padding: A string, either `'VALID'` or `'SAME'`. The padding algorithm. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: An optional string from: "NDHWC", "NCDHW". Defaults to "NDHWC". The data format of the input and output data. With the default format "NDHWC", the data is stored in the order of: [batch, in_depth, in_height, in_width, in_channels]. Alternatively, the format could be "NCDHW", the data storage order is: [batch, in_channels, in_depth, in_height, in_width]. name: A name for the operation (optional). Returns: A `Tensor` of format specified by `data_format`. The max pooled output tensor. MaxPool3DNr$rr+r@rRrrFr)r rGrr<rr(r#s r;r(r(s4 ~~dK%1Tk$//5A1M %M7 ;EGQ yAG      r%znn.max_pool_with_argmaxc |dk7rtd|t|ddd}t|ddd}tj|||||||S)a8Performs max pooling on the input and outputs both max values and indices. The indices in `argmax` are flattened, so that a maximum value at position `[b, y, x, c]` becomes flattened index: `(y * width + x) * channels + c` if `include_batch_in_index` is False; `((b * height + y) * width + x) * channels + c` if `include_batch_in_index` is True. The indices returned are always in `[0, height) x [0, width)` before flattening, even if padding is involved and the mathematically correct answer is outside (either negative or too large). This is a bug, but fixing it is difficult to do in a safe backwards compatible way, especially due to flattening. Args: input: A `Tensor`. Must be one of the following types: `float32`, `float64`, `int32`, `uint8`, `int16`, `int8`, `int64`, `bfloat16`, `uint16`, `half`, `uint32`, `uint64`. 4-D with shape `[batch, height, width, channels]`. Input to pool over. ksize: An int or list of `ints` that has length `1`, `2` or `4`. The size of the window for each dimension of the input tensor. strides: An int or list of `ints` that has length `1`, `2` or `4`. The stride of the sliding window for each dimension of the input tensor. padding: A `string` from: `"SAME", "VALID"`. The type of padding algorithm to use. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: An optional `string`, must be set to `"NHWC"`. Defaults to `"NHWC"`. Specify the data format of the input and output data. output_dtype: An optional `tf.DType` from: `tf.int32, tf.int64`. Defaults to `tf.int64`. The dtype of the returned argmax tensor. include_batch_in_index: An optional `boolean`. Defaults to `False`. Whether to include batch dimension in flattened index of `argmax`. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (output, argmax). output: A `Tensor`. Has the same type as `input`. argmax: A `Tensor` of type `output_dtype`. r"rr,rRrrFr@rrFrDTargmaxinclude_batch_in_indexr9)r5r<rmax_pool_with_argmax)r@rrFrDrE output_dtyperBr9s r;max_pool_with_argmax_v2rE svpF 99D G HH q!W -% '1a 3'  ( (  3  r=c |dk7rtd|td|d|}|tj}t j |||||||S)Nr"rrDrAr@)r5rrint64rrC) r@rrFrDrErAr9rDrBs r;max_pool_with_argmax_v1rHVsvF 99D G HH 'lIw 8' _llG  ( (  3  r=rflopscNtj||jd}|jtj||jd}|jtj||j}|jt |d}t |d}t |d}t |d}t j|jt j} tjd| |z|z|z|zdzS)z3Calculates the compute resources needed for Conv3D.rr+r,rRrrI rtensor_shape_from_node_def_namer@assert_is_fully_definedr9r2rprodr|rGr OpStats) graphnoderLrCr filter_timergrhfilter_in_depth output_counts r;_calc_conv3d_flopsrUws::5$**Q-P+ %%';; TZZ],&&(;;E499M,&&(LO$+l1o&-\!_%, Q(/--/rxx@, W|o= K, -/; <>? @ BBr=rc,tj||jd}|jtj||jd}|jtj||j}|jt |d}t |d}t |d}t j|jt j}tjd||z|z|zdzS)z3Calculates the compute resources needed for Conv2D.rr+r,rrIrK) rPrQrLrCrrgrhrSrTs r;_calc_conv_flopsrWs::5$**Q-P+ %%';; TZZ],&&(;;E499M,&&(l1o&-\!_%, Q(/--/rxx@,  o% 5 DqH KKr=DepthwiseConv2dNativec tj||jd}|jtj||jd}|jtj||j}|jt |d}t |d}t j|jt j}tjd||z|zdzS)zBCalculates the compute resources needed for DepthwiseConv2dNative.rr+rrIr,rKrPrQrLrCrrgrhrTs r;_calc_depthwise_conv_flopsr[::5$**Q-P+ %%';; TZZ],&&(;;E499M,&&(l1o&-\!_%,--/rxx@, W|m;lJQN PPr=rctj||jd}|jt j |j }tjd|S)z,Calculates the computing needed for BiasAdd.rrI) rrLr@rMrrNr|r rO)rPrQrL input_counts r;_calc_bias_add_flopsr_sR::5$**Q-P+ %%' ++-.+ Wk **r=z nn.xw_plus_bc4tj|d|||g5}tj|d}tj|d}tj|d}tj||}t |||cdddS#1swYyxYw)aComputes matmul(x, weights) + biases. Args: x: a 2D tensor. Dimensions typically: batch, in_units weights: a 2D tensor. Dimensions typically: in_units, out_units biases: a 1D tensor. Dimensions: out_units name: A name for the operation (optional). If not specified "xw_plus_b" is used. Returns: A 2-D Tensor computing matmul(x, weights) + biases. Dimensions typically: batch, out_units. xw_plus_brrAweightsbiasesN)r rGrHrmatmulrrrbrcr9mms r;raras  ~~dK!Wf)=>+$ ac*A##G)>> tf.random.set_seed(0) >>> x = tf.ones([3,5]) >>> tf.nn.dropout(x, rate = 0.5, seed = 1).numpy() array([[2., 0., 0., 2., 2.], [2., 2., 2., 2., 2.], [2., 0., 2., 0., 2.]], dtype=float32) >>> tf.random.set_seed(0) >>> x = tf.ones([3,5]) >>> tf.nn.dropout(x, rate = 0.8, seed = 1).numpy() array([[0., 0., 0., 5., 5.], [0., 5., 0., 5., 0.], [5., 0., 5., 0., 5.]], dtype=float32) >>> tf.nn.dropout(x, rate = 0.0) == x By default, each element is kept or dropped independently. If `noise_shape` is specified, it must be [broadcastable](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) to the shape of `x`, and only dimensions with `noise_shape[i] == shape(x)[i]` will make independent decisions. This is useful for dropping whole channels from an image or sequence. For example: >>> tf.random.set_seed(0) >>> x = tf.ones([3,10]) >>> tf.nn.dropout(x, rate = 2/3, noise_shape=[1,10], seed=1).numpy() array([[0., 0., 0., 3., 3., 0., 3., 3., 3., 0.], [0., 0., 0., 3., 3., 0., 3., 3., 3., 0.], [0., 0., 0., 3., 3., 0., 3., 3., 3., 0.]], dtype=float32) Args: x: A floating point tensor. rate: A scalar `Tensor` with the same type as x. The probability that each element is dropped. For example, setting rate=0.1 would drop 10% of input elements. noise_shape: A 1-D integer `Tensor`, representing the shape for randomly generated keep/drop flags. seed: A Python integer. Used to create random seeds. See `tf.random.set_seed` for behavior. name: A name for this operation (optional). Returns: A Tensor of the same shape of `x`. Raises: ValueError: If `rate` is not in `[0, 1)` or if `x` is not a floating point tensor. `rate=1` is disallowed, because the output would be all zeros, which is likely not what was intended. rrc0tjyr)r get_seedrwsr;dummy_rng_stepz"dropout_v2..dummy_rng_stepsr=rurr3rluniform_samplerrzr9r)rDrErrandom_uniform_dropout)rr3rlrrr9r|rzs ` r;rsrs6sAl%%j&?&?dK/ ADk"1!/d( **r=z!nn.experimental.stateless_dropoutc xtjtj||}d}t ||||||dS)a!Computes dropout: randomly sets elements to zero to prevent overfitting. [Dropout](https://arxiv.org/abs/1207.0580) is useful for regularizing DNN models. Inputs elements are randomly set to zero (and the other elements are rescaled). This encourages each node to be independently useful, as it cannot rely on the output of other nodes. More precisely: With probability `rate` elements of `x` are set to `0`. The remaining elements are scaled up by `1.0 / (1 - rate)`, so that the expected value is preserved. >>> x = tf.ones([3,5]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.5, seed=[1, 0]) >>> x = tf.ones([3,5]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.8, seed=[1, 0]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.0, seed=[1, 0]) == x This function is a stateless version of `tf.nn.dropout`, in the sense that no matter how many times you call this function, the same `seed` will lead to the same results, and different `seed` will lead to different results. >>> x = tf.ones([3,5]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.8, seed=[1, 0]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.8, seed=[1, 0]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.8, seed=[2, 0]) >>> tf.nn.experimental.stateless_dropout(x, rate=0.8, seed=[2, 0]) Compare the above results to those of `tf.nn.dropout` below. The second time `tf.nn.dropout` is called with the same seed, it will give a different output. >>> tf.random.set_seed(0) >>> x = tf.ones([3,5]) >>> tf.nn.dropout(x, rate=0.8, seed=1) >>> tf.nn.dropout(x, rate=0.8, seed=1) >>> tf.nn.dropout(x, rate=0.8, seed=2) >>> tf.nn.dropout(x, rate=0.8, seed=2) The difference between this function and `tf.nn.dropout` is analogous to the difference between `tf.random.stateless_uniform` and `tf.random.uniform`. Please see [Random number generation](https://www.tensorflow.org/guide/random_numbers) guide for a detailed description of the various RNG systems in TF. As the guide states, legacy stateful RNG ops like `tf.random.uniform` and `tf.nn.dropout` are not deprecated yet but highly discouraged, because their states are hard to control. By default, each element is kept or dropped independently. If `noise_shape` is specified, it must be [broadcastable](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html) to the shape of `x`, and only dimensions with `noise_shape[i] == shape(x)[i]` will make independent decisions. This is useful for dropping whole channels from an image or sequence. For example: >>> x = tf.ones([3,10]) >>> tf.nn.experimental.stateless_dropout(x, rate=2/3, noise_shape=[1,10], ... seed=[1, 0]) Args: x: A floating point tensor. rate: A scalar `Tensor` with the same type as x. The probability that each element is dropped. For example, setting rate=0.1 would drop 10% of input elements. seed: An integer tensor of shape `[2]`. The seed of the random numbers. rng_alg: The algorithm used to generate the random numbers (default to `"auto_select"`). See the `alg` argument of `tf.random.stateless_uniform` for the supported values. noise_shape: A 1-D integer `Tensor`, representing the shape for randomly generated keep/drop flags. name: A name for this operation. Returns: A Tensor of the same shape and dtype of `x`. Raises: ValueError: If `rate` is not in `[0, 1)` or if `x` is not a floating point tensor. `rate=1` is disallowed, because the output would be all zeros, which is likely not what was intended. )rralgcyrrsrsr=r;rzz)stateless_dropout..dummy_rng_stepr=stateless_dropoutr{)rDrErstateless_random_uniformr~)rr3rrrng_algrlr9r|rzs r;rrsFL%%33$GM/ ADk"1!/d2 44r=znn.experimental.general_dropoutc ,d}t||||||dS)a Computes dropout: randomly sets elements to zero to prevent overfitting. Please see `tf.nn.experimental.stateless_dropout` for an overview of dropout. Unlike `tf.nn.experimental.stateless_dropout`, here you can supply a custom sampler function `uniform_sampler` that (given a shape and a dtype) generates a random, `Uniform[0, 1)`-distributed tensor (of that shape and dtype). `uniform_sampler` can be e.g. `tf.random.stateless_random_uniform` or `tf.random.Generator.uniform`. For example, if you are using `tf.random.Generator` to generate random numbers, you can use this code to do dropouts: >>> g = tf.random.Generator.from_seed(7) >>> sampler = g.uniform >>> x = tf.constant([1.1, 2.2, 3.3, 4.4, 5.5]) >>> rate = 0.5 >>> tf.nn.experimental.general_dropout(x, rate, sampler) >>> tf.nn.experimental.general_dropout(x, rate, sampler) It has better performance than using `tf.nn.experimental.stateless_dropout` and `tf.random.Generator.make_seeds`: >>> g = tf.random.Generator.from_seed(7) >>> x = tf.constant([1.1, 2.2, 3.3, 4.4, 5.5]) >>> rate = 0.5 >>> tf.nn.experimental.stateless_dropout(x, rate, g.make_seeds(1)[:, 0]) >>> tf.nn.experimental.stateless_dropout(x, rate, g.make_seeds(1)[:, 0]) because generating and consuming seeds cost extra computation. `tf.nn.experimental.general_dropout` can let you avoid them. Args: x: A floating point tensor. rate: A scalar `Tensor` with the same type as x. The probability that each element is dropped. For example, setting rate=0.1 would drop 10% of input elements. uniform_sampler: a callable of signature `(shape, dtype) -> Tensor[shape, dtype]`, used to generate a tensor of uniformly-distributed random numbers in the range `[0, 1)`, of the given shape and dtype. noise_shape: A 1-D integer `Tensor`, representing the shape for randomly generated keep/drop flags. name: A name for this operation. Returns: A Tensor of the same shape and dtype of `x`. Raises: ValueError: If `rate` is not in `[0, 1)` or if `x` is not a floating point tensor. `rate=1` is disallowed, because the output would be all zeros, which is likely not what was intended. cyrrsrsr=r;rzz'general_dropout..dummy_rng_stepdrr=general_dropoutr{)r~)rr3r|rlr9rzs r;rr%s&~ ADk"1!/d0 22r=ctj|||g5}t|tj}|r|dks|dk\rt d|tj |d}|j}|jst d||r|dk(r||cdddStj} tj|sH|r8d|z } d| z } tj | |} tj|| } nt d ||jj!d|j} | |k7r:| j#|st d |d | tj$||d }t'j(d|}tj*|tj,||} t/||}||| }||k\}t'j(d|}t1j2|| |} | s| j5|j| cdddS#1swYyxYw)aShared implementation of the various dropout functions. Args: x: same as the namesake in `dropout_v2`. rate: same as the namesake in `dropout_v2`. noise_shape: same as the namesake in `dropout_v2`. uniform_sampler: a callable of signature `(shape, dtype) -> Tensor`, used to generate a tensor of uniformly-distributed random numbers in the range `[0, 1)`, of the given shape and dtype. dummy_rng_step: a callable of signature `() -> None`, to make a dummy RNG call in the fast path. In the fast path where rate is 0, we don't need to generate random numbers, but some samplers still require you to make an RNG call, to make sure that RNG states won't depend on whether the fast path is taken. name: same as the namesake in `dropout_v2`. default_name: a default name in case `name` is `None`. Returns: A Tensor of the same shape and dtype of `x`. rr+zN`rate` must be a scalar tensor or a float in the range [0, 1). Received: rate=rrAzS`x.dtype` must be a floating point tensor as `x` will be scaled. Received: x_dtype=Nrz9`rate` must be a scalar or scalar tensor. Received: rate=zB`x.dtype` must be compatible with `rate.dtype`. Received: x.dtype=z and rate.dtype=r3)rIr)r rGr/numbersRealr5rHrrrrr rrmulrassert_has_rankrrrrreal_divsubrorwhere_v2r~)rr3rlr|rzr9ris_rate_numberx_dtypeis_executing_eagerlyrpscaleret rate_dtype one_tensor random_tensor keep_mask zero_tensors r;r~r~ls?, ~~dL1#.7$gll3N4!8tqy 77;f> ?? ac*AggG     ''.i 1 22$!) -770#446  ! !$ ' H I %%e7;q%(Gv NP P nn&&q)::j w ,,W5##*)+;J<IJ J  wV<''9j  ! !!\%5%5j$%G Hc"1k2K$+WEM%I&&q8K   Y[ 9C  mmAKKM" o777sBH5$FH55H>z math.top_kznn.top_kr+c6tj|||||S)aFinds values and indices of the `k` largest entries for the last dimension. If the input is a vector (rank=1), finds the `k` largest entries in the vector and outputs their values and indices as vectors. Thus `values[j]` is the `j`-th largest entry in `input`, and its index is `indices[j]`. >>> result = tf.math.top_k([1, 2, 98, 1, 1, 99, 3, 1, 3, 96, 4, 1], ... k=3) >>> result.values.numpy() array([99, 98, 96], dtype=int32) >>> result.indices.numpy() array([5, 2, 9], dtype=int32) For matrices (resp. higher rank input), computes the top `k` entries in each row (resp. vector along the last dimension). Thus, >>> input = tf.random.normal(shape=(3,4,5,6)) >>> k = 2 >>> values, indices = tf.math.top_k(input, k=k) >>> values.shape.as_list() [3, 4, 5, 2] >>> >>> values.shape == indices.shape == input.shape[:-1] + [k] True The indices can be used to `gather` from a tensor who's shape matches `input`. >>> gathered_values = tf.gather(input, indices, batch_dims=-1) >>> assert tf.reduce_all(gathered_values == values) If two elements are equal, the lower-index element appears first. >>> result = tf.math.top_k([1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0], ... k=3) >>> result.indices.numpy() array([0, 1, 3], dtype=int32) By default, indices are returned as type `int32`, however, this can be changed by specifying the `index_type`. >>> result = tf.math.top_k([1, 1, 0, 1, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0], ... k=3, index_type=tf.int16) >>> result.indices.numpy() array([0, 1, 3], dtype=int16) Args: input: 1-D or higher `Tensor` with last dimension at least `k`. k: 0-D `Tensor` of type `int16`, `int32` or `int64`. Number of top element to look for along the last dimension (along each row for matrices). sorted: If true the resulting `k` elements will be sorted by the values in descending order. index_type: Optional dtype for output indices. name: Optional name for the operation. Returns: A tuple with two named fields: values: The `k` largest elements along each last dimensional slice. indices: The indices of `values` within the last dimension of `input`. )kr index_typer9)rtop_kv2)r@rrrr9s r;top_krs#|    qJT r=zmath.approx_max_kznn.approx_max_kc <tj||||d|||S)a8Returns max `k` values and their indices of the input `operand` in an approximate manner. See https://arxiv.org/abs/2206.14286 for the algorithm details. This op is only optimized on TPU currently. Args: operand : Array to search for max-k. Must be a floating number type. k : Specifies the number of max-k. reduction_dimension : Integer dimension along which to search. Default: -1. recall_target : Recall target for the approximation. reduction_input_size_override : When set to a positive value, it overrides the size determined by `operand[reduction_dim]` for evaluating the recall. This option is useful when the given `operand` is only a subset of the overall computation in SPMD or distributed pipelines, where the true input size cannot be deferred by the `operand` shape. aggregate_to_topk : When true, aggregates approximate results to top-k. When false, returns the approximate results. The number of the approximate results is implementation defined and is greater equals to the specified `k`. name: Optional name for the operation. Returns: Tuple of two arrays. The arrays are the max `k` values and the corresponding indices along the `reduction_dimension` of the input `operand`. The arrays' dimensions are the same as the input `operand` except for the `reduction_dimension`: when `aggregate_to_topk` is true, the reduction dimension is `k`; otherwise, it is greater equals to `k` where the size is implementation-defined. We encourage users to wrap `approx_max_k` with jit. See the following example for maximal inner production search (MIPS): >>> import tensorflow as tf >>> @tf.function(jit_compile=True) ... def mips(qy, db, k=10, recall_target=0.95): ... dists = tf.einsum('ik,jk->ij', qy, db) ... # returns (f32[qy_size, k], i32[qy_size, k]) ... return tf.nn.approx_max_k(dists, k=k, recall_target=recall_target) >>> >>> qy = tf.random.uniform((256,128)) >>> db = tf.random.uniform((2048,128)) >>> dot_products, neighbors = mips(qy, db, k=20) Trreduction_dimension recall_targetis_max_kreduction_input_size_overrideaggregate_to_topkr9r approx_top_koperandrrrrrr9s r; approx_max_krs0h    -!$A)  r=zmath.approx_min_kznn.approx_min_kc <tj||||d|||S)a5 Returns min `k` values and their indices of the input `operand` in an approximate manner. See https://arxiv.org/abs/2206.14286 for the algorithm details. This op is only optimized on TPU currently. Args: operand : Array to search for min-k. Must be a floating number type. k : Specifies the number of min-k. reduction_dimension: Integer dimension along which to search. Default: -1. recall_target: Recall target for the approximation. reduction_input_size_override : When set to a positive value, it overrides the size determined by `operand[reduction_dim]` for evaluating the recall. This option is useful when the given `operand` is only a subset of the overall computation in SPMD or distributed pipelines, where the true input size cannot be deferred by the `operand` shape. aggregate_to_topk: When true, aggregates approximate results to top-k. When false, returns the approximate results. The number of the approximate results is implementation defined and is greater equals to the specified `k`. name: Optional name for the operation. Returns: Tuple of two arrays. The arrays are the least `k` values and the corresponding indices along the `reduction_dimension` of the input `operand`. The arrays' dimensions are the same as the input `operand` except for the `reduction_dimension`: when `aggregate_to_topk` is true, the reduction dimension is `k`; otherwise, it is greater equals to `k` where the size is implementation-defined. We encourage users to wrap `approx_min_k` with jit. See the following example for nearest neighbor search over the squared l2 distance: >>> import tensorflow as tf >>> @tf.function(jit_compile=True) ... def l2_ann(qy, db, half_db_norms, k=10, recall_target=0.95): ... dists = half_db_norms - tf.einsum('ik,jk->ij', qy, db) ... return tf.nn.approx_min_k(dists, k=k, recall_target=recall_target) >>> >>> qy = tf.random.uniform((256,128)) >>> db = tf.random.uniform((2048,128)) >>> half_db_norms = tf.norm(db, axis=1) / 2 >>> dists, neighbors = l2_ann(qy, db, half_db_norms) In the example above, we compute `db_norms/2 - dot(qy, db^T)` instead of `qy^2 - 2 dot(qy, db^T) + db^2` for performance reason. The former uses less arithmetics and produces the same set of neighbors. Frrrs r; approx_min_kr>s0p    -!$A)  r=c4tj||||S)aFinds values of the `n`-th smallest value for the last dimension. Note that n is zero-indexed. If the input is a vector (rank-1), finds the entries which is the nth-smallest value in the vector and outputs their values as scalar tensor. For matrices (resp. higher rank input), computes the entries which is the nth-smallest value in each row (resp. vector along the last dimension). Thus, values.shape = input.shape[:-1] Args: input: 1-D or higher `Tensor` with last dimension at least `n+1`. n: A `Tensor` of type `int32`. 0-D. Position of sorted vector to select along the last dimension (along each row for matrices). Valid range of n is `[0, input.shape[:-1])` reverse: An optional `bool`. Defaults to `False`. When set to True, find the nth-largest value in the vector and vice versa. name: A name for the operation (optional). Returns: A `Tensor`. Has the same type as `input`. The `n`-th order statistic along each last dimensional slice. )reverser9)r nth_element)r@r7rr9s r;rrs6   q' EEr=znn.fractional_max_poolzM`seed2` and `deterministic` args are deprecated. Use fractional_max_pool_v2.c tjr|r|r|std|d|d|tj||||||||S)aPerforms fractional max pooling on the input. This is a deprecated version of `fractional_max_pool`. Fractional max pooling is slightly different than regular max pooling. In regular max pooling, you downsize an input set by taking the maximum value of smaller N x N subsections of the set (often 2x2), and try to reduce the set by a factor of N, where N is an integer. Fractional max pooling, as you might expect from the word "fractional", means that the overall reduction ratio N does not have to be an integer. The sizes of the pooling regions are generated randomly but are fairly uniform. For example, let's look at the height dimension, and the constraints on the list of rows that will be pool boundaries. First we define the following: 1. input_row_length : the number of rows from the input set 2. output_row_length : which will be smaller than the input 3. alpha = input_row_length / output_row_length : our reduction ratio 4. K = floor(alpha) 5. row_pooling_sequence : this is the result list of pool boundary rows Then, row_pooling_sequence should satisfy: 1. a[0] = 0 : the first value of the sequence is 0 2. a[end] = input_row_length : the last value of the sequence is the size 3. K <= (a[i+1] - a[i]) <= K+1 : all intervals are K or K+1 size 4. length(row_pooling_sequence) = output_row_length+1 Args: value: A `Tensor`. 4-D with shape `[batch, height, width, channels]`. pooling_ratio: A list of `floats` that has length >= 4. Pooling ratio for each dimension of `value`, currently only supports row and col dimension and should be >= 1.0. For example, a valid pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements must be 1.0 because we don't allow pooling on batch and channels dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions respectively. pseudo_random: An optional `bool`. Defaults to `False`. When set to `True`, generates the pooling sequence in a pseudorandom fashion, otherwise, in a random fashion. Check (Graham, 2015) for difference between pseudorandom and random. overlapping: An optional `bool`. Defaults to `False`. When set to `True`, it means when pooling, the values at the boundary of adjacent pooling cells are used by both cells. For example: `index 0 1 2 3 4` `value 20 5 16 3 7` If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice. The result would be [20, 16] for fractional max pooling. deterministic: An optional `bool`. Deprecated; use `fractional_max_pool_v2` instead. seed: An optional `int`. Defaults to `0`. If set to be non-zero, the random number generator is seeded by the given seed. Otherwise it is seeded by a random seed. seed2: An optional `int`. Deprecated; use `fractional_max_pool_v2` instead. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (`output`, `row_pooling_sequence`, `col_pooling_sequence`). output: Output `Tensor` after fractional max pooling. Has the same type as `value`. row_pooling_sequence: A `Tensor` of type `int64`. col_pooling_sequence: A `Tensor` of type `int64`. Raises: ValueError: If op determinism is enabled and either the seeds are not set or the "deterministic" argument is False. References: Fractional Max-Pooling: [Graham, 2015](https://arxiv.org/abs/1412.6071) ([pdf](https://arxiv.org/pdf/1412.6071.pdf)) ztf.compat.v1.nn.fractional_max_pool requires "seed" and "seed2" to be non-zero and "deterministic" to be true when op determinism is enabled. Please pass in such values, e.g. by passing"seed=1, seed2=1, deterministic=True". Got: seed=z, seed2=z, deterministic=)rrr5rfractional_max_poolr6 pooling_ratio pseudo_random overlapping deterministicrrseed2r9s r;rrspl %%'U1>  <=A6B'   8 99  ' '}m(3]D%(, ..r=c t|ttfr9|ddk7s|ddk7rtd||D]}|dks td|n7t|tt fr|dkrtd|td|t |dd d }|dk(r@tjrtd |d tj||||d dd|Stj|\}}tj||||d|||S)a Performs fractional max pooling on the input. Fractional max pooling is slightly different than regular max pooling. In regular max pooling, you downsize an input set by taking the maximum value of smaller N x N subsections of the set (often 2x2), and try to reduce the set by a factor of N, where N is an integer. Fractional max pooling, as you might expect from the word "fractional", means that the overall reduction ratio N does not have to be an integer. The sizes of the pooling regions are generated randomly but are fairly uniform. For example, let's look at the height dimension, and the constraints on the list of rows that will be pool boundaries. First we define the following: 1. input_row_length : the number of rows from the input set 2. output_row_length : which will be smaller than the input 3. alpha = input_row_length / output_row_length : our reduction ratio 4. K = floor(alpha) 5. row_pooling_sequence : this is the result list of pool boundary rows Then, row_pooling_sequence should satisfy: 1. a[0] = 0 : the first value of the sequence is 0 2. a[end] = input_row_length : the last value of the sequence is the size 3. K <= (a[i+1] - a[i]) <= K+1 : all intervals are K or K+1 size 4. length(row_pooling_sequence) = output_row_length+1 Args: value: A `Tensor`. 4-D with shape `[batch, height, width, channels]`. pooling_ratio: An int or list of `ints` that has length `1`, `2` or `4`. Pooling ratio for each dimension of `value`, currently only supports row and col dimension and should be >= 1.0. For example, a valid pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements must be 1.0 because we don't allow pooling on batch and channels dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions respectively. pseudo_random: An optional `bool`. Defaults to `False`. When set to `True`, generates the pooling sequence in a pseudorandom fashion, otherwise, in a random fashion. Check paper (Graham, 2015) for difference between pseudorandom and random. overlapping: An optional `bool`. Defaults to `False`. When set to `True`, it means when pooling, the values at the boundary of adjacent pooling cells are used by both cells. For example: `index 0 1 2 3 4` `value 20 5 16 3 7` If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice. The result would be [20, 16] for fractional max pooling. seed: An optional `int`. Defaults to `0`. If set to be non-zero, the random number generator is seeded by the given seed. Otherwise it is seeded by a random seed. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (`output`, `row_pooling_sequence`, `col_pooling_sequence`). output: Output `Tensor` after fractional max pooling. Has the same type as `value`. row_pooling_sequence: A `Tensor` of type `int64`. col_pooling_sequence: A `Tensor` of type `int64`. Raises: ValueError: If no seed is specified and op determinism is enabled. References: Fractional Max-Pooling: [Graham, 2015](https://arxiv.org/abs/1412.6071) ([pdf](https://arxiv.org/pdf/1412.6071.pdf)) rrrvz\`pooling_ratio` should have first and last elements with value 1.0. Received: pooling_ratio=zC`pooling_ratio` elements should be >= 1.0. Received: pooling_ratio=z:`pooling_ratio` should be >= 1.0. Received: pooling_ratio=zL`pooling_ratio` should be an int or a list of ints. Received: pooling_ratio=r,rRrzmtf.nn.fractional_max_pool requires a non-zero seed to be passed in when determinism is enabled, but got seed=z;. Please pass in a non-zero seed, e.g. by passing "seed=1".Frrrrr9T) r/r0r1r5r2rr<rrrrr ry) r6rrrrrr9elementseed1rs r;fractional_max_pool_v2rszXu .aC=#4#;  %%2O 5 66!8 3''4o 78 88 =3,/s  %%2O 5 66  ##0/ 3 44 q!_E- QY '')  77;f=5 6 77  ) )% *5U/0 FF''-LE5  ) )% *5T/4E NNr=znn.fractional_avg_poolzM`seed2` and `deterministic` args are deprecated. Use fractional_avg_pool_v2.c <tj||||||||S)a Performs fractional average pooling on the input. This is a deprecated version of `fractional_avg_pool`. Fractional average pooling is similar to Fractional max pooling in the pooling region generation step. The only difference is that after pooling regions are generated, a mean operation is performed instead of a max operation in each pooling region. Args: value: A `Tensor`. 4-D with shape `[batch, height, width, channels]`. pooling_ratio: A list of `floats` that has length >= 4. Pooling ratio for each dimension of `value`, currently only supports row and col dimension and should be >= 1.0. For example, a valid pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements must be 1.0 because we don't allow pooling on batch and channels dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions respectively. pseudo_random: An optional `bool`. Defaults to `False`. When set to `True`, generates the pooling sequence in a pseudorandom fashion, otherwise, in a random fashion. Check paper (Graham, 2015) for difference between pseudorandom and random. overlapping: An optional `bool`. Defaults to `False`. When set to `True`, it means when pooling, the values at the boundary of adjacent pooling cells are used by both cells. For example: `index 0 1 2 3 4` `value 20 5 16 3 7` If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice. The result would be [20, 16] for fractional avg pooling. deterministic: An optional `bool`. Deprecated; use `fractional_avg_pool_v2` instead. seed: An optional `int`. Defaults to `0`. If set to be non-zero, the random number generator is seeded by the given seed. Otherwise it is seeded by a random seed. seed2: An optional `int`. Deprecated; use `fractional_avg_pool_v2` instead. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (`output`, `row_pooling_sequence`, `col_pooling_sequence`). output: Output `Tensor` after fractional avg pooling. Has the same type as `value`. row_pooling_sequence: A `Tensor` of type `int64`. col_pooling_sequence: A `Tensor` of type `int64`. References: Fractional Max-Pooling: [Graham, 2015](https://arxiv.org/abs/1412.6071) ([pdf](https://arxiv.org/pdf/1412.6071.pdf)) rA)rfractional_avg_poolrs r;rrts*z  ' '}m(3]D%-1 33r=c |dk(rtj||||ddd|Stj|\}}tj||||d|||S)aPerforms fractional average pooling on the input. Fractional average pooling is similar to Fractional max pooling in the pooling region generation step. The only difference is that after pooling regions are generated, a mean operation is performed instead of a max operation in each pooling region. Args: value: A `Tensor`. 4-D with shape `[batch, height, width, channels]`. pooling_ratio: A list of `floats` that has length >= 4. Pooling ratio for each dimension of `value`, currently only supports row and col dimension and should be >= 1.0. For example, a valid pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements must be 1.0 because we don't allow pooling on batch and channels dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions respectively. pseudo_random: An optional `bool`. Defaults to `False`. When set to `True`, generates the pooling sequence in a pseudorandom fashion, otherwise, in a random fashion. Check paper (Graham, 2015) for difference between pseudorandom and random. overlapping: An optional `bool`. Defaults to `False`. When set to `True`, it means when pooling, the values at the boundary of adjacent pooling cells are used by both cells. For example: `index 0 1 2 3 4` `value 20 5 16 3 7` If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice. The result would be [20, 16] for fractional avg pooling. seed: An optional `int`. Defaults to `0`. If set to be non-zero, the random number generator is seeded by the given seed. Otherwise it is seeded by a random seed. name: A name for the operation (optional). Returns: A tuple of `Tensor` objects (`output`, `row_pooling_sequence`, `col_pooling_sequence`). output: Output `Tensor` after fractional avg pooling. Has the same type as `value`. row_pooling_sequence: A `Tensor` of type `int64`. col_pooling_sequence: A `Tensor` of type `int64`. References: Fractional Max-Pooling: [Graham, 2015](https://arxiv.org/abs/1412.6071) ([pdf](https://arxiv.org/pdf/1412.6071.pdf)) rFrT)rrr ry)r6rrrrrr9rrs r;fractional_avg_pool_v2rsph QY  ) )% *5U/0 FF''-LE5  ) )% *5T/4E NNr= Dilation2Dc tj||jd}|jtj||jd}|jtj||j}|jt |d}t |d}t j|jt j}tjd||z|zdzS)z7Calculates the compute resources needed for Dilation2D.rr+rrIr,rKrZs r;_calc_dilation2d_flopsrr\r=z nn.erosion2dc tj|d||g5}tjt j tj|t j|ddg||||cdddS#1swYyxYw)aComputes the grayscale erosion of 4-D `value` and 3-D `kernel` tensors. The `value` tensor has shape `[batch, in_height, in_width, depth]` and the `kernel` tensor has shape `[kernel_height, kernel_width, depth]`, i.e., each input channel is processed independently of the others with its own structuring function. The `output` tensor has shape `[batch, out_height, out_width, depth]`. The spatial dimensions of the output tensor depend on the `padding` algorithm. We currently only support the default "NHWC" `data_format`. In detail, the grayscale morphological 2-D erosion is given by: output[b, y, x, c] = min_{dy, dx} value[b, strides[1] * y - rates[1] * dy, strides[2] * x - rates[2] * dx, c] - kernel[dy, dx, c] Duality: The erosion of `value` by the `kernel` is equal to the negation of the dilation of `-value` by the reflected `kernel`. Args: value: A `Tensor`. 4-D with shape `[batch, in_height, in_width, depth]`. kernel: A `Tensor`. Must have the same type as `value`. 3-D with shape `[kernel_height, kernel_width, depth]`. strides: A list of `ints` that has length `>= 4`. 1-D of length 4. The stride of the sliding window for each dimension of the input tensor. Must be: `[1, stride_height, stride_width, 1]`. rates: A list of `ints` that has length `>= 4`. 1-D of length 4. The input stride for atrous morphological dilation. Must be: `[1, rate_height, rate_width, 1]`. padding: A `string` from: `"SAME", "VALID"`. The type of padding algorithm to use. name: A name for the operation (optional). If not specified "erosion2d" is used. Returns: A `Tensor`. Has the same type as `value`. 4-D with shape `[batch, out_height, out_width, depth]`. Raises: ValueError: If the `value` depth does not match `kernel`' shape, or if padding is other than `'VALID'` or `'SAME'`. erosion2drr+rN)r rGrrrrr reverse_v2)r6kernelrFrrDr9s r;rrsz^ ~~dK%9 T   ##E*''A7      s AA;;Bc 4|dk7rtd|tj|d||g5}tjt j tj|tj|ddg||||cdddS#1swYyxYw)aComputes the grayscale erosion of 4-D `value` and 3-D `filters` tensors. The `value` tensor has shape `[batch, in_height, in_width, depth]` and the `filters` tensor has shape `[filters_height, filters_width, depth]`, i.e., each input channel is processed independently of the others with its own structuring function. The `output` tensor has shape `[batch, out_height, out_width, depth]`. The spatial dimensions of the output tensor depend on the `padding` algorithm. We currently only support the default "NHWC" `data_format`. In detail, the grayscale morphological 2-D erosion is given by: output[b, y, x, c] = min_{dy, dx} value[b, strides[1] * y - dilations[1] * dy, strides[2] * x - dilations[2] * dx, c] - filters[dy, dx, c] Duality: The erosion of `value` by the `filters` is equal to the negation of the dilation of `-value` by the reflected `filters`. Args: value: A `Tensor`. 4-D with shape `[batch, in_height, in_width, depth]`. filters: A `Tensor`. Must have the same type as `value`. 3-D with shape `[filters_height, filters_width, depth]`. strides: A list of `ints` that has length `>= 4`. 1-D of length 4. The stride of the sliding window for each dimension of the input tensor. Must be: `[1, stride_height, stride_width, 1]`. padding: A `string` from: `"SAME", "VALID"`. The type of padding algorithm to use. See [here](https://www.tensorflow.org/api_docs/python/tf/nn#notes_on_padding_2) for more information. data_format: A `string`, only `"NHWC"` is currently supported. dilations: A list of `ints` that has length `>= 4`. 1-D of length 4. The input stride for atrous morphological dilation. Must be: `[1, rate_height, rate_width, 1]`. name: A name for the operation (optional). If not specified "erosion2d" is used. Returns: A `Tensor`. Has the same type as `value`. 4-D with shape `[batch, out_height, out_width, depth]`. Raises: ValueError: If the `value` depth does not match `filters`' shape, or if padding is other than `'VALID'` or `'SAME'`. r"rrrr+rN) r5r rGrrrrrr)r6rfrFrDrErr9s r; erosion2d_v2r@srF 99D G HH ~~dK%)9: d   ##E*''!Q8      s ABBz math.in_top_kz nn.in_top_kctj|d5tj||||cdddS#1swYyxYw)aSays whether the targets are in the top `K` predictions. This outputs a `batch_size` bool array, an entry `out[i]` is `true` if the prediction for the target class is finite (not inf, -inf, or nan) and among the top `k` predictions among all predictions for example `i`. Note that the behavior of `InTopK` differs from the `TopK` op in its handling of ties; if multiple classes have the same prediction value and straddle the top-`k` boundary, all of those classes are considered to be in the top `k`. More formally, let \\(predictions_i\\) be the predictions for all classes for example `i`, \\(targets_i\\) be the target class for example `i`, \\(out_i\\) be the output for example `i`, $$out_i = predictions_{i, targets_i} \in TopKIncludingTies(predictions_i)$$ Args: predictions: A `Tensor` of type `float32`. A `batch_size` x `classes` tensor. targets: A `Tensor`. Must be one of the following types: `int32`, `int64`. A `batch_size` vector of class ids. k: An `int`. Number of top elements to look at for computing precision. name: A name for the operation (optional). Returns: A `Tensor` of type `bool`. Computed Precision at `k` as a `bool Tensor`. in_top_krAN)r rGr in_top_kv2) predictionstargetsrr9s r;rrs?> ~~dJ'E  gqt DEEEs :Act||||S)aOutputs whether the targets are in the top `K` predictions. This outputs a `batch_size` bool array, an entry `out[i]` is `true` if the prediction for the target class is finite (not inf, -inf, or nan) and among the top `k` predictions among all predictions for example `i`. `predictions` does not have to be normalized. Note that the behavior of `InTopK` differs from the `TopK` op in its handling of ties; if multiple classes have the same prediction value and straddle the top-`k` boundary, all of those classes are considered to be in the top `k`. >>> target = tf.constant([0, 1, 3]) >>> pred = tf.constant([ ... [1.2, -0.3, 2.8, 5.2], ... [0.1, 0.0, 0.0, 0.0], ... [0.0, 0.5, 0.3, 0.3]], ... dtype=tf.float32) >>> print(tf.math.in_top_k(target, pred, 2)) tf.Tensor([False True True], shape=(3,), dtype=bool) Args: targets: A `batch_size` vector of class ids. Must be `int32` or `int64`. predictions: A `batch_size` x `classes` tensor of type `float32`. k: An `int`. The parameter to specify search space. name: A name for the operation (optional). Returns: A `Tensor` with the same shape of `targets` with type of `bool`. Each element specifies if the target falls into top-k predictions. )r)rrrr9s r; in_top_k_v2rsB +w4 00r=znn.quantized_avg_poolznn.quantized_conv2dznn.quantized_relu_xznn.quantized_max_poolznn.isotonic_regressionc tjtjtjtjtjtjtjtjtj tji}t j|} ||jdfd }t|||S#t$rtjY/wxYw)aSolves isotonic regression problems along the given axis. For each vector x, the problem solved is $$\argmin_{y_1 >= y_2 >= ... >= y_n} \sum_i (x_i - y_i)^2.$$ As the solution is component-wise constant, a second tensor is returned that encodes the segments. The problems are solved over the given axis. Consider the following example, where we solve a batch of two problems. The first input is [3, 1, 2], while the second [1, 3, 4] (as the axis is 1). >>> x = tf.constant([[3, 1, 2], [1, 3, 4]], dtype=tf.float32) >>> y, segments = tf.nn.isotonic_regression(x, axis=1) >>> y # The solution. Note that the first solution has two blocks [2] and [1.5, 1.5]. The second solution is constant, and thus has a single segment. These segments are exactly what the second returned tensor encodes: >>> segments Args: inputs: A tensor holding the inputs. decreasing: If set to False, the inequalities in the optimizing constrained are flipped. axis: The axis along which the problems should be solved. Returns: output: The solutions, same shape as type as the input. segments: An int32 tensor, same shape as the input indicating the segments that have the same value. Specifically, those positions that have the same value correspond to the same segment. These values start at zero, and are monotonously increasing for each solution. ctjtj|}r||S|| \}}| |fS)N)rDr9)rDrErisotonic_regression)matrixr9iso_fnrsegments decreasingrDs r;compute_on_matrixz.isotonic_regression..compute_on_matrixsK   &&\NF F^fhWh r=r) rrhalfrint8int16r rHrKeyErrorfloat64r)rrrxtype_promotionsrrDs ` @r;rrs\ nn .. kk ++ oo // kk .. ll .. /   (&""6<<0L 6#4d ;; ">>L"s1CC0/C0)NNNr)NNNNNNN)NNNNNN)NrNNN)NrNNNTNr)r@) NNNNNNNNN)r!NN)rr!NN)r"NN) NNNNrr"NNNN)rr"NN)NN)r$NN) NNNrr$NNNN)rr$NN)rNNN)Nrv)rvN)g?N)FN)NNrvNN)r"N)r!N)r$N)r"NNNF)rvgffffff?rvTN)FFFrrN)FFrN)Trv)rrrDrnumpyrtensorflow.python.eagerrtensorflow.python.frameworkrrrrrr r r rr r tensorflow.python.opsrrrrrrrrrrr tensorflow.python.ops.gen_nn_opstensorflow.python.platformrtensorflow.python.utilrrtensorflow.python.util.compatr"tensorflow.python.util.deprecationrr tensorflow.python.util.tf_exportrlrnlocal_response_normalization frozensetrr<rNrJruadd_dispatch_supportrrrrrrrrrrrewrite_argument_docstringrrr.r0r4rdeprecated_arg_valuesrirHrJrPr_rUrNrYrZrradeprecated_endpointsryr{r}r`rrrrrrrrrrrregister_unary_elementwise_apirrrrrrrrrrrr_XENT_DEPRECATION deprecatedrrrrrr)r$rr*r2r'r/r0r(rGrErHrCRegisterStatisticsrUrWr[r_rarirorursrrr~rrrrrrrrrrrrrrquantized_avg_poolquantized_conv2dquantized_relu_xquantized_max_poolrelurselusoftsignrsr=r;rsWr+.3.32+3<43+1+.,*),6</5.+9>I6 *~~ #$ $V  :zmm`.b ?r"  B*#B*J         M! M#--55  #$  [%[|o"o"d"8&v, ^  !    J"JZ #    $$@??*K**_k; i   N b^'^'B yk    p#p#f 9    ``F  RRj&$R {m """=  #""=      f:  f:R ;2   AAH !   V8"V8r ;2  ! ffR {m     uup *+,  87-87v )*+     <7,<7~ $%&       H'HV R(   R)Rj F '(  "& `A)`AF +,- !!!">?  E@.EP (3,  !!!"MN =O =@ )4-  !!!"NO >P >N %P ;2  " 1 1 {m      E E;K::x4 %%--  $%&      C'CL R(  !'$+"&!E)ER   "#!S Sl = *P*PZ:6 zl > :" //== : (( 1)1D ? (( C)C@ 9 (( 5)5p6Sl <B/ $C0$CN |^,- T#H%PCQ.C$$ !345 (( T#H%PGQ)6G8 /B7 G8G4G  1b9 :::::z 89:     v?vr 8R@ 8/A8/v =./0 >1>B }o./ 9?"(0(V ?r" $#$N ? *4*4Z ? $$P =./0 J1J\ }o    88x ? 5454t ? jj^ ? &&T $,   D-DN ()*     +8#-"A"A"I"I'*B+B$'*K+K$/9 Q: Q 7++,+ ~ + +,.02 |n T$M(*BF/L* /Ld %>B  12 &(#/1#' :3:z  12 &(#/1#' >3>BF< '() T1LM',$)&+!\.M*\.~ #+ */', ! $ mN,mN` '() T1LM',$)&+!;3M*;3| #+ */', ! $ :N,:Nz g. Q/ Q ~ 6 6r >b!  D"DN  ./ E0EB ?Mb1 121D=Y#:";<!H!!*"?"?@99!6 78!H!!*"="=>99!6 78!H!!*"="=>=Y#:";<!H!!*"?"?@  #+ F<,F