AVhdZddlmZddlmZddlmZddlmZddlmZddl m Z ddl m Z dd l m Z dd l m Z dd l mZdd l mZdd l mZddl mZddl mZddl mZddl mZddl mZddl mZddl mZddlmZddlmZddlmZdvdZdZ dZ!dZ"dwdZ#dZ$edg  dxd!Z%ed"g  dxd#Z& dyd$Z'd%Z(ed&g edd' dzd(Z)ed)g  dxd*Z*ed+g  dxd,Z+ed-g  dxd.Z,ed/g  dxd0Z-ed1g  dxd2Z.ed3g  dxd4Z/ed5g  dxd6Z0ed7g  dxd8Z1 d{d9Z2ed:g  dxd;Z3edg  dxd?Z5ed@g  dxdAZ6edBg  dxdCZ7edDg  dxdEZ8edFg  dxdGZ9edHg  dxdIZ:edJg  dxdKZ;edLg  dxdMZ<edNg  dxdOZ=dydPZ>dQZ?dwdRZ@ d{dSZA dxdTZB dydUZC d{dVZDedWg  d|dXZEedYg  d}dZZFed[g  dxd\ZGed]g  dxd^ZHed_g  d~d`ZIddaZJdbZKdcZL dxddZMdeZNedfg eddg dxdhZOedig  dxdjZP dydkZQ dxdlZRedmg  d}dnZSedog eddp d|dqZTedrg  d|dsZUedtg  d~duZVy)z$Implementation of tf.metrics module.)distribute_lib)context)dtypes)ops) sparse_tensor) array_ops)array_ops_stack) check_ops)cond)confusion_matrix)math_ops)nn)sets) sparse_ops) state_ops)variable_scope) variable_v1) variables)weights_broadcast_ops) tf_logging) deprecated) tf_exportNc  tjfddtjjtjj g|t jjt jj|S)apCreate variable in `GraphKeys.(LOCAL|METRIC_VARIABLES)` collections. If running in a `DistributionStrategy` context, the variable will be "sync on read". This means: * The returned object will be a container with separate variables per replica of the model. * When writing to the variable, e.g. using `assign_add` in a metric update, the update will be applied to the variable local to the replica. * To get a metric's result value, we need to sum the variable values across the replicas before computing the final answer. Furthermore, the final answer should be computed once instead of in every replica. Both of these are accomplished by running the computation of the final result value inside `distribute_lib.get_replica_context().merge_call(fn)`. Inside the `merge_call()`, ops are only added to the graph once and access to a sync on read variable in a computation returns the sum across all replicas. Args: shape: Shape of the created variable. dtype: Type of the created variable. validate_shape: (Optional) Whether shape validation is enabled for the created variable. name: (Optional) String name of the created variable. Returns: A (non-trainable) variable initialized to zero, or if inside a `DistributionStrategy` scope a sync on read variable container. c0tjSN)rzeros)dtypeshapesR/home/dcms/DCMS/lib/python3.12/site-packages/tensorflow/python/ops/metrics_impl.pyz!metric_variable..LsiooeU+F) trainable collectionsvalidate_shapesynchronization aggregationname) r VariableV1r GraphKeysLOCAL_VARIABLESMETRIC_VARIABLESrVariableSynchronizationON_READVariableAggregationSUM)rrr$r's`` rmetric_variabler0(scF   + -- ' ')G)G$77??//33   r!c8 tj|}|Ftj||\}}|j j |j ||dfStjj }|j }|dk(r||fS|j }|j }|B|@||z dk(rtjdgn||z dk(rtjdgntj}|tj|z fd |$|jdjdsfd nfd fd}tjtj|dfd |||fS) afSqueeze or expand last dim if needed. Squeezes last dim of `predictions` or `labels` if their rank differs by 1 (using confusion_matrix.remove_squeezable_dimensions). Squeezes or expands last dim of `weights` if its rank differs by 1 from the new rank of `predictions`. If `weights` is scalar, it is kept scalar. This will use static shape if available. Otherwise, it will add graph operations, which could result in a performance hit. Args: predictions: Predicted values, a `Tensor` of arbitrary dimensions. labels: Optional label `Tensor` whose dimensions match `predictions`. weights: Optional weight scalar or `Tensor` whose dimensions match `predictions`. Returns: Tuple of `predictions`, `labels` and `weights`. Each of them possibly has the last dimension squeezed, `weights` could be extended by one dimension. NrcftjtjdfdfdS)Nr3c2tjdgSNr3r expand_dimsweightssrr zN_remove_squeezable_dimensions.._maybe_expand_weights..s)''"6r!cSrr9srr zN_remove_squeezable_dimensions.._maybe_expand_weights..sr!r r equal) rank_diffr:sr_maybe_expand_weightsz<_remove_squeezable_dimensions.._maybe_expand_weightss) YY ..B ' 6IIr!cSrr<r9srr z/_remove_squeezable_dimensions..sgr!c2tjdgSr6)rsqueezer9srr z/_remove_squeezable_dimensions..si&7&7"&Fr!cZtjtjdS)Nr2r=)r@maybe_squeeze_weightsr?sr_maybe_adjust_weightsz<_remove_squeezable_dimensions.._maybe_adjust_weightss( YY ..A &(= !!r!cSrr<r9srr z/_remove_squeezable_dimensions..sr!)rconvert_to_tensorr remove_squeezable_dimensions get_shapeassert_is_compatible_withndimsrrCr8rankdimsis_compatible_withr r r>) predictionslabelsr: weights_shape weights_rankpredictions_shapepredictions_rankweights_rank_tensorrFr@rEr?s ` @@@r_remove_squeezable_dimensionsrWWs.%%k2+ *GG FK55f6F6F6HI _  $$  ! !' *'##%-$$,Q  ''!++-&,,")A&&!+!!'B40g L (A -%%gt4g$..1#inn[&AAII  !    # 6 6q 9-F!ii*A.G fg %%r!c tjdd|f5tjt tj rnt j tjtj|tjjdzfdfdcdddSjj}||jj}|g||k(r cdddS||dzk(r!tjdcdddSt!djd |jd t j tjtj|tjdzfd fd cdddS#1swYyxYw) aIf necessary, expand `labels` along last dimension to match `predictions`. Args: labels: `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN]. The latter implies num_labels=1, in which case the result is an expanded `labels` with shape [D1, ... DN, 1]. predictions: `Tensor` with shape [D1, ... DN, num_classes]. Returns: `labels` with the same rank as `predictions`. Raises: ValueError: if `labels` has invalid shape. N expand_labelsr2cttjtjjdfdS)N)r2rrr')rsparse_reshaperconcat dense_shaperQscopesrr z&_maybe_expand_labels..s3*++$$f&8&8$%?Cr!cSrr<rQsrr z&_maybe_expand_labels..s&r!r3r'zUnexpected labels shape z for predictions shape zS. Predictions rank should be the same rank as labels rank or labels rank plus one .c4tjdS)Nr3rcr7r_srr z&_maybe_expand_labels..s %%fbu=r!cSrr<rbsrr z&_maybe_expand_labels..svr!)r name_scoper"convert_to_tensor_or_sparse_tensor isinstance SparseTensorr r r>rrMsizer^rJrLr8 ValueError)rQrP labels_rankrUr`s` @r_maybe_expand_labelsrms  ~~dOfk-BC!Ou  = =f EF&-445 YY ..nn[)nnV//014 6   !O!O ""$**K$..066  % { *+!O!O, {Q .&&vr>/!O!O0&v'7'7'9&:; **,-.A AB B 99y~~k2 ~~f-1 3=~ O=!O!O!OsBF=:>F=F=+BF==Gc|jjd|jjdtj|||S)a Divides two values, returning 0 if the denominator is 0. Args: numerator: A scalar `float64` `Tensor`. denominator: A scalar `float64` `Tensor`. name: Name for the returned op. Returns: 0 if `denominator` == 0, else `numerator` / `denominator` r2rc)rJwith_rank_at_mostr div_no_nan) numerator denominatorr's r_safe_scalar_divrssG ))!, ++A.   Y $ ??r!ct||gtjd}tj|tj }tj|tj }tj|tj }|j jdkDrtj|dg}|j jdkDrtj|dg}|4|j jdkDrtj|dg}tj||||tj}tj||}||fS)a>Calculate a streaming confusion matrix. Calculates a confusion matrix. For estimation over a stream of data, the function creates an `update_op` operation. Args: labels: A `Tensor` of ground truth labels with shape [batch size] and of type `int32` or `int64`. The tensor will be flattened if its rank > 1. predictions: A `Tensor` of prediction results for semantic labels, whose shape is [batch size] and type `int32` or `int64`. The tensor will be flattened if its rank > 1. num_classes: The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). Returns: total_cm: A `Tensor` representing the confusion matrix. update_op: An operation that increments the confusion matrix. total_confusion_matrixrcr2r3)r:r) r0rfloat64r castint64rJrLrreshaper r assign_add)rQrP num_classesr:total_cm current_cm update_ops r_streaming_confusion_matrixrs%0K &..7OQ( k6<<8+ == .& k6<<8+""Q&##K"6K !   vt ,F  1 1 3 9 9A =".G 00 k;v~~O*""8Z8) 9 r!c\fd}tjj||S)z'Aggregate metric value across replicas.ct|jdr|jj(tjd5|g|}dddn[|jjj |g|}|jjj n |g|}rtjS#1swY#xYw)z;Call `metric_value_fn` in the correct control flow context._outer_control_flow_contextN)hasattrextendedrrcontrol_dependenciesEnterExitadd_to_collections) distributiona metric_valuemetric_value_fnmetrics_collectionss rfnz&_aggregate_across_replicas..fns|$$&CD    : : B  % %d + ;(::, ; ; 99??A&|8a8 99>>@%\6A6l 0,?  ; ;s  C  C)args)rget_replica_context merge_call)rrrrs`` r_aggregate_across_replicasrs0:  + + - 8 8t 9 r!z metrics.mean)v1c:tjr tdtj|d||f5t j |t j}tgt jd}tgt jd}|8t j tj|t j}nut|d|\}}}tjt j |t j|}t j||}t j|}t!j"|t j|} t%j&|g5t!j"||} dddd} t)|| ||} t j*| t j, d d } |rt%j.|| | | fcdddS#1swYkxYw#1swYyxYw) aComputes the (weighted) mean of the given values. The `mean` function creates two local variables, `total` and `count` that are used to compute the average of `values`. This average is ultimately returned as `mean` which is an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `mean`. `update_op` increments `total` with the reduced sum of the product of `values` and `weights`, and it increments `count` with the reduced sum of `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: values: A `Tensor` of arbitrary dimensions. weights: Optional `Tensor` whose rank is either 0, or the same rank as `values`, and must be broadcastable to `values` (i.e., all dimensions must be either `1`, or the same as the corresponding `values` dimension). metrics_collections: An optional list of collections that `mean` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: mean: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `mean_value`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `values`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. @compatibility(TF2) `tf.compat.v1.metrics.mean` is not compatible with eager execution or `tf.function`. Please use `tf.keras.metrics.Mean` instead for TF2 migration. After instantiating a `tf.keras.metrics.Mean` object, you can first call the `update_state()` method to record the new values, and then call the `result()` method to get the mean eagerly. You can also attach it to a Keras model with the `add_metric` method. Please refer to the [migration guide](https://www.tensorflow.org/guide/migrate#new-style_metrics_and_losses) for more details. #### Structural Mapping to TF2 Before: ```python mean, update_op = tf.compat.v1.metrics.mean( values=values, weights=weights, metrics_collections=metrics_collections, update_collections=update_collections, name=name) ``` After: ```python m = tf.keras.metrics.Mean( name=name) m.update_state( values=values, sample_weight=weights) mean = m.result() ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :-------------------- | :-------------- | :------------------------- | | `values` | `values` | In `update_state()` method | | `weights` | `sample_weight` | In `update_state()` method | | `metrics_collections` | Not supported | Metrics should be tracked | : : : explicitly or with Keras : : : : APIs, for example, : : : : [add_metric][add_metric], : : : : instead of via collections : | `updates_collections` | Not supported | - | | `name` | `name` | In constructor | [add_metric]:https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer#add_metric #### Before & After Usage Example Before: >>> g = tf.Graph() >>> with g.as_default(): ... values = [1, 2, 3] ... mean, update_op = tf.compat.v1.metrics.mean(values) ... global_init = tf.compat.v1.global_variables_initializer() ... local_init = tf.compat.v1.local_variables_initializer() >>> sess = tf.compat.v1.Session(graph=g) >>> sess.run([global_init, local_init]) >>> sess.run(update_op) >>> sess.run(mean) 2.0 After: >>> m = tf.keras.metrics.Mean() >>> m.update_state([1, 2, 3]) >>> m.result().numpy() 2.0 ```python # Used within Keras model model.add_metric(tf.keras.metrics.Mean()(values)) ``` @end_compatibility zAtf.metrics.mean is not supported when eager execution is enabled.meantotalrccountNrPrQr:cZtj|tj|ddSNrvaluercr rpmaximum_tcs r compute_meanzmean..compute_means$  H$4$4Q$: IIr!rr~)rexecuting_eagerly RuntimeErrorrr rwrfloat32r0rrjrWrbroadcast_weightsmultiply reduce_sumrrzrrrrprrvaluesr:rupdates_collectionsr'rr num_valuesrupdate_total_opupdate_count_oprmean_tr~s rrr;s@  % &&$$T6FG3DE ]]66>> 2F BW =E BW =E==!7Hj8T7<fa%77 -- 0&:g  1f&&w/j**5(2E2Ef2MNO ! !6( +@!,,UJ?o@J(\5%9F##))/1=KQI 0)< 9 ?"@@#s%E HHA HH HHzmetrics.accuracyctjr tdt|||\}}}|j j |j |j |j k7r tj||j }tjtj||tj}t|||||xsdS)aCalculates how often `predictions` matches `labels`. The `accuracy` function creates two local variables, `total` and `count` that are used to compute the frequency with which `predictions` matches `labels`. This frequency is ultimately returned as `accuracy`: an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `accuracy`. Internally, an `is_correct` operation computes a `Tensor` with elements 1.0 where the corresponding elements of `predictions` and `labels` match and 0.0 otherwise. Then `update_op` increments `total` with the reduced sum of the product of `weights` and `is_correct`, and it increments `count` with the reduced sum of `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose shape matches `predictions`. predictions: The predicted values, a `Tensor` of any shape. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `accuracy` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: accuracy: A `Tensor` representing the accuracy, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `accuracy`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. @compatibility(TF2) `tf.compat.v1.metrics.accuracy` is not compatible with eager execution or `tf.function`. Please use `tf.keras.metrics.Accuracy` instead for TF2 migration. After instantiating a `tf.keras.metrics.Accuracy` object, you can first call the `update_state()` method to record the prediction/labels, and then call the `result()` method to get the accuracy eagerly. You can also attach it to a Keras model when calling the `compile` method. Please refer to [this guide](https://www.tensorflow.org/guide/migrate#new-style_metrics_and_losses) for more details. #### Structural Mapping to Native TF2 Before: ```python accuracy, update_op = tf.compat.v1.metrics.accuracy( labels=labels, predictions=predictions, weights=weights, metrics_collections=metrics_collections, update_collections=update_collections, name=name) ``` After: ```python m = tf.keras.metrics.Accuracy( name=name, dtype=None) m.update_state( y_true=labels, y_pred=predictions, sample_weight=weights) accuracy = m.result() ``` #### How to Map Arguments | TF1 Arg Name | TF2 Arg Name | Note | | :-------------------- | :-------------- | :------------------------- | | `label` | `y_true` | In `update_state()` method | | `predictions` | `y_true` | In `update_state()` method | | `weights` | `sample_weight` | In `update_state()` method | | `metrics_collections` | Not supported | Metrics should be tracked | : : : explicitly or with Keras : : : : APIs, for example, : : : : [add_metric][add_metric], : : : : instead of via collections : | `updates_collections` | Not supported | - | | `name` | `name` | In constructor | [add_metric]:https://www.tensorflow.org/api_docs/python/tf/keras/layers/Layer#add_metric #### Before & After Usage Example Before: >>> g = tf.Graph() >>> with g.as_default(): ... logits = [1, 2, 3] ... labels = [0, 2, 3] ... acc, acc_op = tf.compat.v1.metrics.accuracy(logits, labels) ... global_init = tf.compat.v1.global_variables_initializer() ... local_init = tf.compat.v1.local_variables_initializer() >>> sess = tf.compat.v1.Session(graph=g) >>> sess.run([global_init, local_init]) >>> print(sess.run([acc, acc_op])) [0.0, 0.66667] After: >>> m = tf.keras.metrics.Accuracy() >>> m.update_state([1, 2, 3], [0, 2, 3]) >>> m.result().numpy() 0.66667 ```python # Used within Keras model model.compile(optimizer='sgd', loss='mse', metrics=[tf.keras.metrics.Accuracy()]) ``` @end_compatibility zEtf.metrics.accuracy is not supported when eager execution is enabled.raccuracy) rrrrWrJrKrr rwr>rrr)rQrPr:rrr' is_corrects rrrsZ  / 00"?fg"?+vw 33F4D4D4FG \\[&&&-- V\\:K}}nn[&)6>>;* j'#68K j ""r!c  d}||}n|D]}||vstd|tjtj|t j d|jdtj|t j d|jdg5tt j |tjt j |tj| \}}}dddt|}tj|d d g}tjt j |tjd d g} |j!j#d } | tj$|d } tj&tj(tj*|d gt-j.d | g} t j0tj&tj2||d g| } d |vsd|vrt j4| } tj&| |d g}d|vsd|vrt j4|}|t7j8t j |tj|}tj&tj|d d g|d g}| j!j;|j!nd}i}i}d|vrt=|gtjd}t j t j>|| tj}|||z}tAjB|t jD|d |d<||d<d |vrt=|gtjd}t j t j>| tj}|||z}tAjB|t jD|d |d <||d <d|vrt=|gtjd}t j t j> tj}|||z}tAjB|t jD|d |d<||d<d|vrt=|gtjd}t j t j>| tj}|||z}tAjB|t jD|d |d<||d<||fS#1swYoxYw)aNComputes true_positives, false_negatives, true_negatives, false_positives. This function creates up to four local variables, `true_positives`, `true_negatives`, `false_positives` and `false_negatives`. `true_positive[i]` is defined as the total weight of values in `predictions` above `thresholds[i]` whose corresponding entry in `labels` is `True`. `false_negatives[i]` is defined as the total weight of values in `predictions` at most `thresholds[i]` whose corresponding entry in `labels` is `True`. `true_negatives[i]` is defined as the total weight of values in `predictions` at most `thresholds[i]` whose corresponding entry in `labels` is `False`. `false_positives[i]` is defined as the total weight of values in `predictions` above `thresholds[i]` whose corresponding entry in `labels` is `False`. For estimation of these metrics over a stream of data, for each metric the function respectively creates an `update_op` operation that updates the variable and returns its value. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). includes: Tuple of keys to return, from 'tp', 'fn', 'tn', fp'. If `None`, default to all four. Returns: values: Dict of variables of shape `[len(thresholds)]`. Keys are from `includes`. update_ops: Dict of operations that increments the `values`. Keys are from `includes`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if `includes` contains invalid keys. )tprtnfpNz Invalid key: rzpredictions must be in [0, 1])message?rr3r2rrrrrtrue_positivesrcfalse_negativestrue_negativesfalse_positives)#rkrrr assert_greater_equalr rwrassert_less_equalrWrrboollenrryrJas_listrtiler8constantr stackgreater transpose logical_notrrrKr0 logical_andrrzr)rQrP thresholdsr:includes all_includesincludenum_thresholdspredictions_2d labels_2dnum_predictions thresh_tiled pred_is_pos pred_is_neg label_is_pos label_is_neg weights_tiledr update_opstrue_pis_true_positivefalse_nis_false_negativetrue_nis_true_negativefalse_pis_false_positives r_confusion_matrix_at_thresholdsr}s^*, H4  $= 2334 $$  --;#4#4 513!!  --;#4#4 513 !  $AMM+v~~>}}V6;;7$ K z?.$$[2q':.mmF&++.B9)#,,.668;/oon5a8OI..z:QC@Q013,   nnY((8>1:MN+ hDH,&&{3K NA+>?, hDH,'' 5L #55 gv~~. =GNN'Ar7+na-@BM66!#M &* X  &../?AF}}\;7I -' ++F,4,?,?0@!-EFJtF4L X &../@BG \;7I =( ++G,4,?,?0A1-FGJtF4L X  &../?AF}}\;7I -' ++F,4,?,?0@!-EFJtF4L X &../@BG \;7I =( ++G,4,?,?0A1-FGJtF4L  S  s "AU''U1c"d}t|||S)Nc8|jj|Sr)rread_var)rrs rr z%_aggregate_variable..!s,"7"7"@"@"Gr!)r)vr#fs r_aggregate_variabler sG! #KA 66r!z metrics.auczzThe value of AUC returned by this may race with the update so this is deprecated. Please use tf.keras.metrics.AUC instead.c `tjr tdtj|d|||f5dk7rdk7rt ddd} | t | } t | d zn'td z D cgc]} | d zd zd z z } } d | z g| zd | zgz} t||| |\} } d fdfdfd}t||| }| d| d| d| dd}|rtj||||fcdddScc} w#1swYyxYw)aComputes the approximate AUC via a Riemann sum. The `auc` function creates four local variables, `true_positives`, `true_negatives`, `false_positives` and `false_negatives` that are used to compute the AUC. To discretize the AUC curve, a linearly spaced set of thresholds is used to compute pairs of recall and precision values. The area under the ROC-curve is therefore computed using the height of the recall values by the false positive rate, while the area under the PR-curve is the computed using the height of the precision values by the recall. This value is ultimately returned as `auc`, an idempotent operation that computes the area under a discretized curve of precision versus recall values (computed using the aforementioned variables). The `num_thresholds` variable controls the degree of discretization with larger numbers of thresholds more closely approximating the true AUC. The quality of the approximation may vary dramatically depending on `num_thresholds`. For best results, `predictions` should be distributed approximately uniformly in the range [0, 1] and not peaked around 0 or 1. The quality of the AUC approximation may be poor if this is not the case. Setting `summation_method` to 'minoring' or 'majoring' can help quantify the error in the approximation by providing lower or upper bound estimate of the AUC. The `thresholds` parameter can be used to manually specify thresholds which split the predictions more evenly. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `auc`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). num_thresholds: The number of thresholds to use when discretizing the roc curve. metrics_collections: An optional list of collections that `auc` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. curve: Specifies the name of the curve to be computed, 'ROC' [default] or 'PR' for the Precision-Recall-curve. name: An optional variable_scope name. summation_method: Specifies the Riemann summation method used (https://en.wikipedia.org/wiki/Riemann_sum): 'trapezoidal' [default] that applies the trapezoidal rule; 'careful_interpolation', a variant of it differing only by a more correct interpolation scheme for PR-AUC - interpolating (true/false) positives but not the ratio that is precision; 'minoring' that applies left summation for increasing intervals and right summation for decreasing intervals; 'majoring' that does the opposite. Note that 'careful_interpolation' is strictly preferred to 'trapezoidal' (to be deprecated soon) as it applies the same method for ROC, and a better one (see Davis & Goadrich 2006 for details) for the PR curve. thresholds: An optional list of floating point values to use as the thresholds for discretizing the curve. If set, the `num_thresholds` parameter is ignored. Values should be in [0, 1]. Endpoint thresholds equal to {-epsilon, 1+epsilon} for a small positive epsilon value will be automatically included with these to correctly handle predictions equal to exactly 0 or 1. Returns: auc: A scalar `Tensor` representing the current area-under-curve. update_op: An operation that increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` variables appropriately and whose value matches `auc`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. z@tf.metrics.auc is not supported when eager execution is enabled.aucROCPRz&Curve must be either ROC or PR. Curve z is unknown.Hz>Nr2rrgư>c |ddz |ddz }||z}tj|tj|ddz |ddz dd}|ddtj||ddz }t j tj |ddz dkD|dddkDtj|ddz tj|ddddt j|dd}tjtj|||tj|zzztj|dd|ddzdddS) aInterpolation formula inspired by section 4 of (Davis et al., 2006). Note here we derive & use a closed formula not present in the paper - as follows: Modeling all of TP (true positive weight), FP (false positive weight) and their sum P = TP + FP (positive weight) as varying linearly within each interval [A, B] between successive thresholds, we get Precision = (TP_A + slope * (P - P_A)) / P with slope = dTP / dP = (TP_B - TP_A) / (P_B - P_A). The area within the interval is thus (slope / total_pos_weight) times int_A^B{Precision.dP} = int_A^B{(TP_A + slope * (P - P_A)) * dP / P} int_A^B{Precision.dP} = int_A^B{slope * dP + intercept * dP / P} where intercept = TP_A - slope * P_A = TP_B - slope * P_B, resulting in int_A^B{Precision.dP} = TP_B - TP_A + intercept * log(P_B / P_A) Bringing back the factor (slope / total_pos_weight) we'd put aside, we get slope * [dTP + intercept * log(P_B / P_A)] / total_pos_weight where dTP == TP_B - TP_A. Note that when P_A == 0 the above calculation simplifies into int_A^B{Precision.dTP} = int_A^B{slope * dTP} = slope * (TP_B - TP_A) which is really equivalent to imputing constant precision throughout the first bucket having >0 true positives. Args: tp: true positive counts fp: false positive counts fn: false negative counts Returns: pr_auc: an approximation of the area under the P-R curve. References: The Relationship Between Precision-Recall and ROC Curves: [Davis et al., 2006](https://dl.acm.org/citation.cfm?id=1143874) ([pdf](https://www.biostat.wisc.edu/~page/rocpr.pdf)) Nr2r prec_slopercrecall_relative_ratiopr_auc_incrementinterpolate_pr_auc) r rprrrwherer ones_likerlog) rrrdtppr intercept safe_p_ratiors rrzauc..interpolate_pr_aucs}J "" #bf ,c r'a&&   10nq01AabE91 =jQR&8,,Z12??i__   q!4.1"4591QR519 E   #!#$qua(* ,.7-@-@12-G Il    C)hll<.H"HHIr!"v122% '$ %%r!c H dk(r*dk(rtjdndk(r  |||Stj| z||z z} dk(r!tj|||z z}|}|}n#tj| z||z z} |}| }dvrFtjtj |d dz |ddz |d dz |ddzd z | Sd k(rTtjtj |d dz |ddz tj |d dz |dd| Sd k(rTtjtj |d dz |ddz tj|d dz |dd| Std d)z9Computes the roc-auc or pr-auc based on confusion counts.r trapezoidalziTrapezoidal rule is known to produce incorrect PR-AUCs; please switch to "careful_interpolation" instead.careful_interpolationr)rrNr2g@rcminoringmajoringzInvalid summation_method: z^ summation_method should be 'trapezoidal', 'careful_interpolation', 'minoring', or 'majoring'.) loggingwarningr dividerrminimumrrk)rrrrr'recfp_ratexypreccurveepsilonrrsummation_methods r compute_auczauc..compute_aucs $ } , //B C!8 8#BB/ / OOBL"r'G*; ?   E E""   a 3!!34qu< !4.1"45!"=C E  z )""   a 3!!34qu<&..q1D.12D/EquM O  z )""   a 3!!34qu<&..q1D.12D/EquM O  56F5GH))* *r!c4|d|d|d|ddS)Nrrrrrr<)rrr s rcompute_auc_valuezauc..compute_auc_values+ vd|VD\6$< ""r!rrrrr~) rrrrrksortedrrangerrrr)rQrPr:rrrr r'r rkepsilonirrr auc_valuer~r r rs ` ` ` @@@rrr%sv  % &&$$T5&,k7%CEK  ~%4- ?wG"" ##H*%j:*n #>A#568UcM^a%788j8 .!J.#.1AAJ8 Z2FJG7%r(*(*V"+.8IJt,j.>&t,j.> MI 0)< i WK K 8K K s AD$DA8D$D$$D-zmetrics.mean_absolute_errorctjr tdt|||\}}}t j ||z }t |||||xsdS)a=Computes the mean absolute error between the labels and predictions. The `mean_absolute_error` function creates two local variables, `total` and `count` that are used to compute the mean absolute error. This average is weighted by `weights`, and it is ultimately returned as `mean_absolute_error`: an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `mean_absolute_error`. Internally, an `absolute_errors` operation computes the absolute value of the differences between `predictions` and `labels`. Then `update_op` increments `total` with the reduced sum of the product of `weights` and `absolute_errors`, and it increments `count` with the reduced sum of `weights` If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` of the same shape as `predictions`. predictions: A `Tensor` of arbitrary shape. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `mean_absolute_error` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: mean_absolute_error: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `mean_absolute_error`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zPtf.metrics.mean_absolute_error is not supported when eager execution is enabled.rmean_absolute_error)rrrrWr absr)rQrPr:rrr'absolute_errorss rrrsqd  : ;;"?fg"?+vwLLv!56/ ow(;!4#@+@ BBr!zmetrics.mean_cosine_distancectjr tdt|||\}}}t j ||}t j ||gd}t||dd|xsd\}} t jd|}t jd| } |rtj|||rtj|| || fS)arComputes the cosine distance between the labels and predictions. The `mean_cosine_distance` function creates two local variables, `total` and `count` that are used to compute the average cosine distance between `predictions` and `labels`. This average is weighted by `weights`, and it is ultimately returned as `mean_distance`, which is an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `mean_distance`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` of arbitrary shape. predictions: A `Tensor` of the same shape as `labels`. dim: The dimension along which the cosine distance is computed. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). Also, dimension `dim` must be `1`. metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: mean_distance: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zQtf.metrics.mean_cosine_distance is not supported when eager execution is enabled.rT)axiskeepdimsNmean_cosine_distancer) rrrrWr rrrsubtractrr) rQrPdimr:rrr' radial_diffs mean_distancer~s rrrOsb  5 66"?fg"?+vw"";7,$$ ,",tTF9"8:-##C7-Y/). >. :  !!r!zmetrics.mean_per_class_accuracyctjr tdtj|d|||f5t j |t j}|jjdkDrtj|dg}|jjdkDrtj|dg}|jj|jt|gt jd}t|gt jd}tjtj |gt j} |j"|j"k7r t j ||j"}t j t j$||t j} |b|jjdkDrtj|dg}t j |t j}| |z} | |z} t'j(||| } t'j(||| } d } t+|| ||}t j,| t j.| d d }|rt1j2||||fcdddS#1swYyxYw) a2Calculates the mean of the per-class accuracies. Calculates the accuracy for each class, then takes the mean of that. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates the accuracy of each class and returns them. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` of ground truth labels with shape [batch size] and of type `int32` or `int64`. The tensor will be flattened if its rank > 1. predictions: A `Tensor` of prediction results for semantic labels, whose shape is [batch size] and type `int32` or `int64`. The tensor will be flattened if its rank > 1. num_classes: The possible number of labels the prediction task can have. This value must be provided, since two variables with shape = [num_classes] will be allocated. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `mean_per_class_accuracy' should be added to. updates_collections: An optional list of collections `update_op` should be added to. name: An optional variable_scope name. Returns: mean_accuracy: A `Tensor` representing the mean per class accuracy. update_op: An operation that updates the accuracy tensor. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zTtf.metrics.mean_per_class_accuracy is not supported when eager execution is enabled. mean_accuracyr2r3rrcrNctj|tj|dd}tj|d}|S)Nrrcr!)r rpr reduce_mean)rrrper_class_accuracymean_accuracy_vs rcompute_mean_accuracyz6mean_per_class_accuracy..compute_mean_accuracysB#.. !!%+$8 ,, ?4o r!rr~)rrrrr rwrrxrJrLrryrKr0ronesrjrr>r scatter_addrrprrr)rQrPr{r:rrr'rrr'rrrr&r%r~s rmean_per_class_accuracyr)sW`  : ;;$$T?&167%CE2& ]]66<< 0F!#  ".f$$q(%%kB48k55f6F6F6HI 6>> HE 6>> HE >>9>>&12FNN CD ||{(((MM+v|| 1. predictions: A `Tensor` of prediction results for semantic labels, whose shape is [batch size] and type `int32` or `int64`. The tensor will be flattened if its rank > 1. num_classes: The possible number of labels the prediction task can have. This value must be provided, since a confusion matrix of dimension = [num_classes, num_classes] will be allocated. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `mean_iou` should be added to. updates_collections: An optional list of collections `update_op` should be added to. name: An optional variable_scope name. Returns: mean_iou: A `Tensor` representing the mean intersection-over-union. update_op: An operation that increments the confusion matrix. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zEtf.metrics.mean_iou is not supported when eager execution is enabled.mean_iouc*tjtj|dtj}tjtj|dtj}tjt j |tj}||z|z }tjtjtj|dtj}t jtj|d|t j|}tj||}t jtj|dtj|d|z d}|S)zBCompute the mean intersection-over-union via the confusion matrix.rr2rr+rc) r rwrrrr diag_part not_equalrrrr) rr| sum_over_row sum_over_colcm_diagrrnum_valid_entriesiouresults rcompute_mean_iouz"mean_iou..compute_mean_iouBs2]]   h *FNN zeros_likerrr)rQrP normalizerr:rrr'relative_errorss rr8r8ish  5 66"?fg"?+vw-II:+z 33J4H4H4JKOOnnZ%y';';F'Coohll6K#78*EG/ ow(;!4#@+@ BBr!zmetrics.mean_squared_errorctjr tdt|||\}}}t j ||}t |||||xsdS)a5Computes the mean squared error between the labels and predictions. The `mean_squared_error` function creates two local variables, `total` and `count` that are used to compute the mean squared error. This average is weighted by `weights`, and it is ultimately returned as `mean_squared_error`: an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `mean_squared_error`. Internally, a `squared_error` operation computes the element-wise square of the difference between `predictions` and `labels`. Then `update_op` increments `total` with the reduced sum of the product of `weights` and `squared_error`, and it increments `count` with the reduced sum of `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` of the same shape as `predictions`. predictions: A `Tensor` of arbitrary shape. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `mean_squared_error` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: mean_squared_error: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `mean_squared_error`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zOtf.metrics.mean_squared_error is not supported when eager execution is enabled.rmean_squared_error)rrrrWr squared_differencer)rQrPr:rrr' squared_errors rr=r=sod  5 66"?fg"?+vw--fkB- mW&9;N** ,,r!zmetrics.mean_tensorctjr tdtj|d||f5t j |t j}t|jt jd}t|jt jd}tj|}|vt|d|\}}}tjt j |t j|}t j||}t j||}t!j"||} t%j&|g5t!j"||} dddd} t)|| ||} t j*| t j, d d } |rt%j.|| | | fcdddS#1swYkxYw#1swYyxYw) awComputes the element-wise (weighted) mean of the given tensors. In contrast to the `mean` function which returns a scalar with the mean, this function returns an average tensor with the same shape as the input tensors. The `mean_tensor` function creates two local variables, `total_tensor` and `count_tensor` that are used to compute the average of `values`. This average is ultimately returned as `mean` which is an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `mean`. `update_op` increments `total` with the reduced sum of the product of `values` and `weights`, and it increments `count` with the reduced sum of `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: values: A `Tensor` of arbitrary dimensions. weights: Optional `Tensor` whose rank is either 0, or the same rank as `values`, and must be broadcastable to `values` (i.e., all dimensions must be either `1`, or the same as the corresponding `values` dimension). metrics_collections: An optional list of collections that `mean` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: mean: A float `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `mean_value`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `values`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zHtf.metrics.mean_tensor is not supported when eager execution is enabled.r total_tensorrc count_tensorNrcZtj|tj|ddSrrrs rr zmean_tensor..2s%8#6#6 8  Aq !$1r!rr~)rrrrr rwrrr0rJrrrWrrrrrzrrrrprrrs r mean_tensorrDs^  5 66$$T6FG3DE ]]66>> 2F FNN AE FNN AE$$V,J8T7<fa%77 -- 0&:g  1f$$Z9j**5&9O ! !6( +@!,,UJ?o@1L(\5%9F##))/1=KQI 0)< 9 ?"@@#s%D2G8+G,A G8,G5 1G88Hzmetrics.percentage_belowctjr tdtjtj ||t j}t|||||xsdS)aComputes the percentage of values less than the given threshold. The `percentage_below` function creates two local variables, `total` and `count` that are used to compute the percentage of `values` that fall below `threshold`. This rate is weighted by `weights`, and it is ultimately returned as `percentage` which is an idempotent operation that simply divides `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `percentage`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: values: A numeric `Tensor` of arbitrary size. threshold: A scalar threshold. weights: Optional `Tensor` whose rank is either 0, or the same rank as `values`, and must be broadcastable to `values` (i.e., all dimensions must be either `1`, or the same as the corresponding `values` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: percentage: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `values`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zMtf.metrics.percentage_below is not supported when eager execution is enabled.percentage_below_threshold) rrrr rwlessrrr)r thresholdr:rrr'is_below_thresholds rpercentage_belowrJ@sjZ  5 66 }}mmFI&8  '+>!4#G+G IIr!c tj|tjt gtj d}t j|tj }|tjtj|dtj|ff5t j|tj }t j||}dddt||}tj |t j"|}|rtj$||||fS#1swYZxYw)aSums the weights of cases where the given values are True. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: values: A `bool` `Tensor` of arbitrary size. weights: Optional `Tensor` whose rank is either 0, or the same rank as `values`, and must be broadcastable to `values` (i.e., all dimensions must be either `1`, or the same as the corresponding `values` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. Returns: value_tensor: A `Tensor` representing the current value of the metric. update_op: An operation that accumulates the error from a batch of data. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `values`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. rrcNr)r assert_typerrr0rr rwrrassert_rank_inrrMrrrrzrr)rr:rrr value_tensorr~s r_count_conditionrOws6  , "fnn7 ;% == 0&  ! !9#;#;!Y^^F+,$.#0 12 gv~~6g  1f2 %U,?@,""5(*=*=f*EF). : y  22s &;D::Ezmetrics.false_negativesctjr tdtj|d|||f5t t j |tjt j |tj|\}}}t jt j|dt j|d}t||||cdddS#1swYyxYw)aComputes the total number of false negatives. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: value_tensor: A `Tensor` representing the current value of the metric. update_op: An operation that accumulates the error from a batch of data. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `values`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zLtf.metrics.false_negatives is not supported when eager execution is enabled.rrrTFN rrrrrWr rwrrrr>rO)rQrPr:rrr'rs rrrsF  5 66$$T+<&167%CE 1$AMM+V[[A}}V6;;7$ K!,,vt$hnn[%&HJ -w8K/ 1 1 1 1 B$C((C1z%metrics.false_negatives_at_thresholdsc(tjr tdtj|d|||f5t ||||d\}}t |d|} |rt j||d| |dfcdddS#1swYyxYw)atComputes false negatives at provided threshold values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `false_negatives` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: false_negatives: A float `Tensor` of shape `[len(thresholds)]`. update_op: An operation that updates the `false_negatives` variable and returns its current value. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zZtf.metrics.false_negatives_at_thresholds is not supported when eager execution is enabled.r)rr:rrNrrrrrrrr) rQrPrr:rrr'rrfn_values rfalse_negatives_at_thresholdsrWN  D EE$$T+<&167%CE &8 Z7LFJ#6$<1DEH 0*T2BC Z% % & & & ABBzmetrics.false_positivesctjr tdtj|d|||f5t t j |tjt j |tj|\}}}t jt j|dt j|d}t||||cdddS#1swYyxYw)a+Sum the weights of false positives. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: value_tensor: A `Tensor` representing the current value of the metric. update_op: An operation that accumulates the error from a batch of data. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zLtf.metrics.false_positives is not supported when eager execution is enabled.rrrFTNrQ)rQrPr:rrr'rs rrrsH  5 66$$T+<&167%CE 1$AMM+V[[A}}V6;;7$ K!,,vu%x~~k4'HJ -w8K/ 1 1 1 1rRz%metrics.false_positives_at_thresholdsc(tjr tdtj|d|||f5t ||||d\}}t |d|} |rt j||d| |dfcdddS#1swYyxYw)atComputes false positives at provided threshold values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `false_positives` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: false_positives: A float `Tensor` of shape `[len(thresholds)]`. update_op: An operation that updates the `false_positives` variable and returns its current value. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zZtf.metrics.false_positives_at_thresholds is not supported when eager execution is enabled.r)rrTrNrU) rQrPrr:rrr'rrfp_values rfalse_positives_at_thresholdsr]FrXrYzmetrics.true_negativesctjr tdtj|d|||f5t t j |tjt j |tj|\}}}t jt j|dt j|d}t||||cdddS#1swYyxYw)a*Sum the weights of true_negatives. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: value_tensor: A `Tensor` representing the current value of the metric. update_op: An operation that accumulates the error from a batch of data. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zKtf.metrics.true_negatives is not supported when eager execution is enabled.rrrFNrQ)rQrPr:rrr'rs rrr~sH  D EE$$T+;&167%CE 1$AMM+V[[A}}V6;;7$ K ++vu%x~~k5'IK ,g7J/ 1 1 1 1rRz$metrics.true_negatives_at_thresholdsc(tjr tdtj|d|||f5t ||||d\}}t |d|} |rt j||d| |dfcdddS#1swYyxYw)apComputes true negatives at provided threshold values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `true_negatives` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: true_negatives: A float `Tensor` of shape `[len(thresholds)]`. update_op: An operation that updates the `true_negatives` variable and returns its current value. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zYtf.metrics.true_negatives_at_thresholds is not supported when eager execution is enabled.r)rrTrNrU) rQrPrr:rrr'rrtn_values rtrue_negatives_at_thresholdsraN  D EE$$T+;&167%CE &8 Z7LFJ#6$<1DEH 0*T2BC Z% % & & &rYzmetrics.true_positivesctjr tdtj|d|||f5t t j |tjt j |tj|\}}}t jt j|dt j|d}t||||cdddS#1swYyxYw)a*Sum the weights of true_positives. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that the metric value variable should be added to. updates_collections: An optional list of collections that the metric update ops should be added to. name: An optional variable_scope name. Returns: value_tensor: A `Tensor` representing the current value of the metric. update_op: An operation that accumulates the error from a batch of data. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zKtf.metrics.true_positives is not supported when eager execution is enabled.rrrTNrQ)rQrPr:rrr'rs rrrsH  D EE$$T+;&167%CE 1$AMM+V[[A}}V6;;7$ K ++vt$hnn[$&GI ,g7J/ 1 1 1 1rRz$metrics.true_positives_at_thresholdsc(tjr tdtj|d|||f5t ||||d\}}t |d|} |rt j||d| |dfcdddS#1swYyxYw)apComputes true positives at provided threshold values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` whose shape matches `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `true_positives` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: true_positives: A float `Tensor` of shape `[len(thresholds)]`. update_op: An operation that updates the `true_positives` variable and returns its current value. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zYtf.metrics.true_positives_at_thresholds is not supported when eager execution is enabled.r)rrTrNrU) rQrPrr:rrr'rrtp_values rtrue_positives_at_thresholdsrf rbrYzmetrics.precisionc  tjr tdtj|d|||f5t t j |tjt j |tj|\}}}t|||ddd\}}t|||ddd\}} d fd} t|| ||}  || d } |rtj|| | | fcdddS#1swYyxYw) aComputes the precision of the predictions with respect to the labels. The `precision` function creates two local variables, `true_positives` and `false_positives`, that are used to compute the precision. This value is ultimately returned as `precision`, an idempotent operation that simply divides `true_positives` by the sum of `true_positives` and `false_positives`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `precision`. `update_op` weights each prediction by the corresponding value in `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `precision` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: precision: Scalar float `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_positives`. update_op: `Operation` that increments `true_positives` and `false_positives` variables appropriately and whose value matches `precision`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zFtf.metrics.precision is not supported when eager execution is enabled. precisionrrNrrr'ctjtj||zdtj|||zd|SNrrrr rr)rrr's rcompute_precisionz$precision..compute_precisions> __   27A &BG(DaOOr!c||dSNrr<)rrrrms ronce_across_replicasz'precision..once_across_replicass vw 88r!r~)rrrrrWr rwrrrrrrr)rQrPr:rrr'rtrue_positives_update_oprfalse_positives_update_oprprr~rms @rrhrhXs1d  D EE$$T;&167%CE&$AMM+V[[A}}V6;;7$ K (6   ($F $*9   *&G &O9 ##68L#)7 4A"":";[JI 0)< i.compute_precisions$ __R2!29L MMr!c$|d|ddS)Nrrrr<)rrrms rprecision_across_replicasz:precision_at_thresholds..precision_across_replicass vd|VD\7 CCr!rrr~Nrrrrrrrr)rQrPrr:rrr'rrr{rr~rmr s @@rrurush  D EE$$T+D&167%CE8 Z<IFJGND &6 @D"*T"2Jt4D"-/I 0)< ?- ABB'zmetrics.recallc  tjr tdtj|d|||f5t t j |tjt j |tj|\}}}t|||ddd\}}t|||ddd\}} d fd} t|| ||}  || d } |rtj|| | | fcdddS#1swYyxYw) aComputes the recall of the predictions with respect to the labels. The `recall` function creates two local variables, `true_positives` and `false_negatives`, that are used to compute the recall. This value is ultimately returned as `recall`, an idempotent operation that simply divides `true_positives` by the sum of `true_positives` and `false_negatives`. For estimation of the metric over a stream of data, the function creates an `update_op` that updates these variables and returns the `recall`. `update_op` weights each prediction by the corresponding value in `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: The predicted values, a `Tensor` of arbitrary dimensions. Will be cast to `bool`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `recall` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: recall: Scalar float `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_negatives`. update_op: `Operation` that increments `true_positives` and `false_negatives` variables appropriately and whose value matches `recall`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zTtf.metrics.recall is not supported is not supported when eager execution is enabled.recallrrNrictjtj||zdtj|||zd|Srkrl)rrr's rcompute_recallzrecall..compute_recallR s? __   6G+Q / //&&7"2 3Q>>r!c||dSror<)rrrrs rrpz$recall..once_across_replicasW s FGW 55r!r~)rrrrrWr rwrrrrrrr)rQrPr:rrr'rrqrfalse_negatives_update_oprprr~rs @rrr s-`  D EE$$T8&167%CE&#@MM+V[[A}}V6;;7$ K (6   ($F $*9   *&G &> 6 %167 DC78+GI 0)<  >M&&&rsc6|d||fz}nd|z}|d||fz}|S)Nz%s_at_%dz%s_at_kz %s_class%dr<)r'kclass_ids r _at_k_namere s9] q !D  D  4* *D +r!ctj|}t|tjr4t j |t j|j|Stj|tj}tj|dz }t j|tj|dg}tj |t j"|tj}t%j&||}tj|j(|j|S)aFilter all but `selected_id` out of `ids`. Args: ids: `int64` `Tensor` or `SparseTensor` of IDs. selected_id: Int id to select. Returns: `SparseTensor` of same dimensions as `ids`. This contains only the entries equal to `selected_id`. )out_typer2indicesrr^)rrgrhrir sparse_retainr r>rrrrrxrj reduced_shaperyfillrwrset_intersectionr)ids selected_id ids_shape ids_last_dimfilled_selected_id_shapefilled_selected_idr4s r_select_class_idro s 88=#]//0  # #C 8C*E FFoocFLL9) *Q.,%33I4=4E4E8Dqc5KL !~~&>&.mmK&NP  !3S 9&  # #nnV]]  KKr!c>|||fSt||t||fS)aIf class ID is specified, filter all other classes. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: `int64` `Tensor` of class IDs, with shape [D1, ... DN, k] where N >= 1. Commonly, N=1 and `predictions_idx` has shape [batch size, k]. selected_id: Int id to select. Returns: Tuple of `labels` and `predictions_idx`, possibly with classes removed. )r)rQpredictions_idxrs r_maybe_select_class_idr s3" ? "" 6; / ?K 8 ::r!ctj|d|||f5t|||\}}tjtj ||}t j|tj}|mtjtj||f5t j|tj}t j||}ddd|cdddS#1swYxYw#1swYyxYw)aCalculates true positives for recall@k and precision@k. If `class_id` is specified, calculate binary true positives for `class_id` only. If `class_id` is not specified, calculate metrics for `k` predicted vs `n` label classes, where `n` is the 2nd dimension of `labels_sparse`. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). name: Name of operation. Returns: A [D1, ... DN] `Tensor` of true positive counts. rN)rrfrrset_sizerr rwrrvrrassert_broadcastabler)rQrrr:r'rs r_sparse_true_positive_at_kr s> ~~d,&8: 4V_5=?FO t,,_fE FB r6>> *B  # #%:%O%O 2&% ,--8   r7 +,   ,,  s$B C?%;C3 C?3C< 8C??Dcrtj|td|||||f5}t||||}t j t j |tj}tgtj|} | tj| |dfcdddS#1swYyxYw)aCalculates weighted per step true positives for recall@k and precision@k. If `class_id` is specified, calculate binary true positives for `class_id` only. If `class_id` is not specified, calculate metrics for `k` predicted vs `n` label classes, where `n` is the 2nd dimension of `labels`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. k: Integer, k for @k metric. This is only used for default op name. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). name: Name of new variable, and namespace for other dependent ops. Returns: A tuple of `Variable` and update `Operation`. Raises: ValueError: If `weights` is not `None` and has an incompatible shape. true_positiverrrQrr:rcupdateN) rrfrrr rwrrrvr0rrz) rQrrrr:r'r`rbatch_total_tpvars r$_streaming_sparse_true_positive_at_kr sL ~~dJHM&8: I=B #'  B ]]8#6#6r#:FNNKN "fnn5 9C  $$S.xH H I I I A= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). Returns: A [D1, ... DN] `Tensor` of false negative counts. NrFaminusbrrfrrrset_differencer rwrrvrrrr)rQrrr:rs r_sparse_false_negative_at_kr s: ~~d-&8: 4V_5=?FO  OVUC EB r6>> *B  # #%:%O%O 2&% ,--8   r7 +,   ,,  $B D';C5" D5C> :DD crtj|td|||||f5}t||||}t j t j |tj}tgtj|} | tj| |dfcdddS#1swYyxYw)aCalculates weighted per step false negatives for recall@k. If `class_id` is specified, calculate binary true positives for `class_id` only. If `class_id` is not specified, calculate metrics for `k` predicted vs `n` label classes, where `n` is the 2nd dimension of `labels`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. k: Integer, k for @k metric. This is only used for default op name. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). name: Name of new variable, and namespace for other dependent ops. Returns: A tuple of `Variable` and update `Operation`. Raises: ValueError: If `weights` is not `None` and has an incompatible shape. false_negativerrrcrN) rrfrrr rwrrrvr0rrz) rQrrrr:r'r`rbatch_total_fnrs r%_streaming_sparse_false_negative_at_kr4 L ~~dJ'7XN&8: I=B $'  B ]]8#6#6r#:FNNKN "fnn5 9C  $$S.xH H I I Irzmetrics.recall_at_kc tjr tdtj|t d|||||f5}t j||\} } t|| ||||||cdddS#1swYyxYw)a Computes recall@k of the predictions with respect to sparse labels. If `class_id` is specified, we calculate recall by considering only the entries in the batch for which `class_id` is in the label, and computing the fraction of them for which `class_id` is in the top-k `predictions`. If `class_id` is not specified, we'll calculate recall as how often on average a class among the labels of a batch entry is in the top-k `predictions`. `sparse_recall_at_k` creates two local variables, `true_positive_at_` and `false_negative_at_`, that are used to compute the recall_at_k frequency. This frequency is ultimately returned as `recall_at_`: an idempotent operation that simply divides `true_positive_at_` by total (`true_positive_at_` + `false_negative_at_`). For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `recall_at_`. Internally, a `top_k` operation computes a `Tensor` indicating the top `k` `predictions`. Set operations applied to `top_k` and `labels` calculate the true positives and false negatives weighted by `weights`. Then `update_op` increments `true_positive_at_` and `false_negative_at_` using these values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions`. Values should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. Values outside this range always count towards `false_negative_at_`. predictions: Float `Tensor` with shape [D1, ... DN, num_classes] where N >= 1. Commonly, N=1 and predictions has shape [batch size, num_classes]. The final dimension contains the logit values for each class. [D1, ... DN] must match `labels`. k: Integer, k for @k metric. class_id: Integer class ID for which we want binary metrics. This should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. If class_id is outside this range, the method returns NAN. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: recall: Scalar `float64` `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_negatives`. update_op: `Operation` that increments `true_positives` and `false_negatives` variables appropriately, and whose value matches `recall`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zHtf.metrics.recall_at_k is not supported when eager execution is enabled.rrrQrrrr:rrr'N) rrrrrfrrtop_krecall_at_top_k rQrPrrr:rrr'r`r top_k_idxs r recall_at_krg sV  D EE ~~dJxXF"FG46 9>88K+LAy ! //      ,A<<Bzmetrics.recall_at_top_kctj|td|||||f5t||}t j |t j}t|||||\} } t|||||\} } fd} t|| | | }t j| t j| | d}|rtj||||fcdddS#1swYyxYw)aK Computes recall@k of top-k predictions with respect to sparse labels. Differs from `recall_at_k` in that predictions must be in the form of top `k` class indices, whereas `recall_at_k` expects logits. Refer to `recall_at_k` for more details. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions`. Values should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. Values outside this range always count towards `false_negative_at_`. predictions_idx: Integer `Tensor` with shape [D1, ... DN, k] where N >= 1. Commonly, N=1 and predictions has shape [batch size, k]. The final dimension contains the top `k` predicted class indices. [D1, ... DN] must match `labels`. k: Integer, k for @k metric. Only used for the default op name. class_id: Integer class ID for which we want binary metrics. This should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. If class_id is outside this range, the method returns NAN. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: recall: Scalar `float64` `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_negatives`. update_op: `Operation` that increments `true_positives` and `false_negatives` variables appropriately, and whose value matches `recall`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. rrrrQrrr:c\tj|tj||SNrcr radd)rrrr`s rrz'recall_at_top_k..compute_recall ! __Rb"!5E BBr!rrcN)rrfrrmr rwrrxrrrrrr)rQrrrr:rrr'rr tp_updater fn_updatermetricrr`s @rrr sl ~~dJxXF&8:=B !&/ :F ov||7s B1C##C,zmetrics.recall_at_thresholdscT tjr tdtj|d|||f5t ||||d\}}d fd fd} t || |}  |d|d d } |rt j|| | | fcd d d S#1swYy xYw) aMComputes various recall values for different `thresholds` on `predictions`. The `recall_at_thresholds` function creates four local variables, `true_positives`, `true_negatives`, `false_positives` and `false_negatives` for various values of thresholds. `recall[i]` is defined as the total weight of values in `predictions` above `thresholds[i]` whose corresponding entry in `labels` is `True`, divided by the total weight of `True` values in `labels` (`true_positives[i] / (true_positives[i] + false_negatives[i])`). For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `recall`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. thresholds: A python list or tuple of float thresholds in `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `recall` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: recall: A float `Tensor` of shape `[len(thresholds)]`. update_op: An operation that increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` variables that are used in the computation of `recall`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zQtf.metrics.recall_at_thresholds is not supported when eager execution is enabled.recall_at_thresholds)rrrvrcFtj||z|zd|zS)Nrecall_rcry)rrr'r s rrz,recall_at_thresholds..compute_recallV s$ __R2!2T9I JJr!c$|d|ddS)Nrrrr<)rrrs rrecall_across_replicasz4recall_at_thresholds..recall_across_replicasY s F4L&, @@r!rrr~Nr|)rQrPrr:rrr'rrrrr~rr s @@rrr sd  D EE$$T+A&167%CE8 Z<IFJGKA %3V =Cz$/D1A;OI 0)<  >+r}zmetrics.root_mean_squared_errorc tjr tdt|||\}}}t |||dd|xsd\}}d}t |||} t j|} |rtj|| | | fS)awComputes the root mean squared error between the labels and predictions. The `root_mean_squared_error` function creates two local variables, `total` and `count` that are used to compute the root mean squared error. This average is weighted by `weights`, and it is ultimately returned as `root_mean_squared_error`: an idempotent operation that takes the square root of the division of `total` by `count`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `root_mean_squared_error`. Internally, a `squared_error` operation computes the element-wise square of the difference between `predictions` and `labels`. Then `update_op` increments `total` with the reduced sum of the product of `weights` and `squared_error`, and it increments `count` with the reduced sum of `weights`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: A `Tensor` of the same shape as `predictions`. predictions: A `Tensor` of arbitrary shape. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that `root_mean_squared_error` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: root_mean_squared_error: A `Tensor` representing the current mean, the value of `total` divided by `count`. update_op: An operation that increments the `total` and `count` variables appropriately and whose value matches `root_mean_squared_error`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, or if `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zTtf.metrics.root_mean_squared_error is not supported when eager execution is enabled.rNroot_mean_squared_errorc,tj|Sr)r sqrt)rmses rr z)root_mean_squared_error.. s c(:r!) rrrrWr=rr rrr) rQrPr:rrr'r update_mse_oprprmseupdate_rmse_ops rrrf sd  D EE"?fg"?+vw)&+w*.1D*CE#}; #/ 6$==/..? ~ r!z"metrics.sensitivity_at_specificityctjr tddksdkDrtddt j|d|||f5dt |dz Dcgc]}|dzd z|dz z } }d z g| zd zgz} t ||| |\} } fd fd } t|| | } | d | d| d| dd}|rtj||| |fcdddScc}w#1swYyxYw)a Computes the specificity at a given sensitivity. The `sensitivity_at_specificity` function creates four local variables, `true_positives`, `true_negatives`, `false_positives` and `false_negatives` that are used to compute the sensitivity at the given specificity value. The threshold for the given specificity value is computed and used to evaluate the corresponding sensitivity. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `sensitivity`. `update_op` increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` counts with the weight of each case found in the `predictions` and `labels`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. For additional information about specificity and sensitivity, see the following: https://en.wikipedia.org/wiki/Sensitivity_and_specificity Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. specificity: A scalar value in range `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). num_thresholds: The number of thresholds to use for matching the given specificity. metrics_collections: An optional list of collections that `sensitivity` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: sensitivity: A scalar `Tensor` representing the sensitivity at the given `specificity` value. update_op: An operation that increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` variables appropriately and whose value matches `sensitivity`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, if `weights` is not `None` and its shape doesn't match `predictions`, or if `specificity` is not between 0 and 1, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zWtf.metrics.sensitivity_at_specificity is not supported when eager execution is enabled.rr2zH`specificity` must be in the range [0, 1]. Currently, `specificity` got .sensitivity_at_specificityrrrrc(tj|||zz}tjtj|z d}tj|t j }tj||||||zz|Srk)r rargminrrwrint32) rrrrr' specificitiestf_indexr specificitys r"compute_sensitivity_at_specificityzFsensitivity_at_specificity..compute_sensitivity_at_specificity soob"r'H*<=mmk.I!JANhx6h__R\\BxL88CTKKr!c4|d|d|d|ddSNrrrrrr<)rrrs rsensitivity_across_replicasz?sensitivity_at_specificity..sensitivity_across_replicas - / ,t fTlF4L'KKr!rrrrr~N rrrrkrrrrrr)rQrPrr:rrrr'rrrrr sensitivityr~rrs ` @@rrr sqv  D EE1_ a **5a9 ::$$T+G&167%CE!"H6;NQNI 0)<  !C!"!"!"!"C;)C6>A.C;6C;;Dc |dkrtd|dtj|d|||f5}tj|}t |tj r|dkr8tjtj|j|zdg}n|g}tjtj|jdg|dgtj|j|dgfdd}tj||d }|dk(r |cd d d Stj|dkr|dz n||g|z|cd d d Stj ||dk\r|n|dz d }|dk(r |cd d d Stj"tj$|}tj|d ||f||d fdd } tj&|| |cd d d S#1swYy xYw) a/Slice `tensor` shape in 2, then tile along the sliced dimension. A new dimension is inserted in shape of `tensor` before `dim`, then values are tiled `multiple` times along the new dimension. Args: tensor: Input `Tensor` or `SparseTensor`. multiple: Integer, number of times to tile. dim: Integer, dimension along which to tile. name: Name of operation. Returns: `Tensor` result of expanding and tiling `tensor`. Raises: ValueError: if `multiple` is less than 1, or `dim` is not in `[-rank(tensor), rank(tensor)]`. r2zInvalid argument multiple=z= for expand_and_tile call. `multiple` must be an integer > 0expand_and_tilerr3expanded_shapercexpandr[N multiples)rkrrfrrgrhrirryrjr^r]slicerr\ sparse_concatr8rrr) tensormultiplerr'r`r8rexpandedr'tile_multipless r_expand_and_tiler s &\ 1(>(N ?=@@@sC*G/#%G/(G/A!G//G8c F|dkrtd|tjdd|f5}tj|}t |tj r4tjtj|||cdddStjtjtj|dtj|tj |d}tj|||cdddS#1swYyxYw) aComputes number of relevant values for each row in labels. For labels with shape [D1, ... DN, num_labels], this is the minimum of `num_labels` and `k`. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. k: Integer, k for @k metric. Returns: Integer `Tensor` of shape [D1, ... DN], where each value is the number of relevant values for that row. Raises: ValueError: if inputs have invalid dtypes or values. r2 Invalid k=N num_relevantrcrr3)r)rkrrfrrgrhrir rrrrrwhere_v2 greater_equalrr9)rQrr` num_labelss r _num_relevantrK s(U z!% && ~~dNVI6 7%  = =f EF&-445   dmmF3QU C 7 7$$811&!<$..v6$//7 9 J   J 6 7 7 7sADA=DD cHtjdd||f5}tj|tj d}|j jdk(r td|j jd}| tdt||}tj|dd }t||dd }t||d }tj|dd }tjtj |dd}tj"tj|tj$tj|tj$d} tj&| tj|tj$d} tj(| dd} tjt+||tj$} tj"| | |cdddS#1swYyxYw)aComputes average precision@k of predictions with respect to sparse labels. From en.wikipedia.org/wiki/Information_retrieval#Average_precision, formula for each row is: AveP = sum_{i=1...k} P_{i} * rel_{i} / num_relevant_items A "row" is the elements in dimension [D1, ... DN] of `predictions_idx`, `labels`, and the result `Tensors`. In the common case, this is [batch_size]. Each row of the results contains the average precision for that row. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. Values should be non-negative. Negative values are ignored. predictions_idx: Integer `Tensor` with shape [D1, ... DN, k] where N >= 1. Commonly, N=1 and `predictions_idx` has shape [batch size, k]. The final dimension must be set and contains the top `k` predicted class indices. [D1, ... DN] must match `labels`. Values should be in range [0, num_classes). Returns: `float64` `Tensor` of shape [D1, ... DN], where each value is the average precision for that row. Raises: ValueError: if the last dimension of predictions_idx is not set. Naverage_precisionrrcrz1The rank of `predictions_idx` must be at least 1.r3zIThe last dimension of predictions_idx must be set. Currently, it is None.predictions_idx_per_k labels_per_k)rrr'relevant_per_ktp_per_k)rr'retrieved_per_kprecision_per_krelevant_precision_per_k)r3 precision_sum)rrfr rwrrxrJrLrkrrmrr8rrcumsumrrrvrrr) rQrr`rrrrrrrrrnum_relevant_itemss r"_sparse_average_precision_at_top_krq s@ ~~d/&/17J49mm,=?O  "((A- J KK!!#++-b1Ay 0 11 !&/ :F &11"9;$9L0+2BDN~BZHHooN+";LNOoo h/ ov~~6  O (00 nfnn5 ' ) '' u?DM "}VQ'?P ??=*<5 Io7J7J7Js G4HH!ctj|d|||f5}t||}|Ntjt j |tj|}t j||}tjdd|f5}tgtj|} |:t j tj|dtj} nt j|d} tj| | d} dddtjdd|f5} tgtj| } t j|d }tj| |d}dddd }t!||  }t# |}|rtj$||||fcdddS#1swYxYw#1swYYxYw#1swYyxYw) aX Computes average precision@k of predictions with respect to sparse labels. `sparse_average_precision_at_top_k` creates two local variables, `average_precision_at_/total` and `average_precision_at_/max`, that are used to compute the frequency. This frequency is ultimately returned as `average_precision_at_`: an idempotent operation that simply divides `average_precision_at_/total` by `average_precision_at_/max`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `precision_at_`. Set operations applied to `top_k` and `labels` calculate the true positives and false positives weighted by `weights`. Then `update_op` increments `true_positive_at_` and `false_positive_at_` using these values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. Values should be non-negative. Negative values are ignored. predictions_idx: Integer `Tensor` with shape [D1, ... DN, k] where N >= 1. Commonly, N=1 and `predictions_idx` has shape [batch size, k]. The final dimension contains the top `k` predicted class indices. [D1, ... DN] must match `labels`. Values should be in range [0, num_classes). weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: mean_average_precision: Scalar `float64` `Tensor` with the mean average precision values. update: `Operation` that increments variables appropriately, and whose value matches `metric`. average_precision_at_top_k)rrQNmaxrc batch_maxrr batch_totalct||dS)Nrrc)rs)r total_varmax_vars rr{zO_streaming_sparse_average_precision_at_top_k..precision_across_replicas s iv >>r!)rrfrrrr rwrrvrr0rrjrrrzrrsr)rQrr:rrr'r`r max_scoper r max_update total_scoper r total_updater{mean_average_precisionrs r,_streaming_sparse_average_precision_at_top_kr sd ~~d8&8:(*=B:'8%77 -- 02CEg"++,=wG e&7%9 : Ki  FNNCg MM NN,; ?Q ''kB ''Jj K g(9'; <Q !"fnn;Gi''(9 Nk)))[xPlQ ?86 7LlJU CF 0&9 !6 )Q(*(* K KQQ1(*(*s?A6G)BG G)9A GAG)G G)G& "G))G2cfdfdfd}tjr jn}tjtj||fdS)aReplaces large out-of-range labels by small out-of-range labels. Replaces any value in `labels` that is greater or equal to `num_classes` by -1. Do this conditionally for efficiency in case there are no such values. Args: labels: `int64` `Tensor` or `SparseTensor`. num_classes: `int64` scalar `Tensor`. Returns: An `int64` `Tensor` or `SparseTensor` as `labels` with indices greater or equal to num_classes replaced by -1. cXttjtjfS)z,Returns true is `labels` is a sparse tensor.)rhrriSparseTensorValuerbsr_labels_is_sparsez6_clean_out_of_range_indices.._labels_is_sparse6 s+ f}99,>>@ AAr!ctjtj|dtj|z|S)z/Replaces by -1 any large out-of-range `values`.r3)rrr rr)rr{s r_clean_out_of_rangez8_clean_out_of_range_indices.._clean_out_of_range; s<   h44V[I 9#6#6v#>> HHr!cr8tjjjSS)z9Replaces by -1 ane large out-of-range values in `labels`.r)typerrr^)rrrQsr_clean_labels_out_of_rangez?_clean_out_of_range_indices.._clean_labels_out_of_range@ sD T&\&..!4V]]!C&,&8&8::! ((r!cSrr<rbsrr z-_clean_out_of_range_indices..N sfr!)r reduce_maxrr r)rQr{r max_labelsrrs`` @@r_clean_out_of_range_indicesr( s[A H )""(*fmm8* Z5  r!z%metrics.sparse_average_precision_at_kz"Use average_precision_at_k insteadc &t|||||||S)zDRenamed to `average_precision_at_k`, please use that method instead.rQrPrr:rrr')average_precision_at_kr s rsparse_average_precision_at_kr"Q s&  --  r!zmetrics.average_precision_at_kc tjr td|dkrtd|dt j |t d||||f5}tj||\}} t|tjtj|dtj}t!|| ||||cdddS#1swYyxYw) a Computes average precision@k of predictions with respect to sparse labels. `average_precision_at_k` creates two local variables, `average_precision_at_/total` and `average_precision_at_/max`, that are used to compute the frequency. This frequency is ultimately returned as `average_precision_at_`: an idempotent operation that simply divides `average_precision_at_/total` by `average_precision_at_/max`. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `precision_at_`. Internally, a `top_k` operation computes a `Tensor` indicating the top `k` `predictions`. Set operations applied to `top_k` and `labels` calculate the true positives and false positives weighted by `weights`. Then `update_op` increments `true_positive_at_` and `false_positive_at_` using these values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions`. Values should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. Values outside this range are ignored. predictions: Float `Tensor` with shape [D1, ... DN, num_classes] where N >= 1. Commonly, N=1 and `predictions` has shape [batch size, num_classes]. The final dimension contains the logit values for each class. [D1, ... DN] must match `labels`. k: Integer, k for @k metric. This will calculate an average precision for range `[1,k]`, as documented above. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: mean_average_precision: Scalar `float64` `Tensor` with the mean average precision values. update: `Operation` that increments variables appropriately, and whose value matches `metric`. Raises: ValueError: if k is invalid. RuntimeError: If eager execution is enabled. zZtf.metrics.sparse_average_precision_at_k is not supported when eager execution is enabled.r2rz. `k` should be >= 1.rr3)rQrr:rrr'N)rrrrkrrfrrrrr rwrrrrxr) rQrPrr:rrr'r`rrs rr!r!e sx  D EEU z!$9: ;; ~~dJ':A>"FG469>+q1A ) iook:2> MOF 7'//   s A.CCctjdd|||f5t|||\}}tjtj ||d}t j|tj}|mtjtj||f5t j|tj}t j||}ddd|cdddS#1swYxYw#1swYyxYw)a~Calculates false positives for precision@k. If `class_id` is specified, calculate binary true positives for `class_id` only. If `class_id` is not specified, calculate metrics for `k` predicted vs `n` label classes, where `n` is the 2nd dimension of `labels_sparse`. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). Returns: A [D1, ... DN] `Tensor` of false positive counts. NrTrr)rQrrr:rs r_sparse_false_positive_at_kr% s: ~~d-&8: 4V_5=?FO  OVTB DB r6>> *B  # #%:%O%O 2&% ,--8   r7 +,   ,,  rcrtj|td|||||f5}t||||}t j t j |tj}tgtj|} | tj| |dfcdddS#1swYyxYw)aCalculates weighted per step false positives for precision@k. If `class_id` is specified, calculate binary true positives for `class_id` only. If `class_id` is not specified, calculate metrics for `k` predicted vs `n` label classes, where `n` is the 2nd dimension of `labels`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels], where N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions_idx`. predictions_idx: 1-D or higher `int64` `Tensor` with last dimension `k`, top `k` predicted classes. For rank `n`, the first `n-1` dimensions must match `labels`. k: Integer, k for @k metric. This is only used for default op name. class_id: Class for which we want binary metrics. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). name: Name of new variable, and namespace for other dependent ops. Returns: A tuple of `Variable` and update `Operation`. Raises: ValueError: If `weights` is not `None` and has an incompatible shape. false_positiverrrcrN) rrfrr%r rwrrrvr0rrz) rQrrrr:r'r`rbatch_total_fprs r%_streaming_sparse_false_positive_at_kr) rrzmetrics.precision_at_top_kctjr tdtj|t d|||||f5t ||}tj|tj}t|||||\} } t|||||\} } fd} t|| | | }tj| tj| | d}|rtj ||||fcdddS#1swYyxYw) a Computes precision@k of the predictions with respect to sparse labels. Differs from `sparse_precision_at_k` in that predictions must be in the form of top `k` class indices, whereas `sparse_precision_at_k` expects logits. Refer to `sparse_precision_at_k` for more details. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions`. Values should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. Values outside this range are ignored. predictions_idx: Integer `Tensor` with shape [D1, ... DN, k] where N >= 1. Commonly, N=1 and predictions has shape [batch size, k]. The final dimension contains the top `k` predicted class indices. [D1, ... DN] must match `labels`. k: Integer, k for @k metric. Only used for the default op name. class_id: Integer class ID for which we want binary metrics. This should be in range [0, num_classes], where num_classes is the last dimension of `predictions`. If `class_id` is outside this range, the method returns NAN. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: precision: Scalar `float64` `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_positives`. update_op: `Operation` that increments `true_positives` and `false_positives` variables appropriately, and whose value matches `precision`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zOtf.metrics.precision_at_top_k is not supported when eager execution is enabled.rhrrc\tj|tj||Srr)rrrr`s rr{z5precision_at_top_k..precision_across_replicaserr!rrcN)rrrrrfrrmr rwrrxrr)rrrr)rQrrrr:rrr'rrrr fp_updater{rrr`s @rprecision_at_top_kr-sn  D EE ~~dJ{AI&8:=B !&/ :F ov||7s B1DD zmetrics.sparse_precision_at_kzUse precision_at_k insteadc (t||||||||S)z` and `false_positive_at_`, that are used to compute the precision@k frequency. This frequency is ultimately returned as `precision_at_`: an idempotent operation that simply divides `true_positive_at_` by total (`true_positive_at_` + `false_positive_at_`). For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `precision_at_`. Internally, a `top_k` operation computes a `Tensor` indicating the top `k` `predictions`. Set operations applied to `top_k` and `labels` calculate the true positives and false positives weighted by `weights`. Then `update_op` increments `true_positive_at_` and `false_positive_at_` using these values. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. Args: labels: `int64` `Tensor` or `SparseTensor` with shape [D1, ... DN, num_labels] or [D1, ... DN], where the latter implies num_labels=1. N >= 1 and num_labels is the number of target classes for the associated prediction. Commonly, N=1 and `labels` has shape [batch_size, num_labels]. [D1, ... DN] must match `predictions`. Values should be in range [0, num_classes), where num_classes is the last dimension of `predictions`. Values outside this range are ignored. predictions: Float `Tensor` with shape [D1, ... DN, num_classes] where N >= 1. Commonly, N=1 and predictions has shape [batch size, num_classes]. The final dimension contains the logit values for each class. [D1, ... DN] must match `labels`. k: Integer, k for @k metric. class_id: Integer class ID for which we want binary metrics. This should be in range [0, num_classes], where num_classes is the last dimension of `predictions`. If `class_id` is outside this range, the method returns NAN. weights: `Tensor` whose rank is either 0, or n-1, where n is the rank of `labels`. If the latter, it must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). metrics_collections: An optional list of collections that values should be added to. updates_collections: An optional list of collections that updates should be added to. name: Name of new update operation, and namespace for other dependent ops. Returns: precision: Scalar `float64` `Tensor` with the value of `true_positives` divided by the sum of `true_positives` and `false_positives`. update_op: `Operation` that increments `true_positives` and `false_positives` variables appropriately, and whose value matches `precision`. Raises: ValueError: If `weights` is not `None` and its shape doesn't match `predictions`, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zRtf.metrics.sparse_precision_at_k is not supported when eager execution is enabled.rhrrN) rrrrrfrrrr-rs rr0r0sX  D EE ~~dJ{AI"FG46 9>88K+LAy ! //     rz"metrics.specificity_at_sensitivityctjr tddksdkDrtddt j|d|||f5dt |dz Dcgc]}|dzd z|dz z } }d z g| zd z gz} t ||| |\} } fd fd } t|| | } | d | d| d| dd}|rtj||| |fcdddScc}w#1swYyxYw)a Computes the specificity at a given sensitivity. The `specificity_at_sensitivity` function creates four local variables, `true_positives`, `true_negatives`, `false_positives` and `false_negatives` that are used to compute the specificity at the given sensitivity value. The threshold for the given sensitivity value is computed and used to evaluate the corresponding specificity. For estimation of the metric over a stream of data, the function creates an `update_op` operation that updates these variables and returns the `specificity`. `update_op` increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` counts with the weight of each case found in the `predictions` and `labels`. If `weights` is `None`, weights default to 1. Use weights of 0 to mask values. For additional information about specificity and sensitivity, see the following: https://en.wikipedia.org/wiki/Sensitivity_and_specificity Args: labels: The ground truth values, a `Tensor` whose dimensions must match `predictions`. Will be cast to `bool`. predictions: A floating point `Tensor` of arbitrary shape and whose values are in the range `[0, 1]`. sensitivity: A scalar value in range `[0, 1]`. weights: Optional `Tensor` whose rank is either 0, or the same rank as `labels`, and must be broadcastable to `labels` (i.e., all dimensions must be either `1`, or the same as the corresponding `labels` dimension). num_thresholds: The number of thresholds to use for matching the given sensitivity. metrics_collections: An optional list of collections that `specificity` should be added to. updates_collections: An optional list of collections that `update_op` should be added to. name: An optional variable_scope name. Returns: specificity: A scalar `Tensor` representing the specificity at the given `sensitivity` value. update_op: An operation that increments the `true_positives`, `true_negatives`, `false_positives` and `false_negatives` variables appropriately and whose value matches `specificity`. Raises: ValueError: If `predictions` and `labels` have mismatched shapes, if `weights` is not `None` and its shape doesn't match `predictions`, or if `sensitivity` is not between 0 and 1, or if either `metrics_collections` or `updates_collections` are not a list or tuple. RuntimeError: If eager execution is enabled. zWtf.metrics.specificity_at_sensitivity is not supported when eager execution is enabled.rr2zG`sensitivity` must be in the range [0, 1]. Currently, `sensitivity` is rspecificity_at_sensitivityrrrrctj|||z z}tjtj| z }tjtj| z |}tj |t j}tj|}tj|d}tj |t j}tj||||||z z|S)a&Computes the specificity at the given sensitivity. Args: tp: True positives. tn: True negatives. fp: False positives. fn: False negatives. name: The name of the operation. Returns: The specificity using the aggregated values. r) r r reduce_minrr>rwrrxrargmaxr) rrrrr' sensitivitiesmin_valindices_at_minvalrrrs r"compute_specificity_at_sensitivityzFspecificity_at_sensitivity..compute_specificity_at_sensitivity4soob"r'H*<=m##HLL1L$MNg".. ,,}{2 3W>"--(96<<H"//*;<!2A6hx6h__R\\BxL88CTKKr!c4|d|d|d|ddSrr<)rrr;s rspecificity_across_replicasz?specificity_at_sensitivity..specificity_across_replicasQrr!rrrrr~Nr)rQrPrr:rrrr'rrrrr=rr~r;rs ` @@rr4r4sqv  D EE1_ a ))4 Q8 99$$T+G&167%CE5"H6;NQNI 0)<  !k5"5"5"5"r)TNr)NNNN)NN)NNNrNrN)NNN)NNNNN)NNNNNN)Nr>NNN)rN)W__doc__tensorflow.python.distributertensorflow.python.eagerrtensorflow.python.frameworkrrrtensorflow.python.opsrr r r r r rrrrrrrrtensorflow.python.platformrr"tensorflow.python.util.deprecationr tensorflow.python.util.tf_exportrr0rWrmrsrrrrrrrrrr)r+r8r=rDrJrOrrWrr]rrarrfrhrurrrrrrrrrrrrrrrrrrr"r!r%r)r-r1r0r4r<r!rrGso+7+.+5+1+&2*$&,+0-+7<96,^I&X1Oh @ .b F ~!! b bJ !"#!%!% X"$X"|-1-1 `F7  }o D KL    &f Lf R ,-.!%,0,0! 9B/9Bx -./"&-1-1" F"0F"R 012%)0404!% e&3e&P !"#!%!% c!$c!L ,-.!%,0,0! AB/ABH +,- $+/+/ 9,.9,x $%&$($( Q'Qh )*+")-)- 3I,3In")-)-+!\ ()*!(,(, 01+01f 678+/6:6:'+ 4&94&n ()*!(,(, 11+11h 678+/6:6:'+ 4&94&n '() '+'+ 11*11h 567*.5959&* 4&84&n '() '+'+ 11*11h 567*.5959&* 4&84&n "#$"&"& [%[| 012%)0404!% M3M`  !## Y"YxKD:2)-'+$( *^,02615.2 0Ij*.(,)^4826/3 0If $%&$($(Y'Yx ()*! (,(,P+Pf -./"&-1-1" J0JZ 012%)0404!% C3CL 345(,.13737$(c"6c"L4@n#7LWJx:>EIEI6: Z*z&R 678 D67+/6:6:'+ 89$ /01$(/3/3 $ Q2Ql*.(,)\-13726/3 0If +,- $#+/+/ U.Up ./0 D./$("&.2.2#01( '()!'+'+Z*Zz 345(,.13737$(w"6w"r!