package tflite.evaluation

Get desktop application:
View/edit binary Protocol Buffers messages

Statistics for an accuracy value over multiple runs of evaluation. Next ID: 5

Used in: InferenceProfilerMetrics

• optional float max_value = 1

  Maximum value observed for any Run.

• optional float min_value = 2

  Minimum value observed for any Run.

• optional double avg_value = 3

  Average value across all Runs.

• optional float std_deviation = 4

  Standard deviation across all Runs.

Contains parameters that define how an EvaluationStage will be executed. This would typically be validated only once during initialization, so should not contain any variables that change with each run. Next ID: 3

• optional string name = 1

• optional ProcessSpecification specification = 2

  Specification defining what this stage does, and any required parameters.

Metrics returned from EvaluationStage.LatestMetrics() need not have all fields set.

• optional int32 num_runs = 1

  Total number of times the EvaluationStage is run.

• optional ProcessMetrics process_metrics = 2

  Process-specific numbers such as latencies, accuracy, etc.

Metrics from evaluation of the image classification task. Next ID: 5

Used in: ProcessMetrics

• optional LatencyMetrics pre_processing_latency = 1

• optional LatencyMetrics inference_latency = 2

• optional TfliteInferenceMetrics inference_metrics = 3

• optional TopkAccuracyEvalMetrics topk_accuracy_metrics = 4

  Not set if topk_accuracy_eval_params was not populated in ImageClassificationParams.

Parameters that define how the Image Classification task is evaluated end-to-end. Next ID: 3

Used in: ProcessSpecification

• optional TfliteInferenceParams inference_params = 1

  Required. TfLite model should have 1 input & 1 output tensor. Input shape: {1, image_height, image_width, 3} Output shape: {1, num_total_labels}

• optional TopkAccuracyEvalParams topk_accuracy_eval_params = 2

  Optional. If not set, accuracy evaluation is not performed.

Parameters that define how images are preprocessed. Next ID: 5

Used in: ProcessSpecification

• optional int32 image_height = 1

  Required.

• optional int32 image_width = 2

  Required.

• optional int32 output_type = 3

  Same as tflite::TfLiteType.

• optional float cropping_fraction = 4

  Fraction for central-cropping. A central cropping-fraction of 0.875 is considered best for Inception models, hence the default value. See: https://github.com/tensorflow/tpu/blob/master/models/experimental/inception/inception_preprocessing.py#L296 Set to 0 to disable cropping.

Metrics computed from comparing TFLite execution in two settings: 1. User-defined TfliteInferenceParams (The 'test' setting) 2. Default TfliteInferenceParams (The 'reference' setting) Next ID: 4

Used in: ProcessMetrics

• optional LatencyMetrics reference_latency = 1

  Latency metrics from Single-thread CPU inference.

• optional LatencyMetrics test_latency = 2

  Latency from TfliteInferenceParams under test.

• repeated AccuracyMetrics output_errors = 3

  For reference & test output vectors {R, T}, the error is computed as: Mean([Abs(R[i] - T[i]) for i in num_elements]) output_errors[v] : statistics for the error value of the vth output vector across all Runs.

Latency numbers in microseconds, based on all EvaluationStage::Run() calls so far. Next ID: 7

Used in: ImageClassificationMetrics, InferenceProfilerMetrics, ObjectDetectionMetrics, ProcessMetrics

• optional int64 last_us = 1

  Latency for the last Run.

• optional int64 max_us = 2

  Maximum latency observed for any Run.

• optional int64 min_us = 3

  Minimum latency observed for any Run.

• optional int64 sum_us = 4

  Sum of all Run latencies.

• optional double avg_us = 5

  Average latency across all Runs.

• optional int64 std_deviation_us = 6

  Standard deviation for latency across all Runs.

Average Precision metrics from Object Detection task. Next ID: 3

Used in: ObjectDetectionMetrics, ProcessMetrics

• repeated ObjectDetectionAveragePrecisionMetrics.AveragePrecision individual_average_precisions = 1

  One entry for each in ObjectDetectionAveragePrecisionParams::iou_thresholds, averaged over all classes.

• optional float overall_mean_average_precision = 2

  Average of Average Precision across all IoU thresholds.

Average Precision value for a particular IoU threshold. Next ID: 3

Used in: ObjectDetectionAveragePrecisionMetrics

• optional float iou_threshold = 1

• optional float average_precision = 2

Parameters that define how Average Precision is computed for Object Detection task. Refer for details: http://cocodataset.org/#detection-eval Next ID: 4

Used in: ObjectDetectionParams, ProcessSpecification

• optional int32 num_classes = 1

  Total object classes. The AP value returned for each IoU threshold is an average over all classes encountered in predicted/ground truth sets.

• repeated float iou_thresholds = 2

  A predicted box matches a ground truth box if and only if IoU between these two are larger than an IoU threshold. AP is computed for all relevant {IoU threshold, class} combinations and averaged to get mAP. If left empty, evaluation is done for all IoU threshods in the range 0.5:0.05:0.95 (min:increment:max).

• optional int32 num_recall_points = 3

  AP is computed as the average of maximum precision at (1 + num_recall_points) recall levels. E.g., if num_recall_points is 10, recall levels are 0., 0.1, 0.2, ..., 0.9, 1.0. Default: 100

Proto containing ground-truth ObjectsSets for all images in a COCO validation set. Next ID: 2

• repeated ObjectDetectionResult detection_results = 1

Metrics from evaluation of the object detection task. Next ID: 5

Used in: ProcessMetrics

• optional LatencyMetrics pre_processing_latency = 1

• optional LatencyMetrics inference_latency = 2

• optional TfliteInferenceMetrics inference_metrics = 3

• optional ObjectDetectionAveragePrecisionMetrics average_precision_metrics = 4

Parameters that define how the Object Detection task is evaluated end-to-end. Next ID: 4

Used in: ProcessSpecification

• optional TfliteInferenceParams inference_params = 1

  Required. Model's outputs should be same as a TFLite-compatible SSD model. Refer: https://www.tensorflow.org/lite/models/object_detection/overview#output TODO(b/133772912): Generalize support for other types of object detection models.

• optional int32 class_offset = 2

  Optional. Used to match ground-truth categories with model output. SSD Mobilenet V1 Model trained on COCO assumes class 0 is background class in the label file and class labels start from 1 to number_of_classes+1. Therefore, default value is set as 1.

• optional ObjectDetectionAveragePrecisionParams ap_params = 3

Proto containing information about all the objects (predicted or ground-truth) contained in an image. Next ID: 3

Used in: ObjectDetectionGroundTruth

• repeated ObjectDetectionResult.ObjectInstance objects = 1

• optional string image_name = 2

  Required for ground-truth data, to compare against inference results.

One instance of an object detected in an image. Next ID: 4

Used in: ObjectDetectionResult

• optional int32 class_id = 1

  Required.

• optional ObjectInstance.NormalizedBoundingBox bounding_box = 2

  Required

• optional float score = 3

  Value in (0, 1.0] denoting confidence in this prediction. Default value of 1.0 for ground-truth data.

Defines the bounding box for a detected object. Next ID: 5

Used in: ObjectInstance

• optional float normalized_top = 1

  All boundaries defined below are required. Each boundary value should be normalized with respect to the image dimensions. This helps evaluate detections independent of image size. For example, normalized_top = top_boundary / image_height.

• optional float normalized_bottom = 2

• optional float normalized_left = 3

• optional float normalized_right = 4

Contains process-specific metrics, which may differ based on what an EvaluationStage does. Next ID: 8

Used in: EvaluationStageMetrics

• optional LatencyMetrics total_latency = 1

• oneof stage_metrics

  • TopkAccuracyEvalMetrics topk_accuracy_metrics = 2

  • TfliteInferenceMetrics tflite_inference_metrics = 3

  • ImageClassificationMetrics image_classification_metrics = 4

  • InferenceProfilerMetrics inference_profiler_metrics = 5

  • ObjectDetectionAveragePrecisionMetrics object_detection_average_precision_metrics = 6

  • ObjectDetectionMetrics object_detection_metrics = 7

Defines the functionality executed by an EvaluationStage. Next ID: 7

Used in: EvaluationStageConfig

• oneof params

  • ImagePreprocessingParams image_preprocessing_params = 1

  • TopkAccuracyEvalParams topk_accuracy_eval_params = 2

  • TfliteInferenceParams tflite_inference_params = 3

  • ImageClassificationParams image_classification_params = 4

  • ObjectDetectionAveragePrecisionParams object_detection_average_precision_params = 5

  • ObjectDetectionParams object_detection_params = 6

Metrics specific to TFLite inference. Next ID: 2

Used in: ImageClassificationMetrics, ObjectDetectionMetrics, ProcessMetrics

• optional int32 num_inferences = 1

  Number of times the interpreter is invoked.

Parameters that control TFLite inference. Next ID: 5

Used in: ImageClassificationParams, ObjectDetectionParams, ProcessSpecification

• optional string model_file_path = 1

  Required

• optional TfliteInferenceParams.Delegate delegate = 2

• optional int32 num_threads = 3

  Number of threads available to the TFLite Interpreter.

• optional int32 invocations_per_run = 4

  Defines how many times the TFLite Interpreter is invoked for every input. This helps benchmark cases where extensive pre-processing might not be required for every input.

Used in: TfliteInferenceParams

• NONE = 0

• NNAPI = 1

• GPU = 2

Metrics from top-K accuracy evaluation. Next ID: 2

Used in: ImageClassificationMetrics, ProcessMetrics

• repeated float topk_accuracies = 1

  A repeated field of size |k| where the ith element denotes the fraction of samples for which the correct label was present in the top (i + 1) model outputs. For example, topk_accuracies(1) will contain the fraction of samples for which the model returned the correct label as the top first or second output.

Parameters that define how top-K accuracy is evaluated. Next ID: 2

Used in: ImageClassificationParams, ProcessSpecification

• optional int32 k = 1

  Required.