package io.mrpowers.tsumugi.proto

Get desktop application:
View/edit binary Protocol Buffers messages

Used in: AnomalyDetectionStrategy

• optional double max_rate_decrease = 1

• optional double max_rate_increase = 2

• optional int32 order = 3

Used in: AnomalyDetection, Check.Constraint, VerificationSuite

• oneof analyzer

  Not all the analyzers are compatible with AnomalyDetection!

  • ApproxCountDistinct approx_count_distinct = 1

  • ApproxQuantile approx_quantile = 2

  • ApproxQuantiles approx_quantiles = 3

  • ColumnCount column_count = 4

  • Completeness completeness = 5

  • Compliance compliance = 6

  • Correlation correlation = 7

  • CountDistinct count_distinct = 8

  • CustomSql custom_sql = 9

  • DataType data_type = 10

  • Distinctness distinctness = 11

  • Entropy entropy = 12

  • ExactQuantile exact_quantile = 13

  • Histogram histogram = 14

  • KLLSketch kll_sketch = 15

  • MaxLength max_length = 16

  • Maximum maximum = 17

  • Mean mean = 18

  • MinLength min_length = 19

  • Minimum minimum = 20

  • MutualInformation mutual_information = 21

  • PatternMatch pattern_match = 22

  • RatioOfSums ratio_of_sums = 23

  • Size size = 24

  • StandardDeviation standard_deviation = 25

  • Sum sum = 26

  • UniqueValueRatio unique_value_ratio = 27

  • Uniqueness uniqueness = 28

Used in: Completeness, Compliance, MaxLength, Maximum, MinLength, Minimum, PatternMatch, UniqueValueRatio, Uniqueness

• AnalyzerOptions.NullBehaviour null_behaviour = 1

• AnalyzerOptions.FilteredRowOutcome filtered_row_outcome = 2

Used in: AnalyzerOptions

• NULL = 0

• TRUE = 1

Used in: AnalyzerOptions

• Ignore = 0

• EmptyString = 1

• Fail = 2

Used in: VerificationSuite

• optional AnomalyDetectionStrategy anomaly_detection_strategy = 1

• optional Analyzer analyzer = 2

• optional AnomalyDetection.AnomalyCheckConfig config = 3

Used in: AnomalyDetection

• CheckLevel level = 1

• string description = 2

• map<string, string> with_tag_values = 3

• optional int64 after_date = 4

• optional int64 before_date = 5

Used in: AnomalyDetection

• oneof strategy

  • AbsoluteChangeStrategy absolute_change_strategy = 1

  • BatchNormalStrategy batch_normal_strategy = 2

  • OnlineNormalStrategy online_normal_strategy = 3

  • RelativeRateOfChangeStrategy relative_rate_of_change_strategy = 4

  • SimpleThresholdStrategy simple_thresholds_strategy = 5

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• string column = 1

• double quantile = 2

• optional double relative_error = 3

• optional string where = 4

Used in: Analyzer

• string column = 1

• repeated double quantiles = 2

• optional double relative_error = 3

Used in: AnomalyDetectionStrategy

• optional double lower_deviation_factor = 1

• optional double upper_deviation_factor = 2

• optional bool include_interval = 3

Used in: VerificationSuite

• CheckLevel checkLevel = 1

• string description = 2

• repeated Check.Constraint constraints = 3

Used in: Constraint

• GT = 0

• GET = 1

• EQ = 2

• LT = 3

• LET = 4

Used in: Check

• optional Analyzer analyzer = 1

  In Deequ constraint is built from 1. Analyzer 2. Assertion that is lambda Double -> Bool or Long -> Bool 3. Hint (optional) 4. Name (optional, for a limited amount of Constraints) In this message there is the following: 1. Analyzer 2. Reference value (Double or Long) 3. Comparison sign (>, >=, ==, <, <=) 4. Hint 5. Name TODO: Think about how to extend this message to encode function?

• oneof expectation

  • int64 long_expectation = 2

  • double double_expectation = 3

• ComparisonSign sign = 4

• optional string hint = 5

• optional string name = 6

Used in: AnomalyDetection.AnomalyCheckConfig, Check

• Error = 0

• Warning = 1

Used in: Analyzer

(message has no fields)

Used in: Analyzer

• string column = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• string instance = 1

• string predicate = 2

• optional string where = 3

• repeated string columns = 4

• optional AnalyzerOptions options = 5

Used in: Analyzer

• string first_column = 1

• string second_column = 2

• optional string where = 3

Used in: Analyzer

• repeated string columns = 1

Used in: Analyzer

• string expressions = 1

TODO: This one looks broken; todo revisit the code of Deequ

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• repeated string columns = 1

• optional string where = 2

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• string column = 1

• double quantile = 2

• optional string where = 3

Used in: Repository

• string path = 1

Used in: Analyzer

• string column = 1

• optional int32 max_detail_bins = 2

  TODO: binningUdf; how to serialize it?

• optional string where = 3

• optional bool compute_frequencies_as_ratio = 4

• optional Histogram.AggregateFunction aggregate_function = 5

Used in: Histogram

• oneof aggregate_function

  • AggregateFunction.Count count_aggregate = 1

  • AggregateFunction.Sum sum_aggregate = 2

Used in: AggregateFunction

(message has no fields)

Used in: AggregateFunction

• string agg_column = 1

Used in: Analyzer

• string column = 1

• optional KLLSketch.KLLParameters kll_parameters = 2

Used in: KLLSketch

• int32 sketch_size = 1

• double shrinking_factor = 2

• int32 number_of_buckets = 3

Used in: Analyzer

• string column = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• string column = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• string column = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• string column = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• repeated string columns = 1

• optional string where = 2

Used in: AnomalyDetectionStrategy

• optional double lower_deviation_factor = 1

• optional double upper_deviation_factor = 2

• optional double ignore_start_percentage = 3

• optional bool ignore_anomalies = 4

Used in: Analyzer

• string column = 1

• string pattern = 2

  scala.util.matching.Regex in Deequ; passed as is

• optional string where = 3

• optional AnalyzerOptions options = 4

Used in: Analyzer

• string numerator = 1

• string denominator = 2

• optional string where = 3

Used in: AnomalyDetectionStrategy

• optional double max_rate_decrease = 1

• optional double max_rate_increase = 2

• optional int32 order = 3

Used in: VerificationSuite

• oneof repository

  • FileSystemRepository file_system = 1

  • SparkTableRepository spark_table = 2

Used in: VerificationSuite

• int64 dataset_date = 1

• map<string, string> tags = 2

Used in: AnomalyDetectionStrategy

• optional double lower_bound = 1

• double upper_bound = 2

Used in: Analyzer

• optional string where = 1

Used in: Repository

• string table_name = 1

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• string column = 1

• optional string where = 2

Used in: Analyzer

• repeated string columns = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

Used in: Analyzer

• repeated string columns = 1

• optional string where = 2

• optional AnalyzerOptions options = 3

• optional bytes data = 1

  optional spark.connect.Plan data = 1;

  TODO: Workaround; actually it is a binary representation

• repeated Check checks = 2

• repeated Analyzer required_analyzers = 3

  Analysis in Deequ that is just a wrapper of sequence of Analyzers

• optional Repository repository = 4

  Anomaly detection part

• optional ResultKey result_key = 5

• repeated AnomalyDetection anomaly_detections = 6

• bool compute_row_level_results = 7