package nebula.service

Get desktop application:
View/edit binary Protocol Buffers messages

all service methods are captilized compared to other regular methods

• rpc Echo (EchoRequest, EchoResponse)

  nebula.proto:323

  Sends a greeting

• rpc Tables (ListTables, TableList)

  nebula.proto:325

  Get all available data sources

  message ListTables

  nebula.proto:41

  • uint32 limit = 1

  message TableList

  nebula.proto:45

  • repeated string table = 1

• rpc State (TableStateRequest, TableStateResponse)

  nebula.proto:327

  Get table state for given table

  message TableStateRequest

  nebula.proto:37

  • string table = 1

  message TableStateResponse

  nebula.proto:49

  • int32 blockCount = 1

  • int64 rowCount = 2

  • int64 memSize = 3

  • int64 minTime = 4

  • int64 maxTime = 5

  • repeated string dimension = 6

    columns = dimension columns + metric columns metric column are column with number types, others are dimension columns

  • repeated string metric = 7

  • repeated string hists = 8

    column histograms serialized in JSON string

• rpc Query (QueryRequest, QueryResponse)

  nebula.proto:329

  Query Nebula to get result

  message QueryRequest

  nebula.proto:173

  define query request

  • string table = 1

  • oneof filter

    at the top tree as filter - it can be only one, either AND or OR

    • PredicateAnd filterA = 2

    • PredicateOr filterO = 3

  • int64 start = 4

    every query will specify time range for the data to query - this is enforced start time and end time are unix timestamp with GMT timezone - no delta time range filter will be inclusive on both ends as [start, end]

  • int64 end = 5

  • uint32 window = 6

    window size is used in time serials query that aggregate timestamp into a window as bucket this is in seconds (if the value is 60 -> one minute)

  • repeated string dimension = 7

    dimensions (key columns / group by columns) to pull metrics for

  • repeated Metric metric = 8

    metric list, such as sum(col_a), count(col_b)

  • uint32 top = 9

    usually query only takes top X results based on order spec

  • optional Order order = 10

    define order spec

  • bool timeline = 11

    display type for query result

  • repeated CustomColumn custom = 12

    custom column list

  • uint64 time_unit = 13

    custom time unit

  • int64 tz_offset = 14

    timezone offset in minutes (following javascript getTimezoneOffset convention) timezone behind GMT is positive, ahead is negative (how to bring it back to GMT)

  message QueryResponse

  nebula.proto:244

  define query response from server

  • optional Statistics stats = 1

  • DataType type = 2

  • bytes data = 3

• rpc Load (LoadRequest, LoadResponse)

  nebula.proto:332

  on demand loading specified data with parameters template defined in cluster configuration

  message LoadRequest

  nebula.proto:262

  on-demand data loading request

  • LoadType type = 1

    load type

  • string table = 2

    template table name

  • string json = 3

    different type will utilize this json object for different purpose. configured template uses it as parameters instantiate a template. google sheets will use it store google sheet spec

  • uint32 ttl = 4

    TTL in seconds for the data to live

  message LoadResponse

  nebula.proto:290

  load command response

  • LoadError error = 1

    load state: error=0 indicating success∏

  • uint32 loadTimeMs = 2

    loading time in milliseconds

  • string table = 3

    loaded instance table name

  • string message = 4

    extra message the server wants to share

• rpc Nuclear (EchoRequest, EchoResponse)

  nebula.proto:334

  shut down all work nodes - used for perf profiling

• rpc Url (UrlData, UrlData)

  nebula.proto:336

  URL service - code for raw or raw for code

  message UrlData

  nebula.proto:32

  Url shorten service

  • string code = 1

  • string raw = 2

• rpc Ping (ServiceInfo, PingResponse)

  nebula.proto:338

  Discovery service - register node

  message ServiceInfo

  nebula.proto:307

  • string ipv4 = 1

    IP address

  • uint32 port = 2

    service port

  • string host = 3

    host name (optional)

  • ServiceTier tier = 4

    service role/tier - server, node, web, etc.

  message PingResponse

  nebula.proto:318

  (message has no fields)

define custom column described by a script

Used in: QueryRequest

• string column = 1

• CustomType type = 2

• string expr = 3

define supported custom column type which is a sub set of nebula types

Used in: CustomColumn

• INT = 0

• LONG = 1

• FLOAT = 2

• DOUBLE = 3

• STRING = 4

Used in: QueryResponse

• NATIVE = 0

  use native flat buffer format it has its own serialziation and deserialization logic

• JSON = 1

  JSON string sending in bytes buffer

The request message containing the user's name.

Used as request type in: V1.Echo, V1.Nuclear

• string name = 1

The response message containing the greetings

Used as response type in: V1.Echo, V1.Nuclear

• string message = 1

load error

Used in: LoadResponse

• SUCCESS = 0

• TEMPLATE_NOT_FOUND = 1

• MISSING_PARAM = 2

• MISSING_BUCKET_VALUE = 3

• UNAUTHORIZED = 4

• EMPTY_RESULT = 5

• NOT_SUPPORTED = 6

• IN_LOADING = 7

• PARSE_ERROR = 8

• BAD_SCHEMA = 9

Used in: LoadRequest

• CONFIG = 0

  load command to load data preconfigured (cluster.yml)

• GOOGLE_SHEET = 1

  A spec to load google sheet

• DEMAND = 2

  A spec to load a data file(s) by auto detecting its schema

• UNLOAD = 3

  A simple command to unload a table

A metric is defined by rollup method on a column

Used in: QueryRequest

• string column = 1

• Rollup method = 2

single predicate which is column op [list of values] Operation defines comparions op to form a predicate: =, !=, >, <

Used in: Predicate

• EQ = 0

• NEQ = 1

• MORE = 2

• LESS = 3

• LIKE = 4

• ILIKE = 5

• UNLIKE = 6

• IUNLIKE = 7

Used in: QueryRequest

• string column = 1

• OrderType type = 2

define ordering spec - limiting to order by single column in returning result

Used in: Order

• ASC = 0

• DESC = 1

• NONE = 2

TODO(cao): reference Constraints.js for GROUP/RULE generation to build predicates tree Current definition only supports one group basically. (nested groups) NOTE - based on the code generation rule, we don't put plural term for repeated field so the method will like add_value rather than add_values Based on column type - the value will be converted from string always Adding n_value for integers and d_value for floating values if client uses it. Adding values as customized data list serialzied in some format (data type same as column type)

Used in: PredicateAnd, PredicateOr

• string column = 1

• Operation op = 2

• repeated string value = 3

• repeated int64 n_value = 4

• repeated double d_value = 5

• bytes zip = 6

  allow client to pass a zipped values representing an list of values in current column type. zip format and zip count show what algo used to zip the original array. zip count indicates how many values zipped inside.

• ZipFormat ZipFormat = 7

• int32 zipCount = 8

define filter - a filter is formed by a list of predicates combined by AND or OR

Used in: QueryRequest

• repeated Predicate expression = 1

Used in: QueryRequest

• repeated Predicate expression = 1

This defines all the aggregate metric methods suppored in nebula

Used in: Metric

• COUNT = 0

• SUM = 1

• MIN = 2

• MAX = 3

• AVG = 4

• P10 = 5

  pre-defined percentile

• P25 = 6

• P50 = 7

• P75 = 8

• P90 = 9

• P99 = 10

• P99_9 = 11

• P99_99 = 12

• TREEMERGE = 13

  call stack merge

• CARD_EST = 14

  estimated cardinality

• HIST = 15

Used in: ServiceInfo

• WEB = 0

• SERVER = 1

• NODE = 2

define query processing metrics

Used in: QueryResponse

• uint32 queryTimeMs = 1

  end to end query time in milliseconds

• uint64 rowsScanned = 2

  total rows scanned for the query

• uint64 blocksScanned = 3

  total blocks scanned for the query

• uint64 rowsReturn = 4

  total number of rows in result (before limit)

• uint32 error = 5

  query result - 0=success, other error code to look up

• string message = 6

  may place error message here if failed

Used in: Predicate

• DELTA = 0