package secretflow.spec.v1

Get desktop application:
View/edit binary Protocol Buffers messages

Supported attribute types.

Used in: AttributeDef

• ATTR_TYPE_UNSPECIFIED = 0

  NOTE: ATTR_TYPE_UNSPECIFIED could be used as a child of a union struct with no further attribute(s).

• AT_FLOAT = 1

  FLOAT

• AT_INT = 2

  INT

• AT_STRING = 3

  STRING

• AT_BOOL = 4

  BOOL

• AT_FLOATS = 5

  FLOATS

• AT_INTS = 6

  INTS

• AT_STRINGS = 7

  STRINGS

• AT_BOOLS = 8

  BOOLS

• AT_STRUCT_GROUP = 9

• AT_UNION_GROUP = 10

• AT_CUSTOM_PROTOBUF = 11

• AT_PARTY = 12

  A specialized AT_STRINGS.

The value of an attribute

Used in: AttributeDef.AtomicAttrDesc, Descriptions.Item, NodeEvalParam, Table.Row

• float f = 1

  FLOAT

• int64 i64 = 2

  INT NOTE(junfeng): "is" is preserved by Python. Replaced with "i64".

• string s = 3

  STRING

• bool b = 4

  BOOL

• repeated float fs = 5

  FLOATS

• repeated int64 i64s = 6

  INTS

• repeated string ss = 7

  STRINGS

• repeated bool bs = 8

  BOOLS

• bool is_na = 9

  Indicates the value is missing explicitly.

Describe an attribute. There are three kinds of attribute. - Atomic Attributes: a solid field for users to fill-in. - Struct Attributes: a group of closely related attributes(including atomic, union and struct attributes). - Union Attributes: a group of mutually exlusive attributes(including union, group and dummy atomic attributes). Users should select only one children to fill-in. An atmoic attribute with ATTR_TYPE_UNSPECIFIED AttrType is regarded as dummy, which represents a selection of union without further configurations.

Used in: ComponentDef, IoDef.TableAttrDef

• repeated string prefixes = 1

  Indicates the ancestors of a node, e.g. `[name_a, name_b, name_c]` means the path prefixes of current Attribute is `name_a/name_b/name_c/`. Only `^[a-zA-Z0-9_.-]*$` is allowed. `input` and `output` are reserved.

• string name = 2

  Must be unique in the same level just like Linux file systems. Only `^[a-zA-Z0-9_.-]*$` is allowed. `input` and `output` are reserved.

• string desc = 3

• AttrType type = 4

• optional AttributeDef.AtomicAttrDesc atomic = 5

• optional AttributeDef.UnionAttrGroupDesc union = 6

• string custom_protobuf_cls = 7

  Extras for custom protobuf attribute

Extras for an atomic attribute. Including: `AT_FLOAT | AT_INT | AT_STRING | AT_BOOL | AT_FLOATS | AT_INTS | AT_STRINGS | AT_BOOLS`.

Used in: AttributeDef

• int64 list_min_length_inclusive = 1

  Only valid when type is `AT_FLOATS \| AT_INTS \| AT_STRINGS \| AT_BOOLS`.

• int64 list_max_length_inclusive = 2

  Only valid when type is `AT_FLOATS \| AT_INTS \| AT_STRINGS \| AT_BOOLS`.

• bool is_optional = 3

  If True, when Atmoic Attr is not provided or is_na, default_value would be used. Else, Atmoic Attr must be provided.

• optional Attribute default_value = 4

  A reasonable default for this attribute if the user does not supply a value.

• optional Attribute allowed_values = 5

  Only valid when type is `AT_FLOAT \| AT_INT \| AT_STRING \| AT_FLOATS \| AT_INTS \| AT_STRINGS`. Please use list fields of AtomicParameter, i.e. `ss`, `i64s`, `fs`. If the attribute is a list, allowed_values is applied to each element.

• bool lower_bound_enabled = 6

  Only valid when type is `AT_FLOAT \| AT_INT \| AT_FLOATS \| AT_INTS `. If the attribute is a list, lower_bound is applied to each element.

• optional Attribute lower_bound = 7

• bool lower_bound_inclusive = 8

• bool upper_bound_enabled = 9

  Only valid when type is `AT_FLOAT \| AT_INT \| AT_FLOATS \| AT_INTS `. If the attribute is a list, upper_bound is applied to each element.

• optional Attribute upper_bound = 10

• bool upper_bound_inclusive = 11

Extras for a union attribute group.

Used in: AttributeDef

• string default_selection = 1

  The default selected child.

A list of components

• string name = 1

• string desc = 2

• string version = 3

• repeated ComponentDef comps = 4

The definition of a comp.

Used in: CompListDef

• string domain = 1

  Namespace of the comp.

• string name = 2

  Should be unique among all comps of the same domain.

• string desc = 3

• string version = 4

  Version of the comp.

• repeated AttributeDef attrs = 5

• repeated IoDef inputs = 6

• repeated IoDef outputs = 7

Displays multiple read-only fields in groups.

Used in: Div.Child

• string name = 1

  Name of the Descriptions.

• string desc = 2

• repeated Descriptions.Item items = 3

Used in: Descriptions

• string name = 1

  Name of the field.

• string desc = 2

• string type = 3

  Must be one of bool/int/float/str

• optional Attribute value = 4

A public record for a general distributed data. The type of this distributed data, should be meaningful to components. The concrete data format (include public and private parts) is defined by other protos. Suggested internal types, i.e. - sf.table.vertical_table represent a secretflow vertical table - sf.table.individual_table represent a secretflow individual table

Used in: pipeline.NodeDef, NodeEvalParam, NodeEvalResult

• string name = 1

  The name of this distributed data.

• string type = 2

  Type.

• optional SystemInfo system_info = 3

  Describe the system information that used to generate this distributed data.

• optional google.protobuf.Any meta = 4

  Public information, known to all parties. i.e. VerticalTable.

• repeated DistData.DataRef data_refs = 5

  Remote data references.

A reference to a data that is stored in the remote path.

Used in: DistData

• string uri = 1

  The path information relative to StorageConfig of the party.

• string party = 2

  The owner party.

• string format = 3

  The storage format, support: - csv represent a comma-separated value format file - orc represent a apache orc format file

• repeated string null_strs = 4

  A list of strings that represent NULL value. Only take effect when format is csv

A division or a section of a page.

Used in: Div.Child, Tab

• string name = 1

  Name of the Div.

• string desc = 2

• repeated Div.Child children = 3

Used in: Div

• string type = 1

  Supported: descriptions, table, div.

• optional Descriptions descriptions = 2

• optional Table table = 3

• optional Div div = 4

IndividualTable describes a table owned by a single party.

• optional TableSchema schema = 1

  Schema.

• int64 line_count = 2

  If -1, the number is unknown.

Define an input/output for component.

Used in: ComponentDef

• string name = 1

  should be unique among all IOs of the component.

• string desc = 2

• repeated string types = 3

  Must be one of DistData.type in data.proto

• repeated IoDef.TableAttrDef attrs = 4

  Only valid for tables. The attribute path for a TableAttrDef is `{input\|output}/{IoDef name}/{TableAttrDef name}`.

An extra attribute for a table. If provided in a IoDef, e.g. ```json { "name": "feature", "types": [ "int", "float" ], "col_min_cnt_inclusive": 1, "col_max_cnt": 3, "attrs": [ { "name": "bucket_size", "type": "AT_INT" } ] } ``` means after a user provide a table as IO, they should also specify cols as "feature": - col_min_cnt_inclusive is 1: At least 1 col to be selected. - col_max_cnt_inclusive is 3: At most 3 cols to be selected. And afterwards, user have to fill an int attribute called bucket_size for each selected cols.

Used in: IoDef

• string name = 1

  Must be unique among all attributes for the table.

• string desc = 2

• repeated string types = 3

  Accepted col data types. Please check comments of TableSchema in data.proto.

• int64 col_min_cnt_inclusive = 4

  inclusive

• int64 col_max_cnt_inclusive = 5

• repeated AttributeDef extra_attrs = 6

  extra attribute for specified col.

Evaluate a node. - CompListDef + StorageConfig + NodeEvalParam + other extra configs -> NodeEvalResult NodeEvalParam contains all the information to evaluate a component.

• string domain = 1

  Domain of the component.

• string name = 2

  Name of the component.

• string version = 3

  Version of the component.

• repeated string attr_paths = 4

  The path of attributes. The attribute path for a TableAttrDef is `(input\|output)/(IoDef name)/(TableAttrDef name)(/(column name)(/(extra attributes))?)?`.

• repeated Attribute attrs = 5

  The value of the attribute. Must match attr_paths.

• repeated DistData inputs = 6

  The input data, the order of inputs must match inputs in ComponentDef. NOTE: Names of DistData doesn't need to match those of inputs in ComponentDef definition.

• repeated string output_uris = 7

  The output data uris, the order of output_uris must match outputs in ComponentDef.

• string checkpoint_uri = 8

  If not empty: 1. Component will try to save checkpoint during training if the component supports it. 2. Component will try to reload checkpoint when starting to continue the previous training. If the checkpoint does not exist or cannot be loaded, training will be starting from scratch.

NodeEvalResult contains outputs of a component evaluation.

• repeated DistData outputs = 1

  Output data.

• string name = 1

  Name of the Report.

• string desc = 2

• repeated Tab tabs = 3

• int32 err_code = 4

• string err_detail = 5

  Structed error detail (JSON encoded message).

A StorageConfig specifies the root for all data for one party. - At this moment, only local_fs / S3 compatible object storage is supported - We would support databases like mysql etc. in future.

• string type = 1

  enum[local_fs, s3]

• optional StorageConfig.LocalFSConfig local_fs = 2

  local_fs config.

• optional StorageConfig.S3Config s3 = 3

  s3 config

For local_fs.

Used in: StorageConfig

• string wd = 1

  Working directory.

For S3 compatible object storage

Used in: StorageConfig

• string endpoint = 1

  endpoint https://play.min.io or http://127.0.0.1:9000 with scheme

• string bucket = 2

  the bucket name of the oss datasource

• string prefix = 3

  the prefix of the oss datasource. e.g. data/traindata/

• string access_key_id = 4

  access key

• string access_key_secret = 5

  access secret

• bool virtual_host = 6

  virtual_host is the same as AliyunOSS/AWS S3's virtualhost , default true

• string version = 7

  optional enum[s3v2,s3v4]

Describe the application related to data.

Used in: DistData

• string app = 1

  The application name. Supported: `secretflow`

• optional google.protobuf.Any app_meta = 2

  Meta for application.

A page of a report.

Used in: Report

• string name = 1

  Name of the Tab.

• string desc = 2

• repeated Div divs = 3

Displays rows of data.

Used in: Div.Child

• string name = 1

  Name of the Table.

• string desc = 2

• repeated Table.HeaderItem headers = 3

• repeated Table.Row rows = 4

Used in: Table

• string name = 1

• string desc = 2

• string type = 3

  Must be one of bool/int/float/str

Used in: Table

• string name = 1

• string desc = 2

• repeated Attribute items = 3

The schema of a table. - A col must be one of `id | feature | label`. By default, it should be a feature. - All names must match the regexp `[A-Za-z0-9.][A-Za-z0-9_>./]*`. - All data type must be one of * int8 * int16 * int32 * int64 * uint8 * uint16 * uint32 * uint64 * float16 * float32 * float64 * bool * int * float * str

Used in: IndividualTable, VerticalTable

• repeated string ids = 1

  Id column name(s). Optional, can be empty.

• repeated string features = 2

  Feature column name(s).

• repeated string labels = 3

  Label column name(s). Optional, can be empty.

• repeated string id_types = 4

  Id column data type(s). Len(id) should match len(id_types).

• repeated string feature_types = 5

  Feature column data type(s). Len(features) should match len(feature_types).

• repeated string label_types = 6

  Label column data type(s). Len(labels) should match len(label_types).

VerticalTable describes a virtual vertical partitioning table from multiple parties.

• repeated TableSchema schemas = 1

  The vertical partitioned slices' schema. Must match data_refs in the parent DistData message.

• int64 line_count = 2

  If -1, the number is unknown.