package projectmetis

Get desktop application:
View/edit binary Protocol Buffers messages

• rpc GetCommunityModelEvaluationLineage (GetCommunityModelEvaluationLineageRequest, GetCommunityModelEvaluationLineageResponse)

  controller.proto:11

  Unary RPC. Retrieves community models' metadata related to the models' evaluation.

  message GetCommunityModelEvaluationLineageRequest

  controller.proto:51

  • int32 num_backtracks = 1

    Refers to the number of evaluation request rounds that we need to re-track. If non-positive (x <= 0): reply all, otherwise (x>0) reply current and num-1 latest community models evaluations.

  message GetCommunityModelEvaluationLineageResponse

  controller.proto:57

  • repeated CommunityModelEvaluation community_evaluation = 1

    A list of HashMaps where each position in the list refers to a community model and the HashMap contains the evaluation of the community model across the learners on their local private datasets (train/val/test).

• rpc GetCommunityModelLineage (GetCommunityModelLineageRequest, GetCommunityModelLineageResponse)

  controller.proto:15

  Unary RPC. Controller replies with community models (num_backtracks value 0:all, 1:last, 2:previous to last, etc...).

  message GetCommunityModelLineageRequest

  controller.proto:64

  • int32 num_backtracks = 1

    Number of models that we need to re-track. If non-positive (x <= 0): reply all, otherwise (x>0) reply current and num-1 latest community models.

  message GetCommunityModelLineageResponse

  controller.proto:70

  • repeated FederatedModel federated_models = 1

    Replies a sequence of models.

• rpc GetLearnerLocalModelLineage (GetLearnerLocalModelLineageRequest, GetLearnerLocalModelLineageResponse)

  controller.proto:19

  Unary RPC. Retrieves most recent and historically trained learners local models (num_backtracks value 0:all, 1:last, 2:previous to last, etc...).

  message GetLearnerLocalModelLineageRequest

  controller.proto:93

  • int32 num_backtracks = 1

    Refers to the number of local models that we need to re-track. If non-positive (x <= 0): reply all, otherwise (x>0) reply current and num-1 latest learners' local models.

  • repeated ServerEntity server_entity = 2

    Retrieves the num_backtracks local models for every learner that exists in the provided LearnerEntity collection.

  message GetLearnerLocalModelLineageResponse

  controller.proto:102

  • repeated LearnerLocalModelResponse learner_local_model = 1

    Replies a sequence of LocalModelResponses. One record for each requested learner: [ (LearnerEntity, [LocalModel1, LocalModel2, ...]), (,[]), ... ].

• rpc GetLocalTaskLineage (GetLocalTaskLineageRequest, GetLocalTaskLineageResponse)

  controller.proto:23

  Unary RPC. Retrieves learners local models' metadata related to the models' evaluation (num_backtracks value 0:all, 1:last, 2:previous to last, etc...).

  message GetLocalTaskLineageRequest

  controller.proto:75

  • int32 num_backtracks = 1

    Refers to the number of local tasks that we need to re-track. If non-positive (x <= 0): reply all, otherwise (x>0) reply current and num-1 latest learners' local models evaluations.

  • repeated string learner_ids = 2

    Retrieves the num_backtracks evaluations for every learner that exists in the provided LearnerEntity collection.

  message GetLocalTaskLineageResponse

  controller.proto:86

  • map<string, LocalTasksMetadata> learner_task = 1

    A HashMap containing the evaluation of the locally trained model (i.e., result of run task) of each learner on its local private datasets (train/val/test) and other metadata related to the locally completed training task.

• rpc GetRuntimeMetadataLineage (GetRuntimeMetadataLineageRequest, GetRuntimeMetadataLineageResponse)

  controller.proto:27

  Unary RPC. Replies with runtime metadata related to the federation environment (e.g., number of community update requests).

  message GetRuntimeMetadataLineageRequest

  controller.proto:108

  • int32 num_backtracks = 1

    Refers to the number of runtime metadata we need to request for / retrack. If non-positive (x <= 0): reply all, otherwise (x>0) reply current and num-1 latest runtime metadata.

  message GetRuntimeMetadataLineageResponse

  controller.proto:114

  • repeated FederatedTaskRuntimeMetadata metadata = 1

  • string json_metadata = 2

    TODO No structured response yet, but in a future release this should follow a specific format.

• rpc GetParticipatingLearners (GetParticipatingLearnersRequest, GetParticipatingLearnersResponse)

  controller.proto:30

  Unary RPC. Retrieves a collection of all participating learners in the federation.

  message GetParticipatingLearnersRequest

  controller.proto:120

  (message has no fields)

  message GetParticipatingLearnersResponse

  controller.proto:122

  • repeated LearnerDescriptor learner = 1

    Returns a collection with all the participating learners in the federation. Even though the response is a collection of LearnerDescriptor type, for security reasons the controller only populates the learner id and dataset specifications fields.

• rpc GetServicesHealthStatus (GetServicesHealthStatusRequest, GetServicesHealthStatusResponse)

  controller.proto:33

  Unary RPC. Replies with the health status (beat) of its internal services.

• rpc JoinFederation (JoinFederationRequest, JoinFederationResponse)

  controller.proto:36

  Unary RPC. A new participating learner asks the controller to join the federation.

  message JoinFederationRequest

  controller.proto:129

  • optional ServerEntity server_entity = 2

    TODO This field seems redundant - not used anywhere! Controller generates the learner_ids based on ServerEntity! string learner_id = 1; // If this field is empty/None then this learner is new. If it not empty its (id,token) must match with the existing records inside controller's book keeping collection.

  • optional DatasetSpec local_dataset_spec = 3

  message JoinFederationResponse

  controller.proto:136

  • optional Ack ack = 1

  • string learner_id = 2

    The id of the learner assigned by the controller, see `JoinFederationResponse`.

  • string auth_token = 3

    For security reasons, the controller generates a random 64-characters string that is associated with the joining learner. The (learner_id, token) composite key is required throughout training when a learner contacts the controller.

• rpc LeaveFederation (LeaveFederationRequest, LeaveFederationResponse)

  controller.proto:39

  Unary RPC. An existing learner informs the controller that it leaves the federation.

  message LeaveFederationRequest

  controller.proto:161

  • string learner_id = 1

    The id of the learner assigned by the controller, see `JoinFederationResponse`.

  • string auth_token = 2

    This is associated with the auth_token in `JoinFederationResponse` message.

  message LeaveFederationResponse

  controller.proto:166

  • optional Ack ack = 1

• rpc MarkTaskCompleted (MarkTaskCompletedRequest, MarkTaskCompletedResponse)

  controller.proto:42

  Unary RPC. Receives the local model of a learner when it completes its (locally) assigned task.

  message MarkTaskCompletedRequest

  controller.proto:147

  • string learner_id = 1

    The id of the learner assigned by the controller, see `JoinFederationResponse`.

  • string auth_token = 2

    This is associated with the auth_token in `JoinFederationResponse` message.

  • optional CompletedLearningTask task = 3

  message MarkTaskCompletedResponse

  controller.proto:157

  • optional Ack ack = 1

• rpc ReplaceCommunityModel (ReplaceCommunityModelRequest, ReplaceCommunityModelResponse)

  controller.proto:45

  Unary RPC. Receives a new model to replace the current community model.

  message ReplaceCommunityModelRequest

  controller.proto:170

  • optional FederatedModel model = 1

    TODO: Require some sort of auth to ensure that whoever is replacing the model, e.g., driver, is authorized to do so.

  message ReplaceCommunityModelResponse

  controller.proto:175

  • optional Ack ack = 1

• rpc ShutDown (ShutDownRequest, ShutDownResponse)

  controller.proto:48

  Shuts down the controller service and clears any resources.

• rpc EvaluateModel (EvaluateModelRequest, EvaluateModelResponse)

  learner.proto:12

  Unary RPC. Receives a model and evaluates it at its local dataset (any of [train, validation, test]).

  message EvaluateModelRequest

  learner.proto:25

  • optional Model model = 1

    A single model is sent to the evaluation service of the learner.

  • uint32 batch_size = 2

    We need to provide the batch size for evaluation because a default value will not work for all models, due to memory limitations, cf. CIFAR images to sentences for NER sequence tagging.

  • repeated EvaluateModelRequest.dataset_to_eval evaluation_dataset = 3

  • optional EvaluationMetrics metrics = 4

    The list of metrics we want to evaluate the model, e.g., ["accuracy", "f1_score", "confusion_matrix", etc...]

  message EvaluateModelResponse

  learner.proto:47

  • optional ModelEvaluations evaluations = 1

• rpc GetServicesHealthStatus (GetServicesHealthStatusRequest, GetServicesHealthStatusResponse)

  learner.proto:15

  Unary RPC. Replies with the health status (beat) of its internal services.

• rpc RunTask (RunTaskRequest, RunTaskResponse)

  learner.proto:18

  Unary RPC. Assigns task to be trained locally by the learner.

  message RunTaskRequest

  learner.proto:51

  • optional FederatedModel federated_model = 1

    This is the community model sent by the controller to the learner to run the assigned task.

  • optional LearningTask task = 2

    Description of the assigned task.

  • optional Hyperparameters hyperparameters = 3

    The hyperparameters related to the SGD optimization, i.e., model's optimizer.

  message RunTaskResponse

  learner.proto:62

  • optional Ack ack = 1

• rpc ShutDown (ShutDownRequest, ShutDownResponse)

  learner.proto:21

  Unary rpc. Shuts down all running services of the learner module.

Acknowledgment message.

Used in: JoinFederationResponse, LeaveFederationResponse, MarkTaskCompletedResponse, ReplaceCommunityModelResponse, RunTaskResponse, ShutDownResponse

• bool status = 1

  Set to true if this acknowledges the successful completion of a request, or false otherwise.

• optional google.protobuf.Timestamp timestamp = 2

  Timestamp of the acknowledgement.

• string message = 3

  An optional message.

Used in: OptimizerConfig

• float learning_rate = 1

• float beta_1 = 2

• float beta_2 = 3

• float epsilon = 4

Tensor storing double values.

Used in: Model.Variable

• optional TensorSpec spec = 1

  Tensor metadata.

• bytes values = 2

  Bytes (encrypted) representation of the array.

Used in: ControllerParams

• CommunicationSpecs.Protocol protocol = 1

• optional ProtocolSpecs protocol_specs = 2

Used in: CommunicationSpecs

• UNKNOWN = 0

• SYNCHRONOUS = 1

• ASYNCHRONOUS = 2

• SEMI_SYNCHRONOUS = 3

Used in: GetCommunityModelEvaluationLineageResponse

• uint32 global_iteration = 1

  Records the id of the global iteration, i.e., id of the community model being evaluated.

• map<string, ModelEvaluations> evaluations = 2

  A HashMap with learner as the key and the evaluation of the community model on its local datasets as the value.

Used in: MarkTaskCompletedRequest

• optional Model model = 1

  This is the model trained by the learner locally.

• optional TaskExecutionMetadata execution_metadata = 2

  These are the basic metadata sent by the learner to the controller whenever a locally assigned training task is complete.

• string aux_metadata = 3

  TODO These are additional metadata sent by the learner to the controller. No structured response yet, but in a future release this should follow a specific format.

• optional ServerEntity server_entity = 1

• optional GlobalModelSpecs global_model_specs = 2

• optional CommunicationSpecs communication_specs = 3

• optional FHEScheme fhe_scheme = 4

• optional ControllerParams.ModelHyperparams model_hyperparams = 5

Used in: ControllerParams

• uint32 batch_size = 1

  TODO Shall we replace (batch_size, optimizer) with Hyperparameters message?

• uint32 epochs = 2

• optional OptimizerConfig optimizer = 3

• float percent_validation = 4

  TODO Need to figure out, if this quantity will be used as part of the aggregation scheme or as part of the actual validation evaluation.

Used in: JoinFederationRequest, LearnerDescriptor

• uint32 num_training_examples = 1

• uint32 num_validation_examples = 2

• uint32 num_test_examples = 3

• oneof training_dataset_spec

  • DatasetSpec.ClassificationDatasetSpec training_classification_spec = 4

  • DatasetSpec.RegressionDatasetSpec training_regression_spec = 5

• oneof validation_dataset_spec

  • DatasetSpec.ClassificationDatasetSpec validation_classification_spec = 6

  • DatasetSpec.RegressionDatasetSpec validation_regression_spec = 7

• oneof test_dataset_spec

  • DatasetSpec.ClassificationDatasetSpec test_classification_spec = 8

  • DatasetSpec.RegressionDatasetSpec test_regression_spec = 9

Used in: DatasetSpec

• map<uint32, uint32> class_examples_num = 1

Used in: DatasetSpec

• double min = 1

  TODO(canast02): Need to add histogram support; the following statistics should refer to a histogram bucket.

• double max = 2

• double mean = 3

• double median = 4

• double mode = 5

• double stddev = 6

Tensor storing double values.

Used in: Model.Variable

• optional TensorSpec spec = 1

  Tensor metadata.

• repeated double values = 2

  Flattened representation of the array.

Used in: TaskEvaluation

• uint32 epoch_id = 1

  The id of the epoch. This is an incremental value, i.e., serial number. A learner is training continuously and therefore it can increment its epoch id as it progresses its training.

• optional ModelEvaluation model_evaluation = 2

This is a repeated field since we might request to evaluate a model on more than one datasets.

Used in: EvaluateModelRequest

• TRAINING = 0

• TEST = 1

• VALIDATION = 2

Used in: EvaluateModelRequest, LearningTask

• repeated string metric = 1

Used in: ControllerParams

• bool enabled = 1

• string name = 2

• uint32 batch_size = 3

• uint32 scaling_bits = 4

• string cryptocontext = 5

• string public_key = 6

• string private_key = 7

Used in: OptimizerConfig

• float learning_rate = 1

• float proximal_term = 2

Represents a community model.

Used in: GetCommunityModelLineageResponse, ReplaceCommunityModelRequest, RunTaskRequest

• uint32 num_contributors = 1

  Number of entities that contributed to the aggregation of this model.

• uint32 global_iteration = 2

• optional Model model = 3

  The aggregated community model.

Used in: GetRuntimeMetadataLineageResponse

• uint32 global_iteration = 1

• optional google.protobuf.Timestamp started_at = 2

• optional google.protobuf.Timestamp completed_at = 3

• repeated string assigned_to_learner_id = 4

• repeated string completed_by_learner_id = 5

Tensor storing double values.

Used in: Model.Variable

• optional TensorSpec spec = 1

  Tensor metadata.

• repeated float values = 2

  Flattened representation of the array.

Used as request type in: ControllerService.GetServicesHealthStatus, LearnerService.GetServicesHealthStatus

(message has no fields)

Used as response type in: ControllerService.GetServicesHealthStatus, LearnerService.GetServicesHealthStatus

• map<string, bool> services_status = 1

  A Key-Value services status reply.

Used in: ControllerParams

• GlobalModelSpecs.AggregationRule aggregation_rule = 1

• float learners_participation_ratio = 2

Used in: GlobalModelSpecs

• UNKNOWN = 0

• FED_AVG = 1

• PWA = 2

Used in: RunTaskRequest

• uint32 batch_size = 1

• optional OptimizerConfig optimizer = 2

Tensor storing integer values.

Used in: Model.Variable

• optional TensorSpec spec = 1

  Tensor metadata.

• repeated int32 values = 2

  Flattened representation of the array.

Used in: GetParticipatingLearnersResponse, LearnerState

• string id = 1

• string auth_token = 2

• optional ServerEntity service_spec = 3

• optional DatasetSpec dataset_spec = 4

• string json_response = 1

Used in: GetLearnerLocalModelLineageResponse

• optional ServerEntity server_entity = 1

  The description of the learner entity.

• repeated Model model = 2

  For a single learner we can return a collection of locally trained models. We encapsulate the local models as federation models because there is no difference in the structure of the two model types.

• optional LearnerDescriptor learner = 1

  Describes the learner. It also includes the generated authorization token for the learner.

• repeated Model model = 2

  Learner's model lineage.

Used in: RunTaskRequest

• uint32 global_iteration = 1

  Records the id of the global iteration, i.e., id of the community model sent to the learner for local training.

• uint32 num_local_updates = 2

  This reflects the number of local steps the learner needs to perform. It is similar to epochs if we take |num_training_examples| / batch_size.

• float training_dataset_percentage_for_stratified_validation = 3

• optional EvaluationMetrics metrics = 4

• uint32 num_local_updates = 1

  This reflects the number of local steps the learner needs to perform.

Used in: GetLocalTaskLineageResponse

• repeated TaskExecutionMetadata task_metadata = 1

Represents a machine learning model.

Used in: CompletedLearningTask, EvaluateModelRequest, FederatedModel, LearnerLocalModelResponse, LearnerState

• repeated Model.Variable variables = 1

  Model's variables.

Used in: Model

• string name = 1

  The name of the variable.

• bool trainable = 2

  Set to true if this is a trainable variable. Otherwise, false.

• oneof tensor

  The tensor holding the values of this variable.

  • IntTensor int_tensor = 3

  • FloatTensor float_tensor = 4

  • DoubleTensor double_tensor = 5

  • CiphertextTensor ciphertext_tensor = 6

TODO: Not sure if we need a simple json response or scores! For instance, we might need additional metrics such as confusion matrices...

Used in: EpochEvaluation, ModelEvaluations

• map<string, string> metric_values = 1

  FOR NOW, we need to return the key-value formatted value! That is for every string metric passed through the EvaluationMetrics object we reply with a <metric, value> collection.

Wrapper for multiple model evaluations.

Used in: CommunityModelEvaluation, EvaluateModelResponse

• optional ModelEvaluation training_evaluation = 1

• optional ModelEvaluation validation_evaluation = 2

• optional ModelEvaluation test_evaluation = 3

Used in: OptimizerConfig

• float learning_rate = 1

• float momentum_factor = 2

Used in: ControllerParams.ModelHyperparams, Hyperparameters

• oneof config

  • VanillaSGD vanilla_sgd = 1

  • MomentumSGD momentum_sgd = 2

  • FedProx fed_prox = 3

  • Adam adam = 4

Used in: CommunicationSpecs

• int32 semi_sync_lambda = 1

  Parameters specific to the semi-synchronous protocol.

• bool semi_sync_recompute_num_updates = 2

Used in: ServerEntity

• string server_key = 1

• string server_cert = 2

A server entity that participates in the collaborative learning environment.

Used in: ControllerParams, GetLearnerLocalModelLineageRequest, JoinFederationRequest, LearnerDescriptor, LearnerLocalModelResponse

• string hostname = 1

  Either the hostname or the IP address of the server.

• uint32 port = 2

  The server port that the service is running on.

• optional SSLConfig ssl_config = 3

  For enabling https between the learners and controller

TODO: require some sort of auth key here?

Used as request type in: ControllerService.ShutDown, LearnerService.ShutDown

(message has no fields)

Used as response type in: ControllerService.ShutDown, LearnerService.ShutDown

• optional Ack ack = 1

Used in: TaskExecutionMetadata

• repeated EpochEvaluation training_evaluation = 1

  A list with all training evaluations across all epochs.

• repeated EpochEvaluation validation_evaluation = 2

  A list with all validation evaluations across all epochs.

• repeated EpochEvaluation test_evaluation = 3

  A list of all test evaluations across all epochs.

Used in: CompletedLearningTask, LocalTasksMetadata

• uint32 global_iteration = 1

  Records the id of the global iteration, i.e., id of the global task or of the received community model. Need it for bookkeeping when contacting the controller the completed local task.

• optional TaskEvaluation task_evaluation = 2

• float completed_epochs = 3

  Learner may perform partial epochs, thus the float data type.

• uint32 completed_batches = 4

• uint32 batch_size = 5

• float processing_ms_per_epoch = 6

  Time-per-epoch in milliseconds.

• float processing_ms_per_batch = 7

  Time-per-batch in milliseconds.

Structural info of a tensor.

Used in: CiphertextTensor, DoubleTensor, FloatTensor, IntTensor

• uint32 length = 1

  Length of the tensor, i.e., total number of values it stores.

• repeated uint32 dimensions = 2

  The size of each dimension of the tensor.

• TensorSpec.DType dtype = 3

  (Optional) The data type this tensor holds; may be used for casting the values to a specific type.

Supported data types.

Used in: TensorSpec

• UNKNOWN = 0

• INT = 1

• LONG = 2

• FLOAT = 3

• DOUBLE = 4

Used in: OptimizerConfig

• float learning_rate = 1

• float L1_reg = 2

• float L2_reg = 3