Proto commits in camsas/firmament

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2017-06-16

Replaced MachinePerfSample with ResourceStats. Unified Heapster ResourceStats and MachinePerfSample protobuf files and uniformized all memory/network fields to be KB. Affected modules: base engine messages platforms scheduling sim Change-Id: I60abe631020317c651652b42034c60e4bb1a88b5

The documentation is generated from this commit.

2017-05-20

Updated Task/ResourceStats types to match Heapster's types. Affected modules: base scheduling Change-Id: Idef3ad39d7fe9b908a00196f971827353ba017e6

2017-05-18

Extended scheduler service with task update support. This change exposes a RPC method that can be called to update a task's resource requirements, priority, labels and label selectors. Affected modules: scheduling Change-Id: Ia06ea5244377a3d7be917dac1724bc35198f1c44

2017-05-16

Extended scheduler service with node update support. Previously, there was no way to update a node once it has been added to the Firmament scheduler. This change adds a simple NodeUpdate RPC that can be used to update the a node. It currently only supports node label updates. Affected modules: misc scheduling/firmament Change-Id: Id641929c32a49b527d43ebe8e15f55b2b79e97c2

2017-05-11

Replaced TaskPerfStatisticsSample with TaskStats. TaskPerfStatisticsSample were not used in any cost model. This change unifies the two task statistics protobuf into a single one. This protobuf is either partially populated by TaskLib or fully populated by Heapster via cAdvisor. Affected modules: base engine messages scheduling Change-Id: I690458565431809d71af85b7ebe16a3bb1a2402b

Added hostname to TastStats message. Affected modules: scheduling Change-Id: I8229ac53aa88a784009a116dcceae31771b223bf

Added fields to ResourceStats and TaskStats messages. Affected modules: scheduling Change-Id: Ic556ebe158d7f0ef409e65359760deaaa38b0d51

2017-05-04

Added Firmament GRPC scheduler service. This change exposes the Firmament service as a GRPC service. External cluster managers can use this service to interact with Firmament and get scheduling placements. Affected modules: misc scheduling Change-Id: I1dd7a5ef74294c2fd2640bcbf62b05d50a2d8a08

2017-03-22

Added support for Kubernetes-style labels. This change adds support for Kubernetes-style labels as described here: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ Affected modules: base scheduling Change-Id: I22e2dd1e472699b41260cb6f256a294465353406

2016-11-07

Merge "Added Apache boilerplate notice to source files."

Added Apache boilerplate notice to source files. Change-Id: I069573759cb77aa129ee347492f7bce4a5622bbb

2016-10-26

Make binary field in task descriptor a string. The field does not need to be type "bytes". The change has no substantial effect other than cleanup, since byte and string fields are treated the same in the Python and C++ protobuf generated code. Affected modules: base Change-Id: Ia9fa83c72e31e6d1f8be19535b9db30dfeaf8bf0

2016-10-20

Removed unused messages protobuf. This change removes several protobuf messages that were not used. Affected modules: engine messages storage Change-Id: Ib1a726561838632a8aa74a674682883ec10c150b

Remove unused DataObjectName, TaskInstanceProfile. This change removes two unused protobuf files. Affected modules: base Change-Id: Ib806efc71bfa67611cb33d2cb7bf0d5b7045d26f

Removed unused DIOS protobufs. This changes removes unused DIOS protobufs together with coordinator's code that was handling some of them. Affected modules: engine messages Change-Id: I487911df380884d2b5a901152fe3245f1278d2d5

Port code to protobuf3 APIs. This change ports all C++ code to the new protobuf3 interfaces. They are mostly the same as in protobuf2, but there are two differences that had breaking effects on our code: 1) Singular numeric fields no longer expose has_<field>() method, but are always initialized to the default value (zero). This requires us to compare against the default zero value instead of checking whether an optional field exists (all fields are optional in protobuf3). 2) Singular string fields likewise no longer expose a has_<field>() method either, and are instead initialized to the empty string. This requires us to change all checks for these fields to compare against the empty string instead. Note that this requires the condition to be negated: previously, we checked for has_<field>() being TRUE, and now we need to check for <field>().empty() being FALSE (and vice versa). The good news is that these changes tend to simplify our code: in several places, conditionals can be omitted because default initialization values have the right effect (e.g., with numeric fields being set to zero). Affected modules: base engine misc scheduling scheduling/flow sim storage Change-Id: I291e748f869d5a05c0f98c205b8b1fd9bf958261

2016-10-19

Move protocol buffers to protobuf3 syntax. The new version of Protocol Buffers makes backwards-incompatible changes to the protobuf syntax. This change ports all of our protobufs to the new syntax, but does not (yet) fix the Firmament C++ code to use the correct new protobuf3 APIs. Affected modules: base messages scheduling sim Change-Id: Icbcb6e9bec24a3cd895d6747177dfa4890419a65

2016-10-13

Remove unused protobuf imports. Affected modules: base messages Change-Id: I3060108f1c93e5bdf5be28f77a457f4ddee66d45

2016-09-19

Updated net stats usage to differentiate between rx and tx. Previously, we were using a single value for network bandwidth. This diff changes how we account for network usage by using two values: one for network rx and another one for network tx. The diff also appropriately updated the CoCo and Net cost models. Affected modules: scripts src/base src/engine src/platforms src/scheduling/flow src/sim src/webui Change-Id: I30c52be480df35260d16c01101273db85514f435

2016-05-24

Added scheduling method that returns deltas. Affected modules: scheduling scheduling/flow scheduling/simple Change-Id: Iec6466084d4b83099a63aa838b0beeed60cb8b95

2016-03-27

Pass the priority and scheduling fields to the simulator. Affected modules: sim Change-Id: Ie7fd70443956215dbf45ff8d283badf80e34dbfe

Fixed QuincyCostModel's cost to the unscheduled aggregator. The previous implementation was only considering the total time spent unscheduled before previous task runs. However, for non-repeating tasks this would always be 0. The change updates the code to include the unscheduled time of the current run as well. NOTE: The change resets the submit time after every task eviction. This may lead to some time stats having negative values in the UI. Moreover, the change adds code to print the number of preference arcs to racks and machines. Affected modules: base engine/executors scheduling/flow Change-Id: Ie507e5e6d3b4d5626e40ed6a4de2bb8ecba97923

2016-03-24

Removed legacy actioned field from scheduling deltas. This field is not useful in our current design. We do not have to track if we've actioned on a scheduling event because we currently handle all event types. The deleted code was slowing down the scheduler 3x. Affected modules: scheduling/flow Change-Id: I30aaf9f52eaa7f8b260a7d19d8f09ef841eff0bd

Added support for multiple tasks running on the same PU. Previously, we could only run one task on a PU. This diff removes that limitation with the help of the following additions: 1) For each PU resource descriptor we set the num_running_tasks_below field to a FLAG value rather than setting it to 1. 2) We also set the capacity from the PU to the sink the the same flag value. 3) We replaced ResourceDescriptor's current_running_task with a repeated current_running_tasks field. 4) We changed the logic in which we generate preemption deltas to work with the repeated field. The old logic would have required us to constantly remove from the repeated field. However, this would have been inefficient because repeated fields do not have any remove(element) method. We now create all the preemption deltas and clear the current_running_tasks field before we generate other kinds of deltas. While we generate these other deltas we make sure to repopulate current_running_tasks with the task that haven't been moved. Affected modules: base scheduling scheduling/flow Change-Id: I77faffc746cab7e9d1526eccd221dac17c02060f

2016-03-19

Compute total unscheduled and run time for every task. Quincy cost model requires access to the total run and unscheduled time of a task. This change adds total_run_time and total_unscheduled_time fields to the task descriptor. Moreover, the change makes sure the fields are updated when the state of a task changes. Affected modules: base engine misc scheduling/flow sim Change-Id: Ifc466b78ccae12a9e416b9d4636116b98a0ee1e1

2016-03-05

Moved NodeBindingToSchedulingDelta from SolverDispatcher to FlowGraphManager. NodeBindingToSchedulingDelta was previously located in SolverDispatcher despite not accessing any of SolverDispatcher's field. Moreover, the method requires access to the FlowGraph. This change moves the method to the FlowGraphManager. In order to be able to do that, I had to change FlowGraphNodes to store TaskDescriptors rather than TaskID_ts. Affected modules: scheduling scheduling/flow Change-Id: I647a60837da0ed27c0bfdac478cef1c661f9dc91

2016-03-01

Fixed EquivClassToResourceNode to return the correct capacity type. EquivClassToResourceNode and TaskFitCount were returning the capacity as an int64_t despite the capacity's type being uint64_t. Moreover, TaskFitCount was initializing the number of tasks that fit to MAX_INT64 which would cause the solver to overflow because several arcs with max capacity can point to a node. This change makes sure that FlowGraphManager doesn't overwrite the return we get from EquivClassToResourceNode. Previously, FlowGraphManager would set the capacity to the number of tasks slots when EquivClassToResourceNode was returning a negative capacity. This is dangerous to do because it can change the behaviour of some cost models. The cost models should return the correct capacity. Lastly, the change moves the logic that computes the number of tasks that fit into a resource subtree to the cost models. Affected modules: scheduling/flow scheduling/flow/sim Change-Id: If71d453fe8e1d6a8b779400d3eebbb0eb740954f

2016-02-24

Removed unused flow_node_type.proto. Affected modules: scheduling/flow Change-Id: I9a8aad57a21992033968f2728091a502fcd360b9

2016-02-19

Updated simulator and CoCo to correctly handle resource request. This change updates to correctly set the resource requests in the simulator. Moreover, the change makes sure that CoCo handles well tasks that request 0 CPUs. Such tasks exist in the Google trace. Affected modules: base scheduling/flow sim Change-Id: I97c45c38e04cfcc1fb762186afc154940ba46e7f

2016-02-04

Updated TraceGenerator to print the machine ids for scheduled tasks. Previously, we weren't printing the machine_id field in the task_events table. This change, adds functionality so TraceGenerator can print the machine_id for the TASK_SCHEDULED events. Moreover, the change removes the hack of passing trace_machine_id via the resource's friendly name. Affected modules: base misc scheduling/flow sim Change-Id: I0116bbf3c35af2308f563fddd3e7122e043e2173

2016-02-03

Added trace job and task id fields to TaskDescriptor. In this change I've added task and job trace id fields to the TaskDescriptor protobuf. This allowed us to get rid of the hack we were previously using to pass trace ids to the TraceGenerator. Affected modules: base misc sim Change-Id: I76a5fefb7619eecc5250377d27ca40729a663e31

2016-02-01

Remove deprecated PlatformID argument. The constructors for Node and Coordinator used to take an argument indicating the current platform ID. We never used this for anything and deprecated it a while ago; I've removed all references to it from the code base. Change-Id: I768c2ddea5bf6225cea7cc6c11b93acebeea4d5d

2015-09-23

Removed deprecated Ensemble code. Affected modules: base misc Change-Id: Ie9b750f6d76a87441e008633decf62a699284db5

2015-09-20

Merge "Removed unused SchedulingParameters protobuf."

2015-09-16

Removed old simulator and moved new one to src/sim. This change removes the old simulator and moves the code of the new simulator from sim/trace-extract to sim. Affected modules: engine scheduling/flow sim/dfs sim Change-Id: If4482770ff8d00f502a0bc180b62b3236d4ef83e

Refactored google_trace_bridge to use the scheduler_interface. This change refactors the simulator to make it compatible with different types of schedulers and cost models. Affected modules: engine schedulng scheduling/flow scheduling/simple sim/trace-extract Change-Id: Ibddd4d8f5771658da566df00b05df2e20c04a8c9

2015-09-14

Removed unused SchedulingParameters protobuf. This change removes the deprecated SchedulingParameters protbuf. These parameters are a property of the cost model and not the flow scheduler. Hence, they should not be passed and stored in the FlowScheduler. Affected modules: engine scheduling/flow Change-Id: Ifc6df1b731c347ea903f8f678b8c61517f38f531

2015-08-20

Add initial support for resource reservations. This change introduces the infrastructure necessary to support resource reservations. However, they won't work quite yet: code to set and propagate the reservation will be pushed in a separate change. The current implementation of reservations is *always strict*: in other words, a task will never be placed in a location where the resource reservation is not satisfied in all dimensions. In principle, however, the code is set up to support non-strict fit (opportunistic overcommit) and partial fit. Affected modules: base scheduling/flow Change-Id: If503ae17daa51129d45a61ceb0c84b958d500237

Correctly propagate and assign interference scores. This change completes the CoCo cost model implementation by propagating the interference score vectors (now represented by the CoCoInterferenceScores protobuf) through the topology and by assigning the interference score on the TEC -> resource aggregate arc as the vector's element in the incoming task type dimension. (In other words, the cost is the penalty for running a task of this type under this resource aggregate). When propagating the interference vector, we divide by the number of subordinate resources at each branching point in order to keep the cost within (0, omega_). However, the division only takes place when using the accumulated value *at the next level*, since we don't know whether we have accumulated to the total before then. Affected modules: base scheduling/flow Change-Id: I064bfb37d0e0944f5b5ff82a494b5f01d18c989f

2015-08-18

Track machine resource capacities and use in CoCo. This change adds several features related to tracking overall resource capacity of a machine: 1. We automatically read resource capacities for CPU cores, RAM and network bandwidth from procfs and sysfs at coordinator startup, and store them in the machine resource descriptor's resource_capacity field. 2. We calculate available resources in the CoCo cost model using the known load information from the KB and the extracted capacities. The change also adds a more generic unit conversion header. Affected modules: base engine platforms/unix scheduling/flow Change-Id: I76e2c61f74823de4235765991847373e70a60bc8

Propagate interference scores across topology. This change implements the CoCo cost model's propagation of interference scores from the tasks (at the leaves) through the resource topology. Compared to the written description, a number of bugs have been fixed: 1) The scale factor's directionalty is now correct (more idle cores = lower scale factor = lower cost). 2) The impact of a particular leaf's score *decays* through the topology, as the formula calculates, for vertex u with children v \in V: cost(u) = ( e^( (t-i) / t) * \sum_v( cost(v) / |V| ) ) - \sum_v( cost(v) / |V| ) This leaves only the cost execess caused by non-idle leaves, with an impact that is proportional to the fraction of occupied leaves, but which decays in magnitude (although costs are, of course, summed all the way to the leaves when solving the optimisation). In order to make this work, we now track both the number of occupied leaves below a resource and the total number of leaves below it via the resource statistics mechanism (similar to how the Octopus cost model works). Affected modules: base engine scheduling/flow Change-Id: I5af001d518176b1275b7e1c32936ed68e613161a

2015-08-10

Resource-fitting admission control for CoCo cost model. This change adds multi-dimensional resource fitting admission control to the CoCo cost model (as per cost model description). Arcs are added between TECs and maximally aggregated subtrees of the resource topology. To detect such trees, a partial BFS of the resource topology is completed for each TEC. However, the BFS skips subtrees where tasks from the TEC definitely do or do not fit, and only explores until either of those cases is hit. This means that while it is O(N) in the topology size in principle, it's actually much faster in practice (i.e., << O(N)). This is a large change that brings in significant new code, including resource vector comparators, CoCo's statistics Prepare-Gather-Update cycle and various others. Affected modules: scheduling/flow Change-Id: I640459e9e5b432f2844594dfe8d003fc7f32bc17

Add COORDINATOR node type and use it in FlowGraph. This change replaces the GLOBAL_AGGREGATOR flow node type (which we no longer use) with a COORDINATOR one. This is used to represent the resource corresponding to a coordinator in the flow graph; the CoCo cost model requires this (as the COORDINATOR -> MACHINE arcs contain machine-specific costs). Affected modules: scheduling/flow Change-Id: I893175be58351723f395f0aab27ab661a0550f81

Add statistics and initialise protobufs to zero. This change adds fields to track current disk I/O and network I/O bandwidth in the MachinePerfStatisticsSample. It also initialises fields' default values in relevant protocol buffers to zero, since they may otherwise contain uninitialised values in release mode. Affected modules: base Change-Id: Id66f13eee6d512777a43d8c4ef0e929a5548699b

2015-07-22

Add logic for preference arcs in CoCo cost model. This change brings a simple partial BFS traversal of the resource topology in order to identify maximally aggregated subtrees under which a task can definitely fit. Effectively, we perform a BFS and stop exploring a subtree once we discover that tasks from a TEC can definitely fit under *all* subordinate resources, or cannot ever fit under *any* subordinate resources. As part of this change, the now more generally used ResourceRequest protobuf was renamed to ResourceVector, and additional fields were added to the task descriptor. Affected modules: base scheduling/flow Change-Id: I23deb5498737d478edb692fb784a1f44ce6fd730

2015-07-20

Implement Whare-MCs part of Whare-Map cost model. This change introduces support for the part of the Whare-Map cost model that implements the "Whare-MCs" strategy in the Mars and Tang 2013 ISCA paper: 1) preference arcs from TECs to preferable machines are added (number controlled by a command line flag), with the cost set to the cost of scheduling in the presence of the existing co-located tasks. 2) PsPI values are recorded for < <TEC, co-runners>, MEC> tuples in the xi_map_ (named after the cost model formula, Xi(c_t, L_m, c_m)). We also separately track best and worst-case values for Xi, since they're more specific than the Whare-M ones (Psi). Affected modules: base scheduling/flow Change-Id: I04545cf52d9b8751e04e86e68766d1c25a3edd17

2015-07-05

Correct insertion of remote tasks' final reports to parent coordinator. Before this change, the delivery of delegated tasks' final reports was implemented by adding a second submessage into the message informing the parent coordinator of the task completion (both TaskStateMessage and TaskFinalReport were present). However, since the TaskFinalReport part of the message was only processed *after* the task state change, the cost model's update functions were invoked before the report had been processed, and thus no report was linked in the TD yet. This commit adds an optional field for the report to the TaskStateMessage, allowing it to be processed alongside that. Affected modules: engine messages Change-Id: I90e02b2166a2378eecec5e28355b0b40d771a8f4

Implement Whare-M policy in WhareMap cost model. This change adds all necessary functionality to support the Whare-M cost model: when a task finishes, the psPI value (picoseconds per instruction, an inverse version of Whare-Map's instructions per second, IPS, since the flow network requires greater cost to correspond to worse placements) is stored under a mapping of the form <task equiv class, machine equiv class> -> [psPI val 0, ..., psPI val n] This requires a number of API changes in the cost model and in other parts of the system. Specifically, the TD now also stores copies of the scheduled resource and the task's final report (for easier reference from the cost model), which are stored by the EventDrivenScheduler base once a task has been placed and once it completes. There is more work to be done: the pspi_map_ contents in the WhareMap cost model are not currently used; this will follow in a separate change. Affected modules: base engine scheduling scheduling/flow Change-Id: I84bb6829f79443495e267e038c9d9b5d8fa9365e

2015-06-28

Implement Whare-M policy in WhareMap cost model. This change adds all necessary functionality to support the Whare-M cost model: when a task finishes, the psPI value (picoseconds per instruction, an inverse version of Whare-Map's instructions per second, IPS, since the flow network requires greater cost to correspond to worse placements) is stored under a mapping of the form <task equiv class, machine equiv class> -> [psPI val 0, ..., psPI val n] This requires a number of API changes in the cost model and in other parts of the system. Specifically, the TD now also stores copies of the scheduled resource and the task's final report (for easier reference from the cost model), which are stored by the EventDrivenScheduler base once a task has been placed and once it completes. There is more work to be done: the pspi_map_ contents in the WhareMap cost model are not currently used; this will follow in a separate change. Affected modules: base engine scheduling scheduling/flow Change-Id: I84bb6829f79443495e267e038c9d9b5d8fa9365e

2015-05-31

[scheduling] Reorganize scheduling code, splitting it into a flow scheduler subdirectory and a simple scheduler subdirectory. This addresses the cyclic dependency that made unit test compilation fail (see #15). Issue: #15 Change-Id: Iab17f81d609ca9e367e08bb8ec92ad55b529390b

2015-04-04

[simulator] Add support for scheduling latency online mode

2015-03-29

[base] Track RTND with each ResourceStatus, and add resource load aggregation fields to RD.

2015-03-26

Merge branch 'master' of github.com:ms705/firmament Conflicts: src/base/resource_desc.proto src/scheduling/quincy_scheduler.cc

[base] Add load field to RD (needed for CoCo cost model).

[scheduling] Work on Octopus cost model.

[scheduling] Tested resource statistics gatherer.

2015-03-25

[base] Added whare-map statistics protobuf.

[base] Add resource request protobuf.

Merge branch 'master' of github.com:ms705/firmament

[base] Add priority and resource request fields to TD.

[scheduling] Fixed remove machine.

2015-03-23

[scheduling] Code to add the aggregators for WhareMap cost model.

2015-03-22

Removed superfluous fields from resource_desc.proto.

[messages] Make naming of task_final_report field consistent.

2015-03-21

[base] Make ABORTED state consistent with TaskDescriptor.

2015-03-19

[base] Add task finish time field to TDD.

2015-03-18

[base] Make some fields optional.

2015-03-17

[scheduling] Add new handlers for job completion and fix a few inconsistencies between queue-based and flow schedulers.

[base] Add heartbeat time field to TD.

2015-03-16

[base] Add delegated_to field to task descriptor and seperate it from the heartbeat location field.

2015-03-15

[messages] Restructure sub-messages to be more logically named and ordered.

2015-02-17

[base] Restructure TaskDescriptor fields and add a few more.

2015-02-16

[sim] Untested single priority queue simulator.

2015-02-14

[sim] Added simulator event protobuf.

2015-02-11

[base] Task-specific field to toggle LD_PRELOAD injection.

[messages] Add final report to task messages.

[base] Add extra fields to task descriptor and statistics protobuf.

2014-03-26

[base] [engine] Plumb scheduler statistics through the heartbeat protocol.

2014-02-13

[sim] Clean out old simulator code and add skeleton infrastructure for new simulation mode.

2013-11-11

[base] Add cache reference count field to task report.

2013-05-26

[scheduling] More fixes to min-flow scheduler.

2013-05-25

[base] Protobufs for task profiling.

2013-03-30

[base] Support final report from task on exit.

[base] Add info as to where a delegated task came from.

[messages] Support response message for task delegation attempt.

2013-03-27

[engine] [base] [messages] Add support for success indication on task delegation.

[base] Add TaskFinalReport protobuf.

2013-03-19

[base] [messages] [storage] Make location a singular field.

2013-03-18

[messages] Add protobufs for libDIOS interaction messages.

2013-03-17

[scheduling] Quincy scheduler now applies deltas to schedule tasks.

2013-03-16

[engine] [messages] Working machine statistics as part of heartbeats to parent coordinators.

[base] [engine] Working task kill message dispatch.

[engine] [messages] [scheduling] Add a TaskKillMessage.

2013-03-15

Moving from uint64_t DataObjectIDs to 256-bit UUIDs as per libDIOS. This commit touches almost everything, and rewrites significant portions of code. Breakage is not only expected, but very likely.

Get machine CPU/Mem info.

Gather machine usage.

2013-03-14

Remove use of protobuf extensions (unsupported in protobuf-c).

[scheduling] Add missing file.

[scheduling] Working extraction of scheduling decisions from solver output.

2013-03-13

From Quincy to worker to task mapping.