Proto commits in apache/incubator-nemo

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2023-02-13

java11

2021-05-13

waiting executor init when vm scaling

2021-05-05

logging output

2021-05-03

print latency to client

2021-04-29

fix

source event

[backpressure policy]

2021-04-28

executor metric for backpressure

optimize task scheduling: rm sending task byte again

2021-04-26

[Throttling, fix bugs, R2/R3 protocol]

2021-04-20

[R3 partial agg protocol optimization; reroute again while partial agg]

2021-04-18

[R3 optimization: add shortcut for partial. Directly invoke final within a partial task]

2021-04-17

[Implementation of R3 rule]

2021-04-15

[Optimizing rerouting (state migration and task output stop)]

[Finish rerouting protocol: task output done ACK]

2021-04-13

[correctness guarantee protocol for r2]

2021-04-11

[CR task state migration to transient data path]

[write bytebuf data through pipeManagerWorker]

2021-03-29

[signal crvertex]

2021-03-21

kill job wheb latency is higher than threshold

2021-03-11

[Nemo name server resolver]

[Netty messge environment]

[Refactoring messages. change listener id to enum]

2021-03-09

migration specific stage

2021-03-03

[Partial offloading warmup] [Deactivate lambda control message] [waiting remaining event before deactivation of partial offloading] [fix bug, sampling based on interval]

2021-03-02

[Warm up: task send before handling bursty]

2021-02-27

[Offloading task from master]

2021-02-16

[source throttle, recliam, deoffload]

[Implement yarn offloading requestor]

2021-02-13

[add offloading task and executor with cmd line control]

2021-01-30

[Building Testbed for task migration]

2021-01-29

Task migration implementing.. TODO: task stop and input pipe restart

2021-01-26

register pipe index

Pipe manager worker refactoring

2021-01-24

[Stateless task stop and reschedule on HDFStore]

reschedule a stopped task

2021-01-23

Move task enabled

Testing hdfs state store

2021-01-22

[Graceful shutdown with ctrl+c and aws region set]

driver shutting down

2020-02-03

[Done: scaling from SF to VM. TODO: VM to VM]

2020-02-01

[sf to vm.. mechanism]

[Launching vm workers in master]

2020-01-30

[Implement VM scaling]

2019-09-27

throttling

sf cpu use

2019-09-24

Scaling with input rate and cpu

2019-09-23

[stage input output logging]

[broadcast info for evaluation]

2019-09-22

proactive migration

2019-09-19

prioritizing based on states

2019-09-18

proritized scaling

2019-08-30

local scaling with global sync

2019-08-06

rendevous server impl

2019-08-05

disable stage2 offloading in query5

2019-08-03

scaling in

task location management in master

2019-08-02

vm scaling in multiple nodes

2019-07-15

GlobalExecutorAddressMap and remove logging

2019-07-10

Implement multivm relay server

2019-07-09

[Implement master scaling decision]

2019-06-02

stage-based offloading

2019-05-27

..

2019-05-22

global transfer index

2019-04-01

add taskIndexMaster

2019-03-31

[Preparing multi-stage lambda.. remove protobuf and move classes]

2019-03-30

[NEMO-360] Implementing an 'XGBoostPolicy' (#203) JIRA: [NEMO-360: Implementing an 'XGBoostPolicy'](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-360) **Major changes:** - Adds a python script that runs XGBoost. - as well as a bash script for using the python script. - Uses the Client <-> Driver RPC to run the XGBoost script and return the results to the driver. - Enables environment tag. - Refactors the utility classes accordingly. **Minor changes to note:** - Miscellaneous methods to help the implementation (e.g., getEdgeById) - Consistency in the shell scripts - Appropriately adds files to the gitignore - Javadocs **Tests for the changes:** - MetricUtilsTest confirms that changing an EP to indices and back to EP works correctly (TDD) **Other comments:** - None Closes #203

The documentation is generated from this commit.

[Pipe reverse connection]

2019-02-07

[NEMO-332] Refactor RunTimePass (#191) JIRA: [NEMO-332: Refactor RunTimePass](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-332) **Major changes:** * Refactors the RunTimePass API * Refactors the run-time plan rewriting logic into the NemoPlanRewriter **Minor changes to note:** * Replaces ResourceSkewedDataProperty with ResourceAntiAffinityProperty (generalizing the directive) **Tests for the changes:** * Existing tests continue to pass Closes #191

2018-10-30

[NEMO-8] Implement PipeManagerMaster/Worker (#129) JIRA: [NEMO-8: Implement PipeManagerMaster/Worker](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-8) **Major changes:** - Supports fully-pipelined data streaming for bounded sources (not unbounded sources) - Tasks do 'finish' after processing all input data, as the data is finite - When a tasks finishes, it emits all data it has (e.g., GroupByKey accumulated results) and closes corresponding outgoing pipes, notifying downstream tasks the end of the pipes - For stream-processing unbounded sources, we need watermarks (https://issues.apache.org/jira/browse/NEMO-233) - Introduces PipeManagerMaster/Worker - Shares code with BlockManagerMaster/Worker - Naive, Element-wise serialization+compression+writeAndFlush - Very likely that this will cause some serious overheads. Will run proper benchmarks and fix the issues in a later PR. **Minor changes to note:** - JobConf#SchedulerImplClassName: Batch and Streaming options - StreamingPolicyParallelismFive: The default policy + PipeTransferEverythingPass - Fixes the StreamingScheduler to pass the new streaming integration tests - Fixes a coder bug in the Beam frontend (PCollectionView coder) **Tests for the changes:** - WindowedWordCountITCase#testStreamingFixedWindow - WindowedWordCountITCase#testStreamingSlidingWindow **Other comments:** - Also closes "Implement common API for data transfer" (https://issues.apache.org/jira/browse/NEMO-9) Closes #129

2018-10-26

[NEMO-239] Edit Source Headers for ASF Guideline Compliance (#132) JIRA: [NEMO-239: Edit Source Headers for ASF Guideline Compliance](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-239) **Major changes:** - Edited source headers to meet the standards defined in https://www.apache.org/legal/src-headers.html **Minor changes to note:** - N/A **Tests for the changes:** - N/A **Other comments:** - N/A Closes #132

2018-09-06

[NEMO-211] Move Java package and Maven artifact from edu.snu.nemo to org.apache.nemo JIRA: [NEMO-211: Move Java package and Maven artifact from edu.snu.nemo to org.apache.nemo](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-211) **Major changes:** - Moved Java package from edu.snu.nemo to org.apache.nemo - Moved Maven artifact from edu.snu.nemo to org.apache.nemo **Minor changes to note:** - N/A **Tests for the changes:** - N/A **Other comments:** - N/A Closes #116

2018-09-05

[NEMO-96] Modularize DataSkewPolicy to use MetricVertex and BarrierVertex (#115) JIRA: [NEMO-96: Modularize DataSkewPolicy to use MetricVertex and BarrierVertex](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-96) [NEMO-98: Implement MetricVertex that collect metric used for dynamic optimization](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-98) [NEMO-99: Implement AggregationBarrierVertex for dynamic optimization](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-99) **Major changes:** - Handle dynamic optimization via `MetricCollectionVertex` and `AggregationBarrierVertex` instead of `MetricCollectionBarrierVertex` - For each shuffle edge with main output, `MetricCollectionVertex` is inserted in compile-time at the end of its source tasks, which collects key frequency data - For each shuffle edge with main output, `AggregationBarrierVertex` is inserted in compile-time. It aggregates task-level key frequency data, which is collected via each `MetricCollectionVertex` and emitted as additional tagged output **Minor changes to note:** - Added encoder/decoder factories needed for aggregating dynamic optimization data - in here key frequency data - Modified `PipelineTranslator` to extract key encoder/decoders - Modified `DataSkewRuntimePass` and related code path to handle `Object` type keys, instead of integer type hash index keys **Tests for the changes:** - N/A(unit tests for skew handling and `PerKeyMedianITCase` test the changes) **Other comments:** - N/A Closes #115

2018-08-31

[NEMO-15] Run Spark ALS in distributed mode (#113) JIRA: [NEMO-15: Run Spark ALS in distributed mode](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-15) **Major changes:** - Removes 'sideinput' from IRDAG, and instead introduces BroadcastVariableProperty - InMasterBroadcastVariables/BroadcastManagerWorker: Components that deal with broadcast variables - New RPC messages for exchanging broadcast variables between client->driver, and driver<->executor **Minor changes to note:** - Explicit scala lambdas (update_ms, update_us) to get around serialization errors - Avoid duplicate PlanState completions which can be caused by cloned executions **Tests for the changes:** - SparkScala#testALS **Other comments:** - This PR also closes: [NEMO-14] Re-investigate Broadcast and SideInputs Closes #113

2018-08-19

[NEMO-178] Zero-delay task cloning (#107) JIRA: NEMO-178: Zero-delay task cloning Major changes: * A state machine per task attempt/its output blocks * Randomly choose one of the available attempt output blocks when fetching data * A cloned task, or a retried task is just another task attempt * ClonedSchedulingProperty: Zero-delay, upfront task cloning Minor changes to note: * Removes unused code * RuntimeIdGenerator -> RuntimeIdManager * Removes the first hyphens from the ids (e.g., vertex-4 => vertex4) Tests for the changes: * WordCountITCase#testClonedScheduling Other comments: * ClonedSchedulingPass at the moment just clones source vertices, as cloning Beam sink vertices appears to cause failures due to a duplicate file name conflict (I'll look into this in a later PR) resolves NEMO-178

2018-08-14

[NEMO-64] Fix map stage hang under DataSkewPolicy (#99) JIRA: [NEMO-64: Fix map stage hang under DataSkewPolicy](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-64), [NEMO-181: Fix DataSkewPolicy bug for multiple shuffles](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-181) **Major changes:** - Data for dynamic optimization is aggregated at `DynOptDataHandler` in Scheduler instead of a variable in `MetricCollectionBarrierVertex`. Updating `MetricCollectionBarrierVertex` as data arrives in RuntimeMaster clashed with serializing IRDAG for scheduled Tasks, which caused `ConcurrentModificationException` and silently killed the Scheduler - Identifies the target edge to optimize via MetricCollectionProperty in case multiple shuffles are involved. **Minor changes to note:** - Removed now unused entries in `dataSizeMetricMsg` - Added `DataSkewMetricProperty` and `MetricFactory` to make task hash ranges of shuffle edges as an execution property, so that RuntimePass can optimize the given IR DAG **Tests for the changes:** - N/A(No new features, covered by `PerKeyMedianITCase`) **Other comments:** - Data for dynamic optimization will be aggregated in designated vertex and not in the RuntimeMaster via upcoming PR for [NEMO-99](https://issues.apache.org/jira/browse/NEMO-99) resolves [NEMO-64](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-64), [NEMO-181](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-181)

2018-07-19

[NEMO-62] Support Multiple DAG Submission in a Single User Program (#73) JIRA: [NEMO-62: Support Multiple Jobs Submission in a Single User Program](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-62) **Major changes:** - The client and the driver interacts with each other to notify the status of each other when executing the job, enabling the client to submit DAGs multiple times to the driver. - At the moment, the scheduler does not handle multiple DAGs, but it's possible to submit multiple DAGs to the driver. The changes here are related to the client side. The actual execution will be handled with [NEMO-152](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-152?filter=allopenissues). - Beam launches a DAG a single time, as it collects everything before launching the pipeline, but Spark provides `actions` that we can use multiple times in an application. In order to support the functionality, this change is necessary. **Minor changes to note:** - Handle SonarQube issues and fix minor typos and grammatical errors. **Tests for the changes:** - Add an example test program (IT case) to ensure that it works properly. **Other comments:** - N/A resolves [NEMO-62](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-62)

2018-07-03

[NEMO-20] RESTful APIs to Access Job State and Metric (#61) JIRA: NEMO-20: RESTful APIs to Access Job State and Metric Major changes: * Removed previous MetricCollector. * Add Metric interface to handle easily with metrics. * Add MetricStore which collects all metric data at the master side. * Add HTTP and WebSocket REST API endpoints. Minor changes to note: N/A Tests for the changes: * Simple JSON dump test at MetricStoreTest. Other comments: N/A resolves NEMO-20

2018-06-29

[NEMO-61] Fix lost execution metric collection problem (#60) JIRA: [NEMO-61: Fix lost execution metric collection problem](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-61) **Major changes:** - Add metric flush request and response types in control message type. - `RuntimeMaster#terminate` now wait until all metrics from executors arrive. **Minor changes to note:** - Expose `ExecutorReqistry#viewExecutors` to public. **Tests for the changes:** - MetricFlushTest ensures MetricManagerMaster can send flush requests to MetricManagerWorkers and workers respond appropriately by flushing remaining metric logs. **Other comments:** - Related to NEMO-20. resolves [NEMO-61](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-61)

2018-06-28

[NEMO-16] Implement collection of data from executor to client (#56) JIRA: [NEMO-16: Implement collection of data from executor to client](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-16) **Major changes:** - Gets rid of the file-writing and reading during the `collect` action of Spark. Instead, it brings the data from the executor through the driver, and then back to the client using the RPC implemented with [NEMO-103] (#45). **Minor changes to note:** - I've separated the MR Spark integration tests into a separate file, as the SparkJava integration tests file has been becoming too large and crowded. - A number of typos have been fixed **Tests for the changes:** - I've created a test for the `ContextImpl`, which had no unit tests before. **Other comments:** - N/A resolves [NEMO-16](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-16)

2018-06-19

[NEMO-103] Implement RPC between Client and Driver (#45) JIRA: [NEMO-103: Communication between Driver and Client](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-103) **Major changes:** - Implement RPC between NemoClient and NemoDriver using NettyMessagingTransport - Use the RPC stack to submit DAG to driver, allowing multiple-DAG submission to one REEF instance. **Minor changes to note:** - N/A **Tests for the changes:** - Added 'ClientDriverRPCTest' - Existing integration tests also cover this change. **Other comments:** - N/A resolves [NEMO-103](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-103)

[NEMO-87] Remove unused BlockStates (#49) JIRA: [NEMO-87: Remove unused BlockStates](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-87) **Major changes:** - READY, LOST_BEFORE_COMMIT, LOST, REMOVED => NOT_AVAILABLE - SCHEDULED => IN_PROGRESS - COMMITTED => AVAILABLE **Minor changes to note:** - N/A **Tests for the changes:** - N/A (no new feature) **Other comments:** - N/A resolves [NEMO-87](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-87)

2018-06-15

[NEMO-94] Clean up legacy TODOs (#36) JIRA: NEMO-94: Clean up legacy TODOs Major changes: * Remove TODOs of resolved issues * Re-reference the issue number Minor changes to note: n/a Tests for the changes: n/a Other comments: n/a resolves NEMO-94

2018-06-01

[NEMO-79] Clean up the legacy Task (#24) JIRA: NEMO-79: Clean up the legacy Task Major changes: * Removes the 'old' Task interface and its implementations to make the 'new' Task (formerly TaskGroup) become the only Task in the codebase * Specifically, changes the DAG in the 'new' Task from DAG<Task, RuntimeEdge> to DAG<IRVertex, RuntimeEdge> * Adds SourceVertex#clearInternalStates to clear the huge list of input splits held by BeamBoundedSourceVertex before sending the vertex to remote executors. Between clearing states of an existing vertex and creating a new vertex, I've chosen the former approach to ensure consistent use of the same IRVertex object across the compiler, the master, and the executors. Minor changes to note: * Make BeamBoundedSourceVertex remember sourceDescription for visualization purposes * Changes related method and variable names appropriately * Changes comments appropriately Tests for the changes: N/A (no new feature was added) Other comments: N/A resolves NEMO-79

2018-05-21

[NEMO-76] Rename TaskGroup to Task (#20) JIRA: NEMO-76: Rename TaskGroup to Task Major changes: Appropriately renames things like 'TaskGroup', 'taskGroup', 'Task group', and 'task group' Minor changes to note: Removes the legacy TaskState Tests for the changes: This PR only changes names, and removes TaskState that isn't used anywhere The existing tests continue to pass Other comments: The legacy 'Task' will be handled with https://issues.apache.org/jira/browse/NEMO-79 Once this PR is merged, I'll work on that ASAP to remove the confusion between the two 'Task's. resolves NEMO-76

2018-03-08

resolve conflict

[NEMO-26] Implement SourceLocationAwareSchedulingPolicy (#1) JIRA: [NEMO-26: Implement SourceLocationAwareSchedulingPolicy](https://issues.apache.org/jira/projects/NEMO/issues/NEMO-26) **Major changes:** - Implemented SourceLocationAwareSchedulingPolicy - This scheduling policy schedules TaskGroups with Readables to the one of the executors that have the corresponding input split. - For TaskGroups without any Readables, it uses RoundRobinSchedulingPolicy. - Added unit tests for SourceLocationAwareSchedulingPolicy. - Refactor scheduling thread model on runtime master side (thanks to @johnyangk) - Made only two threads call scheduling-related code: RuntimeMaster thread (RMT), and SchedulerThread(ST) - RMT and ST meet only at two points: SchedulingPolicy, and PendingTaskGroupQueue, which are synchronized(ThreadSafe) - Other scheduler-related classes that are accessed by only one of the two threads are not synchronized(NotThreadSafe) - We use ContainerManager for handling containers only, and not executors (to avoid hanging, we also don’t wait for pending resource requests upon termination) - ExecutorRegistry and ExecutorRepresenter fully classify running/failed/completed Executor/Taskgroups **Minor changes to note:** - None **Tests for the changes:** - Implemented SourceLocationAwareSchedulingPolicy - Added SourceLocationAwareSchedulingPolicyTest - Refactored container management and scheduling on runtime master side (thanks to @johnyangk) - ContainerManagerTest covers the changes **Other comments:** - None

2018-03-07

debug dataskew

2018-02-28

[Minor] Add missing license headers (#788) This PR: - modifies `pom.xml` to use RAT to check copyright header in source files - adds missing copyright notice to some source files

[Minor] Rename Coral to Nemo (#783) Remove all Coral in the codebase.

[Nemo-762] Support multiple byte streams for one block transfer (#765) This commit features the following changes: * Add support for multiple sub-stream for one block transfer context * Complete separation of block management and byte transfer * Lower data plane renamed from 'BlockTransfer' to 'ByteTransfer' * Higher data plane embeds block metadata in 'TransferContext'. Lower data plane does not interpret it. * Enabling Streaming for inter-TaskGroup communication (disabled by BlockManagerWorker) * Fixed InputReader not to introduce deadlock while reading streamized SideInput * Much more concise codebase! Resolves #765

[Minor] Rename onyx to coral (#763) * rename * rename all-cap Nemo

[Nemo-742] Make PhysicalPlan scalable (#746) This PR: - Removes physical-level information from `Task`. - Moves physical-level information from `TaskGroup` (index, for example) to `ScheduledTaskGroup`. Resolves #742.

[Nemo-715] Remote Block Metadata in File (#718) This PR: - makes `RemoteFileMetadata` be stored in remote file instead of memory in driver side. - when write a remote block, it's metadata (including partition information) is maintained in the memory of the executor which runs the task - when the block is committed, the metadata will be flushed into a separate remote file - when the block is read, the metadata will be restored from the file - removes all message passing for metadata between driver & executor Resolves #715

[Nemo-494] Replace Hash Range with Key Range (#665) Resolves #494 . This PR: - Creates an interface `KeyRange`, makes `HashRange` implement the interface. - Replaces all the code that assumes partition keys to be of integer type, introducing a generic type `K`

[Minor] Switch Block with Partition (#663) This PR: - Renames Block and corresponding classes & parameters to Partition - Renames Partition and corresponding classes & parameters to Block

[Nemo-24] pom.xml per Sub-Package (#616) Resolves #24 Resolves #592 This PR: - Basic sub-packaging: `pom.xml` per sub-package with no dependency error - Modified README with running examples(`bin/run.sh`) - `mvn clean install` runs all unit tests/ITs currently located in `tests`

2017-11-13

[Nemo-206] Replace NCS with a proper RPC framework (#590) This PR: - implements grpc based RPC server and client to implement interfaces of `edu.snu.onyx.runtime.common.message` Resolves #206

[Nemo-588] Implement MemoryStore which stores data in serialized form (#589) This PR: - implements a serializing version of MemoryStore - makes lower-level data plane (PartitionTransfer) support byte array transfer - moves data (de)serializing parts to `DataSerializationUtil` class (because many `PartitionStore`s share these codes) - creates `LocalPartitionStore` to represent `PartitionStore`s which contain the (meta)data as a single map in local. Resolves #588.

2017-10-13

[Nemo-548] Rename old name to Nemo (#551) This PR: - renames all remnants of our old name to Nemo. resolves #548