These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)
| Commit: | 0e942f5 | |
|---|---|---|
| Author: | sagewe | |
improve proto buffer generation Signed-off-by: sagewe <wbwmat@gmail.com>
The documentation is generated from this commit.
| Commit: | 10d0aa8 | |
|---|---|---|
| Author: | sagewe | |
fix deepspeed client issue Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 3dee00d | |
|---|---|---|
| Author: | v_wbxiongli | |
merge dev-3.0.0-rc Signed-off-by: v_wbxiongli <740332065@qq.com>
| Commit: | cb0e440 | |
|---|---|---|
| Author: | v_wbxiongli | |
Merge remote-tracking branch 'origin/dev-3.0.0-rc' into feature-3.0.0-new # Conflicts: # conf/eggroll.properties # jvm/cluster_manager/src/main/java/org/fedai/eggroll/clustermanager/dao/impl/StoreCrudOperator.java # jvm/cluster_manager/src/main/resources/eggroll.properties # jvm/core/src/main/java/org/fedai/eggroll/core/config/DeepspeedContainerBuildConfig.java # jvm/core/src/main/java/org/fedai/eggroll/core/containers/container/PythonContainer.java # jvm/core/src/main/java/org/fedai/eggroll/core/pojo/ErStoreLocator.java # python/client/sdk/core/proto/containers_pb2.py # python/client/sdk/core/proto/containers_pb2_grpc.py # python/eggroll/client/cli/commands/task.py # python/eggroll/client/sdk/api/task.py # python/eggroll/computing/tasks/transfer/transfer_service.py # python/eggroll/core/session.py # python/setup.py
| Commit: | 9810ad4 | |
|---|---|---|
| Author: | sagewe | |
refactor: rewrite python side codes 1. mv serdes and partitoner to client side 2. simplify and purge unused codes 3. better config support 4. better logging and tracing support Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 636959c | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix merge conflict Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | d8f8a4c | |
|---|---|---|
| Author: | v_wbxiongli | |
| Committer: | sagewe | |
merge from feature3.0 Signed-off-by: v_wbxiongli <740332065@qq.com>
| Commit: | 0e135f3 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
merge Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 2ce5319 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
enable compress level Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | bad548c | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix store timeout Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 6e2f490 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add rendezvous store destroy Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | f630e9a | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix container killing Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 33d51cb | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add models and logs download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | a1bab5c | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed container start Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | cddd289 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add resource options Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | f03bac9 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
impl job download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | bea4949 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor and impl query Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | e9e14cd | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed and container Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | abfc9d1 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 836d214 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
(feat): add store for pytorch dist rendezvous Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | f2f0475 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | a634461 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource manager Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 8096639 | |
|---|---|---|
| Author: | kaideng | |
| Committer: | sagewe | |
add resource manager Signed-off-by: kaideng <forgive_dengkai@163.com>
| Commit: | 1f30ea9 | |
|---|---|---|
| Author: | weiwee | |
| Committer: | sagewe | |
implement general job submit client, server and containers Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | a73e56d | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor and impl query Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 411c631 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
Refactor Python Serialization and Introduce Manual Partitioning in Eggroll - Undertook a major overhaul of Python modules to phase out pickle serialization in favor of a more robust and secure method. This shift empowers FATE with full control over serialization and partitioning mechanisms. - Integrated manual partitioning logic into Python code, enhancing data management flexibility and efficiency. - Synchronized Java and Python components with the new serialization and partitioning paradigms to ensure seamless operation and system coherence. - Updated SQL configurations to reflect and support these backend enhancements. - Refined Protocol Buffers (proto/) definitions to align with the updated data handling and transfer strategies, enhancing data integrity and efficiency. - Conducted comprehensive enhancements and validations of the test suites across Java and Python components, focusing on the new changes to guarantee system reliability and performance (jvm/ and python/ test directories). This commit marks a pivotal enhancement in Eggroll’s architecture, significantly elevating its efficiency and security by transitioning from pickle serialization to a more controlled serialization approach, and by integrating support for manual data partitioning. These changes ensure greater compatibility, performance, and flexibility across the system's diverse components. Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 3e99687 | |
|---|---|---|
| Author: | v_wbxiongli | |
| Committer: | sagewe | |
merge from feature3.0 Signed-off-by: v_wbxiongli <740332065@qq.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | fdf22f2 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add rendezvous store destroy Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 9963f29 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
enable compress level Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | bee3e77 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix store timeout Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 55b9c5f | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add models and logs download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 510cd3f | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix container killing Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 7d8cefe | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed container start Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 1a7033a | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add resource options Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 233918f | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
impl job download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 67b7001 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 527627a | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed and container Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | b904ebb | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
merge Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 5f11170 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
(feat): add store for pytorch dist rendezvous Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 7f0b1cd | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource manager Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 7e0930a | |
|---|---|---|
| Author: | kaideng | |
| Committer: | sagewe | |
add resource manager Signed-off-by: kaideng <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | c10449c | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 76fb159 | |
|---|---|---|
| Author: | weiwee | |
| Committer: | sagewe | |
implement general job submit client, server and containers Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 6226b66 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
Refactor Python Serialization and Introduce Manual Partitioning in Eggroll - Undertook a major overhaul of Python modules to phase out pickle serialization in favor of a more robust and secure method. This shift empowers FATE with full control over serialization and partitioning mechanisms. - Integrated manual partitioning logic into Python code, enhancing data management flexibility and efficiency. - Synchronized Java and Python components with the new serialization and partitioning paradigms to ensure seamless operation and system coherence. - Updated SQL configurations to reflect and support these backend enhancements. - Refined Protocol Buffers (proto/) definitions to align with the updated data handling and transfer strategies, enhancing data integrity and efficiency. - Conducted comprehensive enhancements and validations of the test suites across Java and Python components, focusing on the new changes to guarantee system reliability and performance (jvm/ and python/ test directories). This commit marks a pivotal enhancement in Eggroll’s architecture, significantly elevating its efficiency and security by transitioning from pickle serialization to a more controlled serialization approach, and by integrating support for manual data partitioning. These changes ensure greater compatibility, performance, and flexibility across the system's diverse components. Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 513bdf3 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
merge Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | cab3ca9 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix store timeout Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 3b3ad1b | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
enable compress level Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | a3a42ee | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add rendezvous store destroy Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 79886bb | |
|---|---|---|
| Author: | v_wbxiongli | |
| Committer: | sagewe | |
merge from feature3.0 Signed-off-by: v_wbxiongli <740332065@qq.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | a41586a | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
fix container killing Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | ecf4180 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add models and logs download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 286c110 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed container start Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 05eb275 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
add resource options Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 2395638 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
impl job download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 682073b | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor and impl query Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 9b20950 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refact deepspeed and container Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 10ef379 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 3da02b2 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
(feat): add store for pytorch dist rendezvous Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 0291a1b | |
|---|---|---|
| Author: | weiwee | |
| Committer: | sagewe | |
implement general job submit client, server and containers Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 956589e | |
|---|---|---|
| Author: | kaideng | |
| Committer: | sagewe | |
add resource manager Signed-off-by: kaideng <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | fab6086 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 86b4b50 | |
|---|---|---|
| Author: | forgive_dengkai | |
| Committer: | sagewe | |
add resource manager Signed-off-by: forgive_dengkai <forgive_dengkai@163.com> Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | ae735d0 | |
|---|---|---|
| Author: | v_wbxiongli | |
update Signed-off-by: v_wbxiongli <740332065@qq.com>
| Commit: | 855ccb0 | |
|---|---|---|
| Author: | v_wbxiongli | |
upodate python Signed-off-by: v_wbxiongli <740332065@qq.com>
| Commit: | a44844b | |
|---|---|---|
| Author: | wmqwxb | |
add api for query cluster info Signed-off-by: wmqwxb <13628607970@163.com>
| Commit: | 516db24 | |
|---|---|---|
| Author: | sagewe | |
Refactor Python Serialization and Introduce Manual Partitioning in Eggroll - Undertook a major overhaul of Python modules to phase out pickle serialization in favor of a more robust and secure method. This shift empowers FATE with full control over serialization and partitioning mechanisms. - Integrated manual partitioning logic into Python code, enhancing data management flexibility and efficiency. - Synchronized Java and Python components with the new serialization and partitioning paradigms to ensure seamless operation and system coherence. - Updated SQL configurations to reflect and support these backend enhancements. - Refined Protocol Buffers (proto/) definitions to align with the updated data handling and transfer strategies, enhancing data integrity and efficiency. - Conducted comprehensive enhancements and validations of the test suites across Java and Python components, focusing on the new changes to guarantee system reliability and performance (jvm/ and python/ test directories). This commit marks a pivotal enhancement in Eggroll’s architecture, significantly elevating its efficiency and security by transitioning from pickle serialization to a more controlled serialization approach, and by integrating support for manual data partitioning. These changes ensure greater compatibility, performance, and flexibility across the system's diverse components.
| Commit: | f925ebe | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor eggroll Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | c04284d | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor roll_pair Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 9f7b891 | |
|---|---|---|
| Author: | v_ndu | |
add API Signed-off-by: v_ndu <598671894@qq.com>
| Commit: | b277a03 | |
|---|---|---|
| Author: | kakawu | |
| Committer: | kakawu | |
reformat code Signed-off-by: kakawu <945844419@qq.com>
| Commit: | 600ff80 | |
|---|---|---|
| Author: | Signed-off-by: wmqwxb | |
| Committer: | wmqwxb | |
add queryNodeMetaInfo Signed-off-by: wmqwxb <13628607970@163.com>
| Commit: | 35d0763 | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
refactor eggroll Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | f23c7a7 | |
|---|---|---|
| Author: | v_wbxiongli | |
merge from feature3.0 Signed-off-by: v_wbxiongli <740332065@qq.com>
| Commit: | 1744168 | |
|---|---|---|
| Author: | Signed-off-by: wmqwxb | |
| Committer: | wmqwxb | |
add resourceEnoughQuery proto file Signed-off-by: wmqwxb <13628607970@163.com>
| Commit: | 5472cd5 | |
|---|---|---|
| Author: | sagewe | |
refactor roll_pair Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 8d17135 | |
|---|---|---|
| Author: | Signed-off-by: wmqwxb | |
| Committer: | kevin | |
gpu processor handle Signed-off-by: kevin <13628607970@163.com>
| Commit: | 86d17d7 | |
|---|---|---|
| Author: | forgive_dengkai | |
merge Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | f7d55ae | |
|---|---|---|
| Author: | forgive_dengkai | |
develop Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | f932a41 | |
|---|---|---|
| Author: | forgive_dengkai | |
develop Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | d165e2e | |
|---|---|---|
| Author: | forgive_dengkai | |
develop Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 9639496 | |
|---|---|---|
| Author: | forgive_dengkai | |
develop Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | f4f2dab | |
|---|---|---|
| Author: | forgive_dengkai | |
add eggroll interface Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 55f4965 | |
|---|---|---|
| Author: | forgive_dengkai | |
fix bug Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 860e0e3 | |
|---|---|---|
| Author: | forgive_dengkai | |
enhance download Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | eccec66 | |
|---|---|---|
| Author: | forgive_dengkai | |
Merge remote-tracking branch 'origin/eggroll-logs' into feature-2.5.2-dev
| Commit: | b48ac99 | |
|---|---|---|
| Author: | forgive_dengkai | |
enhance download Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 2078174 | |
|---|---|---|
| Author: | sagewe | |
move containers working dir to <sessionId>/<rank>/ Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 2cdcae3 | |
|---|---|---|
| Author: | sagewe | |
add rendezvous store destroy Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | b5d4104 | |
|---|---|---|
| Author: | sagewe | |
enable compress level Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 799ec1e | |
|---|---|---|
| Author: | sagewe | |
fix store timeout Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 616b6c5 | |
|---|---|---|
| Author: | sagewe | |
add models and logs download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 6f4f7d9 | |
|---|---|---|
| Author: | sagewe | |
fix container killing Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 6056120 | |
|---|---|---|
| Author: | sagewe | |
refact deepspeed container start Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 50c3eb6 | |
|---|---|---|
| Author: | sagewe | |
add resource options Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 5fb1e6e | |
|---|---|---|
| Author: | sagewe | |
| Committer: | sagewe | |
impl job download Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | ff2b157 | |
|---|---|---|
| Author: | sagewe | |
refactor and impl query Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 7e8e470 | |
|---|---|---|
| Author: | sagewe | |
refact deepspeed and container Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | 517acca | |
|---|---|---|
| Author: | sagewe | |
feat: improve deepspeed support Signed-off-by: sagewe <wbwmat@gmail.com>
| Commit: | a6c446a | |
|---|---|---|
| Author: | forgive_dengkai | |
add resource logic Signed-off-by: forgive_dengkai <forgive_dengkai@163.com>
| Commit: | 8140116 | |
|---|---|---|
| Author: | sagewe | |
Merge remote-tracking branch 'origin/feature/deepspeed-sage' into feature/general-job-submit # Conflicts: # jvm/core/main/scala/com/webank/eggroll/core/resourcemanager/ClusterManagerBootstrap.scala