Proto commits in securefederatedai/openfl

These 34 commits are when the Protocol Buffers files have changed:

2025-04-30

Batched fetch Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

Disable streaming on send task results for consistency Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

2025-04-15

Remove round_number field: never used, also named_tensor has it Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

2025-04-10

Merge branch 'develop' into karansh1/nparray_tobytes

Extract dtype field at relevant points Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

2025-04-08

Collaborator CLI command for pinging the aggregator (#1516) * Adding a collaborator command for pinging the aggregator, without starting any tasks Signed-off-by: Teodor Parvanov <teodor.parvanov@intel.com> * Addressing review comments Signed-off-by: Teodor Parvanov <teodor.parvanov@intel.com> * Additional logging for the "fx collaborator ping" command Signed-off-by: Teodor Parvanov <teodor.parvanov@intel.com> * Additional documentation on the usage of fx collaborator ping Signed-off-by: Teodor Parvanov <teodor.parvanov@intel.com> --------- Signed-off-by: Teodor Parvanov <teodor.parvanov@intel.com>

The documentation is generated from this commit.

2025-04-04

Generic ndarray transformer Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

2025-03-19

Introducing Task Runner workspace to enable interoperability with Flower workloads (#1433) * enable flwr workspace Signed-off-by: kta-intel <kevin.ta@intel.com> * add tmp dir to minimize how much patching is needed Signed-off-by: kta-intel <kevin.ta@intel.com> * add hash verification to setup data Signed-off-by: kta-intel <kevin.ta@intel.com> * remove patch Signed-off-by: kta-intel <kevin.ta@intel.com> * fix save location Signed-off-by: kta-intel <kevin.ta@intel.com> * docstring Signed-off-by: kta-intel <kevin.ta@intel.com> * remove superfluous edit Signed-off-by: kta-intel <kevin.ta@intel.com> * update readme Signed-off-by: kta-intel <kevin.ta@intel.com> * remove debugger Signed-off-by: kta-intel <kevin.ta@intel.com> * update aggregator client Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * remove todo Signed-off-by: kta-intel <kevin.ta@intel.com> * minor functionality fixes Signed-off-by: kta-intel <kevin.ta@intel.com> * update docstrings Signed-off-by: kta-intel <kevin.ta@intel.com> * update docstrings Signed-off-by: kta-intel <kevin.ta@intel.com> * update README instructions Signed-off-by: kta-intel <kevin.ta@intel.com> * do not add connector to settings unless connector exists Signed-off-by: kta-intel <kevin.ta@intel.com> * check attribute for connector availability Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * code cleanup Signed-off-by: kta-intel <kevin.ta@intel.com> * fix readme Signed-off-by: kta-intel <kevin.ta@intel.com> * update plan to reflect refactoring Signed-off-by: kta-intel <kevin.ta@intel.com> * grammar fix Signed-off-by: kta-intel <kevin.ta@intel.com> * remove __all__ Signed-off-by: kta-intel <kevin.ta@intel.com> * remove superfluous init Signed-off-by: kta-intel <kevin.ta@intel.com> * change flwr home dir Signed-off-by: kta-intel <kevin.ta@intel.com> * remove patch comment, .sort() added in flwr 1.16 Signed-off-by: kta-intel <kevin.ta@intel.com> * update name for grpc protocols and components Signed-off-by: kta-intel <kevin.ta@intel.com> * remove Connector abc Signed-off-by: kta-intel <kevin.ta@intel.com> * refactoring local grpc to interop Signed-off-by: kta-intel <kevin.ta@intel.com> * import fixes Signed-off-by: kta-intel <kevin.ta@intel.com> * remove duplicate self.callback Signed-off-by: kta-intel <kevin.ta@intel.com> * fix hashes Signed-off-by: kta-intel <kevin.ta@intel.com> * move into workspace for relative path installation Signed-off-by: kta-intel <kevin.ta@intel.com> * update readme to remove connector ABC Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com>

2025-03-07

Cleanup (#1422) * DCE, rename variable Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Move to module-level loggers Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Plan.logger to logger Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Fixup Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Also for workspaces Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Fix again for logger Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove run_simulation tests Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Header creation is stateless and common across both Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove TASK_REGISTRY code; deprecated with interactive Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * More use_delta_updates changes Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * lets see if this revert works Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove openfl install in notebook Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * revert few flags Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * please please please this should work Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Migrate stateless function to common Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Cleanup outdated docstrings Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove Interactive API RPCs Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove set devices call; not used anywhere Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Replace checks with inline assertions Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Unfold one-liner functions Signed-off-by: Shah, Karan <kbshah1998@outlook.com> * Remove moot tests Signed-off-by: Shah, Karan <kbshah1998@outlook.com> --------- Signed-off-by: Shah, Karan <kbshah1998@outlook.com>

2025-02-24

Add Secure Aggregation Workflow in pipeline (#1396) * WIP - Add Secure Aggregation Workflow in pipeline Signed-off-by: payalcha <payal.chaurasiya@intel.com> * WIP - Add Secure Aggregation Workflow in pipeline Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Remove stale process Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Remove stale process Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Remove stale process Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Fix summary helper Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * name changes Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Review comments Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> --------- Signed-off-by: payalcha <payal.chaurasiya@intel.com> Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com>

2024-12-17

Federated Runtime - Initial Implementation (#1190) * Introduction to FederatedRuntime Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Updated aggregator Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * updated collaborator & added 101_MNIST federated_runtime tutorial Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Updated code & fixed stream_stdout Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Adding testcases for FederatedRuntime Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Fixed formatting issues Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Fix formatting issues Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Incorporated internal review comments Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Fix checkpoint issue Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Updated FederatedRuntime Tutorials Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Updated tutorial Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Incorporated Teo's review comments Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Incorporated Teo's review comments Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Incorporated review comment Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Incorporate review comment Co-authored-by: Patrick Foley <psfoley@gmail.com> * Updated Workflow Interface documentation Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> * Fix certificates for federated_runtime.py Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> --------- Signed-off-by: Ishant Thakare <ishantrog752@gmail.com> Co-authored-by: Patrick Foley <psfoley@gmail.com>

2024-11-28

Reorganization of OpenFL Experimental Directory Structure (#1169) * reorganize directory structure Signed-off-by: refai06 <refai.ahamed06@gmail.com> * reorganize tutorial dir structure Signed-off-by: refai06 <refai.ahamed06@gmail.com> * reorganize workspace dir structure Signed-off-by: refai06 <refai.ahamed06@gmail.com> * reorganize tests dir structure Signed-off-by: refai06 <refai.ahamed06@gmail.com> * reorganize workspace directory Signed-off-by: refai06 <refai.ahamed06@gmail.com> * reorganize dir Signed-off-by: refai06 <refai.ahamed06@gmail.com> * revert testcase update Signed-off-by: refai06 <refai.ahamed06@gmail.com> * update directory structure Signed-off-by: refai06 <refai.ahamed06@gmail.com> * revert copyright changes Signed-off-by: refai06 <refai.ahamed06@gmail.com> * update requirement Signed-off-by: refai06 <refai.ahamed06@gmail.com> * directory restructure changes Signed-off-by: refai06 <refai.ahamed06@gmail.com> * update directory restructure Signed-off-by: refai06 <refai.ahamed06@gmail.com> --------- Signed-off-by: refai06 <refai.ahamed06@gmail.com>

2024-07-09

Federated Runtime + Workspace Builder (#876) * Federated Runtime v1.0 experimental Aggregator and Collaborator in working condition Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added test, and mnist experimental workspaces working prototype Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Working on integration with Ishant's, and Keerti's code Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Second half of integration installment. Remaning: Integration of data.yaml file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary directory my_federation. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * 1. Removed unncessart files. 2. Fix `fx plan freeze` command not working issue. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added function for test workspace Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added include_exclude testcase workspace. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Default execution workspace is mnist. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * data.yaml changes are integrated with this update. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Running a federation with GaNDLF Documentation (#794) * Initial commit of Running the federation with GaNDLF Documentation Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update README.md Removed references to Intel's ownship, given it's now owned by the LF AI and Data. Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix Flake8 C419 for Ubuntu CI (#800) C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update README.md Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix warnings and issues in docs (#825) * Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> * fixup! Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> --------- Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add Logo (#827) * Add Logo * Update README.md Signed-off-by: Patrick Foley <psfoley@gmail.com> * Change OpenFL documentation font to improve accessibility (#809) This replaces the font of OpenFL documents with Intel One Mono font for low vision developers. Known issues: 1. The text font within the images has not been changed. 2. Some icons that do not exist in the new font cannot be displayed properly. Fixes securefederatedai#799 Co-authored-by: Wang, Le <le3.wang@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update unit tests to improve code coverage (#821) * Update ci config Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add unit test for following files - openfl/federated/plan/plan.py - openfl/interface/aggregation_functions/core/adaptive_aggregation.py Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add some test cases for databases module Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Fix bugs for databases module unittest Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Update unit tests for component module Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Restore workflow config and update some comments Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Enable save_ test case. Add yaml under test dir for unit test usage. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add aggregator start test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 2 aggregator test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 1 aggregator test case. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Format code. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Refactor code. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add collaborator start test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 1 collaborator test case. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Format with flake8 Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove TODO comments Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> --------- Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add PyTorch linear regression example (#808) This adds a new tutorial example on distributing a linear regression task over OpenFL cluster. The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters. Fixes #797 Co-authored-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * This prints out the hash of the CSR to disk for both the aggregator and (#813) * This prints out the hash of the CSR to disk for both the aggregator and collaborator. The user then compares and approves this hash with the hash printed out of the file to validate the CSR. In addition, a warning message is pritned if certify is run in silent mode. Fixes securefederatedai#692 Signed-off-by: Grant Baker <grant.baker@intel.com> * Refactor read_csr function to use get_csr_hash Signed-off-by: Grant Baker <grant.baker@intel.com> * Ask to check hashes before prompt --------- Signed-off-by: Grant Baker <grant.baker@intel.com> Co-authored-by: Grant Baker <grant.baker@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Improve workspace requirements import (#810) Remove the dump_requirement_file operation in export_ method. Fixes securefederatedai#767 Co-authored-by: Li, Qingqing <qingqing.li@intel.com> Co-authored-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Issue 506 Added Example using FedProx (#818) * created new ineractive_api dir to hold pytorch fedprox mnist example corrected files changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example renamed FedProx notebook used mode.parrameters() to get pytorch model weights got weights using state_dict changed old wieghts to list (for serialization) and fixed README input wieghts before zero_grad [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: #506 Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: securefederatedai#506 Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * Update README.md Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README. Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> --------- Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * [Bug: 768] FX CLI: Separate create, cert gen commands (#807) This change separates existing command "fx collaborator.py generate-cert-request" command into two commands. "fx collaborator create -n {NAME} -d {DATA_PATH: optional}". "fx collaborator generate-cert-request -n {NAME}". Fixes #768 Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add new tutorial example to OpenFL interactive API (#812) * Add new tutorial example to OpenFL interactive API This adds a new tutorial example on distributing a linear regression task over OpenFL cluster The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters. Fixes #798 Co-authored-by: Beverly Klemme <beverly.klemme@intel.com> Co-authored-by: Grant Baker <grant.baker@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> * reduced requirements.txt in workspace Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> --------- Signed-off-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (#776) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * build(deps): bump tensorflow (#777) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix documentation links. Change path names and templates for CI Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix paths Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix paths Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix breaking tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add compatible onnx version to requirements.txt file Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong csv file name Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong csv file name Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong names in workflow file Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong data path Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint in test_gandlf.py Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint errors Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com> Co-authored-by: akantak <aleksander.kantak@intel.com> Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: xiaoranf <desolatefang@gmail.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (#841) Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fix GaNDLF documentation links (#842) * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix links in GaNDLF Documentation * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix links in GaNDLF Documentation Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fix incorrectly formatted link in docs (#839) Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added licensing information to all experimental workspaces Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Made changes to LocalRuntime, and tested Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Rename collaborator_name variable to name in experimental/component/collaborator/collaborator.py file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump onnx in /openfl-workspace/gandlf_seg_test (#840) Bumps [onnx](https://github.com/onnx/onnx) from 1.12 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](https://github.com/onnx/onnx/compare/v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Selected collaborators functionality added, test workspace of the same is work-progress. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Selected collaborator testcases included with this commit. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testcase private attributes workspace added with this commit. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Instead of remove private attributes from checkpoint, changed the value to string "Private attributes: Not Available." Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added / Modified types of funciton arguments, and their return types. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Aggregator shutdown related changes are done in this commit Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update testflow_subset_of_collaborators.py Updated subset of collaborators testcase Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified subset of collaborators testcase. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Add files via upload Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp (#773) * build(deps): bump tensorflow in /openfl-workspace/keras_nlp Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update RMSProp optimizer import Signed-off-by: Patrick Foley <psfoley@gmail.com> * flake8 Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_mnist (#771) * build(deps): bump tensorflow in /openfl-workspace/keras_cnn_mnist Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * revert experimental Adam to legacy (#863) Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_with_compression (#770) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Adam Optimizer import Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorials/interactive_api/Flax_CNN_CIFAR (#775) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed breaking backages Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add quiet flag back to pip install Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp_gramine_ready (#769) * build(deps): bump tensorflow-cpu Bumps [tensorflow-cpu](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow-cpu dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Use legacy RMSprop optimizer Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility color contrast fixes (#864) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> * Color contrast updates for accessibility Color contrast updates for accessibility - update generic `a` element - update color of links in the toggle-able read-the-docs panel - update the color of the text in search results - update the color of notes headers Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Tweak link color so it’s not so aggressive (#865) Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added following tutorials, and testcase: 1. Vertical FL 2. Vertical FL Two Party 3. Internal Loop Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * plan.yaml modified in 301 watermarking, and subset of collaborators. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifying temp.sh for 2 collaborators. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update aggregator.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * FQDN switched to localhost FQDN switched to localhost Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * While activating experimental OpenFL commands, install depedencies as well. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Install experimental requirements while activating experimental openfl commands. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update experimental-requirements.txt Added ray 2.2.0 to experimental-requirements.txt file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_workflow_interface.txt Metaflow, and Ray dependencies are moved to openfl-workspace/experimental/experimental-requrirements.txt file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Delete openfl/experimental/utilities/utils.py, and removed torch depency from openfl/experimental/utilities/resources.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_workflow_interface.txt Updated experimental tutorials requirements.txt file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updating 201 GPUs tutorial Updated Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb for Aggregator, and Collaborator as stateful actor Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified 101, 102, 103, 301, 401 tutorials as per Aggregator, & Collaborator stateful collaborators Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Excluding 'experimental' directory from workspace templates Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update 101 tutorial data.yaml Updated 101 tutorials data.yaml to correct a mistake by in index parameter of private attribute function. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update resources.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified get_number_of_gpus function Modify get_number_of_gpus function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Removed torch dependency from "get_number_of_gpus" 2. Added keras mnist example notebook 3. Added keras mnist workspace example 4. Removed openfl-workspace/experimental/experimental-requirements.txt 5. "fx aggregator start" command will install libraries from experimental tutorials requirements_workflow_interface.txt file 6. Added testcase_datastore_cli, testcase_reference and testcase_reference_with_include_exclude 7. Updated 101 torch mnist plan.yaml and data.yaml 8. Removed aggregator start flow thread from cli/aggregator.py 9. At last changes made in federated runtime Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/interactive_api_director/experiments/tensorflow_mnist/envoy (#772) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update sd_requirements.txt * revert to legacy SGD and install tensorflow==2.11 for workflow Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Co-authored-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/tf_2dunet (#774) * build(deps): bump tensorflow in /openfl-workspace/tf_2dunet Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements.txt to retrigger CI * Update requirements.txt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Renamed Workflow_Interface_101_Keras_MNIST_with_GPU.ipynb to Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 2. Updated documentation in Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 3. protobuf bumped from 3.19.6 to 3.20.0 in setup.py 4. Incorporating review comments (WIP) Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified 104 Keras MNIST tutorial Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This comming brings following changes: 1. Experimental Aggregator workflow testcases moved to test/github/experimental/workspace directory 2. Added dependecy related changes Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary logs Removing unnecessary logs Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update collaborator.py Removing unnecessary logs Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Avoided deep copying the context by removing private attributes from context before checkpoint Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Incoporated review comments Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added dependecies to each tutorial notebooks Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * pinned tensorboardX (#870) Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * While taking aggregator step remove private attributes before taking checkpoint and set it back to ctx after checkpoint Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary deepcopies from aggregator.py file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary comments Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update stream_redirect.py Removing unnecessary imports Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added color escape sequences to datastore cli localruntime and workspace testcases Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_privacy_meter.txt torch 1.13.1 requires torchvision 0.14.1 not 0.14.0 Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Workflow_Interface_VFL_Two_Party Added actor related changed to Workflow_Interface_VFL_Two_Party Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Files added required to test experimental aggregator based workflow by GitHUB actions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb Removing GPU parameter from collaborator decorator. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update resources.py Updated get_number_of_gpus function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updating import statement In 104 keras workspace utils.py updating import SummaryWriter statement from torch to tensorflow Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * With this commit path of template can be explicitely given, user does not have to choose from available example templates. Only applicable to experimental openfl. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues and typos Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolved flake8 issues, Added warning statements when data.yaml is not found or aggregator / collaborator private attributes are not found in data.yaml, Tested by running all tutorials and tests Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Moving the logs after data.yaml warning, and removing unnecessary blank lines. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding N-level arguments functionality Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update workflow_interface.rst Added section on how to save best and last model from Metaflow CLI Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving last few TODO items. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updated logs to provide exact information to user. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following updates: 1. Remove unnecessary spaces from Workflow_Interface_102_Aggregator_Validation.ipynb 2. Upgraded protobuf to 3.20.3 in setup.py as per tensorboard requirements 3. Modified 101-torch-cnn-mnist workspace for n-level arguments 4. Added a federated runtime workspace for 102-aggregator-validation Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Added 501_pytorch_tinyimagenet_transfer_learning workspace for experimental workflow interface 2. Removed `feature_shape` from openfl/experimental/interface/cli/plan.py 3. Resolved flake8 errors in openfl-workspace/experimental directory Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update data.yaml Correcting a mistake train dataset assigned to test and vice versa. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update flow.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update flow.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding functionality to convert jupyter notebook to workspace (in progress) Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Write flow.py, collaborator_private_attrs.py and aggregator_private_attrs.py automatically Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Find aggregator, and collaborator arguments Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifying arguments help Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removed automatically generated Zone.Identifier file Added Approach2 notebook Modified creator.py Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * workspace_creator renamed to workspace_builder finding values without help of any tags Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Finished exporting data.yaml, plan.yaml. Workspace creator validated for 101 MNIST and 301 Watermarking with MNIST Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing round number constant argument from 301 MNIST WATERMARKING Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * * Created get_collaborator_kwargs function in localruntime class which takes collaborator name as argument and returns name of private attributes function and arguments required to call it. * Changed workspace builder code used get_collaborator_kwargs function to build data.yaml file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified get_collaborator_args function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified workspace_builder/build.py file to resolve a typo Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removed unnecessary files from template_workspace directory Moved print_tree to after generating python script Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added documentation and inline comments for workspace_builder Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added libraries required by WorkspaceBuilder Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Tensorflow, gRPC, Protobuf dependencies (#868) * Update Tensorflow to latest, finally update grpcio/protobuf Signed-off-by: Patrick Foley <psfoley@gmail.com> * Lint issue fix and missing tf reference Signed-off-by: Patrick Foley <psfoley@gmail.com> * pyzmq version fixed * fix taskrunner tests for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix taskrunner test syntax for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * adding user option to workspace pip install requirements for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix windows CI test Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing venv for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * test venv for windows * test venv for windows * Added new KerasSerializer. Fixed other Interactive API experiments * Update taskrunner.yml * Update taskrunner.yml * Update workspace.py * Update workspace.py * Update taskrunner.yml * Remove get_model import from global namespace so dependencies are not loaded into memory unnecessarily (breaking windows build) * Refactoring and cleaning up imports to support Windows install * Fixed logger import paths * Fix missing imports * Fix native import * Fix lint errors * Fix keras optimizer patch. Remove irrelevant unit test * Format logs in UTF-8 for windows * Update interactive-kvasir.yml * Consolidate github actions python versions to single file * Update python versions * Update python versions * Update python versions * Reduce # of DataLoader workers for Pytorch Kvasir CI test * Fix Windows encoding * Fix Windows encoding and limit rounds so Github Actions CI doesn't run out of memory Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix windows encoding * Fix Windows encoding --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Add FL plan description to documentation (#872) * Add plan description to documentation Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix indentation Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Apply suggestions from code review Co-authored-by: Patrick Foley <patrick.foley@intel.com> --------- Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolve Coverity Issues (#874) * Fix coverity issues * Resolve remaining coverity issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Migrate to Ubuntu 22.04 LTS release (supported through 2027) (#875) Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues in test/experimental directory Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving DCO issue fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update EdenPipeline in the documentation (#877) Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added weighted_average aggregation function under openfl.experimental.interface.{keras,torch}.aggregation_funtions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Tested all local-runtime & federated-runtime tutorials and testcases Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving merge conflicts in experimental local-runtme tutorials 101, 103, 201, & 301 fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * WIP: CI Scans (#873) * Initial scans commit for bandit, hadolint, trivy Signed-off-by: Patrick Foley <psfoley@gmail.com> * Address bandit scan results Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix Trivy action Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix linting Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add Coverity Badge Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update ROADMAP.md (#878) Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Renaming workspace_builder module to workspace_export. 2. WorkspaceBuilder class renamed to WorkspaceExport. 3. Added WorkspaceExport.export function for user which wraps generate_requirements, generate_plan_yaml, generate_data_yaml functions. 4. Merged WorkspaceExport module to `fx workspace create` command by adding new following arguments: - `--notebook` Path to jupyter notebook - `--template_workspace_dir` Output path for generated workspace 5. WorkspaceExport removed from 101, and 103 tutorials. 6. setup.py changed accordingly. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. output_dir renamed to output_workspace also the user must pass this parameter (default=/tmp is now removed) or there will be an error. 2. template_workspace_path is not required anymore. 3. Added a new tutorials Workflow_Interface_1001_Workspace_Creation_from_JupyterNotebook.ipynb (copy of Workflow_Interface_301_MNIST_Watermarking.ipynb) explaining how to use WorkspaceExport. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * In Workflow_Interface_1001 tutorial list of collaborator are changed to 5 from 2 Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Uncommeted pip install libraries Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding GPU for aggregator in 201, Global DP & Privacy Meter tutorials Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifed documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixing 'param' attribute not found issue. in 301, 1001 experimental tutrials and 301 mnist watermarking workspace. Sol. imagen library installs param depedency latest version and param package got an upgrde recently. Pinned param to 1.13.0 after installing imagen which fixed the issue. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * VFL 2 party example Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * participant names to lowercase and changes to instructions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Resolving missing packages from watermarking examples that break python 3.9+. Fix linting, spelling. Remove experimental aggregation functions Signed-off-by: Patrick Foley <psfoley@gmail.com> * Remove duplicate files in openfl.experimental Signed-off-by: Patrick Foley <psfoley@gmail.com> * Trigger CI Signed-off-by: Patrick Foley <psfoley@gmail.com> * Moving test to be conditional Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix inconsistent use of enumerate in loops Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fixing tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Accomodate conversion of names to lower case Signed-off-by: Patrick Foley <psfoley@gmail.com> * Missing torchvision dependency in tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint issues Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com> Co-authored-by: akantak <aleksander.kantak@intel.com> Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: xiaoranf <desolatefang@gmail.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Francis Storr <francis.storr@intel.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Co-authored-by: Keerti Prakash Talwar <115972088+KeertiX@users.noreply.github.com> Co-authored-by: kta-intel <kevin.ta@intel.com> Co-authored-by: porteratzo <44075849+porteratzo@users.noreply.github.com> Co-authored-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Amit Portnoy <1131991+amitport@users.noreply.github.com>

2024-03-01

Federated Runtime + Workspace Builder (#876) * Federated Runtime v1.0 experimental Aggregator and Collaborator in working condition Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added test, and mnist experimental workspaces working prototype Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Working on integration with Ishant's, and Keerti's code Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Second half of integration installment. Remaning: Integration of data.yaml file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary directory my_federation. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * 1. Removed unncessart files. 2. Fix `fx plan freeze` command not working issue. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added function for test workspace Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added include_exclude testcase workspace. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Default execution workspace is mnist. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * data.yaml changes are integrated with this update. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Running a federation with GaNDLF Documentation (#794) * Initial commit of Running the federation with GaNDLF Documentation Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update docs/running_the_federation_with_gandlf.rst Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update README.md Removed references to Intel's ownship, given it's now owned by the LF AI and Data. Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix Flake8 C419 for Ubuntu CI (#800) C419 Unnecessary list comprehension passed to any()/all() prevents short-circuiting - rewrite as a generator Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update README.md Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix warnings and issues in docs (#825) * Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> * fixup! Fix warnings and issues in docs Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> --------- Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add Logo (#827) * Add Logo * Update README.md Signed-off-by: Patrick Foley <psfoley@gmail.com> * Change OpenFL documentation font to improve accessibility (#809) This replaces the font of OpenFL documents with Intel One Mono font for low vision developers. Known issues: 1. The text font within the images has not been changed. 2. Some icons that do not exist in the new font cannot be displayed properly. Fixes securefederatedai#799 Co-authored-by: Wang, Le <le3.wang@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update unit tests to improve code coverage (#821) * Update ci config Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add unit test for following files - openfl/federated/plan/plan.py - openfl/interface/aggregation_functions/core/adaptive_aggregation.py Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add some test cases for databases module Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Fix bugs for databases module unittest Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Update unit tests for component module Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Restore workflow config and update some comments Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Enable save_ test case. Add yaml under test dir for unit test usage. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove plan to new dir. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add aggregator start test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 2 aggregator test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 1 aggregator test case. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Format code. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Refactor code. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add collaborator start test cases. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Add 1 collaborator test case. Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Format with flake8 Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> * Remove TODO comments Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> --------- Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add PyTorch linear regression example (#808) This adds a new tutorial example on distributing a linear regression task over OpenFL cluster. The model is defined by Pytorch which is able to run over both cpu (by default) and gpu. The dataset is generated by make_regression from sklearn.datasets with pre-defined parameters. Fixes #797 Co-authored-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * This prints out the hash of the CSR to disk for both the aggregator and (#813) * This prints out the hash of the CSR to disk for both the aggregator and collaborator. The user then compares and approves this hash with the hash printed out of the file to validate the CSR. In addition, a warning message is pritned if certify is run in silent mode. Fixes securefederatedai#692 Signed-off-by: Grant Baker <grant.baker@intel.com> * Refactor read_csr function to use get_csr_hash Signed-off-by: Grant Baker <grant.baker@intel.com> * Ask to check hashes before prompt --------- Signed-off-by: Grant Baker <grant.baker@intel.com> Co-authored-by: Grant Baker <grant.baker@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Improve workspace requirements import (#810) Remove the dump_requirement_file operation in export_ method. Fixes securefederatedai#767 Co-authored-by: Li, Qingqing <qingqing.li@intel.com> Co-authored-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Jiang, Jiaqiu <jiaqiu.jiang@intel.com> Signed-off-by: Li, Qingqing <qingqing.li@intel.com> Signed-off-by: Wang, Le <le3.wang@intel.com> Signed-off-by: Wu, Caili <caili.wu@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Issue 506 Added Example using FedProx (#818) * created new ineractive_api dir to hold pytorch fedprox mnist example corrected files changed to FedProxOptimizer and ran set_old_weights for new FedProx Pytorch example renamed FedProx notebook used mode.parrameters() to get pytorch model weights got weights using state_dict changed old wieghts to list (for serialization) and fixed README input wieghts before zero_grad [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: #506 Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * [Enhancement: 506] Add an example that uses the FedProx optimizer in the interative_api This duplicates the MedNIST_2D example in the interative api but changes it to use the FedProx optimizer. Fixes: securefederatedai#506 Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * Update README.md Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> * addressed comments by psfoley: corrected words in the jupyter notebook metadata and added a link to the FedProx paper in the README. Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> --------- Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * [Bug: 768] FX CLI: Separate create, cert gen commands (#807) This change separates existing command "fx collaborator.py generate-cert-request" command into two commands. "fx collaborator create -n {NAME} -d {DATA_PATH: optional}". "fx collaborator generate-cert-request -n {NAME}". Fixes #768 Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add new tutorial example to OpenFL interactive API (#812) * Add new tutorial example to OpenFL interactive API This adds a new tutorial example on distributing a linear regression task over OpenFL cluster The model is defined by scikit-learn which is able to run over both cpu (by default) and gpu. The dataset is 1-dimensional noisy data of sinusoid with pre-defined parameters. Fixes #798 Co-authored-by: Beverly Klemme <beverly.klemme@intel.com> Co-authored-by: Grant Baker <grant.baker@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> * reduced requirements.txt in workspace Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> --------- Signed-off-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * build(deps): bump tensorflow in /openfl-workspace/tf_cnn_histology (#776) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * build(deps): bump tensorflow (#777) Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update GaNDLF repo location and test CI with master branch Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix documentation links. Change path names and templates for CI Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix paths Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix paths Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix breaking tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add compatible onnx version to requirements.txt file Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong csv file name Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong csv file name Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong names in workflow file Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix wrong data path Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint in test_gandlf.py Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint errors Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com> Co-authored-by: akantak <aleksander.kantak@intel.com> Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: xiaoranf <desolatefang@gmail.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid (#841) Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fix GaNDLF documentation links (#842) * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix links in GaNDLF Documentation * Fixed GaNDLF rst issues. Add sphinxcontrib-mermaid Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix links in GaNDLF Documentation Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fix incorrectly formatted link in docs (#839) Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added licensing information to all experimental workspaces Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Made changes to LocalRuntime, and tested Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Rename collaborator_name variable to name in experimental/component/collaborator/collaborator.py file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump onnx in /openfl-workspace/gandlf_seg_test (#840) Bumps [onnx](https://github.com/onnx/onnx) from 1.12 to 1.13.0. - [Release notes](https://github.com/onnx/onnx/releases) - [Changelog](https://github.com/onnx/onnx/blob/main/docs/Changelog.md) - [Commits](https://github.com/onnx/onnx/compare/v1.12.0...v1.13.0) --- updated-dependencies: - dependency-name: onnx dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Selected collaborators functionality added, test workspace of the same is work-progress. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Selected collaborator testcases included with this commit. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testcase private attributes workspace added with this commit. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Instead of remove private attributes from checkpoint, changed the value to string "Private attributes: Not Available." Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added / Modified types of funciton arguments, and their return types. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Aggregator shutdown related changes are done in this commit Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update testflow_subset_of_collaborators.py Updated subset of collaborators testcase Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified subset of collaborators testcase. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Add files via upload Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp (#773) * build(deps): bump tensorflow in /openfl-workspace/keras_nlp Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update RMSProp optimizer import Signed-off-by: Patrick Foley <psfoley@gmail.com> * flake8 Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_mnist (#771) * build(deps): bump tensorflow in /openfl-workspace/keras_cnn_mnist Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * revert experimental Adam to legacy (#863) Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/keras_cnn_with_compression (#770) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Adam Optimizer import Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.9.3 to 2.11.1 in /openfl-tutorials/interactive_api/Flax_CNN_CIFAR (#775) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.9.3 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.9.3...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Fixed breaking backages Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add quiet flag back to pip install Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow-cpu from 2.8.4 to 2.11.1 in /openfl-workspace/keras_nlp_gramine_ready (#769) * build(deps): bump tensorflow-cpu Bumps [tensorflow-cpu](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow-cpu dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Use legacy RMSprop optimizer Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility color contrast fixes (#864) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> * Color contrast updates for accessibility Color contrast updates for accessibility - update generic `a` element - update color of links in the toggle-able read-the-docs panel - update the color of the text in search results - update the color of notes headers Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Tweak link color so it’s not so aggressive (#865) Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * External Loop Functionality Added External Loop Functionality Added Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added following tutorials, and testcase: 1. Vertical FL 2. Vertical FL Two Party 3. Internal Loop Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * plan.yaml modified in 301 watermarking, and subset of collaborators. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifying temp.sh for 2 collaborators. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update aggregator.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * FQDN switched to localhost FQDN switched to localhost Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * While activating experimental OpenFL commands, install depedencies as well. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Install experimental requirements while activating experimental openfl commands. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update experimental-requirements.txt Added ray 2.2.0 to experimental-requirements.txt file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_workflow_interface.txt Metaflow, and Ray dependencies are moved to openfl-workspace/experimental/experimental-requrirements.txt file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Delete openfl/experimental/utilities/utils.py, and removed torch depency from openfl/experimental/utilities/resources.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_workflow_interface.txt Updated experimental tutorials requirements.txt file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updating 201 GPUs tutorial Updated Workflow_Interface_201_Exclusive_GPUs_with_Ray.ipynb for Aggregator, and Collaborator as stateful actor Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified 101, 102, 103, 301, 401 tutorials as per Aggregator, & Collaborator stateful collaborators Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Excluding 'experimental' directory from workspace templates Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update 101 tutorial data.yaml Updated 101 tutorials data.yaml to correct a mistake by in index parameter of private attribute function. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update resources.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified get_number_of_gpus function Modify get_number_of_gpus function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Removed torch dependency from "get_number_of_gpus" 2. Added keras mnist example notebook 3. Added keras mnist workspace example 4. Removed openfl-workspace/experimental/experimental-requirements.txt 5. "fx aggregator start" command will install libraries from experimental tutorials requirements_workflow_interface.txt file 6. Added testcase_datastore_cli, testcase_reference and testcase_reference_with_include_exclude 7. Updated 101 torch mnist plan.yaml and data.yaml 8. Removed aggregator start flow thread from cli/aggregator.py 9. At last changes made in federated runtime Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /tests/github/interactive_api_director/experiments/tensorflow_mnist/envoy (#772) * build(deps): bump tensorflow Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update sd_requirements.txt * revert to legacy SGD and install tensorflow==2.11 for workflow Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Co-authored-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * build(deps): bump tensorflow from 2.8.4 to 2.11.1 in /openfl-workspace/tf_2dunet (#774) * build(deps): bump tensorflow in /openfl-workspace/tf_2dunet Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.8.4 to 2.11.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.8.4...v2.11.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Update requirements.txt to retrigger CI * Update requirements.txt --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Renamed Workflow_Interface_101_Keras_MNIST_with_GPU.ipynb to Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 2. Updated documentation in Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb 3. protobuf bumped from 3.19.6 to 3.20.0 in setup.py 4. Incorporating review comments (WIP) Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified 104 Keras MNIST tutorial Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This comming brings following changes: 1. Experimental Aggregator workflow testcases moved to test/github/experimental/workspace directory 2. Added dependecy related changes Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary logs Removing unnecessary logs Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update collaborator.py Removing unnecessary logs Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Avoided deep copying the context by removing private attributes from context before checkpoint Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Incoporated review comments Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added dependecies to each tutorial notebooks Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * pinned tensorboardX (#870) Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * While taking aggregator step remove private attributes before taking checkpoint and set it back to ctx after checkpoint Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary deepcopies from aggregator.py file Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing unnecessary comments Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update stream_redirect.py Removing unnecessary imports Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added color escape sequences to datastore cli localruntime and workspace testcases Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update requirements_privacy_meter.txt torch 1.13.1 requires torchvision 0.14.1 not 0.14.0 Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Workflow_Interface_VFL_Two_Party Added actor related changed to Workflow_Interface_VFL_Two_Party Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Files added required to test experimental aggregator based workflow by GitHUB actions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Workflow_Interface_104_Keras_MNIST_with_GPU.ipynb Removing GPU parameter from collaborator decorator. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update resources.py Updated get_number_of_gpus function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updating import statement In 104 keras workspace utils.py updating import SummaryWriter statement from torch to tensorflow Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * With this commit path of template can be explicitely given, user does not have to choose from available example templates. Only applicable to experimental openfl. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues and typos Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolved flake8 issues, Added warning statements when data.yaml is not found or aggregator / collaborator private attributes are not found in data.yaml, Tested by running all tutorials and tests Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Moving the logs after data.yaml warning, and removing unnecessary blank lines. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding N-level arguments functionality Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update workflow_interface.rst Added section on how to save best and last model from Metaflow CLI Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving last few TODO items. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Updated logs to provide exact information to user. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following updates: 1. Remove unnecessary spaces from Workflow_Interface_102_Aggregator_Validation.ipynb 2. Upgraded protobuf to 3.20.3 in setup.py as per tensorboard requirements 3. Modified 101-torch-cnn-mnist workspace for n-level arguments 4. Added a federated runtime workspace for 102-aggregator-validation Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Added 501_pytorch_tinyimagenet_transfer_learning workspace for experimental workflow interface 2. Removed `feature_shape` from openfl/experimental/interface/cli/plan.py 3. Resolved flake8 errors in openfl-workspace/experimental directory Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update data.yaml Correcting a mistake train dataset assigned to test and vice versa. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update flow.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update flow.py Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding functionality to convert jupyter notebook to workspace (in progress) Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Write flow.py, collaborator_private_attrs.py and aggregator_private_attrs.py automatically Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Find aggregator, and collaborator arguments Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifying arguments help Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removed automatically generated Zone.Identifier file Added Approach2 notebook Modified creator.py Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * workspace_creator renamed to workspace_builder finding values without help of any tags Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Finished exporting data.yaml, plan.yaml. Workspace creator validated for 101 MNIST and 301 Watermarking with MNIST Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removing round number constant argument from 301 MNIST WATERMARKING Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * * Created get_collaborator_kwargs function in localruntime class which takes collaborator name as argument and returns name of private attributes function and arguments required to call it. * Changed workspace builder code used get_collaborator_kwargs function to build data.yaml file. Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified get_collaborator_args function Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified workspace_builder/build.py file to resolve a typo Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Removed unnecessary files from template_workspace directory Moved print_tree to after generating python script Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added documentation and inline comments for workspace_builder Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added libraries required by WorkspaceBuilder Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update Tensorflow, gRPC, Protobuf dependencies (#868) * Update Tensorflow to latest, finally update grpcio/protobuf Signed-off-by: Patrick Foley <psfoley@gmail.com> * Lint issue fix and missing tf reference Signed-off-by: Patrick Foley <psfoley@gmail.com> * pyzmq version fixed * fix taskrunner tests for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix taskrunner test syntax for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * adding user option to workspace pip install requirements for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix windows CI test Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing virtual env for windows github actions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * testing venv for windows Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * test venv for windows * test venv for windows * Added new KerasSerializer. Fixed other Interactive API experiments * Update taskrunner.yml * Update taskrunner.yml * Update workspace.py * Update workspace.py * Update taskrunner.yml * Remove get_model import from global namespace so dependencies are not loaded into memory unnecessarily (breaking windows build) * Refactoring and cleaning up imports to support Windows install * Fixed logger import paths * Fix missing imports * Fix native import * Fix lint errors * Fix keras optimizer patch. Remove irrelevant unit test * Format logs in UTF-8 for windows * Update interactive-kvasir.yml * Consolidate github actions python versions to single file * Update python versions * Update python versions * Update python versions * Reduce # of DataLoader workers for Pytorch Kvasir CI test * Fix Windows encoding * Fix Windows encoding and limit rounds so Github Actions CI doesn't run out of memory Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix windows encoding * Fix Windows encoding --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Add FL plan description to documentation (#872) * Add plan description to documentation Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * fix indentation Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Apply suggestions from code review Co-authored-by: Patrick Foley <patrick.foley@intel.com> --------- Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolve Coverity Issues (#874) * Fix coverity issues * Resolve remaining coverity issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Migrate to Ubuntu 22.04 LTS release (supported through 2027) (#875) Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues in test/experimental directory Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Accessibility updates (#861) * Fix incorrectly formatted link in docs Signed-off-by: Francis Storr <francis.storr@intel.com> * Font styling, color contrast, other accessibility updates This update: 1. Restores Roboto and Lato fonts for most body copy, leaving Intel One Mono for code samples. 2. Adds colors (in `colors.css`) 3. Adds a new `accessibility_overrides.css` file containing CSS that improves the accessibility of the documentation and, where possible, Read The Docs. These updates remediate numerous non-conforming WCAG 2.x Level AA bugs. The use of a separate file for this hopefully makes these changes easier to manage and less likely to be accessibility overwritten in the future. Closes #848 Signed-off-by: Francis Storr <francis.storr@intel.com> --------- Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Testflow for verifying stdout redirection to Metaflow datastore (#758) * implemented ray.wait * reverted changes back after testing * adding datastore cli test case * removed unused variables * removed stderr validation * fixed lint suggestions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving DCO issue fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update EdenPipeline in the documentation (#877) Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixed flake8 issues Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Added weighted_average aggregation function under openfl.experimental.interface.{keras,torch}.aggregation_funtions Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Tested all local-runtime & federated-runtime tutorials and testcases Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Resolving merge conflicts in experimental local-runtme tutorials 101, 103, 201, & 301 fixing FedAvg in workflow interface tutorials to be compatible with latest numpy stable release (1.24.3) (#833) * fixing FedAvg averaging in order to be compatible with numpy v1.24+ Signed-off-by: kta-intel <kevin.ta@intel.com> * uncommenting installations for consistency with other tutorials Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarking tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * fixing 301_MNIST_Watermarki ng tutorial FedAvg Signed-off-by: kta-intel <kevin.ta@intel.com> * Switching to py38 kernel and clearing cell outputs Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * WIP: CI Scans (#873) * Initial scans commit for bandit, hadolint, trivy Signed-off-by: Patrick Foley <psfoley@gmail.com> * Address bandit scan results Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix Trivy action Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix linting Signed-off-by: Patrick Foley <psfoley@gmail.com> * Add Coverity Badge Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <psfoley@gmail.com> * Update Hadolint threshold to flag errors only Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Update ROADMAP.md (#878) Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. Renaming workspace_builder module to workspace_export. 2. WorkspaceBuilder class renamed to WorkspaceExport. 3. Added WorkspaceExport.export function for user which wraps generate_requirements, generate_plan_yaml, generate_data_yaml functions. 4. Merged WorkspaceExport module to `fx workspace create` command by adding new following arguments: - `--notebook` Path to jupyter notebook - `--template_workspace_dir` Output path for generated workspace 5. WorkspaceExport removed from 101, and 103 tutorials. 6. setup.py changed accordingly. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * This commit brings following changes: 1. output_dir renamed to output_workspace also the user must pass this parameter (default=/tmp is now removed) or there will be an error. 2. template_workspace_path is not required anymore. 3. Added a new tutorials Workflow_Interface_1001_Workspace_Creation_from_JupyterNotebook.ipynb (copy of Workflow_Interface_301_MNIST_Watermarking.ipynb) explaining how to use WorkspaceExport. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * In Workflow_Interface_1001 tutorial list of collaborator are changed to 5 from 2 Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Uncommeted pip install libraries Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Adding GPU for aggregator in 201, Global DP & Privacy Meter tutorials Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modifed documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Modified documentation in 1001 tutorial Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * Fixing 'param' attribute not found issue. in 301, 1001 experimental tutrials and 301 mnist watermarking workspace. Sol. imagen library installs param depedency latest version and param package got an upgrde recently. Pinned param to 1.13.0 after installing imagen which fixed the issue. Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> * VFL 2 party example Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * participant names to lowercase and changes to instructions Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> * Resolving missing packages from watermarking examples that break python 3.9+. Fix linting, spelling. Remove experimental aggregation functions Signed-off-by: Patrick Foley <psfoley@gmail.com> * Remove duplicate files in openfl.experimental Signed-off-by: Patrick Foley <psfoley@gmail.com> * Trigger CI Signed-off-by: Patrick Foley <psfoley@gmail.com> * Moving test to be conditional Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix inconsistent use of enumerate in loops Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fixing tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Accomodate conversion of names to lower case Signed-off-by: Patrick Foley <psfoley@gmail.com> * Missing torchvision dependency in tests Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix lint issues Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Parth Mandaliya <parthx.mandaliya@intel.com> Signed-off-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Signed-off-by: Patrick Foley <psfoley@gmail.com> Signed-off-by: Aleksander Kantak <aleksander.kantak@intel.com> Signed-off-by: He, Dan H <dan.h.he@intel.com> Signed-off-by: Fang, Xiaoran <xiaoran.fang@intel.com> Signed-off-by: Klemme, Beverly <beverly.klemme@intel.com> Signed-off-by: Baker, Grant <grant.baker@intel.com> Signed-off-by: ELizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Jillela, Emmanuel <emmanuel.jillela@intel.com> Signed-off-by: Beverly Klemme <beverly.klemme@intel.com> Signed-off-by: Elizabeth Simon, Neethu <neethu.elizabeth.simon@intel.com> Signed-off-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Signed-off-by: Yi CAO <yi2.cao@intel.com> Signed-off-by: Francis Storr <francis.storr@intel.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: kta-intel <kevin.ta@intel.com> Signed-off-by: Mansi Sharma <mansi.sharma@intel.com> Signed-off-by: Amit Portnoy <1131991+amitport@users.noreply.github.com> Co-authored-by: Patrick Foley <patrick.foley@intel.com> Co-authored-by: Sarthak Pati <sarthak.pati@pennmedicine.upenn.edu> Co-authored-by: Prashant Shah <40899779+SprashAI@users.noreply.github.com> Co-authored-by: akantak <aleksander.kantak@intel.com> Co-authored-by: wangleflex <106506636+wangleflex@users.noreply.github.com> Co-authored-by: He, Dan H <dan.h.he@intel.com> Co-authored-by: xiaoranf <desolatefang@gmail.com> Co-authored-by: Wang, Wenjie <wenjie2.wang@intel.com> Co-authored-by: Lei5 Chen <lei5.chen@intel.com> Co-authored-by: Beverly Klemme <35578090+bjklemme-intel@users.noreply.github.com> Co-authored-by: Emmanuel Jillela <emmanuel.jillela@intel.com> Co-authored-by: Yi CAO <yi2.cao@intel.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Francis Storr <francis.storr@intel.com> Co-authored-by: Patrick Foley <psfoley@gmail.com> Co-authored-by: Kevin Ta <116312994+kta-intel@users.noreply.github.com> Co-authored-by: Keerti Prakash Talwar <115972088+KeertiX@users.noreply.github.com> Co-authored-by: kta-intel <kevin.ta@intel.com> Co-authored-by: porteratzo <44075849+porteratzo@users.noreply.github.com> Co-authored-by: Parth Mandaliya <parth.mandaliya.007@gmail.com> Co-authored-by: Mansi Sharma <77758170+mansishr@users.noreply.github.com> Co-authored-by: Mansi Sharma <mansi.sharma@intel.com> Co-authored-by: Amit Portnoy <1131991+amitport@users.noreply.github.com>

2023-01-23

Update Copyright to 2023 (#704) * Update copyright to 2023 Signed-off-by: Patrick Foley <psfoley@gmail.com> * Fix .proto files Signed-off-by: Patrick Foley <psfoley@gmail.com> --------- Signed-off-by: Patrick Foley <psfoley@gmail.com>

2023-01-08

Add DIrector service control RPCs Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com>

2023-01-06

add comments to proto files Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com>

2023-01-05

Fixes to protocols Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com>

Initial Director Admin interface implementation Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com>

2022-12-10

Envoys to review plan before experiment starts (#489) * squashed commits Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com> * fix after rebase Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com> Signed-off-by: igor-davidyuk <igor.davidyuk@intel.com> Co-authored-by: igor-davidyuk <igor.davidyuk@intel.com>

2022-12-09

Handling envoy disconnections on getting experiment. (#623) Signed-off-by: Aleksandr Mokrov <aleksandr.mokrov@intel.com> Signed-off-by: Aleksandr Mokrov <aleksandr.mokrov@intel.com>

2022-03-23

Merge branch 'develop' into dockerezation-launch

2022-02-28

Task Assigner initial implementation (#343) * Additional error handling * Getting registered tasks from task registry (task keeper or task interface) * Excluding non-compatible tests * Aggregation function by task name * Backward compatibility (Task Runner API flow) * Redundant logging * Backward compatibility * Backward compatibility * Backward compatibility * Backward compatibility * Backward compatibility * Backward compatibility * Documentation * Return round incrementation

2022-02-10

Director aggregator communication by rpc (#334) * Implementation of director-aggregator gRPC communication * Retry on unavailable aggregator for async client * Check if the experiment is available * Increase timeout * Retrun instead raise error after timeout * Fix test * enforce_image_hw is string * Col exp can be empty * Artifacts were removed, less dependency from aggregator attribute * Wait experiment readiness to get an aggregator client. * Director requests validation * Some enhancements * Doc strings * Update openfl/federated/plan/plan.py Co-authored-by: Igor Davidyuk <igor.davidyuk@intel.com> * Update plan.py * Remove redundant method * Additional error handling Co-authored-by: Igor Davidyuk <igor.davidyuk@intel.com>

2022-02-09

Additional comments for gRPC methods (#315)

2022-01-27

Redesign protocols. Now we have two services with their protos: Aggre… (#296) * Redesign protocols. Now we have two services with their protos: Aggregator and Director, and common base part: base.proto. * Rebuild protos after merge * flake8 * Fix test: UpdateShardInfo returns response with shard_info attribute * compile_proto.sh was returned to protocols directory with correct paths. Basic proto readme was added.

2022-01-14

Basic collaborator failure handling (#252) * Collaborator failure handle mechanism * Filtering authorised collaborator instead deleting. Redundant code has been removed.

2021-12-13

Docs correction (#220) * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Review and edit content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review content. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * Edit and review. * removed unused files, corrected double spacing in notices, improved phrasing in the title. * merging develop * corrections to director workflow * fix spacing after asterisk. * fix spacing after asterisk. * edited flexperiment start method parameters. * edited flexperiment start method parameters. * fixed odd phrasing and added definition for FL plan. * fix spacing after asterisk. * improved phrasing. * fixed missing articles. * fixed capitalization. * fixed missing articles. * defined mtls. * minor formatting. * improved title for option 1. * mention that this is for aggregator-based workflow. * mention that this is for aggregator-based workflow. * mention that this is for director-based workflow. * added link to director service communication diagrams. * review and edit. * improved title. * review and edit. * review and edit. * review and edit. * added link for openfl-tutorials. * removed conda fixed branding * shorten names in content * shorten names in content 2 * reverting changes * install page corrections * correct install guide 2 * Install rst + index corrections * readme corrections * plugin dadpter correction * alexandrs corrections * removed long naming and fixed code blocks * overview remove nested doc * overview and install correction * overview toctree * changes to director based workflow * States correction and alignment (#257) * Components fixes (#255) * components fixes * model is not loaded with shard descriptor * alexander fixes * Update docs/source/openfl/components.rst * Update docs/source/openfl/components.rst Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com> * Update docs/source/openfl/components.rst Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com> * Update docs/source/openfl/components.rst Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com> * Update docs/source/openfl/components.rst Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com> * Docs correction Manual (last topics) (#258) * docs correction * PR fixes * fix README and Makefile (#261) * Docs correction olga (#259) * Update overview.rst Removed figure (duplicates the next figure) * Add files via upload * Update overview.rst Added picture - Centralized versus Federated Learning * Add files via upload * Update overview.rst Update pictures * Update install.rst * Update install.rst * Update install.rst * Update install.rst * Update overview.rst Updated main terms (aggregator, collaborator, removed FL plan) * Update running_the_federation.rst * Add files via upload * Add files via upload * Delete CT_versus_FL.png * Delete director_workflow.png * Delete director_workflow.png * Add files via upload * Update director_based_workflow.rst * Update director_based_workflow.rst * Delete director_workflow.png * Add files via upload * Update director_based_workflow.rst * Delete director_workflow.svg * Add files via upload * Update components.rst * Update README.md * Docs correction. Installation and Run Federation sections (#256) * install doc update running federation director flow update * install doc update running federation aggregator flow update * fix image path * fix structure * fix text * fix text * fix code-blocks * fix text * fix text * fix text * fix text * fix text * fix text * next/prev page fix (#262) * Dagapov/docs fix (#263) * next/prev page fix * fix python version Co-authored-by: Yong <louis.chen.cherng.yong@intel.com> Co-authored-by: igor-davidyuk <igor.davidyuk@intel.com> Co-authored-by: aleksandr-mokrov <76171391+aleksandr-mokrov@users.noreply.github.com> Co-authored-by: alexey-khorkin <89856652+alexey-khorkin@users.noreply.github.com> Co-authored-by: Olga Perepelkina <olga.perepelkina@intel.com> Co-authored-by: Dmitry Agapov <dmitry.agapov@intel.com>

2021-11-10

Get experiments from director (#174) * init * WIP: get experiment from director * fix * add GetExperimentDescription rpc * add get experiment description * add totalRounds and currentRound to message GetExperimentDescriptionResponse * wip * highlight mocks * get user experiments * fix flake8 * add experiment_name mock for envoy * statuses * rename GetExperimentDescriptions to GetExperimentsList * tabs * envoy plugins * flake8 * methods rename * get_cuda_version refactoring * fix flake8 * change director port

2021-10-29

Cuda devices monitoring (#218) * introduce envoy config with gpus added a message to director.proto with gpu info * added cuda version to message * introduced device_monitor plugin * cuda devices included in shard_info * experiment update * fix repeated field assignmet * cuda status updates * fixes * envoy represented as dict * working example * flake8 fixes * Iliya's suggestion for template unpacking Co-authored-by: Ilya Trushkin <ilya.trushkin@intel.com> * Required fixes * enum fix in collaborator * fix envoy client test * avoid passing new device parameter to old tusk runners * moved experiment * removed unsued files * fix envoy cli after rebase * director fix * fix tests * removed additional notebook * added plugin to setup.py and fixed envoy configs in tutorials * fixed default value for device monitor plugin * fixed tensorflow test * fix rebase * initialized docks * update docs * restore setup.py content * shard-config -> envoy-config renaming * more renamings: shard_config -> envoy_config Co-authored-by: Ilya Trushkin <ilya.trushkin@intel.com>

2021-09-03

use client_id instead of cert name in no tls case (#166) * use client_id instead of cert name in no tls case * fix test * rename caller to sender_name * add default client_id == '__default__'

2021-08-23

Configure envoy health check period form director (#153) * Configure envoy health check period form director * Period in seconds comment * Rename listen_ip -> listen_host. Add params to default director.yaml * Refactor health-check

2021-07-30

Director envoy (#104) * Add new log level - METRIC. Write metrics on aggregator to tensorboard * Refactor code. Add tensorboardX to setup.py * fix flake8 * Return back global logger in native.py * Delete raising exceptions in addLoggingLevel function * Modify log level checking * Apply suggestions from code review Rename tensorboard option Co-authored-by: Ilya Trushkin <76161256+itrushkin@users.noreply.github.com> * Remove log level checking * Pass write_metric function through plan * Pass write metric function also in workspace If we dont define this callback in notebook it will be taken from workspace, if exists * Add docs for log metric callback * Clear notebook output * Add cfssl_scheme.md * Add diagram as png * Add description of CA to clients tls connestion * Fix cfssl scheme * Set 'metric' tag to train task result in interactive api It's necessary for logger to parse this tag * Log metric task result on aggregator additionally. Collaborator logs metric task result, but sometimes collaborator and aggregator have different consoles (in interactive api e.g.) * fix flake8 * Remove default tensorboard option in defaults plan.yaml * Remove tensorboardX dependency * Fix some pep8 issues. Remove debugging grpc code. * VOC shard descriptor * Remove debugging grpc code. * First draft working pki with step-ca * Initial test experiments for new python API * Director - envoy initial commit * Base shard descriptor * Pass shard descriptor to get_collaborator and addition API * Reorganize project structure for director and collaborator manager * Move ca commands to separate file * first LL adaption approach * fix flake8 * First working example * Prepare an aggregator workspace and pass initial tensors * fix flake8 * Adapted launching aggregator on director side * requirements fix * Updated the Kvasir notebook * Add tensorboard to torch_cnn_mnist requirements * Fix typo * Initial test experiments for new python API * Director - envoy initial commit * Base shard descriptor * Pass shard descriptor to get_collaborator and addition API * Reorganize project structure for director and collaborator manager * VOC shard descriptor * first LL adaption approach * Prepare an aggregator workspace and pass initial tensors * First working example * CLI commands * Add tensorboard * Flake8 and logging * Flake8 * Setup logging for experiment * Setup logging for experiment * Change requirements for grpcio * Change requirements for grpcio * Change requirements for grpcio * Remove unused kwargs * Resolve aggregator serving * Resolve aggregator serving * flake8 * Working notebook * Create Updated_Kvasir_with_Director.ipynb * lost fixes after merge * fixed notebook experimnet start * Infrastructure for model retrieving * fixes to the director example * added requirements file for envoy * fixed metric logging * pkg-resources bug fix * pkg-resources fix * pip fix * first try to run and stop aggregator async * added workspace cleaning in director and envoy * removing workspace archive on Frontend API in case of set_new_experiment fail * Update pki scheme to step-ca flow * Draft ca integration * Add ca to utils * Move ca funtions to separate component * Add certificates for api layer director and collaborator-manager * Pass disable_tls option to all grpc clients and servers * New CLI and fixes for the step-ca component * flake8 fixes * Pass certificates name to collaborator and aggregator * Fix rank_worldsize value * Fix flake8 * flake8 * Separate install and run command for pki * Stream metrics to frontend RPC * UnpackWorkspace context manager * local changes * Add step-ca pki mermaid scheme to documentation. * flake8 * Resolve merge conflicts * Backward compatibility * Fix cert api * make cert paths absolute * Fix simultaneous requirements installation * Change default and max token and certificate durations * Import order * First implementation of API registry * Added metric logging on frontend * Tensorboard logs: removed hardcoded name * Add description of pki workflow to documentation * Health check and flake8 * Fix pb2 import * Add info about token live time and other. * Add check for bin downloading responce * Change link * Add ca_path as optional to certify command * Add ca_path as optional to certify command * Replaced old shard registry by new * Fixes, renaming and error handling * Rename collaborator_manager -> envoy * Configure a director address,fix certificates path checking, add director tests * Collaborator-manager renamed to envoy. * _read_image_ids is astatic method. No reason to call it from class. * Break after successful installation * Remove extra method. * Additional certificate path checks. * Separate a transport layer from the director. Running server details was hidden. * Director redesign * Fix pr comments * flake8 * Cleanup gitignore * Cleanup * Small fixes by comments * Generate default experiment name with timestamp * Assign only online collaborator to experiment by default. GetRegisteredShards was deleted. * tests for components are added * created default shard_descriptor Rewrited summary writer, plus small cli fixes for ll entities * Refactor tensorboard metrics + default envoy files * Add Dmitriy fixes for ca part * Fixes by pr comments * Fixes by pr comments * Collaborator manager -> Envoy * Up -> up * Collaborator selection for experiment by default changed * Gramma * Comments and doc strings * use delimiter for token * fix typo * typo fix * Fix token encoding * flake8 * Start federation fixes Co-authored-by: igor-davidyuk <76463150+igor-davidyuk@users.noreply.github.com> Co-authored-by: Radionov, Maksim <maksim.radionov@intel.com> Co-authored-by: maradionov <77286045+maradionov@users.noreply.github.com> Co-authored-by: Ilya Trushkin <76161256+itrushkin@users.noreply.github.com> Co-authored-by: Igor Davidyuk <igor.davidyuk@intel.com> Co-authored-by: Dmitry Agapov <dmitry.agapov@intel.com>

2021-01-16

OpenFL initial commit