Proto commits in Tencent/Firestorm

These 9 commits are when the Protocol Buffers files have changed:

2022-05-27

[Feature] Remote storage can apply its own extra config (#143) ## What changes were proposed in this pull request? Add remote storage conf Server can apply the specific remote storage conf to its storage manager (HdfsStorageManager for now) client can apply specific conf of the remote storage it fetched The storage related hadoop conf and the hadoop conf in the spark context / mr jobConf should be isolated ## Why are the changes needed? Firestorm supports multiple remote storage and we may want to apply some extra/sepecific conf to different remote storage. ## Does this PR introduce any user-facing change? No ## How was this patch tested? UT and system test

The documentation is generated from this commit.

2022-05-05

[Feature] Support multiple HDFS in Firestorm (Part2 for Shuffle Server & Driver) (#121) What changes were proposed in this pull request? The PR is target to support multiple HDFS with RSS cluster, and it's the 2nd part for Shuffle Server & Driver Why are the changes needed? To release I/O pressure when with multiple HDFS cluster Does this PR introduce any user-facing change? No How was this patch tested? new UT added

2022-04-27

[Feature] Support multiple HDFS in Firestorm (Part1 for Coordinator &… (#110) * [Feature] Support multiple HDFS in Firestorm (Part1 for Coordinator & Driver)

2022-02-22

[Feature] add the dynamic client conf fetching and updating (#81) ### What changes were proposed in this pull request? Add api to fetch client conf from coordinator Dynamic apply the client conf ### Why are the changes needed? Support dynamic update client conf and avoid the requirement to add some client conf other than shuffle manager, client jar. For now, we only read the conf from local file or hdfs, and we could support db and kv store or leverage a config center in the future. ### Does this PR introduce any user-facing change? No ### How was this patch tested? UT & System test

2022-01-19

[Feature] Add AccessManager (#61) # What changes were proposed in this pull request? Add access manager to handle cron task access requests and add checkers to check the candidate list and cluster load respectively. # Why are the changes needed? We need to add candidates checking and control of the access when the cluster load is high. # Does this PR introduce any user-facing change? Yes, need to update the client to enable this feature. # How was this patch tested? UT and system test.

2021-12-24

Support store shuffle data in memory (#36) What changes were proposed in this pull request? This pr is target to store shuffle data in memory. New flush strategy and read strategy are involved. The benefits with this change are: Huge memory is useful now There has no wait for data flush after stage finish Reduce flush data to storage Why are the changes needed? new RPC interface is involved Does this PR introduce any user-facing change? Yes, user need to update configuration for the new feature How was this patch tested? UT add

2021-11-23

[Feature] Firestorm supports node's health check (#14) ### What changes were proposed in this pull request? Firestorm supports node's health check. There are some full disk nodes in our cluster, they influence the shuffle service. We should exclude them.Currently we only check disk usage, and exclude the unhealthy nodes. ### Why are the changes needed? Because we should have the mechanism to find the servers that can't serve, and screen them ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? newly added UTs Co-authored-by: roryqi <roryqi@tencent.com>

2021-11-16

[Feature] Read index file first in local mode (#6) * add local part * add index read length * resolve comments * Adjust some error handling * Use BytesValue unsafe wrap instead of ByteString copyFrom * Remove some unused imports Co-authored-by: macduan <macduan@tencent.com>

2021-10-29

Initial version