Proto commits in Alluxio/alluxio

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2024-09-05

Add CRC64 checksum check Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#18682 orig-commit: Alluxio/alluxio@1a3fbc9338075b5b8793b9a4a5f6b22afd5a0256 orig-commit-author: elega <445092967@qq.com> pr-link: Alluxio/alluxio#18688 change-id: cid-4eb62e603591f3235b92e4191c5b511b534f6fc3

Add CRC64 checksum check ### What changes are proposed in this pull request?  ### Why are the changes needed? To check the file integrity ### Does this PR introduce any user facing changes? Add a crc64 check cli pr-link: Alluxio/alluxio#18682 change-id: cid-4eb62e603591f3235b92e4191c5b511b534f6fc3

The documentation is generated from this commit.

2024-05-13

Add UfsUrl and its UT ### What changes are proposed in this pull request? Add UfsUrl and its unit test into feature branch. pr-link: Alluxio/alluxio#18606 change-id: cid-f7d535b48a00f2cbdf013425a1e201e9f5ad6937

2023-11-10

Allow rename() to overwrite, fix an uncaught ex and reenable UT ### What changes are proposed in this pull request? As the title states, this PR does a few things: 1. Allow rename() to overwrite an existing path, if specified in the option 2. The method `rename` in `AbstractFileSystem`, now will process the input path and rerun `rename` method when caught `AlluxioException` or `AlluxioRuntimeException`. Instead of log the exception and return end. 3. Also add some path checks during the rename 4. Reenable `FileSystemRenameIntegrationTest` by adding configs to `LocalAlluxioClusterResource` and reenable UT cases. pr-link: Alluxio/alluxio#18263 change-id: cid-2870bf87fea8a3b2419e5b10a05423ff2dede6a2

2023-11-08

Update Load replica job ### What changes are proposed in this pull request? Update load job to adopt multi replicas ### Why are the changes needed? part of PR to support loading multi replicas ### Does this PR introduce any user facing changes? new load option `replicas` pr-link: Alluxio/alluxio#18320 change-id: cid-7d01ca19a28faf4c7773cbf5c355dd6cf070728f

2023-10-30

Update Load replica job ### What changes are proposed in this pull request? Update load job to adopt multi replicas ### Why are the changes needed? part of PR to support loading multi replicas ### Does this PR introduce any user facing changes? new load option `replicas` pr-link: Alluxio/alluxio#18320 change-id: cid-7d01ca19a28faf4c7773cbf5c355dd6cf070728f

2023-10-26

Add regx pattern file filter for distributed load Add regx pattern file filter for distributed load. **Example:** The following request allows us to load the files under `/test-load` directory with "hello" prefix: `curl -X GET http://localhost:28080/v1/load?path=s3a://jiamingmai-test/test-load&opType=submit&verbose=true&fileFilterRegx=^hello.*` pr-link: Alluxio/alluxio#18311 change-id: cid-4ec2bfe58bfba413f6d2925f5b3937bd6f5c2eb1

Update worker api support for load multi replicas ### What changes are proposed in this pull request? Update worker api support for load multi replicas ### Why are the changes needed? part of PR to support load multi replicas ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#18296 change-id: cid-0213f2aba669b7687ac42cf932cdcec911d397a4

2023-10-24

Improve cold read performance by prefetching data on workers In the position reader, preloading pages on workers to improve the cold read performance. To improve the cold read performance. N/A pr-link: Alluxio/alluxio#18317 change-id: cid-97e28711cf7f7b3ce60da346c737505a365d3238

Improve cold read performance by prefetching data on workers ### What changes are proposed in this pull request? In the position reader, preloading pages on workers to improve the cold read performance. ### Why are the changes needed? To improve the cold read performance. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#18317 change-id: cid-97e28711cf7f7b3ce60da346c737505a365d3238

2023-10-23

Improve cold read performance by prefetching data on workers

2023-10-19

Add concurrent data prefetching

2023-10-18

Load big file ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#18285 change-id: cid-675c904a992ef44cd62d1daa51d4672234093483

2023-10-13

Use new worker id on worker and client ### What changes are proposed in this pull request? Use new worker identity for worker registration and client-side cluster management. ### Why are the changes needed? To allow workers to use generated IDs that are not assigned by the master. To allow clients to build its view of the cluster using more efficient and reliable worker identity implementation. ### Does this PR introduce any user facing changes? The worker identity is now persisted in etcd, if the worker uses etcd-based registration. pr-link: Alluxio/alluxio#18210 change-id: cid-b044e5648329715bd59172a6fd39b406ef1cdad3

2023-09-29

Add worker vCPU info to fsadmin report capacity command Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#18185 orig-commit: Alluxio/alluxio@263b96f41e0b2bf776b5ce16a631542b0b6478d1 orig-commit-author: Rico Chiu <Xenorith@users.noreply.github.com> pr-link: Alluxio/alluxio#18229 change-id: cid-b972079ff380e4ac32fa2e013b3034931f94786a

Add worker vCPU info to fsadmin report capacity command provide more information about worker state pr-link: Alluxio/alluxio#18185 change-id: cid-b972079ff380e4ac32fa2e013b3034931f94786a

Add worker vCPU info to fsadmin report capacity command provide more information about worker state pr-link: Alluxio/alluxio#18185 change-id: cid-b972079ff380e4ac32fa2e013b3034931f94786a

2023-09-24

Enable static consistent hash ring By default, we build a dynamic consistent hash with the live worker list that comes from master or ETCD. Sometimes we want to build a static consistent hash ring to make sure we won't write data to other worker node when a worker node is offline temporarily (especially when other worker nodes are running out of disk space). This PR provides allows us to build a static consistent hash ring by setting `alluxio.user.dynamic.consistent.hash.ring.enabled=false`. In this case, client will read from UFS if the worker where the specified file locate is down. pr-link: Alluxio/alluxio#18183 change-id: cid-ccd37436391d62aa7c6a9942c80e2d3c8ffdb170

2023-09-23

Support unmount and remount in one atomic operation ### What changes are proposed in this pull request? Support unmount and mount to another mount point within the same lock ### Why are the changes needed? Support replace ufs for a mount point. The operation is atomic so no user operations should fail due to observing the middle state. ### Does this PR introduce any user facing changes? A new rpc field and cmd options are added. pr-link: Alluxio/alluxio#17984 change-id: cid-d96b4df7d24ab1082e46e6d5a72563534714a48f

2023-09-22

Add new worker identity ### What changes are proposed in this pull request? 1. Add an abstract representation of a worker's identity. The new identity is the same as a `long` based identity that used to be generated by the master. It decouples the concept of an identity and the actual implementation of an identity. It also includes a version field that allows backward compatible evolution of the worker identifier. 2. A new implementation for worker identities that uses UUID. A worker will generate its own identity instead of acquiring one from the master, unless the user explicitly specifies one in the configuration. The generated identity is then persisted in the worker's working directory, and reloaded at the next worker restart. ### Why are the changes needed? For Dora workers, the identities are not generated by a centralized service, but by workers themselves. Therefore, we need something that provides randomness and avoids collision between workers, e.g. a UUID. The V1 parser implementation included in this PR uses a UUID as the worker identity. ### Does this PR introduce any user facing changes? Yes, a new configuration item `alluxio.worker.identity.uuid` is added that allows a user to specify the worker identity. pr-link: Alluxio/alluxio#17820 change-id: cid-3200e78fa24be0126d4d49370f0a266f8c766e12

2023-09-15

Improve distributed load ### What changes are proposed in this pull request? Improve distributed load 1. Configurable job failure criteria 2. Configuration to determine if the load job should be restored from journal or not 3. Add an option to skip existing fully loaded file 4. Add retry count for failed files 5. Bug fixing ### Why are the changes needed? To enhance the distributed load tool ### Does this PR introduce any user facing changes? Yes. The skip-if-exists option is added to the distributed load cli. pr-link: Alluxio/alluxio#18153 change-id: cid-5644da1c09bd6ee48f628552f51cb570de581b93

2023-09-01

Remove tiered identity ### What changes are proposed in this pull request? Remove tiered identity ### Why are the changes needed? This feature is no longer supported ### Does this PR introduce any user facing changes? N/A Entry points have been removed previously. This PR removes the src code pr-link: Alluxio/alluxio#18033 change-id: cid-92a6d61ad51f12b708721e26ceab063359b5f2e5

2023-08-25

Revamp distributed load

2023-08-16

Skip copy if needed ### What changes are proposed in this pull request? Skip copy process if validation fails. Also adding more info to the progress report. ### Why are the changes needed? Feature request. ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17804 change-id: cid-c0affd0be9609f9e43a3c901247d27ce17021e85

Skip copy if needed

Remove the usage of must_cache and async_cache ### What changes are proposed in this pull request? current version didn't support must_cache and async_cache, so remove the usage of must_cache and async_cache usage in code and mark deprecated to prevent any use. ### Why are the changes needed? This change focuses on removing the write type of MUST_CACHE and ASYNC_CACHE, and https://github.com/Alluxio/alluxio/pull/17980 would remove the master-side code about async persist. ### Does this PR introduce any user facing changes? No pr-link: Alluxio/alluxio#17963 change-id: cid-14d40a217134beb8a5e7316b050af01bca18b65e

2023-08-15

Add UfsUrl design and implementation ### What changes are proposed in this pull request? This pr contains the implementation of UfsUrl and UfsUrl version of ls command. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#17869 change-id: cid-7765d24cd56d6c1c7c52d698b8dfa0ced3c98854

2023-08-11

Remove active sync on the master side ### What changes are proposed in this pull request? Remove active sync feature including RPCs and relevant commands. ### Why are the changes needed? The file metadata and all related features are no longer needed at the Master. This is one step is removing FileSystemMaster/BlockMaster. ### Does this PR introduce any user facing changes? The property keys and corresponding commands are removed. pr-link: Alluxio/alluxio#17964 change-id: cid-1830d7c9ca0d6d2e8c773147bea56a84d2c57360

2023-08-01

Add multipart upload interface & hdfs minicluster test ### What changes are proposed in this pull request? Add multipart upload interface & add HDFS unit tests using HDFS minicluster ### Why are the changes needed? To increase code quality ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#17862 change-id: cid-e01bdb47bdeb9218354040f69587a3550972bdc5

2023-07-23

Remove async-write fallback (UFS tier) ### What changes are proposed in this pull request? Remove the experimental feature to fallback async-write when Alluxio space is full. ### Why are the changes needed? This feature has been experimental for a few years, not been able to graduate. Given it adds a good amount of complexity, we plan to remove it. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including - Property key `alluxio.user.file.ufs.tier.enabled` is removed pr-link: Alluxio/alluxio#17796 change-id: cid-72acdb41ec1164f0ec7abdd5284a580271430855

2023-07-16

Remove Short Circuit read/write ### What changes are proposed in this pull request? Remove source code related to Short Circuit read/write ### Why are the changes needed? - In K8s env (increasingly more popular), support short-ciruit I/O is challenging - We shall use SDK instead ### Does this PR introduce any user facing changes? No more Short Circuit read/write pr-link: Alluxio/alluxio#17243 change-id: cid-45ee1b3ba2b26ea1e38b47c60c875ac9a733fbea

2023-07-13

Add filter to copy jobs ### What changes are proposed in this pull request? Add filter to copy jobs. ### Why are the changes needed? To add copy job to policy engine. ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17757 change-id: cid-5fd5d0520e617694d00979277c5c83a1c4cf5fb0

2023-07-12

Remove table module (2nd attempt) ### What changes are proposed in this pull request? This commit represents a second effort at removing the table module along with its associated definitions and properties. Our initial attempt, PR #17220, was reverted due to a conflict with Presto and Trino codebases, as some removed classes were still being referenced. Unlike the previous attempt, this commit ensures that all externally referenced classes (e.g., TableMasterClient.java) remain within the codebase to prevent blocking older Presto/Trino builds with newer Alluxio clients post table module removal. ### Why are the changes needed? Structured data service gets depreciated in 2.9 and should be removed in 300 or later. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. catalog service removed 2. transformation removed 3. table command removed (CLI already removed in 2.9) pr-link: Alluxio/alluxio#17762 change-id: cid-0af2cd1b5849746cb542b189f2f85776304220cc

Add filter to copy jobs

2023-07-05

Use a more efficient way to store content hash ### What changes are proposed in this pull request? Instead of comparing the fingerprint, we compare the fields in the file info object directly, to make it more efficient. ### Why are the changes needed? In https://github.com/Alluxio/alluxio/pull/17458, we introduced a mechanism that populates fingerprint and persists that into metastore so that we can skip invalidating unnecessary page caches when the metadata is updated. Such mechanism is protected under a property key as it does not perform good. Now after the discussion with @dbw9580 , we came up with a more efficient way where we just leverage the fields in the file info object and we believe we are able to get rid of such switch. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#17725 change-id: cid-c0e7898b674cdfc5094e5eb1ee996b3a580c3bf3

2023-06-30

Add move request to job factory ### What changes are proposed in this pull request? Add move job request ### Why are the changes needed? Need for distMv command. ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17712 change-id: cid-86899c3b7d75bb24077c3f19dc2431e27aac293c

Add move request to job factory ### What changes are proposed in this pull request? Add move job request ### Why are the changes needed? Need for distMv command. ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17712 change-id: cid-86899c3b7d75bb24077c3f19dc2431e27aac293c

2023-06-29

Add move request to job factory

2023-06-27

Add the exists and setAttribute client-worker RPCs ### What changes are proposed in this pull request? Implements the exists and setAttribute client-worker RPCs ### Why are the changes needed? Client completes all operations by RPCs to workers. Client only does fallback in case of failures when fallback is enabled. ### Does this PR introduce any user facing changes? No. pr-link: Alluxio/alluxio#17682 change-id: cid-20cd1b0a9f97f372c6ff2adc887eb7b8c6e53afc

2023-06-15

Add get mount id to worker master rpc ### What changes are proposed in this pull request? Add get mount id to worker master rpc ### Why are the changes needed? support get mount id from worker ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17625 change-id: cid-b9e179fad02234bc345532bd70fdd3ec1e61dc6d

2023-06-12

Improve dora load job ### What changes are proposed in this pull request? 1. Introduced an iterable UFS status listing interface 2. Avoid calling getStatus from dora worker. Passing the UfsStatus via grpc request 3. Add integration test for Dora load command 4. Provide users with an option to only load metadata ### Why are the changes needed? Currently during load, for each file, a getStatus() is called and this is inefficient. This PR does the listing on the scheduler side and send it to workers via rpc. ### Does this PR introduce any user facing changes? load command can run with an --metadataOnly option to only load metadata pr-link: Alluxio/alluxio#17538 change-id: cid-be3ea2532d0c7a21ff5d1aa9e9871fb4c0df6fd6

Implement metadata sync V2 Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17242 orig-commit: Alluxio/alluxio@71bd2029b53c3dbd0a1c9055d0f68bfcfee4de75 orig-commit-author: elega <445092967@qq.com> pr-link: Alluxio/alluxio#17589 change-id: cid-8163ac604eaa6f9d93a7099fd9007ee7594fcecd

Implement metadata sync V2 ### What changes are proposed in this pull request? In this PR, we implemented a new metadata sync architecture. <img width="1311" alt="Screen Shot 2023-05-04 at 2 43 22 PM" src="https://user-images.githubusercontent.com/6771554/236129584-adc9635b-0bef-41d7-8ba4-648f6c01e122.png"> In the new architecture, metadata sync requests are processed as tasks. Tasks are executed asynchronously managed by the task tracker. There are 4 major components in the system: Task Tracker - Creates metadata tasks, controls resources and dedups concurrent sync on the same path The Path Loader - Manages the path loading process and breakdown the loading task into batches The UFS Loader - Loads a single batch of UFS status for a path (up to 1000 files) The Update Metadata - Compares the delta between UFS status and inodes in alluxio and updates the alluxio file metadata In addition to the internal change, we also revamped the load metadata cli command backed by the metadata sync v2. The feature is still experimental and can only be triggered by the loadMetadata cli command. S3 is the only supported UFS. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. Better resource control. We redesigned rate limitation mechanism and fetches file from UFS by batch (backed by S3 listObjectsV2 endpoint) to reduce the memory consumption. 2. Better performance. We introduced aws client sdk v2 in this PR where an s3 async client is provided with better performance. Also we redesigned the concurrency to make it more efficient as well as other minor optimizations. Result: We tested the new code path to load a s3 dataset of 5 million files. We observed 50% to 100% performance improvement and less memory consumption. We used to see OOM on a 8gb alluxio master but now the issue is gone. ### Does this PR introduce any user facing changes? Yes. The loadMetadata v2 command is revamped. pr-link: Alluxio/alluxio#17242 change-id: cid-8163ac604eaa6f9d93a7099fd9007ee7594fcecd

2023-06-08

Add delete/rename/createDirectory RPC ### What changes are proposed in this pull request? - Add delete/rename/createDirectory RPC. Invalidate metadata and data if file/dir is deleted or renamed. - Continue operations in DoraFileOutStream.close() when exception happens. - reduce worker block heartbeat interval. - decrease Netty close timeout value. - add PropertyKeys in MultiProcessCluster for Dora. - Add PropertyKeys in AbstractFuseDoraTest for Dora. - use different names to avoid test conflicts in FuseFileSystemMetadataTest ### Why are the changes needed? Client writes data synchronously to worker PagingStore as a cache, and to UFS as write through. So delete/rename/createDirectory RPCs are going to do metadata and data operations and invalidations on worker side. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#17545 change-id: cid-b8905052e043bbf9142b40ca3d1e3c3773013e98

2023-06-07

Add mount table support for move ### What changes are proposed in this pull request? 1. refactor move API for ee 2. add get mount Id API so we can use this API to get correct ufs in ee ### Why are the changes needed? na ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17565 change-id: cid-81489ef469d4134526b38b263dd1ef57f926cc85

2023-06-02

Merge master-2.x commits 2023/05/01~2023/05/31 into main Merge missing commits from master-2.x to main. The commits in 2023/05/01~2023/05/31 from https://github.com/Alluxio/alluxio/compare/main...master-2.x will be included by this PR. pr-link: Alluxio/alluxio#17483 change-id: cid-44e281750e687d6b4a119fbd19e17baf1f5c667e

Deprecate some Raft journal RPCs Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17358 orig-commit: Alluxio/alluxio@4a57bab5a96ff34f7eaa5eca196157e49d905665 orig-commit-author: Arthur Jenoudet <23088925+jenoudet@users.noreply.github.com> pr-link: Alluxio/alluxio#17540 change-id: cid-d69c86f931511b5c85a55a419a9399ba6d9718f9

2023-06-01

Migrate policy related proto to dora ### What changes are proposed in this pull request? Migrate policy related proto to main so journal entry change can be picked up. ### Why are the changes needed? Journal entry ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17523 change-id: cid-d3e4d5814ea4d0fac1c84266f9f2b7afe349c3d5

[SMALLFIX] Fix proto format ### What changes are proposed in this pull request? Fix proto format ### Why are the changes needed? Fix proto format ### Does this PR introduce any user facing changes? NA pr-link: Alluxio/alluxio#17535 change-id: cid-2402ef89435685195f2f67be7f4edaecfc7a98ec

2023-05-31

Deprecate some Raft journal RPCs Following #16998, some messages, fields, and RPCs are no longer in use. This PR marks them as deprecated. pr-link: Alluxio/alluxio#17358 change-id: cid-d69c86f931511b5c85a55a419a9399ba6d9718f9

cherry-pick without resolving conflicts: Enhance embedded journal checkpointing significantly Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#16998 orig-commit: Alluxio/alluxio@8cbcbcd6d3f3451f2bfeb326efadfd1372c9351e orig-commit-author: Arthur Jenoudet <23088925+jenoudet@users.noreply.github.com> pr-link: Alluxio/alluxio#17486 change-id: cid-0db7ec0d037a4476eeb2433059fff58bc970b652

2023-05-30

Add CreateFile/CompleteFile support in worker ### What changes are proposed in this pull request? - client sends CreateFile request to worker - worker checks if the target file already exists - worker calls UFS.create() to create an OutStream - worker returns a FileInfo and UUID back to client. Client should embed this UUID in subsequent requests - client closes OutStream. It sends CompleteFile to worker, and worker closes open file handle, updates local metadata. ### Why are the changes needed? To support write. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#17510 change-id: cid-09d58691194c20f1b6d346e960f111db57f4a77f

2023-05-29

Add initial framework for write support in Dora - CreateFile from client to worker - CompleteFile from client to worker - OpenFileHandle in worker - OpenFileHandles in worker pr-link: Alluxio/alluxio#17505 change-id: cid-8cf626c579dec84284265354ad7026430e8ecb95

2023-05-26

Update scheduler to work with copy ### What changes are proposed in this pull request? make scheduler universally work for copyjob + doraloadjob ### Why are the changes needed? resolve conflict of copyjob to use the new scheduler ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#17368 change-id: cid-e67f2b0a931944e56d35b06f421223e7d8eb2c4f

2023-05-25

Add move job What changes are proposed in this pull request? Adding the move job to the EE main branch. Why are the changes needed? The move job is essential for the PPDM hardening. Pls see the docs: https://docs.google.com/document/d/1RsDr_378Oc7VoXwDIE-7pM4uYOY2iikK2KEYaipIjq8/edit?usp=sharing Does this PR introduce any user facing changes? No pr-link: Alluxio/alluxio#17482 change-id: cid-323c57e87cd4dcc6d73f3de2ca05f109bdaf7fb0

Add move job

2023-05-24

cherry-pick and resolve a small conflict: Support display config hash and timestamp in web page Display the config hash and config updated time <img width="550" alt="image" src="https://user-images.githubusercontent.com/17329931/207004719-8c684f3d-24c7-4fce-bd95-32ff795b6172.png"> <img width="534" alt="image" src="https://user-images.githubusercontent.com/17329931/207027205-865183d3-c288-4c99-bbee-a53c6d4ba579.png"> Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#16642 change-id: cid-075d4f9ea1cee81f23706781dcf5cc5ca155e416

2023-05-22

cherry-pick and resolve location conflicts: Improve proxy register functionality and command Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4081 orig-commit: TachyonNexus/enterprise@ac514e4d5ab33746c100f90564ecb102c9f9c73a orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17225 change-id: cid-d4d2cdbb3dacd37eced812e10c267d0e567917da

cherry-pick and resolve conflicts: Support gracefully shutdown worker Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4058 orig-commit: TachyonNexus/enterprise@85246fe11db2f57d2eddfcd36875dd098f985074 orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17310 change-id: cid-bf474aac505427be6180828a158b5c23825b4d93

cherry-pick and resolve location conflicts: Fix JobServiceMetricsCommandTest Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4063 orig-commit: TachyonNexus/enterprise@e39e948e567d8ec5e05079a9b54001c6900e818d orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17314 change-id: cid-c3cae8f23c6b5695b6908217e815f09fe5edb625

2023-05-19

Show master versions in fsadmin report command ./alluxio fsadmin report summary  edit: updated screenshot pr-link: Alluxio/alluxio#17177 change-id: cid-d8075ccbbcdaed6c4dd488428d20c59718e125c3

Port #16849 worker register to all masters to main(dora) ### What changes are proposed in this pull request? Cherry-pick change https://github.com/Alluxio/alluxio/commit/ebeac49c3cfc9a224cf8772ebe1c3e38ae1a0b5a#diff-bfc619faf02f71e266105e0649b12b06f2bba271e49d1f91894feca6e0c59515 to main(dora) Also manually resolve conflicts and adapt to dependency injection. pr-link: Alluxio/alluxio#17462 change-id: cid-5889a60d1f35d38102d0f546d352d8871f9d07fa

Merge master-2.x commits 2023/03/16~2023/03/31 into main Merge missing commits from master-2.x to main. The commits in 2023/03/16~2023/03/31 from https://github.com/Alluxio/alluxio/compare/main...master-2.x will be included by this PR. **Exceptions:** https://github.com/Alluxio/alluxio/commit/e106cc8bbb3d74d876c9869a0b2fe77240e80af9 is not included because it has been manually cherry-picked to dora. https://github.com/Alluxio/alluxio/commit/71521f631d1bd9c15ffe05634a7251014f4be1a0 is not included due to the same reason as above. https://github.com/Alluxio/alluxio/commit/73b4d67c8b39a6358897ec349370e7037a6fea7f is not included because we missed https://github.com/Alluxio/alluxio/commit/ebeac49c3cfc9a224cf8772ebe1c3e38ae1a0b5a from https://github.com/Alluxio/alluxio/pull/17435. Instead of resolving conflicts, it's better to cherry-pick those impacted PRs one by one in the time order, in order NOT to lose code. https://github.com/Alluxio/alluxio/commit/d680e56180ce9a8fc67a878a4e23c62954ac337c is not included because it will been reworked in https://github.com/Alluxio/alluxio/pull/17419 pr-link: Alluxio/alluxio#17449 change-id: cid-10751d1267eb97a0eca0725e0ff0323e13caf186

2023-05-18

Update copy job ### What changes are proposed in this pull request? Couple fixes for copy on dora 1. support passing check content option from cli 2. support passing ufs full path from cli 3. delete temp file only when exceptional case 4. skip if file is already in dst instead of throw exception 5. cleanup getStatus method in ufs and fix bugs that we don't pass options down to some ufs 6. add real content hash to local ufs and test ### Why are the changes needed? improvement ### Does this PR introduce any user facing changes? new `checkContent`option for copy job pr-link: Alluxio/alluxio#17422 change-id: cid-4f8e32cd028fba27daaf9d6bc02c7070cd49d65c

cherry-pick and resolve conflict with #17315 on file_system_master.proto: Support executing runTests on specific workers Support executing the runTests command on specific workers. e.g. ./alluxio runTests --workers worker1:29998,worker2:29998 If the workers are specified, runTests will be executed for each worker with writes and reads to go a dedicated worker. <img width="1048" alt="Screen Shot 2023-03-30 at 3 45 20 PM" src="https://user-images.githubusercontent.com/6771554/228784580-99a066db-b325-4750-8e84-0a052c9eeb6c.png"> <img width="879" alt="Screen Shot 2023-03-30 at 3 46 44 PM" src="https://user-images.githubusercontent.com/6771554/228784598-680f076c-bdcc-4de3-b42b-47fa95d34add.png"> So that we can verify if workers are up N/A pr-link: Alluxio/alluxio#17170 change-id: cid-08eb445461ff6248def8164cfa529c070b8e5e9a

cherry-pick and resolve conflicts: Add worker decommission step 1 This pr is related to #13758 This change achieves below functionalities: 1. Provide a client shell command to kick out workers directly. 2. Decommissioned worker nodes are not automatically shutdown and are not chosen for new IO requests 3. Decommissioned workers are not accepted on heartbeat or register Note there's a delay in when clients know the workers have been decommissioned. In the window, target worker will still receive requests. There's also no way for the worker to come back to the cluster. This command should only be used when this worker is going to leave the cluster (and never come back). Offline workers more flexible, no need to wait master detection lost worker interval. For instance, We may encounter situations like we want to remove worker nodes, or we want to offline some worker nodes to save costs, this command is useful. Yes pr-link: Alluxio/alluxio#14000 change-id: cid-822a1cad1b23cd381b907705c1b92bd6ac3f16d0

2023-05-17

cherry-pick and resolve conflicts between #16806 and #17305: Reduce time cost of checking bucket path in S3 API - I create `BUCKET_PATH_CACHE` as a static object to `S3RestHandlerService`. - I create property key `PROXY_S3_BUCKETPATHCACHE_TIMEOUT_MS` to determine how long each entry should be automatically removed from the cache. - I create function `checkPathIsAlluxioDirectory(FileSystem, String,@Nullable S3AuditContext, Cache<AlluxioURI, Boolean> bucketPathCache)` ,which plays the same role as `checkPathIsAlluxioDirectory(FileSystem, String,@Nullable S3AuditContext)` but more efficient. - I change S3RestHandlerService.java to call `checkPathIsAlluxioDirectory` with argument `BUCKET_PATH_CACHE`. - I add boolean value `checkS3BucketPath` as a field of `CreateFilePOptions` and `CreateDirectoryPOptions`. This flag is used in `FileSystemMasterClientServiceHandler` to check the bucket path existent if `checkS3BucketPath` is `TRUE`. 1. `BUCKET_PATH_CACHE` highly improves the efficiency while creating objects. For example: Set up an Alluxio cluster with 1 master(aws ec2 m5.2xlarge) and 2 workers(aws ec2 m5.4xlarge); benchmark S3 API with warp cmd. ``` WARP_ACCESS_KEY=minioadmin WARP_SECRET_KEY=minioadmin ./warp put --concurrent=80 --obj.size=64KiB --host=cluster0120-masters-1 --disable-multipart --insecure --md5 --duration=3m ``` Benchmark result is like: |PUT operation|before|after| |-|-|-| | Average Throughput |48.49 MiB/s, 775.82 obj/s|59.66 MiB/s, 954.57 obj/s| |Time percentage: checkPathIsAlluxioDirectory()/createObjectOrUploadPart()|7.70%|0.50%| |Time percentage: checkPathIsAlluxioDirectory()/total_time|3.31%|0.21%| 2. Users can modify `alluxio.proxy.s3.bucketpathcache.timeout` in `conf/alluxio-site.properties` to change the time period of cache expiration. Users can modify `alluxio.proxy.s3.bucketpathcache.timeout` in `conf/alluxio-site.properties` to change the time period of cache expiration. The default value is `1min`. Users can set it `0min` to close the function. |Property Name|Default|Description| |-|-|-| |alluxio.proxy.s3.bucketpathcache.timeout|1min| Expire bucket path statistics in cache for this time period. Set `0min` to disable the cache.| > Note: `create file` and `create bucket` operations are always accurate , but other operations bring incosistent results because of the cache to bucket path. All in all, alluxio makes sure that every modification is cosistent and correct. pr-link: Alluxio/alluxio#16806 change-id: cid-fdaff4b5a6fdd3eca834cfaf8caea3fe38618005

cherry-pick and resolve location conflict: Display more information in WebUI Masters Add more information in WebUI Masters: 1. Start time. 2. Elected / Step-down time. 3. Last checkpoint time. 4. Number of journal entries since checkpoint. 5. Build version. Add fields in meta master RPC: 1. RegisterMasterPOptions * start time * primacy change time * version * revision 2. MasterHeartbeatPOptions * last checkpoint time * journal entries since checkpoint Add metrics keys: 1. Master process start time. 2. Last gain primacy time. 3. Last lose primacy time. `MasterInfo` and WebUI pages are changed accordingly. Make Masters tab in WebUI more informative. Fix: #16709 <img width="1705" alt="WebUI Masters" src="https://user-images.githubusercontent.com/5821159/207555228-f79fc246-692d-459a-9282-413490f7ff00.png"> Fixes https://github.com/Alluxio/alluxio/issues/8765 pr-link: Alluxio/alluxio#16636 change-id: cid-ce162c4a6b2341b974daaa8f0823791ed5c032b2

2023-05-16

Improve LS by allowing to omit UFS and mount info ### What changes are proposed in this pull request? Improve listStatus 5X performance in some scenarios. ### Why are the changes needed? For instance, Under the scenario: 1. Use Hadoop compatible system to access Alluxio(listStatus) 2. There are many mount point such as more than 500. 3. There are more than 2000 files in a directory. The PathUtils.hasPrefix (comes from MountTable.getMountPoint)method will be called at least 10w (500 * 2000) times. But actually we don't need the information of mount point. The test can be reduced from about 400ms to 70ms under the master branch. The test can be reduced from about 700ms to about 100ms under 2.7.1 branch. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including no pr-link: Alluxio/alluxio#16893 change-id: cid-71d9a351744033426fb1b0633d7b194f94f322b5

Support rm directory and reset DirectChildrenLoadedState ### What changes are proposed in this pull request? Support delete file from alluxio only and reset its directChildrenLoadedState of parent. ### Why are the changes needed? ### Does this PR introduce any user facing changes? ```Console $ bin/alluxio fs rm -R --alluxioOnly --syncParentNextTime=true /test/bb $ bin/alluxio fs ls /test/ drwxr-xr-x mbl staff 0 PERSISTED 05-30-2022 14:45:24:142 DIR /test/bb $ bin/alluxio fs rm -R --alluxioOnly --syncParentNextTime=false /test/bb $ bin/alluxio fs ls /test/ # /test/bb will not be loaded because metadata sync on /test is disabled ``` pr-link: Alluxio/alluxio#15647 change-id: cid-95acf6dd9c2ee820821e382a2a7332486c865caa

Change default TTL action from DELETE to DELETE_ALLUXIO ### What changes are proposed in this pull request? This change makes ttl default action be free. And some changes to make tests passed. ### Why are the changes needed? Fix #12316 ### Does this PR introduce any user facing changes? No pr-link: Alluxio/alluxio#16823 change-id: cid-142490712d94004a0303f18399d4637e12d81523

Add allowOnStandbyMasters option for version grpc endpoint ### What changes are proposed in this pull request? Add a rejectOnStandbyMasters on version grpc endpoint ### Why are the changes needed? We added a PR to make standby master return unavailable on this version service endpoint https://github.com/Alluxio/alluxio/pull/16854 However, in addition to the polling master inquire client, the AbstractMasterClient also needs this endpoint. https://github.com/Alluxio/alluxio/blob/master/core/common/src/main/java/alluxio/AbstractClient.java#L172 When a client makes a request to standby master, such check will constantly fail and resulted in failures. So we want the logic done in #16854 only applies to PollingMasterInquireClient and hence we added this boolean field to bypass the check. ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#16890 change-id: cid-379f8af78250d2a230bceff9d7ea75739979e198

2023-05-15

Add security server version ### What changes are proposed in this pull request? Add security server version ### Why are the changes needed? Add security server version so that it is possible to extend. ### Does this PR introduce any user facing changes? NA pr-link: Alluxio/alluxio#17434 change-id: cid-8afc03475aa28c7fb2819121d6f30f9625f6fcea

2023-05-12

Support display config hash and timestamp in web page ### What changes are proposed in this pull request? Display the config hash and config updated time <img width="550" alt="image" src="https://user-images.githubusercontent.com/17329931/207004719-8c684f3d-24c7-4fce-bd95-32ff795b6172.png"> <img width="534" alt="image" src="https://user-images.githubusercontent.com/17329931/207027205-865183d3-c288-4c99-bbee-a53c6d4ba579.png"> ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#16642 change-id: cid-075d4f9ea1cee81f23706781dcf5cc5ca155e416

2023-05-11

Add a command to set DirectChildrenLoaded on dir ``` $ bin/alluxio fs syncDirNextTime true /data Successfully marked the dir /data to trigger metadata sync on next access $ bin/alluxio fs syncDirNextTime false /data Successfully marked the dir /data to skip metadata sync on next access ``` pr-link: Alluxio/alluxio#16542 change-id: cid-78744676d83727f380d4c7a4ef7206044001a5fa

2023-05-09

Implement metadata sync V2 ### What changes are proposed in this pull request? In this PR, we implemented a new metadata sync architecture. <img width="1311" alt="Screen Shot 2023-05-04 at 2 43 22 PM" src="https://user-images.githubusercontent.com/6771554/236129584-adc9635b-0bef-41d7-8ba4-648f6c01e122.png"> In the new architecture, metadata sync requests are processed as tasks. Tasks are executed asynchronously managed by the task tracker. There are 4 major components in the system: Task Tracker - Creates metadata tasks, controls resources and dedups concurrent sync on the same path The Path Loader - Manages the path loading process and breakdown the loading task into batches The UFS Loader - Loads a single batch of UFS status for a path (up to 1000 files) The Update Metadata - Compares the delta between UFS status and inodes in alluxio and updates the alluxio file metadata In addition to the internal change, we also revamped the load metadata cli command backed by the metadata sync v2. The feature is still experimental and can only be triggered by the loadMetadata cli command. S3 is the only supported UFS. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. Better resource control. We redesigned rate limitation mechanism and fetches file from UFS by batch (backed by S3 listObjectsV2 endpoint) to reduce the memory consumption. 2. Better performance. We introduced aws client sdk v2 in this PR where an s3 async client is provided with better performance. Also we redesigned the concurrency to make it more efficient as well as other minor optimizations. Result: We tested the new code path to load a s3 dataset of 5 million files. We observed 50% to 100% performance improvement and less memory consumption. We used to see OOM on a 8gb alluxio master but now the issue is gone. ### Does this PR introduce any user facing changes? Yes. The loadMetadata v2 command is revamped. pr-link: Alluxio/alluxio#17242 change-id: cid-8163ac604eaa6f9d93a7099fd9007ee7594fcecd

2023-05-04

Support gracefully shutdown worker Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17310 orig-commit: Alluxio/alluxio@141ee0e567a4c6c60907f85ea66b828704bd761d orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17345 change-id: cid-bf474aac505427be6180828a158b5c23825b4d93

Merge branch 'master' of https://github.com/Alluxio/alluxio into feature/metadata-sync-redesign-2

Fix standby master version not shown ### What changes are proposed in this pull request? Fix a bug where standby masters don't send the version info to primary. This bug only emerges in 2.8 and was caused by cherry picking. pr-link: Alluxio/alluxio#17370 change-id: cid-44b5fe9bf0e021f8ce2351027d1a3717736a2572

2023-05-03

Fix file fingerprint to use atomic get content hash of uploaded file Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#16597 orig-commit: Alluxio/alluxio@00da77cf69c0ca29acecc4c37aed6ba53b9e93d2 orig-commit-author: Tyler Crain <tycrain@gmail.com> pr-link: Alluxio/alluxio#17361 change-id: cid-64723be309bdb14b05613864af3b6a1bb30cba6d

2023-05-02

Reorg dora directories ### What changes are proposed in this pull request? ``` re-org the dora directories: /common/transport /dora/integration /job /shaded /stress /table /generated /underfs /pom.xml /core /examples /shell /minicluster /tests /lib /microbench ``` ### Why are the changes needed? Re-org directories to indicate dora explicit ### Does this PR introduce any user facing changes? NA pr-link: Alluxio/alluxio#17355 change-id: cid-e762b1ce623383e02a327f8da9ff2662f9f2d9b2

2023-04-28

Port change [#17310] to branch-2.8 ### What changes are proposed in this pull request? Based on the existing `decommissionWorker` command, changes are: 1. `decommissionWorker` command is moved from fs command to fsadmin command 2. `decommissionWorker` command now accepts a batch of workes 3. This command now is idempotent, meaning it can be fully retried even when some workers have been decommissioned. 4. This command now blocks for a while and waits for the batch of workers to become idle. 5. Endpoints are added on teh worker to monitor idleness. 6. By default workers decommissioned by `decommissionWorker` can join the cluster when they re-register. `-d` can be used to specify workers cannot register again. 7. `enableWorker` is added, as the reverse operation of `decommissionWorker -d`. So after `enableWorker` command, those workers can register again. Example: ``` $ bin/alluxio fsadmin decommissionWorker --addresses 192.168.3.8:30000 Decommissioning worker 192.168.3.8:30000 Set worker 192.168.3.8:30000 decommissioned on master Sent decommission messages to the master, 0 failed and 1 succeeded Failed ones: [] Clients take alluxio.user.worker.list.refresh.interval=2min to be updated on the new worker list so this command will block for the same amount of time to ensure the update propagates to clients in the cluster. Verifying the decommission has taken effect by listing all available workers on the master Now on master the available workers are: [] Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 Polling status from worker 192.168.3.8:30000 There is no operation on worker 192.168.3.8:30000 for 20 times in a row. Worker is considered safe to stop. Waited 0 minutes for workers to be idle All workers are successfully decommissioned and now idle. Safe to kill or restart this batch of workers now. ``` ### Why are the changes needed? This improves the existing decommission feature in that it blocks until the workers are idle. So the admin can directly kill those workers once this command returns 0. ### Does this PR introduce any user facing changes? Yes pr-link: Alluxio/alluxio#17346 change-id: cid-9d7509a3c94ee4341703a2236f3ee19eefcecf40

2023-04-27

Support gracefully shutdown worker Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4058 orig-commit: TachyonNexus/enterprise@85246fe11db2f57d2eddfcd36875dd098f985074 orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17310 change-id: cid-bf474aac505427be6180828a158b5c23825b4d93

Support gracefully shutdown worker Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4058 orig-commit: TachyonNexus/enterprise@85246fe11db2f57d2eddfcd36875dd098f985074 orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17310 change-id: cid-bf474aac505427be6180828a158b5c23825b4d93

Fix listStatus from ufs ### What changes are proposed in this pull request? Fix the issue that when Copy job using ufsIterable, the listStatus doesn't apply to the file. So add an API to fetch file information as well. Also, use the full ufs path through the whole copy process. ### Why are the changes needed? na ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17340 change-id: cid-9eea57866018d37e996c7812cd43517a4634800b

small fixes

Port change [#17314] to branch-2.8 Manually cherry pick https://github.com/Alluxio/alluxio/pull/17314 ### What changes are proposed in this pull request? 1. Standby job masters register with the primary and report info like version. 2. A command to display all job masters in cluster. 3. Adds supporting code for the above. 4. All job workers report version to the primary job master. 5. Display in command all job worker versions too. ``` ✗ bin/alluxio fsadmin report jobservice Master Address State Start Time Version Revision localhost:20001 PRIMARY 20230425-110043 2.10.0-SNAPSHOT ac6a0616 localhost:20021 STANDBY 20230425-110130 2.10.0-SNAPSHOT ac6a0616 localhost:20011 STANDBY 20230425-110117 2.10.0-SNAPSHOT ac6a0616 Job Worker Version Revision Task Pool Size Unfinished Tasks Active Tasks Load Avg 192.168.3.8 2.10.0-SNAPSHOT ac6a0616 10 0 0 9.01904296875, 11.87939453125, 9.30322265625 Status: CREATED Count: 0 Status: CANCELED Count: 0 Status: FAILED Count: 90 Status: RUNNING Count: 0 Status: COMPLETED Count: 0 10 Most Recently Modified Jobs: ... 10 Most Recently Failed Jobs: ... 10 Longest Running Jobs: ... ``` ### Does this PR introduce any user facing changes? The command output is improved pr-link: Alluxio/alluxio#17334 change-id: cid-ee867212deb3082ab5fa895fe7fd8176337e0cdd

Support getting real content hash from hdfs when using getFileStatus ### What changes are proposed in this pull request? Support getting real content hash from hdfs ### Why are the changes needed? for data integrity while moving data between object store & HDFS ### Does this PR introduce any user facing changes? na pr-link: Alluxio/alluxio#17207 change-id: cid-b2dbbde6082a3fd771b988f3ba4430fb2506f559

2023-04-26

Show master versions in fsadmin report command Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17177 orig-commit: Alluxio/alluxio@9a4e154e7293667572fec25c63136462fb58a345 orig-commit-author: elega <445092967@qq.com> pr-link: Alluxio/alluxio#17339 change-id: cid-d8075ccbbcdaed6c4dd488428d20c59718e125c3

Show master versions in fsadmin report command ./alluxio fsadmin report summary  edit: updated screenshot pr-link: Alluxio/alluxio#17177 change-id: cid-d8075ccbbcdaed6c4dd488428d20c59718e125c3

Port changes of [#17170] to branch-2.8 ### What changes are proposed in this pull request? Support executing the runTests command on specific workers. e.g. ./alluxio runTests --workers worker1:29998,worker2:29998 If the workers are specified, runTests will be executed for each worker with writes and reads to go a dedicated worker. <img width="1048" alt="Screen Shot 2023-03-30 at 3 45 20 PM" src="https://user-images.githubusercontent.com/6771554/228784580-99a066db-b325-4750-8e84-0a052c9eeb6c.png"> <img width="879" alt="Screen Shot 2023-03-30 at 3 46 44 PM" src="https://user-images.githubusercontent.com/6771554/228784598-680f076c-bdcc-4de3-b42b-47fa95d34add.png"> ### Why are the changes needed? So that we can verify if workers are up ### Does this PR introduce any user facing changes? N/A pr-link: Alluxio/alluxio#17338 change-id: cid-ff52daafd92744069279a15242f7605f3c8df7cb

Port changes of [#16521] to branch-2.8 Manual cherry pick of https://github.com/Alluxio/alluxio/pull/16521 pr-link: Alluxio/alluxio#17335 change-id: cid-633063176cb03bd9d4e0b32472c949063d4a17a1

Address remaining issues

Fix JobServiceMetricsCommandTest Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17314 orig-commit: Alluxio/alluxio@8da59539201cff376c33256846bd565d976a148e orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17331 change-id: cid-c3cae8f23c6b5695b6908217e815f09fe5edb625

Improve proxy register functionality and command Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17225 orig-commit: Alluxio/alluxio@dbac084c1e1d89c1e9253da036746e2df9e1ef14 orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17330 change-id: cid-d4d2cdbb3dacd37eced812e10c267d0e567917da

Fix JobServiceMetricsCommandTest Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4063 orig-commit: TachyonNexus/enterprise@e39e948e567d8ec5e05079a9b54001c6900e818d orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17314 change-id: cid-c3cae8f23c6b5695b6908217e815f09fe5edb625

Improve proxy register functionality and command Cherry-pick of existing commit. orig-pr: Alluxio/alluxio#17225 orig-commit: Alluxio/alluxio@dbac084c1e1d89c1e9253da036746e2df9e1ef14 orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17327 change-id: cid-d4d2cdbb3dacd37eced812e10c267d0e567917da

Improve proxy register functionality and command Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4081 orig-commit: TachyonNexus/enterprise@ac514e4d5ab33746c100f90564ecb102c9f9c73a orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17225 change-id: cid-d4d2cdbb3dacd37eced812e10c267d0e567917da

2023-04-25

Fix JobServiceMetricsCommandTest Cherry-pick of existing commit. orig-pr: TachyonNexus/enterprise#4063 orig-commit: TachyonNexus/enterprise@e39e948e567d8ec5e05079a9b54001c6900e818d orig-commit-author: Jiacheng Liu <jiacheliu3@gmail.com> pr-link: Alluxio/alluxio#17314 change-id: cid-c3cae8f23c6b5695b6908217e815f09fe5edb625

Copy on dora ### What changes are proposed in this pull request? Please outline the changes and how this PR fixes the issue. ### Why are the changes needed? Please clarify why the changes are needed. For instance, 1. If you propose a new API, clarify the use case for a new API. 2. If you fix a bug, describe the bug. ### Does this PR introduce any user facing changes? Please list the user-facing changes introduced by your change, including 1. change in user-facing APIs 2. addition or removal of property keys 3. webui pr-link: Alluxio/alluxio#17315 change-id: cid-79d7a506307961a64df8bae613397271af3a0e69