Proto commits in opencurve/curve

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2023-12-12

curvefs: add a new distributed transaction model to improve rename performance Signed-off-by: wanghai01 <seanhaizi@163.com>

The documentation is generated from this commit.

2023-12-08

curvefs: support get mdsAddrsOverride from active mds Signed-off-by: h0hmj <h0hmjcn@gmail.com>

curvefs/mds: add rpc SetClientMdsAddrsOverride Signed-off-by: h0hmj <h0hmjcn@gmail.com>

2023-12-05

curvefs: fix trash will delete file data more than once Signed-off-by: wanghai01 <seanhaizi@163.com>

2023-11-23

add poolset for clone

add protect & unprotect & children

add clone readonly feature

add some test case & fix some bugs

add localsnapshot restful api

add local clone client & mds logical & unit test & integration test

mds,client: optimize clone latency, move clone segment logical from Clone interface to GetOrAllocateSegment

fix concurrent apply for clone

add flatten function

remove virtualid logic

tools-v2: add delete snapshot

add test case and add verison support and fix bug add memory sanity check switch

fix bugs

create snapshot use snapfilename

replace cloneno with fileId & remove originchunkid

Refactor chunkserver to generate layers of snapshots A chunk might have multiple snapshots when we remove the assumption that snapshots are always exported and deleted afterwards. Changes are: 1. Introduce CSSnapshots to manage snapshots; 2. `correctedSn' no longer used for CoW logic. 3. Two new fields are introduced to `ChunkRequest': - snapSn, the snapshot sequence number; - snaps, all snapshot sequence numbers. TODO: 1. managing of `snaps' field in client side. 2. CRUD of `internal' snapshots. 3. storage consumption and statistics - take care of CoW files. 4. Recover & clone of `internal' snapshots. 5. caching of meta objects, and take care of memory footprint. 6. optimize deletion - no need to merge if intended to delete both file and snapshots. 7. Remove `correctedSn' from `ChunkFileMetaPage' and 'CSChunkInfo'. UT: 1. `test/chunkserver/datastore/curve_datastore_unittest' still fails. 2. new unit tests is still under development. Signed-off-by: David Lee <live4thee@gmail.com> fix all ut failure in chunkserver/datastore 1. fix bitmap wrong size bug in CSSnapshots::Read method 2. Deletion of snapshot is only allowed when chunk.sn larger than latest snapshot sn 3. fix ut failure Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Add ut case for local multi-snapshot feature 1. add manual ut case for multi-snapshot datastore and all passed 2. fix some bug Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Support multi-snapshot in client side Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> Support read snapshot in curve client for the purpose of snapshot consistency check Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> Add CreateSyncSnapshot method in snapshotclone and multi-snapshot consistency check tool Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> Code improvement about client read-snapshot implementation 1. handle cases where snapshot sequence num in invalid 2. do not call IncreaseEpoch for snapshot file 3. other stuff Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> Fix nebd-part1 block during exit Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> eliminate the requirement of S3 store Signed-off-by: David Lee <live4thee@gmail.com> (cherry picked from commit 28751eab2cd56761fd5a01d6710a96e56cfae192) Support multi-level snapshot deletion orderly Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com> Fix ReadChunkSnap bug Implement OnChunkNotExist for ReadChunkSnapClosure to handle cases where chunk does not exist in chunkserver Signed-off-by: Zhengwei Zhu <zhengwei.zhu@zstack.io> Signed-off-by: David Lee <live4thee@gmail.com>

2023-11-17

[feat] curvefs: merge two rpc into one when delete dentry Signed-off-by: swj <1186093704@qq.com>

2023-10-25

curvefs/mds: accept fs usage delta message from client and write to etcd Signed-off-by: h0hmj <h0hmjcn@gmail.com>

curvefs/client: update fs delta bytes when refresh mds session Signed-off-by: h0hmj <h0hmjcn@gmail.com>

2023-10-24

Chunk pool format asyn Signed-off-by: yyyyufeng <fyu80612@gmail.com>

curvefs/mds: add update fsinfo capacity Signed-off-by: h0hmj <h0hmjcn@gmail.com>

2023-10-09

feat(fs): metaserver support asynchronous snapshot close: #1617 old metaserver snapshot that contain `inode` and `dentry` incompatible with new version Signed-off-by: NaturalSelect <2145973003@qq.com>

2023-09-12

[feat] curvefs: merge two rpc into one when call makenode Signed-off-by: swj <1186093704@qq.com>

2023-07-21

[feat]curvefs: change hostname to optional Signed-off-by: swj <1186093704@qq.com>

2023-07-20

curve/tools-v2:bs 1. add go_package in proto/*.proto 2. add bs proto in mk-proto.sh 3. add bs query file Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

[feat] tools-v2: add update leader-schedule Signed-off-by: montaguelhz <1443171175@qq.com>

[tools-v2] Add tools-v2 to version v1.2.7 Signed-off-by: caoxianfei1 <caoxianfei@corp.netease.com> Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

curvefs: support space deallocate for curvebs volume as backend Signed-off-by: ilixiaocui <ilixiaocui@163.com>

2023-07-06

curvebs: support 512 aligned IO Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

CurveBs:Add size of chunkfilepool to optimize whether the logical pool is allocated Signed-off-by: Zhao ZeYu <2302370185@qq.com> Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2023-07-03

curvebs: support 512 aligned IO Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2023-06-25

[feat] tools-v2: add update leader-schedule Signed-off-by: montaguelhz <1443171175@qq.com>

2023-06-12

curvebs: support poolset Co-authored-by: jolly-sy <757050468@qq.com> Co-authored-by: Hanqing Wu <wuhanqing@corp.netease.com> Signed-off-by: jolly-sy <757050468@qq.com> Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

curvefs: recovey recyclebin file

[feat] add status chunkServer 1. add report version in chunkServer heartbeat 2. save chunkServer version in mds 3. add bs status chunkserver in tools-v2 Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

2023-06-08

curvebs: support poolset Co-authored-by: jolly-sy <757050468@qq.com> Co-authored-by: Hanqing Wu <wuhanqing@corp.netease.com> Signed-off-by: jolly-sy <757050468@qq.com> Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2023-05-31

[WIP] Some bug fixes during testing Signed-off-by: ilixiaocui <ilixiaocui@163.com>

curvefs: support space deallocate for curvebs volume as backend Signed-off-by: ilixiaocui <ilixiaocui@163.com>

2023-05-30

fix_enableSumInDir Signed-off-by: jbk <2634358021@qq.com> add_atomic Signed-off-by: jbk <2634358021@qq.com>

2023-05-19

curvefs: support space deallocate for curvebs volume as backend Signed-off-by: ilixiaocui <ilixiaocui@163.com>

2023-05-18

[feat] add status chunkServer 1. add report version in chunkServer heartbeat 2. save chunkServer version in mds 3. add bs status chunkserver in tools-v2 Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

2023-05-09

curvebs/mds: add getChunkserverInCluster rpc api Signed-off-by: wanghai01 <seanhaizi@163.com>

2023-04-25

curvefs: change the data organization format Signed-off-by: swj <1186093704@qq.com>

2023-02-20

curvebs/mds: add getChunkserverInCluster rpc api Signed-off-by: wanghai01 <seanhaizi@163.com>

2023-02-10

curvefs: clean up resources after failing to create fs Signed-off-by: Ziy1-Tan <ajb459684460@gmail.com>

2023-02-02

fix getxattr return wrong length Signed-off-by: wanghai01 <seanhaizi@163.com>

2023-02-01

curvefs: fix getxattr return wrong length Signed-off-by: wanghai01 <seanhaizi@163.com>

curvefs: fix getxattr return wrong length Signed-off-by: wanghai01 <seanhaizi@163.com>

2023-01-16

fix getxattr return wrong length Signed-off-by: wanghai01 <seanhaizi@163.com>

2022-12-18

add curvefs recycle bin Signed-off-by: chenwei <461432360@qq.com>

2022-12-09

curvefs: support memcache cluster 1. add define in topology.proto 2. add RegistMemcacheCluster in topology service 3. add define in topology_item 4. add AllocOrGetMemcacheCluster in client/mdsclinet 5. add ListMemcacheCluster in topology service Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

2022-12-08

curvefs: support client to memcached cluster Signed-off-by: ilixiaocui <ilixiaocui@163.com>

2022-11-18

CurveBs:Add size of chunkfilepool to optimize whether the logical pool is allocated Signed-off-by: Zhao ZeYu <2302370185@qq.com>

2022-11-04

curve/toolsv2 develop Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

2022-10-31

CurveBs:Add size of chunkfilepool to optimize whether the logical pool is allocated Signed-off-by: Zhao ZeYu <2302370185@qq.com>

2022-09-08

curvefs: remove some code not used anymore Signed-off-by: wanghai01 <seanhaizi@163.com>

curve/tools-v2:bs 1. add go_package in proto/*.proto 2. add bs proto in mk-proto.sh 3. add bs query file Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

curvefs/mds: check fsname 1. skip check s3info if fs has been created 2. add check fsname in create fs 3. add check fsname in toolsv-2 Signed-off-by: Cyber-SiKu <Cyber-SiKu@outlook.com>

2022-09-07

curvefs/metaserver: fix inode create time inconsistently Signed-off-by: fan <yfan3763@gmail.com>

curvefs: remove some code not used anymore Signed-off-by: wanghai01 <seanhaizi@163.com>

2022-08-30

curvefs/client: fix cto bugs Signed-off-by: ilixiaocui <ilixiaocui@163.com> related to issue: 1213/1582/1413 1. All mount points on the same file system are either enableCto, or they do not guarantee consistency. 2. If enableCto, attr and dentry cache will auto set invalid. 3. Consistency of multi-mount nodes under delete operation is not guaranteed. If one mount point deletes a file, the behavior of the other mount point for all operations on that file is unknown.

Revert "curvefs: Merge updateInode and GetOrModifyS3ChunkInfo 2 rpcs in background flush process" This reverts commit 8585222fe4e70a7be4272856b33e927247df3a79.

2022-08-05

curve-fuse: update attr and extent in single rpc (#1779) Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2022-08-04

curvefs/client: fix cto bugs Signed-off-by: ilixiaocui <ilixiaocui@163.com> related to issue: 1213/1582/1413 1. All mount points on the same file system are either enableCto, or they do not guarantee consistency. 2. If enableCto, attr and dentry cache will auto set invalid. 3. Consistency of multi-mount nodes under delete operation is not guaranteed. If one mount point deletes a file, the behavior of the other mount point for all operations on that file is unknown.

2022-07-28

curve-fuse: auto get volume size when creating fs and extend volume when space used up Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2022-07-20

curvebs client、mds、chunkserver: add io fence Signed-off-by: xuchaojie <xuchaojie@corp.netease.com>

curvefs: fix truncate file size becomes larger, chunkid duplicate question Signed-off-by: baijiaruo <baijiaruo@126.com>

curvefs: Merge updateInode and GetOrModifyS3ChunkInfo 2 rpcs in background flush process Signed-off-by: baijiaruo <baijiaruo@126.com>

curvebs client、mds、chunkserver: add io fence Signed-off-by: xuchaojie <xuchaojie@corp.netease.com>

curvefs: Merge updateInode and GetOrModifyS3ChunkInfo 2 rpcs in background flush process Signed-off-by: baijiaruo <baijiaruo@126.com>

2022-07-19

Revert "curvefs: Merge updateInode and GetOrModifyS3ChunkInfo 2 rpcs in background flush process" This reverts commit a6cb57471232b4d1304fda30f1256ef73cbf9981.

2022-07-11

curvefs: Merge updateInode and GetOrModifyS3ChunkInfo 2 rpcs in background flush process Signed-off-by: baijiaruo <baijiaruo@126.com>

2022-07-06

metaserver: add STORAGE_CLOSED status code Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2022-07-05

metaserver: add STORAGE_CLOSED status code Signed-off-by: Hanqing Wu <wuhanqing@corp.netease.com>

2022-06-18

curvefs: umount fs automatically after client crash

2022-06-15

curvefs client: fix bug of "bugfix: rmdir cause parent inode nlink = 0, that report an EIO"

bugfix: rmdir cause parent inode nlink = 0, that report an EIO

curvefs/metaserver: now we removed several containers which takes a lot of memory and optimized some implementations: 1. use dentry vector instead of seek operation, it can speed up getting latest version dentry. 2. use rocksdb default BytewiseComparator instead of RocksDBStorageComparator, it can speed up lookup key. 3. support dumpfile v2 which not require the number of dentrys. 4. added rocksdb perf.

Revert "bugfix: rmdir cause parent inode nlink = 0, that report an EIO" This reverts commit 2944a9960b038c60112414b2e329730d443def0c.

2022-06-11

bugfix: rmdir cause parent inode nlink = 0, that report an EIO

2022-06-10

curvefs: add delete partition in tool and mds

2022-06-07

curvefs: improve performance of command ls -*. 1. Get inodeAttr when need attr only intead of get inode. 2. Impliment readdirplus api. 3. Add iAttrCache for fast get some xattr when ls -l.

2022-06-02

curvefs: fix check mountpoint exist Use message moupoint instead of string

2022-06-01

curvefs fix load copyset stuck heartbeat #1468

2022-05-30

curvefs/topology: add fs_filetype inodeNum metric 1. add file type in partition 2. add filetype2inodeNum in heartbeat 3. add isdeleted and old in inode_manager.updateInode

2022-05-23

curvefs: separate volume extent from inode

2022-05-16

curvefs: support rename by parallel for multiple mountpoints, you MUST set client configure item fuseClient.enableMultiMountPointRename to true for gurantee the consistent of file after rename in multiple mountpoints. (#1369)

curve/tool: basic framework 1. It can be used like this: $ curve fs list fs Name Id Status Type MountNum test_1 1 INITED TYPE_S3 1 test_2 2 INITED TYPE_S3 1 $ curve fs list fs --format json { "error":{ "code":0, "message": "sucess", }, "detail":{ "fsInfo":[ { "fsId":2, "fsName":"test_2", "status":"INITED", "rootInodeId":"1", "capacity":"18446744073709551615", "blockSize":"1048576", "mountNum":1, "mountpoints":[ "localhost:9004:/curvefs/client/mnt/home/nbs/temp/mnt2" ], "fsType":"TYPE_S3", "detail":{ "s3Info":{ "ak":"minioadmin", "sk":"minioadmin", "endpoint":"http://172.17.0.2:9000", "bucketname":"test", "blockSize":"4194304", "chunkSize":"67108864" } }, "enableSumInDir":false }, { "fsId":1, "fsName":"test_1", "status":"INITED", "rootInodeId":"1", "capacity":"18446744073709551615", "blockSize":"1048576", "mountNum":1, "mountpoints":[ "localhost:9002:/curvefs/client/mnt/home/nbs/temp/mnt1" ], "fsType":"TYPE_S3", "detail":{ "s3Info":{ "ak":"minioadmin", "sk":"minioadmin", "endpoint":"http://172.17.0.2:9000", "bucketname":"test", "blockSize":"4194304", "chunkSize":"67108864" } }, "enableSumInDir":false } ] } } If the command entered is incomplete: $ curve fs Usage: curve COMMAND [OPTIONS] curve is a tool for managing curvefs ands curvebs Commands: fs Manage curve fs cluster Options: -f, --format string Output format(json|plain) -h, --help Print usage -v, --version Print curve version Run 'curve COMMAND --help' for more information on a command. 2. If the command entered is not supported: $ curve ds Error: unknown command "ds" for "curve" Did you mean this? fs Run 'curve --help' for usage. 3. If the input flag is not supported: $ curve fs --error Error: unknown flag: --error See 'curve fs --help' 4. Want to know how the command is used: $ curve fs --help Usage: curve COMMAND [OPTIONS] curve is a tool for managing curvefs ands curvebs Commands: fs Manage curve fs cluster Options: -f, --format string Output format(json|plain) -h, --help Print usage -v, --version Print curve version Run 'curve COMMAND --help' for more information on a command.

2022-05-13

curvefs: add support creating hybrid fs 1. add hybrid fs type in proto 2. add create hybrid fs in mds 3. add create hybrid fs in curvefs/tool

2022-05-05

curvefs_tool: query-inode and list-partition 1. use GetOrModifyS3ChunkInfoResponse to query inode 2. add list-partition for locating inodes 3. use GeIndode to get inode baseInfo

2022-04-27

curvefs: fix a compatible issue which caused by modify FsFileType

curvefs: fix a compatible issue which caused by modify FsFileType

2022-04-25

curvefs/metaserver: speed up getting inode by padding inode's s3chunkinfo which small enought instead of invoke RefreshS3ChunkInfo().

curvefs/metaserver: speed up getting inode by padding inode's s3chunkinfo which small enought instead of invoke RefreshS3ChunkInfo().

curvefs: add Owner attr to FsInfo 1. Add Owner to FsInfo 2. Add Owner and Capacity to CreateFsRequest 3. Add -user (deafult: anonymous) and -capacity (default: 100GB) to curvefs_tool create-fs

2022-04-21

curvefs: improve meta data performance add applied index in client read rpc requests to avoid the cost of raft.