Proto commits in Gan-Tu/cppGFS2.0

These 19 commits are when the Protocol Buffers files have changed:

2020-06-06

[feat] - WriteRequest and ApplyMutation implementation for chunk server file service (#54) * implement write request and apply mutation * add definition for apply mutation grpc * fix checksum in grpc to use bytes * check sent data checksum * nit * ConcurrentRegisterAndAllocateChunkServers test

The documentation is generated from this commit.

2020-06-02

Implement InitFileChunk, ReadFileChunk, and AdvanceFileChunkVersion (#48) * [feat] add chunk server impl class * [feat] add revoke lease logic * [docs] add logging comments throughout implementation * add chunk version setter/getter; add const for function param declaration * finish implementingn grant lease, as well as bug fixes * [fix] dynamically create grpc protocol clients, instead of creating ones at startup * [refactor] rename functions and hashmap code, with constructor behaviors * [fix] fixes typo * [refactor] improve logging * [feat] add destructor to chunk server impl * [chore] print all test outputs, detailed * make chunk server public, and clarify version mock functions * add chunk server impl test * minor refactor * [tests] add integration test for chunk server lease implementation * fix typo * fix typo * use strcat instead of + for string building * use file manager for get version * use config manager for initializing file manager database * use file manager for creatintg chunks * add disk db name into end2end config file generation library * style fixes * [feat] add initial server logic for InitFileChunk * add template for file service implementation test * implement file read handler * improve and document when to (or not to) prepare redo/undo logs * uncomment all the commented out code * make test default pass * implement advance file version * [docs] add easier to read commet on different conditional branches * fix typo * add init file chunk test * add test for file service impl * set new chunk version in advance reply * use new ports for the file impl test * add comments about placeholder vars * change status enums * fix typo

2020-05-31

[feat] Implement Chunk Server class, and lease RPC handler, with E2E test (#42) * [feat] add chunk server impl class * [feat] add revoke lease logic * [docs] add logging comments throughout implementation * add chunk version setter/getter; add const for function param declaration * finish implementingn grant lease, as well as bug fixes * [fix] dynamically create grpc protocol clients, instead of creating ones at startup * [refactor] rename functions and hashmap code, with constructor behaviors * [fix] fixes typo * [refactor] improve logging * [feat] add destructor to chunk server impl * [chore] print all test outputs, detailed * make chunk server public, and clarify version mock functions * add chunk server impl test * minor refactor * [tests] add integration test for chunk server lease implementation * fix typo

Continue to implement MasterMetadataService to include chunk allocation, and add caching of server locations on client side (#41) * [feat] Start shaping up Client impl. Add a ClientImpl object, which serves as an impl object to manage units such as cache manager and to issue rpcs with master client and chunk server client * [feat] Add implementation of open funtion to create a file, and its server-side implementation using MetadataManager * [fix] address review feedback * [refactor] Merge version, primary_location into FileChunkMetadata; furthermore, add a repeated field named locations to FileChunkMetadata, as we need to send this information over the wire (using protobuf); although it was stored by ChunkServerManager internally, we ultimately need to send it as a reply to a OpenFileRequest, and this is why we need to add it back * [feat] shaping up the master_metadata_service_impl to allocate chunk when file is created; make clientImpl cache the file chunk metadata * [nit] minor improvement * [fix] revision per feedback Co-authored-by: Gan Tu <github@tugan.dev> Co-authored-by: Gan Tu <tugan@berkeley.edu>

[feat] - File chunk manager (#40) * file chunk mgr * nit * nit

2020-05-26

[feat] - Complete chunkserver manager and its services, and chunkserver control service for heartbeat (#37) * big commit: chunkserver complete implementation with its services * fix build and lil comment * some changes * more comment for priority lsit * todo comment

2020-05-25

[feat] Add ClientImpl and use it to implement the client-side APIs. File creation with open() gets implemented. (#38) * [feat] Start shaping up Client impl. Add a ClientImpl object, which serves as an impl object to manage units such as cache manager and to issue rpcs with master client and chunk server client * [feat] Add implementation of open funtion to create a file, and its server-side implementation using MetadataManager * [fix] address review feedback Co-authored-by: Gan Tu <github@tugan.dev>

2020-05-23

Refactor ChunkServerServiceClient to two separate classes, one for the client side and one for the master side (#33) * [feat] Cleanup comments in client's header and add OpenFlag enum type * [feat] Add client cache manager which stores a local mapping form chunk handle to sever locations, and some other auxillary information * [fix] add more documentation and minor change in unit test * [refactor] Refactor helper function for testing into a tests/utils module * [refactor] Refactor ChunkServerServiceClient to two separate classes, one for the client side and one for the master side * [refactor] rename according to rename feedback, and add SendFileChunkRequest which sends file chunk data and revise WriteFileChunkRequest that instructs replicas to apply the data to disk * refactor out send data flow and write request flow (#34) Co-authored-by: Gan Tu <tugan@berkeley.edu>

2020-05-20

[feat] Extend MetadataManager to include ChunkMetadata (#31) * [feat] Introduce thread_safe_flat_hash_map and use it for the internal resource management of LogManager * [feat] Use thread_safe_flat_hash_map for managing FileMetadata in MetadataManager * [feat] Add more stress test cases to exercise metadata manager; style fix of comment lines etc. * [fix] Move initialization of filename before the emplace function gets called. * Check in extension of Metadata Manager to manage chunk metadata (version, primary server location and chunk server locations), tests pending * [feat] Enahnce the stress test for concurrent chunk creation, added a set of reader threads that exercise GetFileChunk to read from FileMetadata; added first unit test for chunk metadata read and write * [feat] Revise AdvanceChunkVersion name, and revise semantics of SetPrimaryServerLocation, which appends to the locations if the primary location is not in the list, and add test cases for read / write chunk metadata * [doc] Add some additional comments clarifying design decision * [fix] Implement submaps ourselves to manage concurrent read / write to the shared lookup tables as there are issues when we use the parallel hashmap library but do not have explicit control over submaps * [fix] Refactor the management of submaps for parallel hashmap into a customerized class and encapsulate atomic operations in newly added methods * [fix] Purge out server locations from the FileChunkMetadata as it gets owned by the ChunkServerManager Co-authored-by: Gan Tu <tugan@berkeley.edu>

2020-05-18

Make chunk version a part of AdvanceFileChunkVersionRequest (#29)

2020-05-17

[feat] - Master chunk server manager service v1 (#28) Co-authored-by: Gan Tu <tugan@berkeley.edu>

2020-05-16

[fix] make enum's first type be throwable unknown type, as suggested by proto3 (#24) This is because the first enum type is indistinguishable from unnset or default value, and thus is not sent over the wire by serialization, nor in debug string

2020-05-15

[fix] fix typo: Requeast -> Request

2020-05-14

[feature + refactor] Added MetadataManager and some refactoring (#21) Added MatadataManger which provides thread-safe interfaces to - Access reference to fileMetadata - Create fileMetadata - Create new file chunk - Delete a file (placeholder) - Assign new chunk handle

[feat] chunk server gRPC protocol definitions (#20) More gRPC protocol definitions for chunk servers regarding lease grant, extension, revocation chunk creation, chunk version advancements (for writes) file read and write

2020-05-11

[feature]: Chunk server manager v1 (#15) * chunk server manager v1 * chunk server manager test * use protobuf for chunkserver and location * nit * format master build file * build file formatting * make travis run test

2020-05-09

[docs] fix typo to say we use UUID instead

[fix] simplify gRPC for create and open (#13) * [fix] rename Create gRPC request/reply proto to be consistent with gRPC naming * [feat] combine create and open gRPC

[feat] add master metadata service gRPC proto definitions (#12) * [feat] add master metadata service gRPC proto definitions * [fix] update int32 to uint32