Proto commits in near/nearcore

These 85 commits are when the Protocol Buffers files have changed:

2025-02-13

feat(optimistic_block): pass optimistic block to producers over T1 (#12888) Pass the optimistic block to all producers over the T1 network.

The documentation is generated from this commit.

2025-01-24

[spellcheck] Part 1: Spell check directories benchmarks and chain (#12785)

2024-12-20

fix: increment and refactor number of hops for routed messages (#12188) Co-authored-by: Stefan Neamtu <stefan@Stefans-MacBook-Pro.local>

2024-10-18

[Epoch Sync] Change EpochSyncRequest/Response to direct network messages (#12236) It was previously a mistake to use routed messages. There's no reason to do that. This also applies rate limiting to the network messages. Closes #11937 Closes #12091

2024-10-11

.

.

refactor and test num_hops

2024-09-20

feat(network): overhaul state part request (#12110) In this PR we implement the ability to request state parts from arbitrary peers in the network via routed messages. Previously, state parts were requested via a PeerMessage which can only be sent to directly connected peers of the node. Because the responses to these requests are large and non-time-sensitive, it is undesirable to send them over the tier1/tier2 connections used for other operations of the protocol. Hence we also introduce a new connection pool tier3 used for the sole purpose of transmitting large one-time payloads. A separate PR will follow which overhauls the state sync actor in accordance with these changes. The end-to-end behavior has been built and tested in #12095.

2023-11-06

Publish information about state snapshots to the network (#10099) Adds a new cache SnapshotHosts which contains information about state snapshots hosted by peers in the network. It will enable state part requests to be made directly to peers having the desired state. Whenever a node produces a new snapshot, it will sign and publish information to the network specifying: - Sync hash - Epoch height - The shards included in the snapshot The published information is spread to all network nodes: - Whenever a node receives new information, it broadcasts it to all of its neighbors - Whenever a new connection is formed, the two nodes perform a full sync of available information An RPC endpoint `/debug/api/snapshot_hosts` is provided for debugging purposes, as well as a display in the debug UI: <img width="1279" alt="image" src="https://github.com/near/nearcore/assets/3241341/1757bff4-c7c7-4d82-8dda-b5bc54d3451f">

2023-08-10

feat(state-sync) convert state sync messages to peer messages (#9390) Convert the state sync messages to direct messages.

2023-07-18

RoutingTable V2: Distance Vector Routing (#9187) ### Suggested Review Path 1. Browse the (relatively small) changes outside of the `chain/network/src/routing` folder to understand the external surface of the new RoutingTableV2 component. 2. Check out the architecture diagram and event flows documented below. 3. Read the documentation for the EdgeCache component and understand the 3 purposes it serves. The primary role of this component is to support efficient implementation of the routing protocol. 4. Review the RoutingTableV2 component and understand how DistanceVectors are ingested and created. This is the core of the new routing protocol. 5. Return to the EdgeCache and review its implementation. 6. Revisit the call-sites outside of the routing folder. ### Architecture  ### Event Flows - Network Topology Changes - Three Kinds: Peer Connected, Peer Disconnected, received a PeerMessage with new DistanceVector - These are triggered by PeerActor and flow into PeerManagerActor then into the demux - Demux sends batches of updates (up to every 1 second) to the RoutingTableV2 - RoutingTable processes entire batch, expires any outdated routes (relying on too-old edges), then generates updated RoutingTableView and local DistanceVector - If the local DistanceVector changes, it is then broadcast to all peers - Handle RoutedMessage - Received by the PeerActor, which calls into PeerManagerActor for routing decisions - Record the "previous hop" (the peer from which we received this message) in the RouteBackCache - Select a "next hop" from the RoutingTableView and forward the message - Handle response to a RoutedMessage - Received by the PeerActor, which calls into PeerManagerActor for routing decisions - Fetch the "previous hop" from the RouteBackCache and relay the response back to the originating peer for the original message - Connection started - When two nodes A and B connect, each spawns a PeerActor managing the connection - A sends a partially signed edge, which B then signs to produce a complete signed edge - B adds the signed edge to its local routing table, triggering re-computation of routes - B broadcasts its updated DistanceVector, which provides A (and other nodes) with the signed edge - Connection stopped - Node A loses connection to some node B (either B stopped running, or the specific connection was broken) - Node A executes fix_local_edges and notices the lost connection, triggering re-computation of routes - A broadcasts its updated DistanceVector, informing other nodes of the latest routes it has - If B is still running, it will go through the same steps described for A - If B is not running, the other nodes connected to it will process a disconnection (just like A) ### Configurable Parameters To be finalized after further testing in larger topologies: - Minimum interval between routing table reconstruction: 1 second - Time after which edges are considered expired: 30 minutes - How often to refresh the nonces on edges: 10 minutes - How often to check consistency of routing table's local edges with the connection pool: every 1 minute ### Resources - [Design document](https://docs.google.com/document/d/192NdoknskSLavttwOZk40TSYvx2R1if4xNZ51sCNFkI/edit#heading=h.j4e0bgwl42pg) - [Zulip thread](https://near.zulipchat.com/#narrow/stream/297663-pagoda.2Fnetwork/topic/Updated.20thoughts.20on.20TIER2.20routing) with further design discussion #### Future Extensions - [ ] Set up metrics we want to collect - [ ] Implement a debug-ui view showing contents of the V2 routing table - [ ] Implement pruning of non-validator leafs - [ ] Add handling of unreliable peers - [ ] Deprecate the old RoutingTable - [ ] Deprecate negative/tombstone edges

2023-06-16

Revert "Include signed ip address in TIER2 handshake. #8902 (#9100)" (#9191) * Revert "Include signed ip address in TIER2 handshake. #8902 (#9100)" This reverts commit bc9ba978f5a583318a734dd9f141ddc56c4a38c9. * reserved field number for backward compatibility of protocol buffer * Added comment on purpose for reservation

2023-06-15

Revert "Include signed ip address in TIER2 handshake. #8902 (#9100)" (#9191) * Revert "Include signed ip address in TIER2 handshake. #8902 (#9100)" This reverts commit bc9ba978f5a583318a734dd9f141ddc56c4a38c9. * reserved field number for backward compatibility of protocol buffer * Added comment on purpose for reservation

2023-05-29

Include signed ip address in TIER2 handshake. #8902 (#9100) * Algorithm Design: Sign, Verify Ip Address With Interface SignedIpAddress * Implemented Protocol Buffer Serialization Deserialization For std::net::IpAddr and SignedIpAddress * Implemented serialization deserialization of Handshake message containing SignedIpAddress for Protocol Buffer * Implemented and unit tested verifying ip address is properly signed and properly handles errors otherwise * Tested backwards compatibility with lack of signature for ip address

2023-03-09

Ban nodes sending excessively large PeersResponse (#8690) * ban if PeersResponse too large * fix config verification * add max_peers and max_direct_peers to PeersRequest

2023-02-24

PeersResponse: always include connected peers (#8619) * ensure PeersResponse includes connected Peers * include direct_peers as separate field of PeersResponse * clean unused imports * fix tests * include some indirect_peers in tests * actually don't include it bc of borsh * name fields in network_protocol::PeersResponse

2023-02-03

Make TIER2 connections survive node restart (#8318) Adds ConnectionStore, which maintains an LRU cache of outbound connections and persists them to the database. Adds behavior in PeerManagerActor to re-establish stored outbound connections upon starting a node. Adds behavior in PeerManagerActor to re-establish an active outbound connection if it is in the connection store and the connection is broken. Adds ConnectionStore data to the TIER2 network debug page. Adds a boolean flag remove_from_connection_store to Disconnect messages.

2022-12-05

revived UpdateNonceResponse message. Dropping it in this release was backward incompatible (#8159) Force pushing - as it fails the proto backwards compatibility (as we're re-adding old field)

2022-11-29

Added AccountKey to Handshake message. (#8106) * Added a proof of AccountKey -> PeerId mapping to the Handshake message. It is needed to convince the inbound end of connection that the connecting peer actually belongs to TIER1 network. * Separated TIER2 Handshake from TIER1 Handshake message, to make it impossible for the peers to confuse which network the given connection belongs to. In particular the binary from the previous release won't be able to accept TIER1 connections.

2022-11-25

Separated Tier1Handshake message from Tier2Handshake message (#8118) Separated TIER2 Handshake from TIER1 Handshake message, to make it impossible for the peers to confuse which network the given connection belongs to. In particular the binary from the previous release won't be able to accept TIER1 connections.

2022-11-22

revamped TIER1 discovery protocol (#8085) * added some extra fields to AccountData, renamed a bunch of them for better readability. This PR is not backward compatible in a sense that the old binary won't accept new AccountData and vice versa (protobuf compatibility is preserved, however the custom validation logic will treat different fields as required). That's ok as the broadcasted data is not interpreted yet. * changed identifier of AccountData from (EpochId,AccountId) to AccountKey, which simplifies a lot of stuff and generally makes more sense * changed version ID of AccountData from an UTC timestamp to an integer: this allows to avoid bad impact of a wall clock time shift, which can be especially dangerous since we do not version AccountData by EpochId any more. The creation timestamp is still present in AccountData to be able to debug data freshness. We can additionally implement an expiration policy in the future based on the timestamp. * rearranged the code related to broadcasting AccountData to align with the TIER1-related code which comes next after this PR

2022-11-21

Measure the number of hops of routed messages (#8050) * Measure the number of hops for routed messages. * Test output * Clarify * max_num_hops

2022-11-11

feat(o11y): Inter-process tracing (#8004) Serialize TraceId and SpanId to a new field of `PeerMessage`. This lets the receiving node link traces to a trace that generated the network request. `TextMapPropagator` is the interface designed to solve a similar problem, but given that: 1) we have to invoke it manually 2) our custom propagator is stateless ... following the `TextMapPropagator` interface doesn't add value. Setting a global text map propagator doesn't add value for the same reason. Because we want the inter-process tracing to be enabled at the debug level: * We need handling of `SendMessage` to enable tracing at the debug level * And a corresponding `PeerManagerActor` tracing needs to be enabled at the debug level. https://pagodaplatform.atlassian.net/browse/ND-172

2022-11-08

replaced ResponseUpdateNonce with SyncRoutingTable (#8009) SyncRoutingTable functionality is a superset of ResponseUpdateNonce functionality.

2022-10-24

removed messages of unimplemented EpochSync (#7911) EpochSync was never implemented, there is just a bunch of stubs left here and there. Removing them.

2022-08-05

Implemented demultiplexing for rate limiting broadcasts. (#7182) * Implemented demultiplexing for rate limiting broadcasts. * snapshot * applied comments * added test for rate limiting * moved config verification inside PeerManagerActor::new * applied comments * fixed failing integration tests * yet another test fix

2022-08-03

Implemented sending signed AccountData by validators (#7299) With this PR, an MVP of TIER1 peers discovery is complete (hopefully).

2022-07-19

renamed Validator to AccountData (#7206) https://nearinc.atlassian.net/browse/CPR-94

2022-06-20

Measure latency of network messages assuming all nodes have synchronized clocks (#7050) This PR measures latency of all messages by adding a timestamp at the lowest level - at the protobuf level. cc: @mm-near @mzhangmzz CPR-86

2022-06-17

Implemented proto translation for IP announcement messages (#7023) * implemented conversion from protos for IP announcements * split the proto_conv into files * make_signed_validator * implemented test of the RoutingTableUpdate.validators field

2022-06-13

Removed the abandoned IBF support for the routing table (#7014) Co-authored-by: near-bulldozer[bot] <73298989+near-bulldozer[bot]@users.noreply.github.com>

2022-06-06

proto and config for the Validator IP announcement (#6887) [Announcing Validator IPs - design](https://docs.google.com/document/d/1RDd9ETfLQL_JfnEePmUPjkLvIxAtLV-c5AKWV8IMJEo/edit) Added relevant protobuf message definitions (currently unused). Added relevant fields to config.json (currently unused, not visible to the end users yet). Refactored NetworkConfig - grouped together fields relevant to being a Validator. Made RoutingTableAgent an implementation detail of the PeerManager.

2022-05-04

Implemented protobuf equivalent of network_protocol.rs (#6672) See the design: https://docs.google.com/document/d/1gCWmt9O-h_-5JDXIqbKxAaSS3Q9pryB1f9DDY1mMav4/edit Abstracted away the network protocol format for smooth borsh -> protobuf transition. Implemented tests of cross-version communication.

2019-08-28

Merkle proofs for receipts (#1216) * Merkle proofs for receipts The problem is to make sure that received receipts are trusty. The common idea is to deliver a pair of (receipts, receipts_proof) where receipts_proof is a Merkle proof of the receipts are received. While chunk is producing, we calculate Merkle proof root and put it into chunk header (check ClientActor::receipts_proof for more details). Then we should check proofs in two cases: 1. Anyone who cares about shard and applying chunks completely must compare Merkle root with proof which is stored into chunk header. 2. Sending ChunkOnePart, we must calculate proofs for each subset of receipts related to some shard, as many as num_shards we have. Receiving ChunkOnePart, we must check that all proofs are given and all proofs guarantee that each subset contains in chunk receipts. TEST PLAN === - sanity tests in chain and client folders - cargo test --all-features --package near-client --test cross_shard_tx tests::test_cross_shard_tx -- --exact --nocapture (10 iterations) - cargo test --all-features --package near --test run_nodes run_nodes_2 -- --exact --nocapture - cargo test --package near --test stake_nodes test_validator_join -- --exact --nocapture LOGIC REVIEWER === Alex Skidanov

2019-08-23

Implement kickout for chunk producers (#1204) * Implement kickout for chunk producers * fix tests * address comments * Better error message

2019-08-19

Implement StateRecord for Action Receipts and Received Data (#1163) * Implement StateRecord for Action Receipts and Received Data * Rename rpc.rs -> views.rs

This commit does not contain any .proto files.

2019-08-16

Remove protos, replace with custom byte serialization (#1170) * Step 1. Remove protos. * Step 2. View for Account and AccessKey instead of special encoding on them * Step 3. Public/SecretKey/Signature moved into views as well. Tests in runtime pass * Step 4. CryptoHash View * Adding Readable/Writable for custom deserialization * Step 5. Finish BlockHeader -> BlockHeaderInner. Fixups for View usages in tests * NBOR * NBOR * Adding change checker for Transaction * Switch to using NBOR instead of serde. Compiles * Fix issue with unnamed fields in enums in NBOR * Clean up, making tests pass * Fix usize suffix warning * Use u8 instead of u32 for enum variants * Add transaction view for RPC and a bit of KeyFile changes to make serialize properly * Remove #update-as from dependencies (#1168) * Remove #nightshade from near-shell dependency (#1169) * AccessKey changes are done * Replace drain with into_iter * Rename to_vec/from_slice -> try_to_vec/try_from_slice * Fix comments * Add verification for CryptoHash * Switch to external repo borsh * Remove nbor * Replace the rest of nbor usage

Fixing bug #1183: state root should not be indexed by the chunk hash Also fixing a bug we discovered earlier that if the last block of an epoch doesn't have a chunk for a particular shard, we do not do updated necessary for the epoch switch. Fixing it by forcefully executing `apply_transactions` with no txs and receipts for the last block. Conveniently, with the fix for #1183 it is possible (before we would not be able to store two different post-state roots for the same chunk)

2019-08-15

Replace access keys (implements NEP#0005) (#1167) Implement NEP#0005: - Remove public key from account. Replaced with the "full access" access keys - Remove nonce from account. Each access key now has nonce.

2019-08-14

Word towards supporting fast syncing for multiple shards, and various bug fixes. This change has initial implementation of fast syncing with multiple shards. It currently doesn't fully work due to the following issue: - If the validator that joins is syncing the epoch that is behind the epoch that online validators are in, the syncing validator won't have the info about the latest epoch (since we do not apply headers during state sync, and thus can't apply info for the current epoch), and thus won't be able to verify signature for account announces. Without account announces the validator cannot find whom to download state from. Changes implemented in this PR: - Requesting the state for proper shards during fast sync (was hardcoded to 0 before) - For state sync we were mistakenly sending the previous block with the response for each shard; It is both redundant if multiple shards are requested, and is not sufficient if no shards are requested; Made the block request separate from the shard state requests; Bugs encountered and fixed: - State changes in the `ChainStoreUpdate` were only recorded for a single shard, causing `Node not found` errors - Account announce was pushing itself to the end of the route vector if the *first* element was not matching the current peer (needs to be last) - Account announce verification was not properly passing the current hash in the fold - We were banning a peer if the signature verification for the account announcement was failing, including the situation when it was failing because we didn't know the epoch for the account yet. Separated the two reasons for signature verification and made them to be handled differently - [Initial account announce] and [routing account announce] were following the same code path in `PeerManager` that would only actually send the announcement if it updates the local rounting map. Such a routing map check only makes sense for the routing, but not for the announcement of myself; added `force` flag for announcement of myself so that it is always sent.

2019-08-13

Fixing `sync_state_nodes` and one issue in `sync_nodes` - `stake_state_nodes` was failing for several reasons: 0. Rewrote the test to have the first node actually generate 101 blocks. Creating them manually doesn't work anymore since non-trivial state transition happens at the epoch boundary; 1. State sync was not sending the last block, which was OK pre-chunks, but the block is needed now; 2. State sync was not proceeding if the most weighted peer was ahead, but state sync blocks header sync, effectively deadlocking. Made state sync not care about the height of the peers; - `sync_nodes` was failing because `num_total_parts` was using `get_epoch_Id` instead of `get_epoch_id_from_prev_block`. All the interfaces in the `RuntimeAdapter` expect the epoch defined by the prev block. There are new failures in `sync_nodes` after a rebase, they will be addressed separately

2019-08-12

Batched transactions (implements NEP#0008) (#1140) Implements NEP#0008

2019-08-10

Release v0.2.7 (#1155) * Fix stake double return bug (#1084) * fix stake double return bug and update tests * use cache properly * properly implement stake returning * refactor test utils * fix gensis config * Fix block producer assignment (#1088) * fix block producer assignment * trying with test lock * tweak test parameters * Add epoch hash to block headers (#1089) * add epoch hash to block header * rework runtime adapter API * address comments * Check validator signature (#1100) * check validator signature * ignore flaky test * fix (#1101) * Telemetry (#1090) * Add telemetry cargo, which sends RPC to given url. Add serde attributes to transaction to correctly serialize u128. Moved out information into a separate module called from client * Added signature to info telemetry. Default url for telemetry. Better showing mem and cpu * Add formatting for Option<BaseEncode>, fix comments * bump version to 0.2.6 (#1091) * Fix stake double return bug (#1084) * fix stake double return bug and update tests * use cache properly * properly implement stake returning * refactor test utils * fix gensis config * Fix block producer assignment (#1088) * fix block producer assignment * trying with test lock * tweak test parameters * bump version to 0.2.6 * Update docker to have git version and use cache (#1092) * Disable test_kickout * Remove trailing slash in DEFAULT_TELEMETRY_URL * Adding node id to the info/telemetry * Return stake 3 epochs later (#1105) * Return stake 3 epochs later * remove print * fix test * remove clutch * Networking AccountId tracking (#1058) * Remove account id from handshake * Add AnnounceAccount + Communication between client and network on trigger Missing trigger condition + Routing table as mechanism to support conection through hops * Improve signed data on announce & check conditions * reformat announce account into simpler pieces * Handle client side. * Fix tests * Provide `get_epoch_offset` for mock runtime * Address small issues * Use hash for epoch instead of height * Verify signature from account announcement * Verify route while announcing account * nit * Fix mock test * Add missing verify * Update NewtorkInfo api for ease to use on tests * Add test for announce account. Add routing information in NetworkInfo for testing. * Exact cause for ban in invalid acc announcement. * Add test for announce account with different graph * Avoid extra clone and queries * Fix up merge conflict * Account deletion (#1107) * Account deletion implementation * Stakers can't be deleted, but must have more rent on their account (4 * epoch_length). Also added check_rent method to call after transaction execution to make sure there is still enough rent on the account * Actually delete data when deleting an account and test that * Add support for default values in runtime config (expected that we will put reasonable defaults in the binary), and fixed test_deserialize test / testnet.json * Moved helper functions for transactions into test-utils User trait * Reuse check_stake function from runtime and system * Address comments * Merge master into staging (#1116) * bump version to 0.2.6 (#1091) * Fix stake double return bug (#1084) * fix stake double return bug and update tests * use cache properly * properly implement stake returning * refactor test utils * fix gensis config * Fix block producer assignment (#1088) * fix block producer assignment * trying with test lock * tweak test parameters * bump version to 0.2.6 * Update docker to have git version and use cache (#1092) * neartest.com has been retired (#1098) * Change how docker port mapping works for macos (#1086) * Apply Peter's fix to docker image start up code * fix port mapping in nodelib * fix #1042: Ban peer doesn't save to storage (#1093) * Add registers to Wasm bindings * Remove some unreachables (#1108) * Revert back to cranelift backend for now (#1109) Singlepass backend is crashing * Fix stake test (#1095) * Fix stake test * Check expected validators. * Specify expected validator order. * No need to move anymore. * Add slashing in validator manager (#1112) * Add slashing in validator manager * pass slashed_validators as a parameter to add_proposals * change runtime adapter to include slashing * refactor slashing * Finish kickout test after everything gets verified (#1123) * Finish kickout test after everything gets verified * Remove println * typo (#1124) * Remove merge conflict issue from store/lib.rs * Bump version to 0.2.7 * Fix test_deserialize issue * Enabled sending telemetry data over https (#1149) * Update testnet genesis (#1150) * HARD FORK: update testnet genesis * Dedup validators in get_epoch_producers * Correct test and reduce size of docker by removing files after docker is done * Zero out storage_used_at * Validator proposals fix (#1154) * Fix issue that validator proposals were not propagated to Block::produce * Speed up sync_state_stake_change and remove extra print * Update testnet genesis to 1859314 block * Hot fixes to syncing issues * Fix issue with orphan resolution * Fix genesis with fisherman to 0 for now * Fix rushed fix * Fix validator_join and remove warning * Fix total approvals formula when there are 2 validatos * Bump up the timelimits for announce_account tests * Check what happened on CI for the announce_account test * Add ignore to account_announce tests as they are currently not fully propagating account into. Will be fixing separtely

2019-08-04

Merge remote-tracking branch 'origin/validator-reward' into chunks-validators-2

Rename staked to stake in Account

2019-08-03

Merge remote-tracking branch 'origin/chunks-validators-2' into validator-reward

validator reward calculation

Merge remote-tracking branch 'origin/chunks' into chunks-validators-2

Send all the extras with the state sync; New epoch assignments in KeyValueRuntime

2019-08-02

towards validator reward

2019-07-31

Chunks + Validators (#1118) * Adding gas usage and limit to the header to track relevant information for rewards. * Remove usage of ..Default::default in proto conversions, which is hiding issues * Account deletion (#1107) * Account deletion implementation * Stakers can't be deleted, but must have more rent on their account (4 * epoch_length). Also added check_rent method to call after transaction execution to make sure there is still enough rent on the account * Actually delete data when deleting an account and test that * Add support for default values in runtime config (expected that we will put reasonable defaults in the binary), and fixed test_deserialize test / testnet.json * Moved helper functions for transactions into test-utils User trait * Reuse check_stake function from runtime and system * Address comments * Step 1. Add validator proposals are called from process_block. Additional issue that verify_chunk_sginatures calling and that it may result in error are fixed. * Merge master into staging (#1116) * bump version to 0.2.6 (#1091) * Fix stake double return bug (#1084) * fix stake double return bug and update tests * use cache properly * properly implement stake returning * refactor test utils * fix gensis config * Fix block producer assignment (#1088) * fix block producer assignment * trying with test lock * tweak test parameters * bump version to 0.2.6 * Update docker to have git version and use cache (#1092) * neartest.com has been retired (#1098) * Change how docker port mapping works for macos (#1086) * Apply Peter's fix to docker image start up code * fix port mapping in nodelib * fix #1042: Ban peer doesn't save to storage (#1093) * Add registers to Wasm bindings * Remove some unreachables (#1108) * Revert back to cranelift backend for now (#1109) Singlepass backend is crashing * Fix stake test (#1095) * Fix stake test * Check expected validators. * Specify expected validator order. * No need to move anymore. * Step 2. Move post_state_root into ChunkExtra with gas_used and validator_proposals. Removed post state root from ShardMsg * Step 3. Gas limit/usage & validator proposals propagate * Cleaned up tests in validator manager. First steps to make sure routing tables are populated. * Fix chain tests * Change 100ms to 250ms block production for chunk_manager tests

Step 3. Gas limit/usage & validator proposals propagate

2019-07-30

Merge branch 'rewards' into chunks-validators

Merge remote-tracking branch 'origin/staging' into rewards

Step 2. Move post_state_root into ChunkExtra with gas_used and validator_proposals. Removed post state root from ShardMsg

Merge branch 'staging' into chunks In particular, merging all the work on validator rotation that was happening in master (fixing stake returns, kicking validators out etc) with the work on validator rotation for the shard chains. Some aspects of the merged code still appear a bit flaky, more testing and fixes are coming

Account deletion (#1107) * Account deletion implementation * Stakers can't be deleted, but must have more rent on their account (4 * epoch_length). Also added check_rent method to call after transaction execution to make sure there is still enough rent on the account * Actually delete data when deleting an account and test that * Add support for default values in runtime config (expected that we will put reasonable defaults in the binary), and fixed test_deserialize test / testnet.json * Moved helper functions for transactions into test-utils User trait * Reuse check_stake function from runtime and system * Address comments

Remove usage of ..Default::default in proto conversions, which is hiding issues

Adding gas usage and limit to the header to track relevant information for rewards.

2019-07-29

State sync for shard state between epochs (a.k.a. Catchup) 1. Initiating state download (aka Catchup) on each first block of each epoch for the *next* epoch; 2. Reusing the state sync machinery we already have (and fixing various bugs in it) to actually sync state for chunks; 3. Immediately applying state transition for the next epoch if the state is downloaded, putting block into a queue if not; 4. Processing the queue once the state is downloaded; 5. Orhpaning the block if the state still not downloaded when the next epoch starts, unorphaning on (4); Addresses #1046

2019-07-27

Networking AccountId tracking (#1058) * Remove account id from handshake * Add AnnounceAccount + Communication between client and network on trigger Missing trigger condition + Routing table as mechanism to support conection through hops * Improve signed data on announce & check conditions * reformat announce account into simpler pieces * Handle client side. * Fix tests * Provide `get_epoch_offset` for mock runtime * Address small issues * Use hash for epoch instead of height * Verify signature from account announcement * Verify route while announcing account * nit * Fix mock test * Add missing verify * Update NewtorkInfo api for ease to use on tests * Add test for announce account. Add routing information in NetworkInfo for testing. * Exact cause for ban in invalid acc announcement. * Add test for announce account with different graph * Avoid extra clone and queries * Fix up merge conflict

2019-07-24

Add epoch hash to block headers (#1089) * add epoch hash to block header * rework runtime adapter API * address comments

2019-07-20

Extending genesis spec for shards, access keys, etc. Switching to use protos in state storage (#1079) * Update protobuf to 2.7, change GenesisSpec to contain StateRecords: accounts, contracts, access keys and callbacks per shard. Added Account proto * Update state-viewer to dump StateRecord * Update to store account/access keys/callbacks as protos in storage. Add new testnet spec from prev POC. Bunch of other changes * Fix a hack * Fix test nearlib to use master again. Fix readme to point at the latest version of docs * Fix comments * Nearlib test to use yarn

2019-07-18

All the machinery to make the stress test work with 20ms blocks, and erly work on validator rotation for chunks - Changing validator rotation such that the current block is sufficient to know the epoch of the next block - Fixing more issues with handling receipts - Orphaning chunk one parts if the epoch is not known - Re-requesting chunk one parts if they are missing when block is received - Fixing some issues with using incorrect height/prev block when computing number of total parts - For now making header sync to commit after every block, doing otherwise breaks the current interaction with the validator manager

2019-07-10

Squashing all the chunks-related commits.

2019-07-02

State sync (#1007) * Starting to implement fast sync * Adding block fetch horizon, after which to sync state instead of downloading blocks * Add genesis to handshake to prevent nodes from wrong network connecting * Progress on state syncing: requests and responds with state at given block hash / merkle root. Applies it and progresses sync state * Added Grin to attributions * Continue state sync: sync into block t-5 to have few recent blocks as well, updating tip of block head and unlock orphans * Sync state for one shard works without receipts * Add receipts to state syncing * Remove extra print * Add error propagation and retry for state sync * Clean up validator rotation: remove requirement to rewrite genesis hash into CryptoHash::default() * Fixing nits

2019-06-26

Include originator_id and public_key into receipts and callbacks (#1027)

2019-06-11

Nightshade v3 -- New sharding (#928) * WIP Whale Fishing * First stab at single chain * Robust chain, still missing weight calcualtion and integration with runtime * Adding scaffolding for basic pool, network actor, client actor * Serving + better performance + more test coverage * Adapters between Pool and Chain. * formatting * Update primitives: sigs can convert to/from vectors; display peer message * Move store into separate cargo * Adding actix implementation of the network * Update chain to use callback instead of trait to avoid Arcs. * Add validator actor cargo * Init chain code and initial test (1/2) * Add proto encode/decode for chain and make init chain work. (2/2) * Add basic block production helper * Remove validator actor/cargo as block production is light * Chain handles orphans + test * Update RuntimeAdapter with apply transactions and fork choice rule computation * Client produces blocks * Client prints periodic summary. Network peer connect. * Block production works between two authorities. Add client test to accept external blocks. * Rename NetworkMessages to NetworkClientMessages. Small cleanups * Add myself to ghost code owner * Add integration test with 2 nodes, add test that fetches block on receival of header. * Initial propagation of peer chain state information from peers to client. * Bug fixing * Add stopping condition for two nodes test * Added banning peers if they sent invalid state (no tests yet) * Move WaitOrTimeout into test utils for peers. Move out peer manager into its own file * Clean up error reporting from process block/header * Added height/weight to the handshake * Adding basic cascade for JSONRPC server with subscriptions * Adding block producer signature to the block * Adding invalid state blocks test * Fix compile error in simple chain test * Fix tests or comment tests that are not applicable * Fix doc test * Basic approval collection for next block * Clean up peer manager tests. Adding first command line flag for the near bin * Actually start RPC server and print # authorities * Make sure coverage doesn't run forever * Playing with JsonRPC (not working) * Added timeout for waiting for other block to arrive and producing next block * Trie changes - change structure, make an storage object for temporary mutable trie nodes - normalize nodes after delete, a test for unique representation - non-recursive trie methods, an evil contract test - correctly compute refcounts - TrieChanges to support storing multiple versions in the future - Cache for storage, will be more useful later, also fix cached crate Performance got better but still a lot of time is spent allocating. * refactor refcount changes * Progress on robust node syncing * First step in integrating runtime in near cargo * More work on header syncing: added processing of sync headers to the chain, updated chain store to have cache / chain store update provides back interface to read just added items for consistent view on the 'database'. * Small fixes to prev commit * Reorg configuration for rutime/node. * Added handling for block index and test for fork change/skip blocks * Sync test passes: one peer syncs with the other one * First working JSON RPC with actix-web: send tx and query methods. * Remove unused deps * Default folder for configs * Starting to add runtime to NEAR * Moving primitives/protos/store into core/ folder * Move primitives, store and protos into core/ * Rename primitives to near-primitives * Clean up all the errors after moving things * WIP: Runtime integration * Replace GenesisConfig instead of ChainSpec * Clearning trie cache in tests * Make json rpc query work * Println cleanup * Helper scripts + report bls and tps in cli * Added support for receipts and tx status in chain. Running benchmarking: ~650-690 tps * Remove old chain, rename ghost folder to chain * Cleaning up configuration. Removed bls from InMemorySigner * Correct matching in main * Fix typo in two_nodes test and clean up unused imports * Fix tests, clean up cache usage * ClientActor into it's own file. Move out view handling from ClientActor into ViewClientActor * Fix up nearmint to work with new APIs * Remove b58 to base64 for hash, support tx status in rpc, fix up errors in nearmint * Pass chain info for status query to the Client. Correct folder for nearmint test * Added tx verification at runtime and calling it on transaction receival in client * Cleaning up: b58, config, subcommands: * Finish removing bs58: switched to base64 for keys. * Check genesis matching between config and storage. * Subcommand for near binary to clear data dir (unsafe_reset_data), * Added config option to not produce empty blocks (makes cleaner debug output) * Add retry to block production * Move protos, primitves, storage to core folder. Storage is store. Refactored how storage works. Added near/config for new Genesis config definition. Preparation for #928 merge * Disable nonworking test in jsonrpc around broadcast_tx_commut * Return back mana calculation that got removed in refactoring * Move chain/near into high level near/ * Load/store network peers in storage * Remove commented code * Use single place to call base64 in case we want to switch encoding * Print id of the node to connect, read boot nodes from config * WIP. Starting to return intergration tests. * Make Box<std::error::Error> default for TryFrom aross primitives * Remove sharding file leaked from another PR * Fix up mismerge + output logs from unittests when tx failed * Restore master's version of Trie * Make sure to delete near-* binaries for coverage * Restored test_simple for 2 peers. Added network peer exchange (5 star connection test works now). * Fix comments * Fix comments * Test for runtime handling reorgs/forks * Remove block traits/hain/consensus from primitives * Runtime return TrieChanges instead of StoreUpdate to apply insertions only for fork choice rules * Simple intergration tests pass (2, 4, 7 nodes): moved out PeerStore into separate file, fixed retrieving blocks wasn't sending the last one, sync headers wasn't validating them properly * Allocate different ports for rpc and nodes to work properly in multi-test env * Fix sync_tests and runtime_tests * Add a helper tool that runs some number of nodes in one process for quick spot check and load testing * Restore test_tps_regression * Enable test_cases_testnet_rpc integration tests * Continue working on rpc / thread node integration tests * Unify base64_format and b64_format, use it for StatusResponse. Added view_state RPC to nearmint, near, rpcuser. A bit of cleanup * Implemented jsonrpc/send_tx_commit (#975) * Implemented jsonrpc/send_tx_commit * [jsonrpc] Added polling configuration options for send_tx_commit * Remove usage of just base64::* and always use to/from base64 (prep to switch back to base58) * Ref #833: Removed ToBytes trait, cleaned up usage of BaseEncoder/From/Into traits. * Fix issues with timeouts in client. Clean up merge with master * Refactored jsonrpc to use futures 0.3 + async/await (#976) * Refactored jsonrpc to use futures 0.3 + async/await * Upgraded Rust compiler for async/await support * [CI] Replace wget with curl to avoid unnecessary dependency * Introduce timeout helper for jsonrpc async/await * Extracted the async/await helpers into async-utils crate * Continue working on #965: restored process node / rejoin integration test in progress * Working on #963: ValidatorManager does validator tracking and election. Renamed to Validators everywhere to make sure we use consistent language. * Fix not renamed parameters * Ref #963: Added ValidatorManager to the NightshadeRuntime, and hooked to the validator proposals from the transactions. Added integration test where new node becomes validator after staking. * Removed runtime/sharding folder as @SkidanovAlex working in different branch and this is outdated, clean up more namings. * Fix tests * Added test for errors on invalid tx and runtime error. Invalid tx that didn't get into mempool now returnn from broadcast_tx_commit. * Remove nearmint to speed up tests * Add HTTP interface to jsonrpc module to serve /status page by HTTP GET. Split tests into http_query, rpc_query and rpc_transactions * Ref #964: Step 1 -- start / kill NEAR node. Support not producing blocks * Ref #964: Process empty blocks works with receipts, better errors on failing to parse public keys, query rpc data takes as base58 string, test seed as an argument for near for testing * Updated nearlib in branch to work with Nightshade client * Fix issue that client wasn't finding state after restart * Balance is 128 bit number (#985) * First step toward u128 for balances: wrap u128 with Balance struct and started fixing all the usages * Finalize u128 conversion for balance * Moved out Balance into its own file. Restored wasm exports splitting u128 into 2 u64s. * Switched back Balance from newtype to alias. Moved From/TryFrom for proto into proto crate * Serialize u128 in Hex in JSON * Remove unneeded clones for amounts * Replace back Balance::default() to 0 * Fixes @nearmax suggested * Remove flatten on accountinfo as that doesn't work * Official testnet configuration (#986) * First step toward u128 for balances: wrap u128 with Balance struct and started fixing all the usages * Finalize u128 conversion for balance * Moved out Balance into its own file. Restored wasm exports splitting u128 into 2 u64s. * Add official testnet configuration + unsafe_reset_all and detecting existing configuration in home dir * Switched back Balance from newtype to alias. Moved From/TryFrom for proto into proto crate * Serialize u128 in Hex in JSON * Remove unneeded clones for amounts * Replace back Balance::default() to 0 * Fixes @nearmax suggested * Remove flatten on accountinfo as that doesn't work * Fix issue with epoch increment in validator manager * Fix wasm for u128 (#994) * Fixing WASM usage (switch to returning pointers instead of tuple) * Convert everything to points in WASM runtime * Fix rust wasm for new APIs * Switch back to debug for start_near to speed up dev (bs58 is compiled in optimized mode for debug as well). Better error when account id is invalid. * Force lock file for hello.wasm to fix CI * Add hello/out/main.wasm for now that is compiled locally as CI doesn't build * Adding proper file for hello wasm

2019-05-17

Introducing Economics. Update the API of the promises. Replace mana/gas with balance. Expand balance API. (#952) * Init EconomicsConfig * Fix methods of Runtime * Remove mana from AsyncCall and charge transaction cost * Remove ManaAccounting. Introduce liquid_balance, frozen_balance, leftover_balance * Remove transaction stake * Remove file * Replace AccountingInfo with a single AccountId used for the refund * Replace Gas and Mana with Balance * Fixing tests * Fix tests * Fix nearlib tests * Typos * Nit * Remove branches

2019-04-24

Ability to add and remove AccessKeys (#905) * Ability to add and remove AccessKeys * Adding access_key adding/deleting tests

2019-04-22

Refactor the codebase into runtime and chain (#894) * refactor into chain and runtime * make tests compile * fix bench * remove tmp bench * fix

2019-04-18

Add optional public key to the SignedTransaction (#876) First step to add access keys is to be able to provide public key that is used for the signature, so we can check access_key in addition to the public keys on the account.

2019-04-13

Unify runtime user with node user (#858) * Unify runtime user with node user * Fix merge conflict * Nit

2019-04-05

Remove locks from mempool (#838) * remove mempool locks * Fix tests

2019-04-04

Fetch blocks in chunks (#827) * Fetch blocks in chunks * add block fetch limit to config * fix clippy warnings * remove hardcoded block fetch limit * fix

2019-03-26

Towards better mempool (#768) * Towards better mempool * Nonce * fix bench

2019-03-25

Proto cleanup (#770) * Include wrapper.proto in the repo * Remove auto-generated proto files from git * fix * another attempt to fix * yet another attempt * add readme

2019-03-21

More protos (#744) * More protos * use tryfrom to avoid force unwrap * joint bls

2019-03-18

Remove bls key from account (#736) * remove bls key from account * add tests for staking

2019-03-17

Use proto for network (#733) * Use proto for network * use wrapper types * implement From instead of Into * Nit * include google protobufs on CI * typo

2019-03-02

Fix authority keys, NAT network, sync & produce blocks integration works (#653) * Piping proper block signing keys in proper places via Signer trait. InMemorySigner::from_seed creates keys from a seed. Block producing key files (the one that loads by default) contain both BLS and ED25591 keys for block signing. * Fix receipt block propagation * Finished piping signer. Split Signer into TransactionSigner and BlockSigner. * Clean up old types * Remove UID!!!! * Add ability to handle nodes without public addresses * Fix rest of tests/benches. * Fix tests * Towards 3 nodes working * Remove devnet receipt channel * https://github.com/nearprotocol/nearcore/issues/654 * Revert "https://github.com/nearprotocol/nearcore/issues/654" This reverts commit 18dff8c03e65f714ae6d9bdb3e4d71666186f79e. * #654 * Fix DevNet after changes with ReceiptBlocks * Nit * Send tx to alice - 3 nodes work * Fix devnet wrong sig used for the test * Fix devnet test to succed if it manages to produce more blocks * Fix tests in runtime

2019-03-01

add_bls_key transaction type (#619)

2019-02-14

Enable adding keys to and deleting keys from accounts (#570) * add key * delete key * fix * fix add existing key bug

2019-02-12

Resolve deploy security issue (#546) * change deploy * fix tests * remove extraneous fields from DeployContractTransaction

2019-01-29

protobuf for TransactionBody (#431)