Proto commits in uber/kraken

These 12 commits are when the Protocol Buffers files have changed:

2018-07-26

Add mutual connection threshold to handshake Summary: Peers now exchange the bitsets of the peers that they are connected to during the handshake. For now, these bitsets are not used, but the peers themselves are used to drop incoming connections from peers that are connected to >threshold (defaults to 5) peers that we are also connected to. In the future, the bitsets maybe be used to more intelligenty determine who to connect with. Reviewers: #kraken, codyg Reviewed By: #kraken, codyg Subscribers: codyg, jenkins Differential Revision: https://code.uberinternal.com/D1918385

The documentation is generated from this commit.

2018-07-02

Improve Makefile Docker efficiency Summary: Build Linux binaries into output files independent of the `PROGS` output files, such that we aren't constantly removing / re-building binaries. This allows the integration / devcluster targets to reuse already built binaries and avoid costly cross-compile steps (origin / build-index). Also fixes the protobuf target to no-op if the protobuf files have already been generated. This was forcing agent / origin to be rebuilt every time. Overall, simplify and clean up the Makefile in a few ways: - Remove individual `run` targets for components. Just use `devcluster` now, it works great. - Remove intermediate build steps now that everything is more efficient. - Factor out redundant build commands. Reviewers: #kraken, evelynl, yiran Reviewed By: #kraken, evelynl, yiran Subscribers: jenkins Differential Revision: https://code.uberinternal.com/D1864089

2018-06-13

Revert "Add mutual connection threshold to handshake" This reverts commit df01dd4ce498edc42f4dc4812a962ccf27d5f80f.

2018-06-08

Add mutual connection threshold to handshake Summary: Peers now exchange the bitsets of the peers that they are connected to during the handshake. For now, these bitsets are not used, but the peers themselves are used to drop incoming connections from peers that are connected to >threshold (defaults to 5) peers that we are also connected to. In the future, the bitsets maybe be used to more intelligenty determine who to connect with. Reviewers: #kraken, codyg Reviewed By: #kraken, codyg Subscribers: jenkins Differential Revision: https://code.uberinternal.com/D1799237

2018-04-02

Refresh missing blobs in origin on handshake Summary: Fixes T1665275 If origin suffers data loss but metainfo is still cached in tracker, agent torrent downloads will be able to start since metainfo is available, but will timeout because the blob does not exist in origin. In this scenario, the agent is forced to wait for metainfo to expire in tracker, such that the metainfo can be re-requested and trigger a remote blob download. This diff solves this issue by plumbing namespace through handshake, and having the OriginTorrentArchive re-fetch blobs (asynchronously of course) if they receive a handshake for a non-existent blob. The handshake will fail, but the agent will retry after the blacklist duration passes. The implementation for this functionality breaks out the concept of refreshing a blob in a deduplicated fashion into a new package, `blobrefresh`, which is injected into both the blob server and the origin torrent archive. Also lowers the metainfo TTL in tracker cache from 1hr to 5mins. Reviewers: #kraken, #protobuf_reviewers, yiran Reviewed By: #kraken, yiran Subscribers: yiran, jenkins Maniphest Tasks: T1665275 Differential Revision: https://code.uberinternal.com/D1620879

2018-03-29

Add complete message to p2p protocol Summary: When a peer finishes a torrent, it blasts out a "complete" message to all established connections. For a number of reasons, announce messages are unreliable -- however, a complete message will immediately resolve any announce failures. This has a number of benefits: - Can quickly close connections if both peers are complete. - Can quickly alert incomplete peers that all pieces are available. Migration plan: If a peer receives an unknown message type, it simply logs an error and ignores the message. Thus, the migration should be easy. Reviewers: #kraken, #protobuf_reviewers, evelynl Reviewed By: #kraken, evelynl Subscribers: pedge, jenkins Differential Revision: https://code.uberinternal.com/D1614235

2018-01-31

Switch bitfield to real bitmap Summary: So we can use bitwise operation for piece selection logic, which will be in next diff Reviewers: #kraken, codyg Reviewed By: #kraken, codyg Subscribers: codyg, jenkins Differential Revision: https://code.uberinternal.com/D1470329

2017-10-18

Expose origin peers through tracker announce Summary: Integrates tracker and origin such that tracker announce always returns origin peers for the given info hash. To do this, blob servers now expose the peer context of the p2p client running on the same host. Tracker uses an origin cluster client to lookup the peer contexts of origins and caches the results in redis. This eliminates redis as a single point of failure because if redis is down, tracker can always provide peers from the origin cluster. Reviewers: #kraken, igor Reviewed By: #kraken, igor Subscribers: igor, jenkins Differential Revision: https://code.uberinternal.com/D1255939

2017-10-05

Kraken: move libraries under client/ to lib/ Reviewers: #kraken, evelynl Reviewed By: #kraken, evelynl Subscribers: jenkins Differential Revision: https://code.uberinternal.com/D1229557

2017-08-18

Implement torrent storage Summary: Replace scheduler.Torrent with storage.Torrent Change bitfield proto to include torrent name Reviewers: #kraken, yiran, codyg Reviewed By: #kraken, yiran, codyg Subscribers: yiran, codyg, jenkins Differential Revision: https://code.uberinternal.com/D1115189

2017-08-02

Introduce scheduler package to client/torrent Summary: See https://docs.google.com/document/d/13w4p3wMTEiepJypwMXgyRql4VTauDG4TcmvjBbEzcXg/edit# for high-level design. There is still a lot missing from the Scheduler implementation, including seeding policies, preemption, smarter announcing policies, etc. Most of these unfinished areas are marked with TODOs. A massive amount of credit is due to @evelynl, who paired programmed most of this diff with me and wrote several large chunks independently. Test Plan: Tested with the seeder / leecher program in client/bin/scheduler/main.go I plan to add unit tests to this diff if we agree on the overall design / interfaces. Reviewers: yiran, igor, evelynl Reviewed By: igor, evelynl Subscribers: jenkins, evelynl Differential Revision: https://code.uberinternal.com/D1062197

2017-06-23

P2P client: first blood Summary: This is an initial version of P2P client that supports basic P2P sharing scenarios, it's leaking proper connection management, needs to be properly protected from data races, missing recovery sceraios and tests Test Plan: Add more test Reviewers: #kraken, codyg Reviewed By: #kraken, codyg Subscribers: codyg, jenkins Differential Revision: https://code.uberinternal.com/D1004881