Proto commits in postmates/cernan

These 4 commits are when the Protocol Buffers files have changed:

2017-08-12

Integrate histogram (#293) * Integrate quantiles/Histogram This commit is the start to the integration of quantiles' Histogram. It's big and painful and does not work yet. You'll notice that we're pegged on an in-progress version of quantiles and that's needed while we figure out what cernan needs exactly. Heh, this one is painful to write. Telemetry is probably the worst part of cernan to change, second to Bucket. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * First edition that builds This commit has all the hot changes you need to make cernan build in the new Histogram world. There's still stuff commented out and surely there will be things that do not and cannot work. But! It's progress of a kind. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * First compiling tests run A good chunk of the tests fail to run but, well, at least it compiles. Ugh what a nightmare. Changing anything to do with Telemetry hurts hurts hurts Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Another day, another block of changes to Telemetry This commit introduces fixes for many of the operations on Telemetry and resolves some of the test failures. We wrap Value in an Option not because there might be a None in a valid state but so that we can avoid allocing another large structure when we need to do AddAssign. Annoying. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Fix the buckets tests This commit fixes the buckets tests, the hardest of the tests to satisfy. Mostly the issue was 1. not generating the right data and 2. not thaw/hardening correctly. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Correct more tests This commit corrects more of the tests, primarily though the modification of histogram itself. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Fix all non-doc tests, save one This commit gets us to a situation where the tests all pass again, save the most difficult one in buckets. It's no uncommented and we're not building anymore. That's fine: we know right where we need to pick back up. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * All tests now pass Holy hell, look at this. We're now passing all tests and this series of commits is ready for review. The primary changes here are all fiddling with APIs so we can build up the quickcheck model for bucket insertion. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Remove value.rs This has no more use. I forgot to nuke it. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Peg quantiles to 0.6.0 This commit pegs quantiles to an actual release, off from the git SHA we were using. The SHA was only in use because of development needs. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Correct longstanding binning problem in buckets This commit corrects a longstanding problem in buckets whereby a metric with the same name but different tags will be lumped together in aggregate. This is resolved by taking the hash of the name and tags only as we must lump different aggregates together. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Ensure persists remain across resets The buckets aggregation needs to maintain a persist across resets. This commit adds a test -- test_reset_maintains_persist -- to ensure that we keep on the straight and narrow. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Swap is_zeroed boolean's nature We swapped is_zeroed's meaning to be true if the underlying value is zero or None. Unfortunately after these tests were already passing and this implies some major defect in the way wavefront padding functions. We'll have to dig into this further. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Correct padding Previously our padding implementation in the wavefront sink was faulty across flushes _and_ to timestamps sent with timestamps in the future. This commit resolves these problems by constructing an explicit Pad type. Added benefit is a reduction in copies as the padding iterator is now explicitly over references. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * More fiddling with zero padding There are still low-probability conditions that we fail on. This commit addresses the ones I've found so far by making sure we feed last seen through, allowing us, as the comments note, to pad into the past. This commit suffers from out-of-order delivery which should not be possible. I think this may have existed prior to this commit, but I'm unsure. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Correct further test issues with padding This commit corrects further issues with the padding implementation caught by high-test quickcheck runs. There are manual tests to verify specific cases. I'll be running higher capacity tests shortly but I wanted to checkpoint these changes. For the most part we're just adapting to corner cases in the sequence of Telemetry. Signed-off-by: Brian L. Troutwine <blt@postmates.com> * Finish off padding tests There's still a known issue with this implementation but it's a deeper problem. Namely this: if you have two points with the same timestamp arrive into a sink across a flush you'll get two distinct points, rather than one unified, aggregated point. This is not addressable in the context of this work and will be a follow-on project. Signed-off-by: Brian L. Troutwine <blt@postmates.com>

The documentation is generated from this commit.

2017-01-06

Extend the native protocol to have explicit persist option (#174) In #172 we identified that cernan's method of persisting statsd gauges is a little weak. The ambition for the native protocol is to reflect all of the subtlties of the statsd protocol--which is by far the most expansive of cernan's sources--while making them explicit and clear. The intention--noted in #171--is that this should drag cernan's internal model along with it. The native protocol now has an explicit 'persisted' flag in Telemetry. A telemetry object which has been tagged as persisted will persist across binning windows. One which is not persisted will not. Signed-off-by: Brian L. Troutwine <blt@postmates.com>

2017-01-05

Introduce prometheus sink (#170) This commit introduces a sink for the Prometheus monitoring system. This sink is a little weird in that it will accumulate points until such a time as Prometheus scrapes the HTTP endpoint exposed. There's no standard push mechanism for Prometheus, you see. Signed-off-by: Brian L. Troutwine <blt@postmates.com>

2016-12-26

Introduce native protocol, CPU resource improvements (#167) This commit includes a number of broad changes to cernan with the intention of drastically reducing the CPU load that it requires at high levels of ingestion. This has been done through smarter use of memory--we've reduced small allocations through the judicious application of Arc--and through a much more efficient protobuf based ingestion protocol. Text based sinks have been improved as well through push-based building of their strings. The self-telemetry of the wavefront sink is reduced. The problems that needed addressing have been and the CPU drain was non-trivial. A cheap method for self-telemetry should be investigated. Signed-off-by: Brian L. Troutwine <blt@postmates.com>