Proto commits in tokio-rs/console

These 40 commits are when the Protocol Buffers files have changed:

2024-11-05

feat: add the WatchState API (#582) This pull request introduces a new stream API named "WatchState." Currently, it only provides the current temporality, which will enable clients to check if the server has been paused.

The documentation is generated from this commit.

2023-10-10

docs: fix typo in proto (#472)

2023-09-29

feat(console): add task scheduled times histogram (#409) Display the scheduled time percentiles and sparklines for the histogram of scheduled times. The schduled time is the time between when a task is woken and when it is next polled. The scheduled time, which was already calculated, is now stored in a histogram and sent over the wire in together with the task details. This is used to draw percentiles and sparklines on the task details view, in the same way that is done for the poll times histogram. The refactoring done in #408 has been used to more easily display two sets of durations (percentiles and histogram where possible). ## PR Notes The PR depends on both #406, which adds initial support for recording the scheduled (wake-to-poll) time, and #408, which refactors the percentile and histogram widgets to make them easier to reuse. It shouldn't really be reviewed in depth until those two have been merged as it contains a lot of duplication and will need to be rebased. Here are some examples of the scheduled times durations on the task detail view: <img width="1037" alt="task detail view for the sender task in the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232608774-d8ac48a7-3fe7-4742-a75b-e11bdb23abaa.png"> <img width="1043" alt="task detail view for the burn task in the app example" src="https://user-images.githubusercontent.com/89589/232608864-637f4f52-d4a6-468d-88fc-8fe1d53fdff9.png">

feat: add scheduled time per task (#406) Each task displays the sum of the time it has been idle and busy, as well as the total. The idle time includes the time between when a task is woken, and when the runtime actually polls that task. There are cases where a task may be scheduled for a long time after being woken, before it is polled. Especially if many tasks are woken at the same time and don't yield back to the runtime quickly. To add visibility to this, the total time that a task is scheduled before being polled has been added. Additionally, a new task state `Scheduled` has been added. This is displayed in both the tasks table and in the task detail view. In the `console-api`, the total `scheduled_time` for the task has been added to the `TaskStats` message in `tasks.proto`. This is the first of two parts. In the second part (#409), a histogram for scheduled time will be added, the equivalent of the poll time histogram which is already available on the task detail screen. To show a pathological case which may lead to needing to see the scheduled time, a new example has been added to the `console-subscriber` ## PR Notes This PR does something adjacent to what is described in #50, but not quite. The unicode character used for a `SCHED` task is ⏫. The second PR (#409) will record a scheduled time histogram the same as it recorded for poll times. These two changes should go in together (and certainly shouldn't be released separately). However, this PR is already quite big, so they'll be separated out. The idea is that this PR isn't merged until the next one has also been reviewed and approved. It would be good to get some feedback at this stage though. The task list view with the new column for `Sched` time: <img width="1032" alt="a tasks table view for the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232456977-2921f884-4673-420f-ba4f-3646627d44db.png"> The `Task` block in the task detail view showing the new `Scheduled` time entry. <img width="510" alt="The task block on the task detail view for the rx task in the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232457332-e455e086-9468-42c9-8fda-7965d8d1e6f8.png">

2023-04-21

feat(console): add task scheduled times histogram (#409) Display the scheduled time percentiles and sparklines for the histogram of scheduled times. The schduled time is the time between when a task is woken and when it is next polled. The scheduled time, which was already calculated, is now stored in a histogram and sent over the wire in together with the task details. This is used to draw percentiles and sparklines on the task details view, in the same way that is done for the poll times histogram. The refactoring done in #408 has been used to more easily display two sets of durations (percentiles and histogram where possible). ## PR Notes The PR depends on both #406, which adds initial support for recording the scheduled (wake-to-poll) time, and #408, which refactors the percentile and histogram widgets to make them easier to reuse. It shouldn't really be reviewed in depth until those two have been merged as it contains a lot of duplication and will need to be rebased. Here are some examples of the scheduled times durations on the task detail view: <img width="1037" alt="task detail view for the sender task in the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232608774-d8ac48a7-3fe7-4742-a75b-e11bdb23abaa.png"> <img width="1043" alt="task detail view for the burn task in the app example" src="https://user-images.githubusercontent.com/89589/232608864-637f4f52-d4a6-468d-88fc-8fe1d53fdff9.png">

2023-04-19

feat: add scheduled time per task (#406) Each task displays the sum of the time it has been idle and busy, as well as the total. The idle time includes the time between when a task is woken, and when the runtime actually polls that task. There are cases where a task may be scheduled for a long time after being woken, before it is polled. Especially if many tasks are woken at the same time and don't yield back to the runtime quickly. To add visibility to this, the total time that a task is scheduled before being polled has been added. Additionally, a new task state `Scheduled` has been added. This is displayed in both the tasks table and in the task detail view. In the `console-api`, the total `scheduled_time` for the task has been added to the `TaskStats` message in `tasks.proto`. This is the first of two parts. In the second part (#409), a histogram for scheduled time will be added, the equivalent of the poll time histogram which is already available on the task detail screen. To show a pathological case which may lead to needing to see the scheduled time, a new example has been added to the `console-subscriber` ## PR Notes This PR does something adjacent to what is described in #50, but not quite. The unicode character used for a `SCHED` task is ⏫. The second PR (#409) will record a scheduled time histogram the same as it recorded for poll times. These two changes should go in together (and certainly shouldn't be released separately). However, this PR is already quite big, so they'll be separated out. The idea is that this PR isn't merged until the next one has also been reviewed and approved. It would be good to get some feedback at this stage though. The task list view with the new column for `Sched` time: <img width="1032" alt="a tasks table view for the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232456977-2921f884-4673-420f-ba4f-3646627d44db.png"> The `Task` block in the task detail view showing the new `Scheduled` time entry. <img width="510" alt="The task block on the task detail view for the rx task in the long-scheduled example" src="https://user-images.githubusercontent.com/89589/232457332-e455e086-9468-42c9-8fda-7965d8d1e6f8.png">

2022-05-23

feat(api): add optional histogram outlier details (#351) Currently, the protobuf `TaskDetails` message sends only the raw binary representation of an `hdrhistogram` histogram serialized with the V2 serializer. When we add an upper bound to our histograms to solve issue #350, though, we may want to additionally record data about how many outliers exceeded the upper bound. This branch modifies the `TaskDetails` message to send a new `DurationHistogram` message, which includes the binary representation of the histogram in addition to an outlier count and the value of the highest outlier (if there was one). This change was performed by changing the histogram field to a `oneof`, so clients using the new API are still compatible with previous `console-subscriber` versions that send only the raw histogram. BREAKING CHANGE: This is a breaking change *to the Rust bindings* (the `console-api` crate) due to changing a field from an `Option` to a protobuf `oneof` (introducing a new enum type). This is **not** a breaking change to the protobufs themselves --- the actual wire format change is backwards-compatible, but the generated Rust code changes in a breaking way. Signed-off-by: Eliza Weisman <eliza@buoyant.io>

2022-02-09

chore(api): move timestamp proto file (#277) This PR moves `console-api/proto/google/protobuf/timestamp/timestamp.proto` to `console-api/proto/google/protobuf/timestamp.proto` to match the directory structure in https://github.com/protocolbuffers/protobuf. This may not be important for generating code for Rust, however other languages such as Javascript need the path to match the official definitions in order to work with the official [google-protobuf](https://www.npmjs.com/package/google-protobuf) library.

2021-12-16

feat(subscriber): count dropped events due to buffer cap (#211) See also #209 Currently, we don't really have any way of surfacing potential data loss due to the event buffer capacity limit --- so, if events are dropped, the user may not be *aware* that they're seeing an incomplete picture of their application. It would be better if we had a way to surface this in the UI. This branch adds support for counting the number of dropped events in the `ConsoleLayer`. This data is now included in the `ResourceUpdate`, `TaskUpdate`, and `AsyncOpUpdate` messages, respectively. We track a separate counter for dropped events of each type, to make it easier to determine what data may be missing. The console UI doesn't currently *display* these counts; that can be added in a separate PR. We may want to use the warnings interface for displaying this information?

2021-12-15

feat: add resource detail view (#188) This PR introduces the resoruce detail view. It also brings a couple of additional changes: - we now rely on context of entered spans rather than explicit parent - child relationships - we expose internal resources (`BatchSempahore`) - we show the relationships between resources (`Mutex` -> `BatchSemaphore`) - we show the async ops that are live on a specific resource - we also show the tasks that are awaiting on these async ops, which allows the user to draw a relationship between tasks and resources - there are more examples added in the examples folder (semaphore and mutex) Some screenshots: <img width="1050" alt="Screenshot 2021-11-23 at 19 12 28" src="https://user-images.githubusercontent.com/4391506/143072764-f940ed45-350d-4be4-941b-95ef5650c4d8.png"> <img width="1388" alt="Screenshot 2021-11-23 at 19 12 38" src="https://user-images.githubusercontent.com/4391506/143072791-ee5d485f-ea1e-4609-8946-16a61bf5776f.png"> <img width="1371" alt="Screenshot 2021-11-23 at 19 12 50" src="https://user-images.githubusercontent.com/4391506/143072807-4b577d3f-17d6-4ade-8dcc-006b44fc1014.png"> Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

docs(api): console-api docs (#197) * Added documentation for each of the protobuf definitions. * Add docs for the top-level modules (each is just a shim around a protobuf definition). * Apply suggestions from code review Co-authored-by: Eliza Weisman <eliza@buoyant.io>

2021-09-20

feat(subscriber): use `Location` for tasks and resources (#154) This PR modifies the proto schema and the subscriber crate to allow us to capture the spawn location of a task and the instantiation location of a resource into a `Location` proto message rather than a field. Note that the special treatment given to the `spawn.location` filed, found on tasks has not been touched in order not to break apps that use tokio versions, not emitting this data as structured fields. Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-09-13

chore(api): move protobuf definitions into `console-api` dir (#141) Currently, the protobuf definitions for the console API are symlinked into the `console-api` directory. This is intended to support vendoring the `console-api` crate (see #128). However, some users have reported that the symlink isn't the ideal solution. Windows users have had issues following the Unix symlink, so #128 broke the build for people trying to develop the console on Windows (I suspect it may still work in WSL?). There's no _real_ reason that the `proto/` directory needs to be in the repo root, rather than the `console-api` crate; the symlink was only added to avoid moving it around. Therefore, it's fine to just move it instead. This branch moves the `proto/` dir into `console-api/proto`, hopefully fixing the build issues for Windows users.

2021-09-10

feat(console): add `retain-for` cmd line arg (#119) This PR adds a `retain-for` duration argument to the console. Additionally, I have modified the proto to not include the total_time for tasks,resources and async ops and instead contain `created_at` and `dropped_at` fields. A separate `console-util` crate has been added that contains the logic for parsing a duration string. Close #108 Signed-off-by: Zahari Dichev <zaharidichev@gmail.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

2021-09-06

feat: emit and show self-wake counts for tasks (#112) This counts all wakes of a task that happen while the task is "currently polled". This could be from `task::yield_now()`, from `tokio::coop`, or any other pattern that triggers the task to wake even though it's currently being polled. This implementation builds upon @seanmonstar's branch #55. However, rather than detecting self-wakes by simply looking at _whether_ the task is being polled at all, we instead detect them by checking whether the task's span is on the current thread's stack of entered spans. This approach should _not_ produce self-positives when a task is woken by another thread while it is currently entered (as discussed in https://github.com/tokio-rs/console/pull/55#discussion_r659223228). Testing this with the "burn" subcommand for the example app, which spawns one task that does a lot of self-wakes via `task::yield`, indicates that it is, in fact, working properly. Here are the stats for the `burn` task:  Note the very large number of self-wakes. In comparison, here are the task stats for a "normal" task, which does not wake itself (and is instead woken by timers):  Note that no self-wakes stat is displayed. Closes #55. Signed-off-by: Eliza Weisman <eliza@buoyant.io> Co-authored-by: Sean McArthur <sean@seanmonstar.com>

2021-09-01

chore(docs): fix links not found in readme and some typos (#104) * chore(docs): fix links not found in readme and some typos Signed-off-by: wuaoxiang <wuaoxiang@stargraph.cn> * Update console-subscriber/src/visitors.rs Co-authored-by: wuaoxiang <wuaoxiang@stargraph.cn> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

2021-08-31

feat(proto): represent readiness as bool (#103) Based on some [earlier feedback][1], this PR changes the proto definition so readiness of a poll op is represented via a bool. The relevant changes have been made to: https://github.com/tokio-rs/tokio/pull/4072 [1]: https://github.com/tokio-rs/console/pull/77#discussion_r695031627 Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-08-26

feat(subscriber): resource instrumentation (#77) This PR adds the pieces needed to get resource instrumentation data into the console. This includes: - changes to the proto definitions - changes to the subscriber The UI part will come as a follow up PR. This branch uses a patched `tokio` that emits these tracing spans and events for the `Sleep` resource. You can look at the raw data by running: ``` cargo run --example app cargo run --example dump ``` The information piped through includes: - data describing the resource lifecycle, namely when resources are created and dropped - data describing the async operations that take place on these events and their associationg with tasks - data reflecting the state updates that take place on resources (e.g. resetting timer's duration, adding permits to a semaphore, etc) Signed-off-by: Zahari Dichev <zaharidichev@gmail.com> Co-authored-by: Eliza Weisman <eliza@buoyant.io>

2021-08-25

feat: add pause and resume (#78) A first step to address #70. You can now press the spacebar to "pause" the console. The subscriber will continue to process events, but as along as the client is "paused", it won't send any task updates. Resuming will bring the client back to "live", getting all updates again.  I've tried a few different UI things, from making the whole background of the top red, to just making the word red... It's fine if others feel there's a better or prettier way to show this, but to keep moving forward, I just picked something that seemed good enough. Closes #85

2021-08-24

Merge branch 'main' into zd/res-schema

2021-08-23

feedback Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-08-03

feat(console): use sequential `u64` task IDs (#75) Closes #61 ### Subscriber - Assign sequential ID numbers to tasks, keep a `HashMap` of span ID --> task ID. - Change the task id type to `uint64` in protos. ### Console - Use the `u64` task IDs, display them in decimal. - Remove now-unused `Task::id_hex()` - Shrink the TID column to a minimum of 4 chars. Co-authored-by: Eliza Weisman <eliza@buoyant.io>

2021-08-02

feedback Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-07-28

resource instrumentation Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-07-13

subscriber: correctly reflect busy and idle times (#60) This PR enables us to track `last_poll_started` and `last_poll_ended` times in order to be able to correctly calculate the busy time of a task. Namely, the busy time is the sum across all the time spent executing `poll` plus the time we have spent so for in a poll method that has been invoked but has not returned:  Note that task 1 has been polled only ones, its polled method has not returned and as a result of that its idle time is very low while its busy time is close to its total time. You can test this change with the example program provided in the issue. Fixes #59 Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-07-02

feat(subscriber): Record and send poll times with HdrHistogram (#47) As suggested in the issue, this PR uses HdrHistogram to record task poll times as nanoseconds. The histogram is serialized into bytes using the V2Serializer. The subscriber now offers a second RPC stream called TaskDetails for a given task ID. The console app starts a details stream when you view the details of a task and stops the stream when you leave it. To demonstrate that it works, I also added some stats and a small chart to the task view using the histogram data. But I won't insist on merging this part, since you may have other plans for the task details view. Closes #36

2021-06-25

subscriber: emit self-wakes stat when task wakes itself

2021-06-14

subscriber: emit waker stats (#44) Closes #42

2021-06-09

add proto schema for resources instrumentation Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-05-25

feat: send structured fields on the wire (#26) This PR adds structured fields to the wire format. Fixes #6 Signed-off-by: Zahari Dichev <zaharidichev@gmail.com>

2021-05-22

fix: detect completed tasks even if console connects after (#29) The "completion" of a task would result in that being sent to all watchers in a task update. However, after a flush, that status wasn't stored in a way that new watchers could detect. This changes it so that a task is determined "completed" when it includes a "total" duration, instead of being in a separate part of the `TasksUpdate` proto message.

2021-05-04

wip

2021-05-03

include a timestamp in each `TaskUpdate` Signed-off-by: Eliza Weisman <eliza@buoyant.io>

protobuf docs Signed-off-by: Eliza Weisman <eliza@buoyant.io>

Merge branch 'main' of github.com:hawkw/consolation

rename service (whoops copy-paste) Signed-off-by: Eliza Weisman <eliza@buoyant.io>

2021-05-01

build with optional

don't have string fields be a oneof Signed-off-by: Eliza Weisman <eliza@buoyant.io>

wip in-process model Signed-off-by: Eliza Weisman <eliza@buoyant.io>

2021-04-30

initial commit