Proto commits in google/perf_data_converter

These 48 commits are when the Protocol Buffers files have changed:

2024-10-08

Internal change PiperOrigin-RevId: 663807973

Add doc URL to profile format and use it display help link. PiperOrigin-RevId: 668565523

Pick up changes from pprof's copy of profile.proto PiperOrigin-RevId: 669350670

The documentation is generated from this commit.

2024-09-09

Add PERF_RECORD_ID_INDEX support. PiperOrigin-RevId: 662957586

2024-08-29

Update read_format to add the lost field. PiperOrigin-RevId: 656211182

2024-06-13

Distinguish between `cache_latency` and `total_latency` coming from x86 machines with WEIGHT or WEIGHT_STRUCT. Both newer Intel machines and AMD machines with IBS populate both fields, so we propagate them accordingly as Profile proto labels. Older Intel machines only populated the `weight` field, with the "total" issue-to-retirement latency, but we mistakenly used to convert that to `cache_latency`, now it is in `total_latency`. PiperOrigin-RevId: 641530291

2024-05-13

Update to latest internal version (#162) * Add quipper support for PERF_RECORD_KSYMBOL event. PiperOrigin-RevId: 628137192 * Enable quipper parsing of HEADER_GROUP_DESC for PERF_RECORD_HEADER_FEATURE events. PiperOrigin-RevId: 631961661 --------- Co-authored-by: ghanz <ghanz@google.com> Co-authored-by: skanev <skanev@google.com>

2024-01-17

Make quipper support parsing hybrid topology metadata in the perf header More info on Intel hybrid support in perf: tools/perf/Documentation/intel-hybrid.txt PiperOrigin-RevId: 597399015

2023-08-15

Update to latest internal version (#144) * Internal Code Change PiperOrigin-RevId: 548287810 * Add checks for the num_(core|thread)_siblings to avoid allocating large vectors We have tighter checks when actually reading the data, this is just additional checks to prevent allocating large vectors based on the numbers in the metadata. PiperOrigin-RevId: 551683479 * Update description of WeightStruct. On SapphireRapids and newer, var1 has the cache latency, and var2 has the instruction latency, not the other way around. https://github.com/TrenchBoot/linux/blob/master/arch/x86/events/intel/ds.c#L1691-L1711 PiperOrigin-RevId: 552497379 * Add weight label support PiperOrigin-RevId: 552830068 * Use LOST_SAMPLE event to count lost samples for perf data with version 6.1 and above PiperOrigin-RevId: 553162733 * Scan all file_attrs for any comm_exec bit set to avoid incorrectly flagging non-exec comm events PiperOrigin-RevId: 555559068 --------- Co-authored-by: Googler <noreply@google.com> Co-authored-by: shantuo <shantuo@google.com> Co-authored-by: skanev <skanev@google.com> Co-authored-by: yelouis <yelouis@google.com>

2023-04-26

Use buildid-mmap event value and add corresponding build ID source stats PiperOrigin-RevId: 524934334

2023-03-29

Add buildid-mmap field support in quipper perf.data de/serialization PiperOrigin-RevId: 513882868

2023-01-17

Save the MD5 hash for the root path of filenames PiperOrigin-RevId: 501900924

2022-11-30

Populated necessary flags in quipper to support branch samples for Zen4 PiperOrigin-RevId: 489077940

2022-08-05

Add weight struct support PiperOrigin-RevId: 465137180

Fix casing for "ID" in the comments to enhance consistency PiperOrigin-RevId: 459802490

2022-05-23

Support profiles from perf 5.15. - Handle "HW index" field in a branch stack sample; - Add "size" field in build_id_event struct and PerfBuildID proto message; - Update perf binary in testdata (new profiles are not compatible with the old perf). Fixed the following error from the perf.data profile synthesized with perf-5.15: quipper/sample_info_reader.cc:747: Read/skipped 280 bytes, expected 288 bytes. quipper/perf_reader.cc:1023: Couldn't read event PERF_RECORD_SAMPLE The new branch sample type was added in https://lore.kernel.org/lkml/20200228163011.19358-3-kan.liang@linux.intel.com/. Build id size was added in https://lore.kernel.org/all/20201013192441.1299447-8-jolsa@kernel.org/. PiperOrigin-RevId: 449816934

2021-09-27

Update to latest internal revision (#103) * Update goldens for tests hidden behind TEST_CALLGRAPH and TEST_LARGE_PERF_DATA. Because these tests are disabled by default, their goldens have rotted over time. PiperOrigin-RevId: 375787759 * Retain raw data when doing a round-trip through quipper. The raw field is now populated in test files. PiperOrigin-RevId: 375792550 * Clarify raw/raw_size comments. PiperOrigin-RevId: 376000906 * Add support for binary proto format to perf_converter. This makes converting large profiles more efficient. It was documented with format="proto", but wasn't actually implemented. Update testdata to add missing .pb_data files and to fix .pb_data files that don't match their .pb_text. PiperOrigin-RevId: 377584601 * Remove non-existent dependency from conversion_utils. PiperOrigin-RevId: 382162084 * When combining mappings, compare only map type flags. PiperOrigin-RevId: 382618285 * Add configurations for testing in Docker. PiperOrigin-RevId: 383514991 * Handle MMAP2 with build-id Recently the kernel can add build-id of the binary in the MMAP2 record if available. Let's handle it in the quipper so that we can eliminate unnecessary disk lookups for build-id. PiperOrigin-RevId: 384498791 * Internal change. PiperOrigin-RevId: 384945190 * Check build-id size in the MMAP2 A fuzzer test found that MMAP2 with invalid build-id can cause buffer overflow. Check the max size and drop the invalid ones. PiperOrigin-RevId: 385217492 * Update the cross endian test with MMAP2 PiperOrigin-RevId: 385899195 * Check MMAP2 + BUILD_ID properly The PERF_RECORD_MISC_MMAP_BUILD_ID should be in MMAP2 only. PiperOrigin-RevId: 388536374 * Use a fake mapping for synthesized lost samples, so that they can be differentiated from actual unmapped samples in a profile. PiperOrigin-RevId: 390649419 * Update some github links to codeload so Bazel doesn't have to redirect. PiperOrigin-RevId: 392039747 * Don't assume that the first mapping is always for the kernel. We can have profiles collected by non-root users where the system is set up not to allow profiling of the kernel space, that don't have any kernel mappings. And we can have profiles where some user mappings appear before the kernel mapping. We handle the kernel mapping differently, by resetting its file offset to zero. This breaks symbolization if the first mapping is for a user space program and its file offset is not initially zero. PiperOrigin-RevId: 392538404 * Support streaming sample events in PerfReader. This CL adds two relatively simple functionalities: 1. A flag to ignore events of specified types in the proto returned by PerfReader. 2. Support for passing a callback to be invoked for each sample event. Together, they allow processing perf.data files in a two-pass manner: 1. Parse it using PerfParser, but instructing the underlying PerfReader to ignore PERF_RECORD_SAMPLE; construct auxiliary data structures based on the obtained data (e.g. mmap ranges). 2. Scan the perf.data file again, this time using only PerfReader, again ignoring sample events in the proto, but invoking a user-provided callback instead. PiperOrigin-RevId: 396897205 * When creating an mmap2_event, initialize two fields later for gcc compatibility. PiperOrigin-RevId: 396903499 * Add full support for large struct perf_record_time_conv added in Linux kernel 5.10. We serialize and deserialize the content of the new perf_record_time_conv fields depending on the size of the event. Support writing perf.data files including either the small or the large version of the perf_record_time_conv struct. PiperOrigin-RevId: 397753690 * Add data/code page size sampling support PiperOrigin-RevId: 398238738 * Have struct time_conv_event match the definition in the perf tool. PiperOrigin-RevId: 398296367 * Replace Travis CI with Github Actions. PiperOrigin-RevId: 398833899 * Avoid using non-trivial designated initializer lists in mmap2_events. PiperOrigin-RevId: 398838900 Co-authored-by: igorginzburg <igorginzburg@google.com> Co-authored-by: gmx <gmx@google.com> Co-authored-by: namhyung <namhyung@google.com> Co-authored-by: kpszeniczny <kpszeniczny@google.com>

2021-02-25

Update to latest internal version (#95) * Add some VLOG(2) logging, helpful for debugging. PiperOrigin-RevId: 346106261 * Store the filename and the build ID in PerfDataHandler::Mapping by value rather than by pointer. PiperOrigin-RevId: 346147984 * Make sure that created fake mappings are alive for the lifetime of the handler as the pointer value is used as a hash map key by the converter. PiperOrigin-RevId: 346178814 * Add perf_data_converter option to process data addresses. PiperOrigin-RevId: 346577578 * Add quipper support for PERF_RECORD_CGROUP event. PiperOrigin-RevId: 346997787 * Optimize calls to std::string::find() and friends for a single char. The character literal overload is more efficient. PiperOrigin-RevId: 347848925 * Use a constant for the special "kernel PID" value of -1. PiperOrigin-RevId: 350167544 * Unify perf_data_converter/testdata/* file naming. PiperOrigin-RevId: 350369267 * Do not perform any changes on the parser output in tests before validating it against golden files. PiperOrigin-RevId: 350435915 * Use SetMmapFilename to set the filename for an mmap event instead of setting it manually. PiperOrigin-RevId: 350828548 * Update WORKSPACE dependencies to fix builds. Unpins protobuf dependency from 3.9 back to master, adds a dependency on rules_python, and moves the bazel_skylib declaration to the top of the WORKSPACE file to override any internally-declared versions in other dependencies. PiperOrigin-RevId: 353577129 * Enable combining of file backed mappings followed by anonymous mappings that are contiguous and have compatible protection flags. PiperOrigin-RevId: 353797482 * Internal changes PiperOrigin-RevId: 355754697 * Remove double space in the log message. PiperOrigin-RevId: 356358808 * Do not attempt to serialize the reader when the perf.data read failed. PiperOrigin-RevId: 356359812 * Add cgroup info in the SampleEvent in quipper. PiperOrigin-RevId: 357121501 * Remove gflags pinning to an older version as the gcc-4.8 patch was accepted. PiperOrigin-RevId: 358898320 * Update Travis to use gcc-5. PiperOrigin-RevId: 359350610 Co-authored-by: aalexand <aalexand@google.com> Co-authored-by: gmx <gmx@google.com> Co-authored-by: namhyung <namhyung@google.com> Co-authored-by: ckennelly <ckennelly@google.com> Co-authored-by: mkruskal <mkruskal@google.com>

2019-04-20

Remove support for the PERF_RECORD_READ perf event. (#78) PiperOrigin-RevId: 244373367

2019-03-21

Add a dummy kernel mmap as the first mmap in the perf_data.proto for heap profiles. (#73) Quipper does some special processing on the first seen mmap in a perf input under the assumption that this mmap is a kernel mmap. However, there is no kernel mmap in heap profile. PiperOrigin-RevId: 239513159

2018-08-02

Add quipper support for the perf user event PERF_RECORD_STAT_ROUND. PiperOrigin-RevId: 202976699

Add quipper support to read perf user event PERF_RECORD_STAT. PiperOrigin-RevId: 201210989

Add quipper support to read sample type PERF_SAMPLE_PHYS_ADDR. PiperOrigin-RevId: 201039019

Add support for the PERF_RECORD_STAT_CONFIG perf user event. PiperOrigin-RevId: 200454780

2018-06-07

Add support for the PERF_RECORD_THREAD_MAP perf user event. PiperOrigin-RevId: 196733856

Add quipper support for PERF_RECORD_AUXTRACE_INFO perf user event. PiperOrigin-RevId: 196758341

Add quipper support for PERF_RECORD_AUXTRACE_ERROR. PiperOrigin-RevId: 197965826

2018-04-16

Copybara staging (#52) * PiperOrigin-RevId: 192633430 * CheckPerfDataAgainstBaseline and PartiallyEqualsProto take an optional diff output parameter. PiperOrigin-RevId: 192824729 * Add quipper support for PERF_RECORD_NAMESPACES event. PiperOrigin-RevId: 193069966 * Changed bool flag to enable writing new golden files to a char array. Flag must be set to output folder of new golden files to enable saving them. Tests don't pass by default anymore when writing new golden files is enabled. PiperOrigin-RevId: 193070337 * Remap the first MMAP so that its file offset is zero even if it is not the first processed event. Recent perf versions can include user events before the first MMAP event. Add test that remapped kernel offsets are zero. Update impacted golden files. PiperOrigin-RevId: 193078625 * Check that kernel file offsets are remapped to zero in the piped case. PiperOrigin-RevId: 193086673

2018-04-07

Rolling up latest changes. (#51) * Remove duplicate golden files. PiperOrigin-RevId: 190480000 * Modify build mechanism in github to use bazel. PiperOrigin-RevId: 190552987 * Drop PERF_RECORD_HEADER_ATTR record if no IDs present in the event and the file_attr is already seen. PiperOrigin-RevId: 190646055 * Add quipper support for PERF_RECORD_SWITCH and PERF_RECORD_SWITCH_CPU_WIDE events. PiperOrigin-RevId: 191476086 * Move the contents of quipper/kernel/perf_event.h inside quipper namespace and modify macros to constants. This should enable including quipper's headers along-side include/uapi/linux/perf_events.h without conflicting definitions. PiperOrigin-RevId: 191501990 * Rename the class testing::ExampleSwitchEvent to testing::ExampleContextSwitchEvent for unanimity. PiperOrigin-RevId: 191512528 * Add quipper support for the perf userspace event PERF_RECORD_TIME_CONV. PiperOrigin-RevId: 191811857 * Update the struct perf_event_attr with the new binary fields introduced in and after perf version 4.1. PiperOrigin-RevId: 191923312

2018-03-21

Rollup latest changes from upstream (#50) * Add quipper support to read Core IDs and Socket IDs from CPU Topology metadata. PiperOrigin-RevId: 189059499 * Refactor ReadMetadata to use it with header feature events generated in pipe mode. PiperOrigin-RevId: 189388426 * Add quipper support for PERF_RECORD_HEADER_FEATURES event. PiperOrigin-RevId: 189658375 * Remove unused test, constant, and a function. PiperOrigin-RevId: 189818260 * Modify the perf events used in perf_recorder_test. PiperOrigin-RevId: 189953355

2018-03-14

Rolling up latest changes (#49) * Add quipper support for perf event PERF_RECORD_LOST_SAMPLES PiperOrigin-RevId: 188668820 * Move files under mybase/base/ to base/. PiperOrigin-RevId: 188776389

2018-03-10

PiperOrigin-RevId: 177332957

PiperOrigin-RevId: 166206258

Add quipper support for the below perf_branch_entry flags and field. Flags: in_tx, abort Field: cycles PiperOrigin-RevId: 188518918

Add support for perf event PERF_RECORD_ITRACE_START. PiperOrigin-RevId: 187044864

PiperOrigin-RevId: 166395340

PiperOrigin-RevId: 178028884

PiperOrigin-RevId: 182950993

2018-03-02

Restructure perf_data_converter source files/directories for bazel. (#46) (#47)

2018-02-24

Add support for perf events PERF_RECORD_AUX and PERF_RECORD_AUXTRACE in quipper. - Add support for these events in both file and piped mode - Add support to read the trailing trace data from PERF_RECORD_AUXTRACE event - Add new testdata files containing intel_pt// event PiperOrigin-RevId: 186704497

Formatted files. PiperOrigin-RevId: 185032883

2017-12-01

Add PMU Mappings and Group Desc metadata features support for quipper. Remove unused function arguments. PiperOrigin-RevId: 177480940

2017-08-24

Remove third_party/chromiumos-wide-profiling and third_party/kernel as they are moved to quipper and quipper/kernel respectively.

PiperOrigin-RevId: 165228015

2017-07-12

Project import generated by Copybara. (#13) PiperOrigin-RevId: 160971937

2017-01-13

Update quipper to f48e135cdc6b5aa37ebed12b2e7611abe8412964

2016-09-30

Remove profile_wrappers and unused functions Eliminate unnecessary proto and unused functions.

2016-09-21

first commit