Proto commits in UofT-EcoSystem/rlscope

These 34 commits are when the Protocol Buffers files have changed:

2021-01-20

Delete old unused files

The documentation is generated from this commit.

2021-01-16

- Better usability: add paths automatically at container start, better rlscope_help documentation - Naming updates: iml -> rlscope

2020-10-16

In the midst of renaming 'iml' -> 'rls'

In the midst of renaming 'iml' -> 'rls'

In the midst of renaming 'iml' -> 'rls'

In the midst of renaming 'iml' -> 'rls'

2020-06-10

Start using "common_util" and "range_sampling" libraries; delete duplicate files. cmake is confusing as hell... lessons learned: - don't try to use "aliases" for library targets since find_package only appears to run once. - store *_LIBRARY and *_INCLUDE_DIRS in cached variables so they defined on subsequent builds / cmake configures.

2020-06-09

Update cmake / setup.sh build for sanity; docker builds and install everything into local.docker (host uses local.host). Import range_profiling and common_util library code "as-is" from my custom CUPTI-Samples repo; need integrate/use within RLScope project.

2019-10-25

Trying to get minigo from mlperf_training repo running again (branch=iml). Realizing that in order to trace and analyze multiple generations, we need to support the same "training phase" and "process name" being launched multiple times. To do that, we need some notion of "phase_ident" and "process_ident" that is unique (even if phase_name and process_name are identical). Otherwise it becomes difficult to know how to "group" events within phases/processes at analysis time.

2019-09-19

Record the number of overhead events so we can subtract them later on. Was attempting to implement subtraction "later on" by modifying the venn_js files directly, but realized that this is a "hacky"/error-prone way implement handling profiling overhead. Next steps: - "inject" profiling overhead, and add a new CPU category CATEGORY_PROFILING_OVERHEAD. - Need to think about how to handle this in our plots; ideally we will want to "subtract it entirely"... won't we end up with the same problem? A: No; we can handle it during the overlap computation, by doing: overlap[overhead, CPU, GPU] -> overlap[GPU] overlap[overhead, CPU] -> None - Need to parallelize overlap computation and reduce memory usage by performing it in "chunks" that we grab from the SQL database. We can have a pool of worker threads that run a chunk at a time.

2019-09-09

Handle python profiling overhead. Looks like there's still "inconsistent bias" when handling CUPTI overhead; need to investigate whether this is due to CUDA API-calls "becoming slower" over time.

2019-08-26

Trace GPU kernel start/end timestamps. $ iml-prof --cuda-activities python train.py Trace CUDA API start/end timestamps. $ iml-prof --cuda-api-events python train.py

2019-08-23

Import some code for recording GPU start/end timestamps and dumping it async; doesn't compile, needs refactoring first.

2019-08-22

Revert "In the midst of porting over remaining TensorFlow GPU event and" This reverts commit 0bd126b03f87f4291952711f3576943e060e21c4.

In the midst of porting over remaining TensorFlow GPU event and CPU-side-event tracing code. NOTE: does not compile.

2019-08-21

Found that using LD_PRELOAD to intercept cudaLaunchKernel adds less overhead than using libcupti to intercept it; hard to say what CUDA is doing beneath the covers to add overhead (perhaps correlation ID, or extra memory allocations for tracking things). Used LD_PRELOAD interception of cudaLaunchKernel to collect # of cudaLaunchKernel calls and total usec spent in cudaLaunchKernel. Rough number appear to show consistent time spent in cudaLaunchKernel across measured runs (8.5 usec), which suggests we will be able to get meaningful "average overhead" measurements. $ iml-prof --cuda-api-profile python train.py Next steps: - fuzz CUDA API for other API calls we need to trace - Port tensorflow protobuf stat-gathering to my tracer - make a graph of CUDA API times with/without instrumentation (TensorFlow tracing, libcupti) on.

2019-08-13

Trying out minimal PC sampling calls by printing out PC sampling records. $ iml-prof --pc-sampling python train.py Need to switch development to mel-17 (Quadro P4000) which has PC sampling support. Added some boiler-plate code for recording sampling state once PC sampling is running; haven't tested it at all, and still need to hook it up to the PC sampling cupti callbacks.

2019-08-12

Enable CUDA API profiling by doing: $ iml-prof --cuda-api-calls python train.py Need to start dumping traces to protobuf files.

2019-08-07

Get wrapper library compiling. Added backward-cpp stack-trace library to help with debug failed runtime checks. Need to debug api stat collection and printing.

2019-07-31

- Account for and subtract pyprof overhead. If we run with pyprof enabled, and subtract pyprof overhead, we estimate 2.5% (less) than the uninstrumented training time. Still need to check if this 2.5% is within standard error of measurement, but this is promising. - Next: record libcupti profiling overhead numbers, and see if it accounts for the 40% in the PPT slides.

2019-07-24

- Collect profiling overhead plot for all the OpenAI workloads; it's high across the board (~ 200%). - Next steps: dig into what part of IML is causing the profiling overhead by incrementally disabling features.

2019-07-19

Was in the progress of getting train_minigo.sh working (not quite there yet, early agent is struggling to play games during loop_train_eval's since it keeps resigning during all the games it plays). Context switch: trying to debug issues Sri encounters on long training runs.

2019-07-12

- Working on collecting CPU/GPU memory utilization

2019-07-10

- In progress: working on adding multi-machine support, but still to finish the iml-analyze part, since that part generally assumed a single machine.

2019-05-27

Add support for running on AMD ROCm GPUs. Dockerized all the ROCm stuff as well. Am a bit foggy where I left off on this though...

2019-05-07

Create a .whl package using setup.py. Still need to do this for iml-drill. import iml_profiler # All APIs for profiling are accessed @ iml_profiler.* # e.g. with iml_profiler.prof.operation('func'): ...

2019-04-29

Create basic unit-test: Atari pong. Still need to make one for minigo.

2019-03-04

- Create script for sampling CPU/GPU utilization. - Working on figuring out how to use matplotlib to generate "heat-scale" from utilization data. Figured out how to generate heat-scale squares; however it looks hard to precisely generate pixel-to-second accuracy with matplotlib, so we may want to use html/css instead (which isn't actually that hard I don't think).

2019-03-01

Debugging stuff / adding utilization metrics

2019-02-26

In the midst of adding phases.

2019-01-22

Store tracing results into an SQLite database. This is in preparation for supporting tracing multiple processes. Need to re-write some plot generation code to read from SQLite files instead of raw protobuf files, and separate data on a per-operation basis based on overlap with "Operation" events (e.g. q_forward).

2019-01-17

Working on benchmarking minigo; need to support tracing multiple concurrent python scripts. Need to remove the step-centric idempotent-centric code.

2019-01-03

Add pyprof events to timeline. Clearly they are off by ~ 1 second from the GPU/CUDA events. Still not sure why... going to look into tensorflow code and whether it does anything to the raw timestamps (e.g. adjusting them) before outputting them for tfprof.

2019-01-02

Create overlap plot that factors in overlap between CPU/GPU execution.