Proto commits in abyss7/dist-clang

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2019-02-01

Save the arguments order (#124) Also upgrade `args_parser` to print generated command line from daemon

The documentation is generated from this commit.

2017-08-24

Add support for -frewrite-includes flag while preprocessing (#116) * Added support for rewrite-includes flag while preprocessing * Added rewrite-macros to ignored flags * Fixed review issues * Fix typo in comment * Updated tests * Fix dead pointer

2017-08-11

Implemented strict sharding (#110) A lot of changes in the |LockedQueue| logic

2017-06-27

Do not send "hash_match", if emitter doesn't provide "handled_hash".

Calculate unhandled hash only once and verify hash on absorber (#111) * Calculate unhandled hash only once. Also verify hash on absorber * Added tests for cache/hash flags in absorber response

2017-06-06

Support PCH & PTH files (#103) There are still some issues. Will fix them after this commit: * Don't render included headers in `FillFlags()` to remove duplication of code in tests and production. * `a call` → `calls` * Simplify and rename test `StoreDirectCacheForLocalResultAfterCacheMiss` → `StoreDirectCacheForLocalResultWithAndWithoutIncludedHeaders`. * Fix deps contents in tests to contain absolute header path instead of absolute PCH path.

2017-05-24

Add more metrics Also add more tests on `Emitter`

2017-03-12

Properly check and use the number of total shards. Total shards can't be less than 2.

Properly check and use the number of total shards. Total shards can't be less than 2.

2017-03-06

Coordinator now sets total shards for emitter.

Coordinator now sets total shards for emitter.

Implement a sharding of tasks between remote hosts (#96) * Added distribution support to LockedQueue Now Emitter finds out a hash of tasks for remote execution and puts this info to LockedQueue. Also DoRemoteExecute now has assigned distribution value that is used to pop appropriate task from queue. * Factored Index class out of LockedQueue Also updated distribution/range wording to shard/shards_number. Clang version used in travis builds also updated. * Use hash of |HandledSource| to compute shard * Add check on generated source when using shards. * Calculate shard for task - using handled source. * Use a separate worker pool for coordinators. Otherwise they will hang forever. Also don't use progressive sleep in |DoPoll()|. * Fix CoordinatorNoConnection test. * Fix bug with reverse index in LockedQueue. * Added tests for LockedQueue<T>::Index (#97) * Remove task counting from |ThreadPool| interface. It was used only in tests. * Move |LockedQueue::Index| implementation into separate header. * Add pop timeout to |LockedQueue| interface. And take it from configuration in emitter. * Implement |LocketQueue::Pop()| that respects external pool shutdown. * Add more tests on Coordinator rotation logics. * Fix Coorditanor rotation tests. * Update QtCreator project. * Update 'total_shards' from coordinator. * Clean-up emitter tests. * Implement EmitterTest.ConfigurationUpdateFromCoordinator.

2016-12-24

Implemented coordinator role. (#93) Also introduced changes to emitter to make him poll known coordinators to receive new remote hosts. Also added distribution field for remotes, but this one is not used for now.

2016-08-12

Support sanitizer blacklist file (#70) * Support sanitize blacklist file * Fix tests compilation * Correct hash computation * Get rid of ExtraFile type, use Immutable instead. * Remove std::transform and std::accumulate calls. * Refactor hash computation * Review fixes * Use HashMap instead of List for extra files. * Move ExtraFiles type alias to FileCache, more tests. * Add more tests * Remove fsanitize-blacklist flag when preprocessing + one more test.

2016-07-18

Issue 23: implement connection timeout (#62) * ISSUE-23: pass connect timeout to network service. * ISSUE-23: implement Linux part of connect with timeout. * ISSUE-23: implement Mac part. * ISSUE-23: enhance EndPoint a little, cover with test. * ISSUE-23: fix EndPoint::FromPassive and a test. * ISSUE-23: add a test for NetworkService::Connect. * ISSUE-23: fix EndPoint test, use dummy solution for NetworkServiceTest.ConnectTimedOut. * ISSUE-23: fix tests and timeout on Mac. * ISSUE-23: unify zero timeout with Mac, cleanup unused code. * ISSUE-23: s/indefinite/default/. * ISSUE-23: do not include a port if it is empty. * ISSUE-23: cleanup. * ISSUE-23: fix CHECK condition and move it to proper place. * ISSUE-23: fix buffer length for AF_INET6 address. * ISSUE-23: reorder parameters in code and protobufs. * ISSUE-23: remove tests and some EndPoint methods. * ISSUE-23: fix argument order, use clang-format. * ISSUE-23: fix confusing comment. * ISSUE-23: fixed compilation of clang/main.cc

2016-02-28

Do store cache index by default.

Do store cache index by default.

2016-01-26

ISSUE-49: Take the process run timeout from config for Absorber.

2016-01-21

Isolate client configuration logic in a separate class.

2015-11-07

ISSUE-28: implement 'disabled' option for the client.

2015-10-27

ISSUE-32: Implement the storing of the file cache index on disk. No tests for now.

2015-10-14

ISSUE-24: Store the size of an entry in the manifest v1.

ISSUE-24: implement |Migrator|.

2015-07-27

ISSUE-14: Auto-generate C++ sources from proto files. Also a major refactoring is done. Need to fix packaging and python files auto-generation.

2015-07-20

Fix bug with the cache not get actually cleaned. DCHECK() -> DCHECK_O_EVAL()!

2015-06-12

ISSUE-17: Add a couple of new cache size stats.

2015-05-17

ISSUE-6: Add more stats.

2015-05-13

ISSUE-6: Implement the collector and the statistics service.

2015-05-09

ISSUE-6: split |BaseDaemon| into |BaseDaemon| and |CompilationDaemon|. New |BaseDaemon| only does some networking stuff.

2015-04-14

Rename MACOS -> MACOSX

2015-04-07

Load plugins only for an actual platform.

2015-03-27

Use plugin path from the client configuration, if available.

2015-03-03

Clean-up the cache periodically. Every 10 minutes by default.

Use the version of the internal Clang library for a direct hash.

Allow to specify number of threads for the cache checking.

2015-02-16

Specify default read/send timeout values if no config present.

2015-02-13

Make the read and send connection timeouts configurable.

2015-01-01

Totally remove the option |cache.sync| from everywhere.

2014-12-29

Allow client to use configuration file instead of environment variables. Configuration file name should be ".distclang" It's searched in the current working directory and it's parents.

2014-12-10

Turn off IPv6 by default. Since it doesn't work right now.

2014-12-02

Add support for IPv6.

Rename |file_cache| to |cache|. Also, move |FileCache| into namespace |cache|.

2014-11-02

Update protobuf. They have moved to git repo - at last.

2014-11-01

ISSUE-19: Use precalculated hash inside |LocalExecute| message. Also distinguish different string types: sources, hashes, etc.

ISSUE-19: Revert adding and using |hash| field. Also fix |RemoveEntry| test.

ISSUE-19: [WIP] Implement Absorber and Emitter.

ISSUE-19: [WIP] Remove deprecated stuff. Also fix some bugs in Emitter.

Zip cached objects with Snappy.

Remove deps on unused staff like |Statistics| and |Histogram|.

Substitute --resource_dir with one, based or real Clang path. This should fix problems on Mac.

Allow to disable cache via config option.

Move all cache params to a separate config message.

Fix problem with python bindings. Forgot to regenerate and add them to repository.

ISSUE-25: Implement the direct cache mode. No tests for now.

Instrument test binaries with AddressSanitizer.

ISSUE-8: Try to implement something like direct mode.

Move protobuf/src to protobuf/exported

Move config.proto to daemon/ Also some CC.

ISSUE-23: Refactor clang arguments parsing. Now using libclang C++ API.

ISSUE-23: Some stable state with new arguments scheme. The client part is already refactored, but not tested yet.

Fix deps file handling by file cache.

Major refactoring of the file cache.

Add manifest description to file cache. Also, a little refactoring.

Move |dependenies| to |non_cached| in proto-file. And add |deps_file| instead.

Allow to do local compilation only on failed tasks. Also, double the default jobs number.

Reduce number of original 'clang' invocations. The experiment shows that the '-###' and '-### -E' differs in a single param: '-mrelax-all'.

Issue-10: Implement the remote cache store. With unit-tests. Also, this commit may fix Issue-13.

Create object files with correct user id - on local side.

Allow daemon to run as another user.

Pass maximum cache size via daemon config.

Major refactoring within third-party libraries.

Refactor client code - to be able to test it. Also, implement simple test for |ClangFlagSet|.

Implement simple logging.

Add the cache size limit field to configuration.

Separate flags, that shouldn't affect cacheability.

Respect configuration about visible logging levels.

Implement some Statistic collection stubs. Also do a gyp-files CC.

Revert "Implement TCP Fast Open protocol usage." This reverts commit 35d0e9b44a40a48df28c9e1d311234233c8f88ed. Silly me, TCP Fast Open doesn't work with multiple connections.

Make |EpollEventLoop| behave like |KqueueEventLoop|. Also bring back connection tests.

Now local execution thread pool handles can all tasks. But the model is still not perfect.

Implement TCP Fast Open protocol usage.

Add 'disabled' option to remotes in configuration. It allows to disable remotes temporarily.

Refactor |Connection| sending functions.

Add protobuf library to third-party.

Fix potencial leak in case |DeferCompilation()| fails to push task.

Add separate field for language flag "-x". Not used yet.

Make field |Compiler.version| obligatory.

Add more unit-tests. Fix problems found by those tests.

Improve handling of clang flags.

Properly handle clang plugins.

Move execution logics to commands.

Implement the caching of compilation results.

One more refactoring. Also, fix some problems within the class |Process|.

Refactor protocol one more time. Also move connection tests to using a test message.

Present messages as extensions to top-level message. It allows to make |Connection::Send*()| interface polymorphic - to compose and send custom messages.

Local compilation starts to work.

A lot of changes. Among them are: - removal of 'futex' part; - removal of lock-free queue; - reimplementation of thread pool - a simpler variant; - sockets are created non-block and close-on-exec; - some protocol adjustments.

Major refactoring of existing code. Add some unit-tests for the class |Connection|.

Daemon listens and accepts local messages. But does nothing special with them - only prints.

Initial implementation on C++. Lock-free thread-pool added. Gyp-based build system added. Python implementation removed.