Proto commits in sbt/zinc

These 41 commits are when the Protocol Buffers files have changed:

2024-09-06

Remove the Protocol Buffer usage **Problem/Solution** We currently have three (3) implementations of Analysis persistence. Given that the "consistent" binary is the best, we should remove the protobuf one, so we don't need to update three code bases as we refactor for 2.x.

This commit does not contain any .proto files.

2023-12-27

Handle types touched during Macro macro expansion We still don't have a good way to get touched types, in the future this can be provided by compiler via attachment

The documentation is generated from this commit.

2023-05-15

Upgrade to util 1.9.0-RC2-2

2022-10-03

Util 1.7.2

2020-06-22

Shade protobuf runtime

2020-06-21

Switch to Java-based sbt-protobuf sbt-protobuf brings in significantly fewer transitive dependencies.

2020-05-26

Add provenance to data

2020-02-13

Add VirtualFile / VirtualFileRef / FileConverter This implements Zinc support for passing source code as a VirtualFile datatype. This allows build tools to pass source files as in-memory datatype. Another motivation for doing so is to remove machine dependence from the internal state of incremental compilation (Analysis). By making Analysis free of absolute paths, we should be able to cache the file and resume compilation from another machine. Notes: Anlyzer needs to reverse guess the source file from a `*.class` file, which is currently done by comparing the package names against the directory names of source. This supports the behavior by adding `names()` method to VirtualFileRef. scalac would report `/private/var/folders/hg/.../classes/S.class`, but javac thinks it's at `/private/var/folders/hg/..../classes/S.class`. ``` [info] [debug] > classToSouce.get(/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/S.class) = None [info] [debug] > classToSource = List((/private/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/S.class,S), (/private/var/folders/hg/2602nfrs2958vnshglyl3srw0000gn/T/sbt_c2a71c77/target/scala-2.12/classes/JJ.class,JJ)) ```

2019-10-01

sbt.internal.inc.Problem: add "rendered" field This is a backport of d9ac92123930bae056ab118e594d47d2a28c394b which was itself a forward-port of ebe5c2adfab92cbad086f173b19acbb8682de345

2019-09-22

sbt.internal.inc.Problem: add "rendered" field This is a forward-port from 1.2.x to 1.3.x of ebe5c2adfab92cbad086f173b19acbb8682de345

2018-09-20

Detect changes in private trait definitions This pull request builds upon @Duhemm's work on https://github.com/sbt/sbt/pull/2441, but changes slightly the way it's solved. It instead adds a specific api change that only invalidates changes via inheritance (both global and local) and doesn't invalidate member references, which gives us more correctness and precision. As explained in https://github.com/sbt/sbt/issues/2490, the previous fix violated some of the invariants of the incremental algorithm. As such, we introduce the notion of an extra hash which is independent of the normal one. This extra hash, in the case of traits, is computed from the private definitions found, and otherwise its value defaults to the public api hash (which represents the emptiness). As a result, we can tell apart the cases where there are changes in the normal public API hash and the extra one. I've called it extra instead of `trait private hash` or something similar because it can accommodate other fixes in the future and we cannot afford adding a new hash per `AnalyzedClass` every time we need to have `HashAPI` be special cased for a particular encoding. Right now, when the extra hash is different and the classes involved in the hash computations are traits we treat it as a breakage in the trait encoding and recompile appropriately. If the classes involved are not hash computations, we fail because no more cases have been added yet. Fixes: 1. `source-dependencies/trait-private-val` 1. `source-dependencies/trait-private-var` 1. `source-dependencies/trait-private-object` 1. `source-dependencies/trait-super` And it also adds another two tests to complement `source-dependencies/trait-member-modified` in checking that member references to private trait definitions do not trigger unnecessary compilations and that local inheritance does trigger an extra compile. Note that the code in `SameAPI` is rotten and will need to go at some point in the future. It's outdated and the only one-liners used in the incremental compiler should be inlined, and the rest of the code ditched. Right now, it tries to emulate the semantics of `HashAPI` but it hasn't been up-to-date ever since we released Zinc 1.0.

2018-08-30

sbt.internal.inc.Problem: add "rendered" field And bump util to 1.2.2 to include https://github.com/sbt/util/pull/175

2018-08-27

Remove secondary string table I realized how stupid it was the idea of keeping two string tables intead of one, since it's unlikely we'll ever have more than 2147483647 unique string instances and the JVM doesn't allow to instantiate such a big array in normal heap sizes. So we remove this part of the design that we inherited from pprof.

2018-08-26

Add profiling invalidation utils (zprof) zprof is the name I've chosen for this small profiler (or tracker if you will) of the invalidation logic. The profiled data is formalized in an internal format that is not supposed to be used by normal users, but rather by us (Zinc) and related tools (Bloop). The current profiled data exposes details of how the incremental compiler works internally and how it invalidates classes. This is the realization of an idea I registered here: https://github.com/sbt/zinc/pull/550 With this idea, this data will not only be useful for debugging but for providing an automatic way of reporting bugs in Zinc. The infrastructure is far from finished but it's already in a usable state for libraries that depend on Zinc directly and have direct access to `Incremental`. By default, no profiler is used. Only people that change the profiler argument for `Incremental.compile` will be able to get the run profiles.

2018-08-22

[Backport] sbt.internal.inc.Position: record range information This is a backport of #571 to the 1.2.x branch.

2018-08-18

sbt.internal.inc.Position: record range information Follow up to https://github.com/sbt/util/pull/173

2018-08-10

Detect changes in private trait definitions This pull request builds upon @Duhemm's work on https://github.com/sbt/sbt/pull/2441, but changes slightly the way it's solved. It instead adds a specific api change that only invalidates changes via inheritance (both global and local) and doesn't invalidate member references, which gives us more correctness and precision. As explained in https://github.com/sbt/sbt/issues/2490, the previous fix violated some of the invariants of the incremental algorithm. As such, we introduce the notion of an extra hash which is independent of the normal one. This extra hash, in the case of traits, is computed from the private definitions found, and otherwise its value defaults to the public api hash (which represents the emptiness). As a result, we can tell apart the cases where there are changes in the normal public API hash and the extra one. I've called it extra instead of `trait private hash` or something similar because it can accommodate other fixes in the future and we cannot afford adding a new hash per `AnalyzedClass` every time we need to have `HashAPI` be special cased for a particular encoding. Right now, when the extra hash is different and the classes involved in the hash computations are traits we treat it as a breakage in the trait encoding and recompile appropriately. If the classes involved are not hash computations, we fail because no more cases have been added yet. Fixes: 1. `source-dependencies/trait-private-val` 1. `source-dependencies/trait-private-var` 1. `source-dependencies/trait-private-object` 1. `source-dependencies/trait-super` And it also adds another two tests to complement `source-dependencies/trait-member-modified` in checking that member references to private trait definitions do not trigger unnecessary compilations and that local inheritance does trigger an extra compile. Note that the code in `SameAPI` is rotten and will need to go at some point in the future. It's outdated and the only one-liners used in the incremental compiler should be inlined, and the rest of the code ditched. Right now, it tries to emulate the semantics of `HashAPI` but it hasn't been up-to-date ever since we released Zinc 1.0.

2017-07-14

Address Stu's feedback

2017-07-13

Add version to binary format Versioning the binary format is important if in the future we want to change protobuf-based readers and writers on the fly in a transparent way, without users noticing. We start versioning with `V1`, honouring the fact that new Zinc goes public as 1.0.0 soon and it's the first iteration of the binary format. Note that we reuse the same `schema.Version` and we don't convert from it back and forth since the version is not supposed to be public to the users. It's an implementation detail that users shouldn't rely on.

Add writer for Analysis and support the missing API This commit adds support to write the analysis instance with Protobuf. The API was not properly covered before, missing some of the most important protos: `Relations` et al. The current implementation of the `Relations` writer does not sort the results as the previous text analysis format did because order is not required for efficient serialization/deserialization.

Add `ClassDefinition` protos, readers and writers The following commit adds all `ClassDefinition`, method parameters, parameter lists and parameter modifiers protos that were missing to actually read and write `ClassDefinition`s. It also fixes an oversight with the cached classes in `ProtobufDefaults`: instances of classes were not the appropriate ones.

Add readers for `ClassDefinition` and `TypeParam` The reason why we add the new `Modifiers` constructor is to have an efficient way to instantiate `Modifiers` without passing bit per bit which flags are set. The new constructor is `protected` and hence the user of it in `converters` (`InternalApiProxy`) is placed in the `xsbti.api` package. Comments and docs have been added to ensure that no downstream user depends on it.

Don't write APIs if `apiStore` is false

Enable protobuf binary format and add tests This commit finally makes the binary Protobuf format accessible via `binary` in `FileBasedStore`. The commitadds the readers required to read the analysis file, then adds a new proto to serialize/deserialize everything at once without keeping track of the size, and finally it creates the test infrastructure. Note that this commit uses `???` as the stub for better error messages. I'm planning to make severe improvements in the logging facilities, so that overhaul will take care of filling those Nothing in.

Fix `Structure` and add protobuf writers The definition of `Structure` missed `repeated` for every argument. The following commit also adds an object with all the already computed defaults so that it's clearer that we're using the same default values all the time from the converters.

Add writers for `Type` et al This commit adds most of the writers for `Type`, though some writers are still pending and will be added in future commits. This also includes a refactoring of `Super` to use a recursive reference to `Path`.

Add `EmptyType` to `Type` protobuf `EmptyType` cannot be replaced by the default `Empty` type generated by Protobuf because we need to differentiate a missing value from an actual empty type used to fill in the whole type API. This PR adds the empty type support in both readers and writers, as well as puts the defaults for writers and readers in concrete objects that can be scoped and imported selectively.

Use sint32 for `Position` `Position` uses negative values to represent the lack of an option (like `line`, `offset` and `pointer` that are optional and whose default protobuf value is 0). Since 0 is a valid option for most of those options, we use `-1` to represent `None`, which is a sensible choice given the fact that `-1` is not in the value domain of those fields. As we use negative values and `int32` is not good to represent them, we lean towards the use of `sint32` instead.

Add reader and writer for `Stamps` + change the original protobuf definition to be more idiomatic.

Remove `ParameterRef` proto

Finish up `ClassDefinition` proto

Add protobuf Type support

Add protobuf support for `MiniSetup` The whole `MiniSetup` can now be read/written by Protobuf. The feedback error messages have changed the design to be more Scala idiomatic and distinguish between those that show up when writing and those that do when reading.

Use sint32 for hash fields Hash fields are generated through `hashCode` that produces both positive and negative numbers. Negative numbers are more efficiently encoded with sint32 rather than int32. See language spec: https://developers.google.com/protocol-buffers/docs/proto3.

Add `AnalyzedClass` protobuf

Add `ClassLike` proto

Add support for Problem, Position and SourceInfo Add partial support for these protobufs + the converters for Position.

Add support for Compilations protobuf The following changes allow Zinc to serialize and deserialize Compilations using protobuf. It adds: * Protobuf schema. * Protobuf writers class to keep all the converters to write. * Protobuf readers class to keep all the converters to read.

Add first iteration of protobuf stamps writer This commit adds the basic infrastructure that proves that we can successfully write stamps information without removing the previous serialization facilities, that will try to stay as long as we can.

Add `Path` and `AnnotationArgument` protos This is an initial design of the `Path` Java API.

Add protobuf configuration and Stamp protos