Proto commits in JasonSteving99/claro-lang

These 36 commits are when the Protocol Buffers files have changed:

2023-12-27

Drop concrete class codegen (#41) Unfortunately, there is not currently time to fully commit going down this road of migrating Claro's nested types to be codegen'd to use concrete classes that fully specify their runtime types. As I'm reaching the end of my year of full time work on Claro this is just a more pervasive change than I actually have time to work through for now. At the moment it's a much higher priority to truly work through documenting the current state of Claro than trying to work through this more idealized approach to more efficient runtime characteristics. # Oversight in Original Planning I can't just leave things partially migrated to using this approach, as this is something of an all or nothing change. In particular, Claro's type system is designed to allow the following: ``` var nestedOneof: tuple<oneof<int, string>, int> = (10, 10); ``` Currently, Claro's unable to correctly codegen the rhs initialization as tuple<oneof<int, string>, int> and will instead codegen as tuple<int, int> because the LHS asserted type is not being used to override the implied type of the RHS for the sake of codegen. This is certainly something that can be worked around, but in my estimation would (in combination with the remaining migration of struct, list, map and user-defined-type) take so much time that I would lose my opportunity to put sufficient effort into documentation that this project needs more than anything at this point in time. # Maintaining history I'm intentionally reverting these commits in order to maintain the history so that I can potentially revisit these changes to make redoing this at some point in the future easier. ------------------------------------------------- * Revert "Complete Migration of All Structs to Use Generated Monomorphic Java Class" This reverts commit e74fd2aa237024847f9bb26f29d9f3332edc8989. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit eb488e14e30651d2e689335577f4ceb7b2b960a5. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit cffc5f56dff03e909179e999df02e6059655a957. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit 39faa6c788bb5cb97a640c192873998c43512c22. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit 4f7ec92de099b54ba01767c75fc7e15bb558e7c5. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit d65970ffd28e03a3fcfc540429fa01e903395d45. * Revert "Codegen Concrete Classes for Every Concrete Struct Type" This reverts commit 54f09ddd82bc5645c2006978954e48628276ba81.

The documentation is generated from this commit.

2023-12-14

Codegen Concrete Classes for Every Concrete Struct Type Claro's builtin struct type follows this idea of being "polymorphic" in the sense that `struct {foo: int, bar: string}` can just be referenced inline as if it just exists already. Before this change, under the hood all structs were represented at runtime with the single $ClaroStruct Java type where there was more or less an array of Object such that field accesses were codegen'd into something like ((ExpectedType) s.values[i]) which is quite ridiculous. It additionally created additional runtime overhead (casting) on every single struct field access when this is obviously not necessary. This new approach avoids all of that overhead and should have the added benefit of making it much less likely to run into any compiler bugs where Java codegen is missing necessary casts. It'll take an update to the bootstrapping compiler for dep module monomorphization to operate properly with this new approach so this is a breaking change.

2023-12-11

Add Support for Missing Primitive Types: `long`, `double`, `char` These are all just oversights that should've always existed but I moved past during the very early stages of Claro development. These should all be properly supported now including basic mathematical operator support and auto-promotion (not on function calls which could arguably be a nice feature add in keeping with the Java universe way of things). Still need to come back and change the return type of exponentiation and log to return `double` instead of `float`. I also need to change the Java representation for the Claro `float` type from `java.lang.Double` to `java.lang.Float`. Left it as is for now so that I can start by getting the bootstrapping compiler to acknowledge the new types.

The documentation is generated from this commit.

2023-11-17

Update Claro's Dep Module Monomorphization Codegen Strategy Setup SerializedClaroModule.proto to include new field to include all monomorphizations codegend in the entire dep subgraph rooted at the current claro_module(). The following CL(s) will further leverage (abuse) this knowledge to actually start including the set of monomorphizations produced by the dependency sub-graph beneath each .claro_module so that any of its deps can actually completely skip codegen on any monomorphizations that any of it's direct or transitive dependencies have already generated.

2023-11-16

Update Claro's Dep Module Monomorphization Codegen Strategy Setup SerializedClaroModule.proto to include new field to include all monomorphizations codegend in the entire dep subgraph rooted at the current claro_module(). The following CL(s) will further leverage (abuse) this knowledge to actually start including the set of monomorphizations produced by the dependency sub-graph beneath each .claro_module so that any of its deps can actually completely skip codegen on any monomorphizations that any of it's direct or transitive dependencies have already generated.

2023-11-03

Ready Claro Project for Adding to BCR This is a large CL but it mostly moves the example claro programs from //src/java/com/claro/claro_programs/... to //examples/claro_programs and updates build references as appropriate. I've then added a test Bazel module at //examples/bzlmod/... that can be later included in the presubmit.yml that I'll upload with claro-lang's BCR pull request. Lastly I've made significant updates to the README so that potential users will be able to learn the current steps required to work with Claro and create their own project. Lastly, update the Bzlmod version to 0.1.316. This should ideally be the first version pushed to the BCR if everything goes to plan.

2023-10-19

Add a New Synthetic JavaType Proto This will be used by the internal, blessed stdlib Modules that provide access to Java ecosystem types via Claro user-code. This will only ever be used in the definition of internal implementation of exported opaque types, so all interaction with these synthetic Java types will be handled by the blessed internal module implementation.

Add a New Synthetic JavaType Proto This will be used by the internal, blessed stdlib Modules that provide access to Java ecosystem types via Claro user-code. This will only ever be used in the definition of internal implementation of exported opaque types, so all interaction with these synthetic Java types will be handled by the blessed internal module implementation.

Add a New Synthetic JavaType Proto This will be used by the internal, blessed stdlib Modules that provide access to Java ecosystem types via Claro user-code. This will only ever be used in the definition of internal implementation of exported opaque types, so all interaction with these synthetic Java types will be handled by the blessed internal module implementation.

2023-10-18

Add Initial Support for Modules Exporting Opaque Type Defs Pivot the approach for supporting opaque types. Instead of the SerializedClaroModule.proto containing a field dedicated to these opaque types, instead, I'll just introduce a new synthetic Type `SyntheticOpaqueTypeWrappedValueType` that'll be used as the declared wrapped type of all opaque types. This wrapped type will hold the mutability declaration of the actual implementation's wrapped type. Then, I can enforce the restrictions on opaque type usage by checking if the wrapped type is this opaque type sentinel. This synthetic opaque type will also track the actual wrapped type for the sake of Java codegen being able to use correct Java type annotations (i.e. `$UserDefinedType<%s>`)

Add Initial Support for Modules Exporting Opaque Type Defs Pivot the approach for supporting opaque types. Instead of the SerializedClaroModule.proto containing a field dedicated to these opaque types, instead, I'll just introduce a new synthetic Type `SyntheticOpaqueTypeWrappedValueType` that'll be used as the declared wrapped type of all opaque types. This wrapped type will hold the mutability declaration of the actual implementation's wrapped type. Then, I can enforce the restrictions on opaque type usage by checking if the wrapped type is this opaque type sentinel. This synthetic opaque type will also track the actual wrapped type for the sake of Java codegen being able to use correct Java type annotations (i.e. `$UserDefinedType<%s>`)

Add Initial Support for Modules Exporting Opaque Type Defs Pivot the approach for supporting opaque types. Instead of the SerializedClaroModule.proto containing a field dedicated to these opaque types, instead, I'll just introduce a new synthetic Type `SyntheticOpaqueTypeWrappedValueType` that'll be used as the declared wrapped type of all opaque types. This wrapped type will hold the mutability declaration of the actual implementation's wrapped type. Then, I can enforce the restrictions on opaque type usage by checking if the wrapped type is this opaque type sentinel.

2023-10-17

Add Initial Support for Modules Exporting Opaque Type Defs This is going to be a powerful encapsulation feature that allows Modules to hide the definition of a custom type's wrapped value type so that it's only known within the implementation of the module. This will serve to define types that may only be instantiated *AND* used by the module that defined it. Other consumers of the module may receive instances from the defining module, but they will only be able to pass them around or aggregate them in some way, and in order to "do" anything with the values, they'll need to be handed back to the defining module. For now, this CL is just setup. You can now define opaque types in a .claro_module_api file, and Claro will validate that they're correctly defined within the module's impl srcs. However, they can only be used by the defining module. A followup CL will ensure that the defining module actually includes these opaque types in the serialized .claro_module and then imports them during dep module setup.

2023-10-13

Add ExportedFlagDefinitions Field to SerializedClaroModule.proto This is just a setup CL. This field will be populated with any direct exported flag defs along with the transitive closure of all of this module's deps' exported flags. Claro will validate that flag names are globally unique as they need to be in a single global namespace in order for them to be passed on the cli.

Add ExportedFlagDefinitions Field to SerializedClaroModule.proto This is just a setup CL. This field will be populated with any direct exported flag defs along with the transitive closure of all of this module's deps' exported flags. Claro will validate that flag names are globally unique as they need to be in a single global namespace in order for them to be passed on the cli.

Add ExportedFlagDefinitions Field to SerializedClaroModule.proto This is just a setup CL. This field will be populated with any direct exported flag defs along with the transitive closure of all of this module's deps' exported flags. Claro will validate that flag names are globally unique as they need to be in a single global namespace in order for them to be passed on the cli.

2023-10-11

Update SerializedClaroModule.proto to Declare Exported Static Values This is a trivial setup CL so that the bootstrapping compiler build can also have access to this field def.

2023-09-29

Include Required Contract Impl Details in MonomorphizationRequest Proto Some more setup was necessary before the dep module monomorphization subprocesses could make use of the defining module namespacing information for required contract impls. The biggest change here is that the internal Types.$ContractImplementation representation has been updated to include the namespacing information necessary to fully disambiguate where the implementation originated as this information is necessary to codegen the appropriate target. One subtle bug that got fixed that I'll call out explicitly here, is that the IPC proto serialization encoding between monomorphization coordinator -> subprocess was accidentally using an encoding alphabet that contained invalid chars for use in an http url. This was a simple fix once I discovered it, but now it's important that both sides of the encoding/decoding process use the BaseEncoding.base64Url encoding rather than BaseEncoding.base64 to ensure that this IPC scheme continues to behave properly. The next CL should hopefully be able to actually start working on the dep module subprocess itself to register required contract impls.

2023-09-28

Include Required Contract Impl Details in MonomorphizationRequest Proto This is just a simple setup CL that will enable following CLs to start communicating this necessary data that should enable calling generic dep module procedures using contract impls that are unknown to that dep module (for example this would be necessary if the current module implemented the contract, in which case the dep module definitely wouldn't know about it on its own, as that would have required circular deps).

2023-09-27

Serialize Concrete Type Signature of Contract Procedures for Exported Contract Impls This avoids needing to duplicate a massive amount of logic relating to deriving the contract impls' concrete signatures based on the contract impl's concrete type params.

2023-09-26

Modules Can Now Export Contract Implementations A .claro_module_api file may now use the following syntax: ``` implement SomeContract<Foo, Bar>; ``` to export a contract implementation and Claro will validate that the declared implementation is actually implemented within the module's srcs. Included in this CL is the update that a module may implement a locally defined contract over *any* types, whether they were defined in the current module or not. This is safe because every other compilation unit that "knows about" the contract definition itself, will automatically know the contract implementations exported in the same module which avoids any risks of incoherency. This models the other side of Rust's "Orphan Rules" that weren't addressed yet. Again, it's possible that Claro may deviate from Rust's approach in the future, but for now I think it's better to start with a more constrained feature where constraints can potentially be loosened in the future rather than the other way around.

2023-09-20

Dep Module Exported Contracts Are Now Setup Now, the exported contracts from any dep modules should be setup in memory of the compilation unit's depending on them. The following CLs will address being able to actually implement contracts from dep modules and to call their procedures.

Serialized Procedures Now Encode the Defining Module Disambiguator for Required Contracts I'll be migrating away from the old `required_contract_names_to_generic_type_params` fields instead of this in the coming CLs as the new field is utilized. This preliminary work is necessary to separate any naming hackery that the compiler does from the actual .claro_module representation as this should eventually reach a point that it becomes more stable than the compiler implementation itself.

2023-09-15

First Step Towards Supporting Modules Exporting Contract Defs This first step just enables the use of the contract def syntax in .claro_module_api files, allows the contract to be used within the module's impl files, and serializes the exported contract in the resulting .claro_module file. This does not yet provide any support for referencing contracts exported by dep modules, exporting procedures that actually require the contract, or for exporting an impl of the contract. Those will come in following CLs.

2023-09-13

Re-add `transitive_dep_module_monomorphization_requests` to `MonomorphizationResponse` Proto w/ Updated Type This is a migration CL. Unfortunately, working around Claro's multiple compiler versions (latest vs bootstrapping) requires staging changes to even this unused field across multiple CLs. This CL updates the type of the field now that it's free. The next CL will have the dep module subprocess set the field so that the Monomorphization Coordinator can use it in a followup CL.

Temporarily Drop `transitive_dep_module_monomorphization_requests` From `MonomorphizationResponse` Proto This is a migration CL. Unfortunately, working around Claro's multiple compiler versions (latest vs bootstrapping) requires staging changes to even this unused field across multiple CLs. The next CL will update the type of the field once it's free.

2023-09-05

Update the IPC MonomorphizationRequest Proto to Include UserDefinedType Metadata This is necessary since a generic procedure from a dep module may be called with concrete args that were not present in the dep subgraph of the given module. So, in order for the dep module subprocess to successfully monomorphize for these unknown types, their metadata must be passed along.

Add Error Field to IPC MonomorphizationResponse Proto This will be used to catch any Exceptions during handling of a dep module MonomorphizationRequest, so that the subprocess can communicate errors back to the coordinator in a way that makes debugging easier.

2023-09-01

Begin Integrating Dep Module Monomorphization Support Into Compiler This is an extremely hairy CL. I very complicated side-effect of restarting compilation in subprocesses (as well as using Claro's builtin http support) within the compiler's own Dep Module Monomorphization support is that I have somewhat unintentionally entered the world of bootstrapping Claro. So in addition to all of the added complexity of the multiprocessing IPC, I now have to stage changes across releases as local changes won't be available in the "bootstrapping compiler" which is pulled from the latest release on clarolang.com. For now, I'm pushing this initial commit that triggers recompilation of dep modules, and then ends up erroring out when the subprocess compilation unit attempts to overwrite the .claro_module file of the dep-module. This change includes some staged updates to the JavaSourceCompilerBackend so that after this release the Dep Module Monomorphization subprocess can configure it to avoid writing files. All I want the subprocess to do is to recompile, and then keep AST/Symbol-table in memory for handling monomorphization reqs.

2023-08-31

Begin Integrating Dep Module Monomorphization Support Into Compiler This is an extremely hairy CL. I very complicated side-effect of restarting compilation in subprocesses (as well as using Claro's builtin http support) within the compiler's own Dep Module Monomorphization support is that I have somewhat unintentionally entered the world of bootstrapping Claro. So in addition to all of the added complexity of the multiprocessing IPC, I now have to stage changes across releases as local changes won't be available in the "bootstrapping compiler" which is pulled from the latest release on clarolang.com. For now, I'm pushing this initial commit that triggers recompilation of dep modules, and then ends up erroring out when the subprocess compilation unit attempts to overwrite the .claro_module file of the dep-module. This change includes some staged updates to the JavaSourceCompilerBackend so that after this release the Dep Module Monomorphization subprocess can configure it to avoid writing files. All I want the subprocess to do is to recompile, and then keep AST/Symbol-table in memory for handling monomorphization reqs.

2023-08-28

Finish Prototyping IPC Implementation of Dep Module Generic Procedure Monomorphization This last bit added support for dep module monomorphization subprocesses indicating to the coordinator that they actually themselves need a monomorphization from a transitive dep module. I'm now relatively confident that this approach will structurally work when applied to the real compiler. One thing that I'm postponing for now that could actually end up being a great performance optimization would be supporting truly parallel monomorphization across subprocesses. For now, I've very intentionally always used a direct executor everywhere (so that the current thread is used to execute future transforms) and made liberal use of blocking get() calls on all Futures in order to intentionally sequentialize this work. I want to start off with at least *some* aspect of this already complex solution being simplified. I don't want this aspect of Claro's compilation model getting bogged down in completely runaway complexity. If and when I get generic procedure exports working in Claro's module system using this present non-parallel approach, I'll consider moving to a more complicated parallelized implementation. Functionality first.

2023-08-25

Continue Prototyping IPC Implementation of Dep Module Generic Procedure Monomorphization This now adds prototyping handling of triggering monomorphization subprocesses for multiple dep modules, and caching monomorphizations to avoid making unnecessary IPC calls, and finally outputing a trivial example pseudo-codegen to represent what the final result of all of this work will be. The last bit to add support for in this prototype will be the ability to handle dep module monomorphization subprocesses indicating to the coordinator that they actually themselves need a monomorphization from a transitive dep module. If this can be implemented, then I would be relatively confident that this approach will structurally work when applied to the real compiler.

2023-08-04

Minor Update to Procedure Proto Representations to Enable Use of Custom Types w/ Generic Type Params in Module Deps Prior to this there was an internal compiler error if you attempted to initialize a custom type with generic type params if that type came from a dep module. The issue was specifically related to missing information in the proto representation of procedure types where a custom type's constructor type would not be able to correctly reference it's generic type params.

2023-08-02

Remove .claro_module_api File Contents From SerializedClaroModule.proto The migration to depending on the serialized proto representations has been completed, so the contents of the .claro_module_api file itself isn't needed by module consumers. This is intended to avoid AST nodes for definitions exported by modules needing to handle both source code and dep module cases. One nice unintended side effect of this is that I've technically now decoupled the dependency system itself from actual .claro_module_api syntax, so conceivably in the future different Claro versions could use different syntax and then as long as it could still land on the SerializedClaroModule.proto it wouldn't prevent cross-version module deps.

Massive Overhaul of Existing Progress on Claro's Module System. Now, in order to avoid the AST nodes and parsers needing to support multiple use cases in handling both direct sources and deps, I've migrated the module system to make use of Protobuf representations of module apis and written logic in the JavaSourceCompilerBackend to manually configure the symbol table with all dep module exported bindings. This change also went too far in that it also adds various other minor features like the ability to export atoms. Still much more work to be done on the module system but this is a massive reworking of a better foundation to continue making progress atop.

2023-07-12

Progress Towards First-Class Module System Now, when using the claro_library Bazel macro (which needs to be upgraded to a rule) as following to request compilation of a Claro Module from a .claro_module_api file and some .claro source files: ``` claro_library( name = "foo_module", module_api_file = "FooModule.claro_module_api", src = ["FooImpl.claro", ...] ) ``` Claro will compile the given files into a .claro_module which (in the future) will be able to be depended upon by other `claro_library(...)` targets or more importantly by a `claro_binary(...)` target in order to compose a complete executable program from a DAG of modules. For a bit of added detail, this .claro_module file actually just contains a serialized proto message of SerializedClaroModule.proto for convenient serialization/deserialization. This choice to use proto for this purpose will, I hope, turn out to be a good choice in the long run as I can eventually (once backwards compatibility concerns apply) actually utilize Proto's powerful backwards compatibility features.