Proto commits in PolymerLabs/arcs

These 75 commits are when the Protocol Buffers files have changed:

2021-05-27

Updating encoders to support converting Recipes to manifest protos. - updated tests - added 'query' direction types to proto. PiperOrigin-RevId: 376070231

The documentation is generated from this commit.

`mediapipe_arcs` converts HandleProtos from mediapipe graphs. PiperOrigin-RevId: 376055453

The documentation is generated from this commit.

2021-03-03

Rename storage/Reference to RawReference and provide some comments describing the difference between RawReference and entity/Reference. PiperOrigin-RevId: 360596882

add writeOnly boolean to StoreOptions, to signal to storage service that it can instantiate a write-only optimized stack PiperOrigin-RevId: 360583783

2021-02-25

Nullable proto parcelable (#6938) Implements the core of Nullable support in Kotlin (conversion to/from Referencable, to/from Proto and Kotlin, Parcelable and Proto representations of Nullable Types). Also turns on Nullable Support TypeScript flag (needed to ensure parsing and code generation of Nullable is available for Kotlin testing). This should not have an impact on Kotlin runtime behaviour and so is deemed safe. Closes #6938 PiperOrigin-RevId: 359628137

2021-02-12

Add support for FieldType Annotations in DefaultSchemaSerializer PiperOrigin-RevId: 357249282

Remove CrdtOperation interface and rename CrdtOperationAtTime to CrdtOperation PiperOrigin-RevId: 357097811

2021-02-03

[low precision] Duration support for Kotlin (#6881) Added both Shane and Cameron with the intent of dividing the review workload Shane for review of Typescript changes Cameron for review of Kotlin changes Note: @shans I think this is all the different places that currently need to be updated to add a type. I'm planning to come back to this to try to reduce the workload (per type added). Closes #6881 PiperOrigin-RevId: 355328215

2020-11-23

Propagate the cause message when serializing CrdtException to proto. Before this change, only the CrdtException message and stack trace were printed. After this change also the cause message is printed which is normally the underlying problem we want to address. PiperOrigin-RevId: 343936408

2020-10-27

Devtools proto (#6453) - Add blanket DevToolsMessage proto - Only send across DevToolsMessageProtos - Refactor creation of protos to simplify cc: @piotrswigon @jasonwyatt Closes #6453 COPYBARA_INTEGRATE_REVIEW=https://github.com/polymerlabs/arcs/pull/6453 from SHeimlich:devtools-proto d7161b1736ab9c084f06ba751c5e1cb15fbd9376 PiperOrigin-RevId: 339168407

Init DevTools Proto (#6450) - Create DevTools Init Proto - Send a DevTools Init Proto when a websocket is connected. cc: @piotrswigon @jasonwyatt Closes #6450 COPYBARA_INTEGRATE_REVIEW=https://github.com/polymerlabs/arcs/pull/6450 from SHeimlich:devtools-protos2 32df92baa3f4c2b6a2429f55d5564ed0cf91f3a4 PiperOrigin-RevId: 339155446

2020-10-26

Add initial ProtoBuf for DevToolsMessages. (#6438) - Add devtools_message.proto file - Create and send message in DevToolsService with the new ProtoBuf. cc: @piotrswigon @jasonwyatt Closes #6438 COPYBARA_INTEGRATE_REVIEW=https://github.com/polymerlabs/arcs/pull/6438 from SHeimlich:devtools-protos 226b0ab0a62e6ca2216389b3b89f636a7850b9b7 PiperOrigin-RevId: 339127009

2020-10-06

Defines StoreOptionsProto and converter functions for StoreOptions. Will be used for the upcoming IMuxedStorageService AIDL interface. Also updated some existing proto conversion tests to use the JUnit4 runner instead of the AndroidJUnit4 runner. These tests did not really need any Android-specific functionality, and swapping the test runners reduces test times from ~25s down to ~4s. PiperOrigin-RevId: 335738979

2020-10-05

Defines an AIDL interface for IMuxedStorageService, and provides the shell for the implementation. This service is for proxying MuxedStorageProxyMessages to/from the storage service. It doesn't follow the old StorageService's format for registering callbacks, etc. and instead shows a new path forward involving "channel" objects, i.e. a way of communicating with a store (or DirectStoreMuxer in this case) without binding to it every time. Will migrate the old service to this new format too if it looks good. PiperOrigin-RevId: 335518145

2020-10-01

Define MuxedProxyMessageProto for serialising MuxedProxyMessages Will be used to transport these messages over AIDL PiperOrigin-RevId: 334804120

add an overload for the collection remove function that just takes the id. Afterwards we should migrate all usages of remove and retire the old method. This changes the crdt operation to just have the id as well. PiperOrigin-RevId: 334742337

2020-09-15

Add Instant primitive type (#5829) * Add support for Instant to Kotlin * Update the parser and ast types to better support numbers and units, also add a function for viewing source code locations * Make entity internals available to refinement expressions without creating cycles * Update code generation for Instant and other number types * Add tests for sql Instant generation and rename some tests with non-standard naming patterns * Update sql refinement generator for Instant * Update goldens for Instant * Check cgtest results one by one so that test failures print the diffs line by line * Add sanity checks on creation & expiration time to test refinement generator * Add platform independent facades for BigInt, Instant and Duration * Supress unused args * Suppress issues with minAPI level * Introduce explicit drop function

2020-09-09

Add an hard reference boolean field to references. This is set by Ent… (#6091) * Add an hard reference boolean field to references. This is set by EntityBase if the schema field has the @hardRef annotation, and persisted in the database in the entity_refs table. It will be later used to bulk delete those entities. * fix db migration * review comments, upgrade test

2020-08-27

Deletes PolicyOptions, it's not needed anymore. (#6012) It has been replaced by the `@ingress` annotation.

2020-08-26

Uses the `@ingress` annotation to identify ingress handles. (#6001) * Uses the `@ingress` annotation to identify ingress handles. No longer needs to use the PolicyOptions object to identify ingress handles (via store IDs). Now the ingress points are computed by looking at the particle specs that are marked as having ingress. PolicyOptions will be deleted in a follow up PR. Also updated PolicyConstraints to talk about schemas instead of store IDs, which meant updating the proto representation too. * Deletes unmarkedProtectedStoresRestrictUsage * `map` handles are ingress, unannotated ingress-like particles are an error * More tests, removed error for unannotated ingress * TypeClaim -> SchemaClaim, PartialClaim -> SelectorClaim * lint

2020-08-10

Encoding / Decoding Max Type Variables via Protos (#5893) * Squashed commit of the following: commit f12f0cb46b0f15bdba263d2430074f6c56720871 Author: Alex Rosengarten <alxr@google.com> Date: Fri Aug 7 15:21:55 2020 -0700 Increase maxType tests commit 94843029ea535e39bac723993d171988ba624f4b Author: Alex Rosengarten <alxr@google.com> Date: Fri Aug 7 15:09:07 2020 -0700 Added `maxType` field to manifest.proto. Impl TS --> proto serialization * Added maxType flag to dataclass * Updated proto --> Kt * Squashed commit of the following: commit 546019a9cac74ba54c15240b8ec87b18c65d7c86 Author: Alex Rosengarten <alxr@google.com> Date: Mon Aug 10 12:07:07 2020 -0700 var { maxType } --> var { constraint { unconstrained } } commit 65160327486fc30cd4a82520fd8b7ec5e9e8fab8 Author: Alex Rosengarten <alxr@google.com> Date: Fri Aug 7 16:49:32 2020 -0700 cleaned up indentation commit f12f0cb46b0f15bdba263d2430074f6c56720871 Author: Alex Rosengarten <alxr@google.com> Date: Fri Aug 7 15:21:55 2020 -0700 Increase maxType tests commit 94843029ea535e39bac723993d171988ba624f4b Author: Alex Rosengarten <alxr@google.com> Date: Fri Aug 7 15:09:07 2020 -0700 Added `maxType` field to manifest.proto. Impl TS --> proto serialization * Updated proto decode to support new representation * rm extra cruft from merge * maxAccess: ts --> proto * maxAccess: proto --> kt * adding require to tv decoding * nit, fixed compile * Impl nits

2020-08-06

Define PolicyOptionsProto with encode/decode functions (#5876) We'll need this to serialise PolicyOptions to disk for compile-time verifications. PolicyOptions is still just a temporary measure, we'll replace it with some annotations later on.

2020-08-04

Deletes policy EgressType enums and allows it to be an arbitrary string (#5856)

2020-07-31

Adds an egress type parameter to the `@egress` particle annotation (#5828) * Adds an egress type parameter to the `@egress` particle annotation * Kotlin egressType decoding * TS boolean getters renamed from foo to isFoo, store annotations in proto, compute getters * Undelete deprecated arcId * fix test

2020-07-30

Allow refinements and queries to use DSL by default (#5696) * Allow refinments and queries to use DSL by default * add doc * lint * rebase * rebased * add type widening/narrowing to operations * rebase and add tests for widening * fix merge issue * fix tests * remove type * buildifier * strictdeps/buildifier * add DSL library to SDK * strictdeps * strictdeps * strictdeps

Plumbs serialized particle expression through data structures (#5827)

2020-07-29

Policy Verification: allow egresses of policy to be explicitly specified (#5807) Adds a map to the PolicyOptions data class that specifies the allowed egresses of a policy. We also remove the PolicyOptions from policy.proto as we may not serialize PolicyOptions in it current form..

2020-07-24

Proto encoding/decoding for Policy and PolicyConstraints classes (#5717) * Proto encoding/decoding for Policy and PolicyConstraints classes * deps * Revert

Add AccessPath.Root.Store to proto file (#5783)

2020-07-22

Proto encoding for Type, FieldType, Schema (#5709) * Rename TypeProtoDecoders.kt to TypeProto.kt, since it is about to get some encoder functions * Encode Type and Schema to proto A lot of proto decoding functionality already exists, but most encoding functionality is missing. We're going to need to be able to encode lots of things to proto in order to serialise PolicyConstraints properly. This is the first step in adding that proto encoding, starting at the bottom of the stack (types) and working upwards. * Missing changes to SchemaProto.kt * Re-renamed files

Policy Translation: Defines a new PolicyOptions class and corresponding proto. (#5769) This is used to customise the policy enforcement algorithm. Customers will define their own config textproto file and use that when invoking the policy algorithm. Currently this just has the storeMap field, but we can add things in future (e.g. whitelist of particle specs). Co-authored-by: Gogul Balakrishnan <bgogul@google.com>

2020-07-20

encode kotlin types to proto and back (#5742)

2020-07-15

Add proto encoding and decoding for ordered lists and inline entities (#5715)

2020-07-10

End-to-end Inline entities test (#5686)

2020-07-09

Support both Number and BigInt refinements (#5480) * Use operator functions instead of statics * Enforce type safety (i.e. avoid autocasting) * Support BigInt refinements * Update to newer chrome driver to allow tests to pass with up-to-date chrome * Expose segments for testing * Update goldens with BigInteger imports * Fix language name in schema2cpp s/kotlin/cpp/ * Add BigInteger imports to schema2kotlin * Add BigInteger support to kotlin geneators * Using golden_kt to test bigint codegen and refinement support * Test support for refinements that containing subexpressions using both bigint and number * Open up SchemaUnionType and SchemaTupleType types * Add some more tests and de-duplicate some code

2020-07-06

Add a CrdtSet.Clear operation. (#5630) * Add a CrdtSet.Clear operation. This allows collection.clear() to execute without needing to read the current contents of the collection, which in turn will allow write-only handles to avoid syncing any data. * Fix import ordering * Review comments * Use constant for unitialised timestamp in unit test

2020-07-01

Plumb ParticleSpec.isolated property through ParticleSpecProto to Kotlin (#5618)

2020-06-30

manifest2proto: Added type to handle connection instance (contained in particle instance within a recipe). (#5613) * Handle connection instance has type * writing handle connection instance to proto * Updating textproto -- kotlin proto deserialization works * lint issues * Impl feedback

2020-06-22

pass handle annotations (instead of capabilities) to KT classes and protos (#5556) * pass handle annotations (instead of capabilities) to KT classes and protos * address comments: rename method, add test * address comments

2020-06-19

Encoding constraints on type variable protos. (#5536) * update jsdoc * Type vars have constraint info: TS --> proto * updated m2p: handles unconstrained type variables * kotlin can decode type variables * lint fixes * fixing test * Fixed type variable data class * Passing proto test * comment clarifying constraint * right level of variable resolution + tests passing * impl suggestsions * minor mistake * random lint issue * cleanup (lint) * minor fix * updating textproto * update kt side * json formatting for readability * copy editing * tidying up * simplification * undo

2020-06-18

use annotations for arcId in recipe and policy protos (#5538) * replace recipe arcId with annotations in protos merge policy and manifest annotations * fix build * address comments

2020-06-15

Implement database support for ordered lists (#5503)

2020-06-11

Support for converting ReferencableLists to/from protos (#5467)

2020-06-02

Define AnnotationRefProto and use it in PolicyProto. (#5415) Annotation definitions themselves aren't encoded in protos, yet. Also the only parameter type supported so far is strings. This is just to unblock work on policy2proto, the other features will be added later.

Policy proto definition (#5390) * Policy proto representation This is a proto representation of the policy specification language. It will be used to serialize policies after being parsed from a manifest file, to be used in Kotlin. * Review comments * More review * Fix tests * Fix more tests

2020-05-26

Adds "implication" DFA checks, of the form `(A => B)` (#5360) e.g. `check input (derives from store 'x' => is redacted)` means the `redacted` tag is required if the input derives from store 'x'. This PR includes support for DFA implications in both the manifest language and in the DFA algorithm, and also in the manifest proto output.

2020-05-18

fix a problem with ReferenceProto, timestamps were being lost during the roundtrip (#5315)

2020-05-15

Updates to proto pathway for Schemas (#5302) * impl hash decode * proto encoding of refinement * Schema normalizeForHash supports tuples * Fix test (queries) + added tuple test * reverting refinementexpression on schema proto * simplified test -- inlined schema * Impl + tested Schema Decoding * lint * impl feedback * ParcelableSchemaFields supports Tuples (iter 0) * Builds; added TODOs * better comments in tests * fixed funky merge * proto text is more compact

2020-05-14

Decode ClaimProto in Kotlin. (#5299)

2020-05-13

Add Checks and Claims to manifest.proto and serialize claims. (#5294) * Create a proto for checks & claims and convert checks to protos. * Remove Literal message type from Predicate.

2020-05-08

Furthering manifest.proto to Kotlin conversion (#5272) * serializing more types * encodes tuple * Proto Type for Count * TypeProto Decoding w/ Tests * minor fix: err msg

Adding query to manifest.proto (#5257)

2020-05-01

Kill RecipeEvelopeProto with Fire (#5237)

Replaces ParcelableCrdtException with a proto (#5230)

2020-04-22

Convert ParcelableProxyMessage to protos (#5187) * Convert ParcelableProxyMessage to protos I've left the ParcelableProxyMessage class around as a thin wrapper for now. It's used in a few AIDL functions, which I'll migrate in a follow-up. * fix * strict deps * Delete crdtType param * Comment

Manifest to proto serializer (#5170)

Replace ParcelableCrdtData and ParcelableCrdtOperation with protos (#5186) * Replace ParcelableCrdtData and ParcelableCrdtOperation with protos This means I can finally delete all the Parcelable classes for all the crdt subtypes * Review * lint

Convert ParcelableCrdtEntity to protos (#5175) * Convert ParcelableCrdtEntity to protos * Review

2020-04-21

Convert ParcelableCrdtSingleton to protos (#5174)

Convert ParcelableCrdtCount to protos (#5173)

2020-04-15

Convert ParcelableCrdtSet classes to protobufs (#5145)

2020-04-10

Adds some missing fields to recipe.proto and reuses HandleMode (#5128)

2020-04-09

Convert all ParcelableReferencable implementations to protobufs (#5115) * Convert all ParcelableReferencable implementations to protobufs * strict deps * Review comments * Revert VersionMapProto.toVersionMap() to fromProto(VersionMapProto)

2020-03-04

XXS CL to add a hash field to the Schema proto (#4752)

2020-02-28

Move recipe.proto to java/arcs/core/data/proto/ (#4789)

2020-02-27

A proto encoding for an example recipe utilizing joins feature (#4786)

Fix a typo in an comment in the recipe.proto file. (#4784)

Makes it possible to decode a proto from the text format (#4768)

2020-02-26

Add a simple serialization of refinement expressions into a proto (#4766)

2020-02-25

[Cleanup] Remove a trailing whitespace in recipe.proto. (#4739)

2020-02-21

Introduces a proto to store a Recipe (#4713)

2020-02-20

Add example and tests for TS -> Proto -> Kotlin path (#4702)

2020-02-10

Add VersionMap proto (#4637) * Add VersionMap proto This proto will be used in the SQLite database to store serialized version maps as BLOBs. I've also deleted the ParcelableVersionMap class and have replaced it with proto serialisation. This has reduced a ton of code. I'm hoping to replace the other handwritten Parcelable classes with protos too, assuming we like this new approach. * Fixes * unused deps * Make nullable and non-nullable versions of readProto * lint fix

2018-11-14

Remove early asylo exploration. (#2242)

This commit does not contain any .proto files.

2018-07-12

secure storage PoC (enclaved via Asylo) (#1504) A small PoC of a secure storage mechanism For more information see enclave/README.md.