Proto commits in Pometry/Raphtory

These 40 commits are when the Protocol Buffers files have changed:

2025-03-31

Decimal support (#1943) * add support for decimal128 # Conflicts: # pometry-storage-private # Conflicts: # pometry-storage-private * minor fixes * add missing file * add support for Utf8View * add some more support for utf8view * fix multiple issues with utf8view * added validation for bigdecimal so it fits in parquet decimal(38,x) * parquet decoder has issues * fixing things * dependency issues * fix dependencies and tests * fix the bigdecimal conversion from Graph to disk Graph * generate properties other than list and dict * fix post rebase * formatting and test * add big decimal to proto and make unification more strict for Decimal * rename DecimalTimeCol * fmt * fix compilation issues after rebase

The documentation is generated from this commit.

2025-01-27

no manual variant management for proto enums (#1934) avoid having to manually maintain the mapping between int enum values and property type

fix protobuf errors (#1933)

2025-01-15

remove Graph and PersistentGraph from Prop (#1902) * add Wrapper for ArrayRef in properties add support for Array in Prop and PropType remove Graph and PersistentGraph from Prop * use Arrow for some Prop * added support for Arrow * tests pass * fix graphql issues * use Arrays for graphql * add comprehensive test for previous proto format * fix compilation issue with wrappers/prop.rs

2024-08-21

Feature/cached graph (#1718) * add support for tracking id creation in the internal apis * node type resolution should not create the type if it is illegal to set * Clean up the InternalAdditionOps and implement it for the CachedGraph * move cache implementation into storage * Use Storage inside Graph and PersistentGraph * Expose caching interface * remove old unused docstring macros * rename protoc generated module to "proto" * move protobuff code to top-level serialise module * implement the caching support using traits and a bit more refactoring * fix rerun path in build script * remove ssh key action from workflows that don't need it * implement cache for python and clean up * nuke the old bincode * fix storage feature * fixes for the proto feature * update the type stubs * enable incremental updates in GraphQL server * fix the proto feature * update submodule * remove the rest of the old bincode serialisation * constructor for DFView is never used * clean up unused dependencies * bump some dependencies * bump more dependencies and fix python test * fix the opentelemetry dependencies * add test for incremental writing for Graph * add test for incremental caching for persistent graph * make sure the caching code actually gets called

2024-08-08

Feature/incremental protobuf (#1706) * Graph temporal properties should not be able to change type (also clean up InternalMaterialize trait and impl) * New proto format * wip serialise impl * Implementation of proto serialise and deserialise that passes the old tests * tweak the mutation apis * clean up the debug prints * fix storage and clean up warnings * mark more doctests as no_run * poc test for appending protobuffs * update rust version * attempt to get the rust_version from toml file * try defining it as an action instead * try again * use the new setup action in all the places * one more place * remove old setup code * fixes from review * simplify the pickling implementation * fix time parsing in python * remove unused lifetime * rust fmt workflow has no outputs * fix all the warnings and merge issues --------- Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>

2024-07-30

Pre multi graph (#1695) * Move all APIs to GraphStorage Make the OS Raphtory support GID * changes post rebase * fix GitRef in non storage file * fix is_immutable * fix various python things * fmt * fix node conflict tests in rust * fixes to Immutability, removing todos * fix warning and deactivate storage * fix serialisation issues * increment BINCODE_VERSION * fix python errors * deactivate storage * enable storage test in python via test_graphdb.py * fix benches * fix notebook.ipynb * attempt to fix Windows permission errors * inline some things and avoid cloning the string when looking up GID as string * avoid string clones on resolve node * drop i64 from GID, refactor NodeRef, remove internal module * put back the iterator loading bars when calling from python * fix more loader tests * update pometry storage * clean up python impls in raphtory-api * fix doc warning * keep the proto ids compact for now as we do not consider this frozen yet * turn tqdm switching into a macro so we don't have the feature switches everywhere * fmt --------- Co-authored-by: Lucas Jeub <lucas.jeub@pometry.com>

2024-07-10

Stable serialise (#1683) * serialise one node graph with proto and prost * add minimal support for properties * add better support for properties and start using ids * can encode props and encode and decode edges * added graph const and temporal properties * add test for edge properties * add test for edge deletions * arrange dependencies * add string interning and add protoc to the build * add protoc to rust doctest * try adding github token to binstall and protoc install * add protoc to benchmarks * attempt to fix the github token issues with protoc * more protoc * Graph and Persistend graph now work with pickle and the format is stable * move protobuf in a separate feature * fix the python workflow * fmt * add more protoc in yml * attempt to fix the disk storage * fix python failing * fix graphql lib import issues * change the proto format so it can be easily extended * try ubuntu 20.04 to avoid 143 error * some changes as per review * refactor StableDecode trait * use assert_graph_equals * fix compile issues

2023-04-24

docbrown cleanup

This commit does not contain any .proto files.

docbrown cleanup

This commit does not contain any .proto files.

2023-01-17

Add SqlEdgeSource and SqlVertexSource (#762) * Add SqlSource and create unit tests for it * Simplify GraphAlterations message structure * Replace doobie by vanilla JDBC * Add support for source and target properties on edge source * Simplify IngestionExecutor implementation * Add support for Type parameter * Simplify SqlSource API * Add support for float, double and boolean types * Add more detailed unit tests for SqlSource * Remove unused commented code * Move telemetry to the IngestionExecutor itself * Bring back GraphBuilder companion object * Add Python wrappers * Add python test for SQLSource * Fix result set releasing missing * Add scaladoc for sql sources and sql connections * Add Postgres example to ingestion notebook * Fix compilation errors * Fix bug caused by late flush * Rerun ingestion notebook after fixing bug * Fix markdown bad syntax in docs * Apply requested changes after review * Fix bug in test_pyraphtory.py * Fix bug omitting the name property when not using a vertex source * Fix SqlSourceTest * Adding a little bit to the tutorial * Making sure it all runs Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk>

Add SqlEdgeSource and SqlVertexSource (#762) * Add SqlSource and create unit tests for it * Simplify GraphAlterations message structure * Replace doobie by vanilla JDBC * Add support for source and target properties on edge source * Simplify IngestionExecutor implementation * Add support for Type parameter * Simplify SqlSource API * Add support for float, double and boolean types * Add more detailed unit tests for SqlSource * Remove unused commented code * Move telemetry to the IngestionExecutor itself * Bring back GraphBuilder companion object * Add Python wrappers * Add python test for SQLSource * Fix result set releasing missing * Add scaladoc for sql sources and sql connections * Add Postgres example to ingestion notebook * Fix compilation errors * Fix bug caused by late flush * Rerun ingestion notebook after fixing bug * Fix markdown bad syntax in docs * Apply requested changes after review * Fix bug in test_pyraphtory.py * Fix bug omitting the name property when not using a vertex source * Fix SqlSourceTest * Adding a little bit to the tutorial * Making sure it all runs Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk>

2023-01-12

impl disruptor based pojo partition (#766) * impl disruptor based pojo partition * ref * add review changes * add review changes * wait for ingestion to complete before querying from client * switch isruptor maps to fastutils * lazy vals * fix querysupervisor race condition between ingestion and query * halt disruptors when graphs are destroyed * increase twitter test timeout * batch edge updates * ref * comments * restruct * ref

impl disruptor based pojo partition (#766) * impl disruptor based pojo partition * ref * add review changes * add review changes * wait for ingestion to complete before querying from client * switch isruptor maps to fastutils * lazy vals * fix querysupervisor race condition between ingestion and query * halt disruptors when graphs are destroyed * increase twitter test timeout * batch edge updates * ref * comments * restruct * ref

2022-12-08

DEV-1310 / Drop removals and sync between writers (#723) * Drop removals and sync between writers * Fix bug caused by Partitioner not reading the conf * Drop unused code * Drop commented out tests from ArrowStorageSuite * Fully drop references to deleteVertex/deleteEdge from doc * Add three different versions for addEdge in GraphPartition * Add comments and increase TwitterTest timeout * Remove commented out code

DEV-1310 / Drop removals and sync between writers (#723) * Drop removals and sync between writers * Fix bug caused by Partitioner not reading the conf * Drop unused code * Drop commented out tests from ArrowStorageSuite * Fully drop references to deleteVertex/deleteEdge from doc * Add three different versions for addEdge in GraphPartition * Add comments and increase TwitterTest timeout * Remove commented out code

2022-12-02

DEV-1236 / Akka clean-up (#717) * Remove Watermarker * Remove communication classes * Translate query update messages into gRPC * Remove unuseful traits * Reorder classes under Messages.scala

DEV-1236 / Akka clean-up (#717) * Remove Watermarker * Remove communication classes * Translate query update messages into gRPC * Remove unuseful traits * Reorder classes under Messages.scala

2022-12-01

Refactor (#709) * change earliest and latest time to cats ref * refactor submit query * rid blocking sources in query and querysender * ren block unblock ingestion to start and end ingestion * rid dead references in comments * rid dead query executory and handler * rid topic repository * all tests passed * moved it to its own module * fix dynamic class loader tests * fix connectors tests * intro testkit * rid connectors build.sbt * rid build.sbt from it module * ref * comments * moved all integration tests to it

Refactor (#709) * change earliest and latest time to cats ref * refactor submit query * rid blocking sources in query and querysender * ren block unblock ingestion to start and end ingestion * rid dead references in comments * rid dead query executory and handler * rid topic repository * all tests passed * moved it to its own module * fix dynamic class loader tests * fix connectors tests * intro testkit * rid connectors build.sbt * rid build.sbt from it module * ref * comments * moved all integration tests to it

2022-11-28

DEV-1302 / QueryExecutor (#686) * ref * ren ingestionserviceinstance * add builder to lotr * ref servicedescriptoir * ren servicerepository to registry * impl query service * rid rechecks in queryhandler * rid clustercomms topic * rid watermarker topic * rid submissions topic * rid messages and topics * retry service fetch * ingestion executor calling queryservice to block unblock * catsfied * review changes * fix blocking/unblocking ingestion\= * getting query sink now returns exception if not found * rid dead code * add timeout for python test * impl earliest time seen in graphbuilders and querysender * Start working on the QueryHandler * fix multiple unblocking ingestion calls in ingestion executor * Start working on the QueryHandler * Start implementation of query executor * impl raphtoryservice unit tests * impl unit tests for QueryProgressTracker * impl unit test for QueryProgressTrackerWithIterator * impl unit test for QueryProgressTrackerWithIterator * impl query supervisors inst var as cats effect refs * ref * Add QueryHandler and QueryExecutor new implementation * change getgraph to graphid * queryservice submitquery returns stream * impl ingestion executor as io * ref * rid isblocking * Minor changes and fixes * fix compilation errors * fix tests * Drop topic repository from code base * ref * rid unblocking ingestion from query sender * Implement table tracker and table function handling * Fix table output bug * Fix bug with graph state * Fix bug caused by shared graph states between executors * Fix bug caused by the row pool * Add support for max iterations * Start processing vertex messages in chunks * remove logs * ordering load query requests * Fix definition of several unit tests * Minor changes * AdjPlus now works * Queries is still broken * impl unit test for query supervisor * Refactor the processPerspective function * Fixing random issues with perspectives * all tests passed * Gave sbt 4GB of memory * Added G1GC * Moved set too high up * Ignoring twitter test Co-authored-by: Shivam Kapoor <mailtoshivamk@gmail.com> Co-authored-by: Haaroon Yousaf <Haaroon@users.noreply.github.com> Co-authored-by: Shivam Kapoor <4599890+iamsmkr@users.noreply.github.com> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>

DEV-1302 / QueryExecutor (#686) * ref * ren ingestionserviceinstance * add builder to lotr * ref servicedescriptoir * ren servicerepository to registry * impl query service * rid rechecks in queryhandler * rid clustercomms topic * rid watermarker topic * rid submissions topic * rid messages and topics * retry service fetch * ingestion executor calling queryservice to block unblock * catsfied * review changes * fix blocking/unblocking ingestion\= * getting query sink now returns exception if not found * rid dead code * add timeout for python test * impl earliest time seen in graphbuilders and querysender * Start working on the QueryHandler * fix multiple unblocking ingestion calls in ingestion executor * Start working on the QueryHandler * Start implementation of query executor * impl raphtoryservice unit tests * impl unit tests for QueryProgressTracker * impl unit test for QueryProgressTrackerWithIterator * impl unit test for QueryProgressTrackerWithIterator * impl query supervisors inst var as cats effect refs * ref * Add QueryHandler and QueryExecutor new implementation * change getgraph to graphid * queryservice submitquery returns stream * impl ingestion executor as io * ref * rid isblocking * Minor changes and fixes * fix compilation errors * fix tests * Drop topic repository from code base * ref * rid unblocking ingestion from query sender * Implement table tracker and table function handling * Fix table output bug * Fix bug with graph state * Fix bug caused by shared graph states between executors * Fix bug caused by the row pool * Add support for max iterations * Start processing vertex messages in chunks * remove logs * ordering load query requests * Fix definition of several unit tests * Minor changes * AdjPlus now works * Queries is still broken * impl unit test for query supervisor * Refactor the processPerspective function * Fixing random issues with perspectives * all tests passed * Gave sbt 4GB of memory * Added G1GC * Moved set too high up * Ignoring twitter test Co-authored-by: Shivam Kapoor <mailtoshivamk@gmail.com> Co-authored-by: Haaroon Yousaf <Haaroon@users.noreply.github.com> Co-authored-by: Shivam Kapoor <4599890+iamsmkr@users.noreply.github.com> Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com> Co-authored-by: Ben Steer <b.a.steer@qmul.ac.uk>

2022-11-16

Dev 1269 fs2 ingestion (#673) * wip * attempt at making the ingestion parallel and based on fs2.Stream * Fix compilation errors and issues with the order * Twitter ingestion is down to 40s * made Writer take a batch of updates * Move gRPC interface from writers to partition services * Fix first chunk missing * Fix error caused by unordered chunks and minor refactoring * Fix bug caused by unordered ingestion on multi-spout graphs * Minor fixes Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com>

Dev 1269 fs2 ingestion (#673) * wip * attempt at making the ingestion parallel and based on fs2.Stream * Fix compilation errors and issues with the order * Twitter ingestion is down to 40s * made Writer take a batch of updates * Move gRPC interface from writers to partition services * Fix first chunk missing * Fix error caused by unordered chunks and minor refactoring * Fix bug caused by unordered ingestion on multi-spout graphs * Minor fixes Co-authored-by: Fabian Murariu <murariu.fabian@gmail.com>

2022-10-28

Dev 1265/ Orchestrator service super class (#646) * Add common super class for orchestrator services * Add id management to instances in ServiceRepository * Prepare OrchestratorService for partitions * Add comment to BaseRaphtoryAlgoTest

Dev 1265/ Orchestrator service super class (#646) * Add common super class for orchestrator services * Add id management to instances in ServiceRepository * Prepare OrchestratorService for partitions * Add comment to BaseRaphtoryAlgoTest

2022-10-26

Dev1255 getgraphimpl (#637) * impl getgraph * intro raphtoryservice get graph tests * docs readme * fix test

Dev1255 getgraphimpl (#637) * impl getgraph * intro raphtoryservice get graph tests * docs readme * fix test

Newgraphapis (#630) * config ref * Raphtory.getDefaultConfig should be used in projects and tests and not internally * build config via config builder fluent api * merged all contexts into raphtorycontext * runwithgraph impl * ref * graph exceptions * impl raphtoryapp * ref * fix tests * fix tests * fix tests * ref * fix tests * fix tests * rid close * intro local/remote traits * intro defaultconfig * ref * impl pyraphtory context * ref tests * ref tests * ref tests * ref tests * dep incr * fix tests * fix py test Co-authored-by: Shivam Kapoor <shivamkapoor@POM-MBP7.local> Co-authored-by: Shivam Kapoor <shivamkapoor@pom-mbp7.home>

Newgraphapis (#630) * config ref * Raphtory.getDefaultConfig should be used in projects and tests and not internally * build config via config builder fluent api * merged all contexts into raphtorycontext * runwithgraph impl * ref * graph exceptions * impl raphtoryapp * ref * fix tests * fix tests * fix tests * ref * fix tests * fix tests * rid close * intro local/remote traits * intro defaultconfig * ref * impl pyraphtory context * ref tests * ref tests * ref tests * ref tests * dep incr * fix tests * fix py test Co-authored-by: Shivam Kapoor <shivamkapoor@POM-MBP7.local> Co-authored-by: Shivam Kapoor <shivamkapoor@pom-mbp7.home>

2022-10-25

DEV-1214 and DEV-1234 / Ingestion service and healthcheck (#625) * Add functional interface for the ingestion service * Add ServiceRepository and fix cluster mode * Add gRPC healthcheck to RaphtoryService * Clean TODO comments

DEV-1214 and DEV-1234 / Ingestion service and healthcheck (#625) * Add functional interface for the ingestion service * Add ServiceRepository and fix cluster mode * Add gRPC healthcheck to RaphtoryService * Clean TODO comments

2022-10-07

DEV 1213 / RaphtoryService (#624) * Add RaphtoryService definition * Start using raphtory service trait locally * Unify remote and local deployments under the same interface

DEV 1213 / RaphtoryService (#624) * Add RaphtoryService definition * Start using raphtory service trait locally * Unify remote and local deployments under the same interface

2022-09-22

Removing client zookeeper dependency (#595) * pausing to fix another issue * Zookeeper is dead * Fixed pedros suggestions

Removing client zookeeper dependency (#595) * pausing to fix another issue * Zookeeper is dead * Fixed pedros suggestions

2022-09-16

Move protocol definition to core project (#583)

Move protocol definition to core project (#583)

2022-09-13

Dev-1142 / gRPC Middleware (#576) * Add protobuf RpcServer and RpcClient * Removed scooba as it was conflicting with mu, will update later once it has been fixed by them * Ported akka fixes from Pedro's other branch * Remove deploymentID parameter from Raphtory.connect * Change sbt-mu-srcgen plugin version Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk>

Dev-1142 / gRPC Middleware (#576) * Add protobuf RpcServer and RpcClient * Removed scooba as it was conflicting with mu, will update later once it has been fixed by them * Ported akka fixes from Pedro's other branch * Remove deploymentID parameter from Raphtory.connect * Change sbt-mu-srcgen plugin version Co-authored-by: miratepuffin <b.a.steer@qmul.ac.uk>