Proto commits in salesforce/apollo

These 64 commits are when the Protocol Buffers files have changed:

2024-08-13

misc 10 (#217) * uni kerl prophylactic. Concrete reification * refine add assertion api to indicate whether assertion was added or previously existed. expose moar primitive api to allow more sophisticated stored procedure capability required * refine add assertion api to indicate whether assertion was added or previously existed. expose moar primitive api to allow a more sophisticated stored procedure capability * Extend Gorgoneion protocol to provide an establishment provisioning package as part of the cycle * update README.md * update README.md * update README.md * add netty vm socket channel * remove outer context server. * add session encryption key for provisioning * fix pending block, use initial block for beacon if none * remove filter api from Verifier * fix keystate accessor

The documentation is generated from this commit.

2024-07-07

🚫 Ring Communications/Iterator (#215) * revamp FF gossiping. Add failure detection monitoring for future needs. * revamp rbc gossiping * revamp gossiping * revamp Bootstrapper gossiping pt 1. resolve long-standing async issue with GenesisAssembly * revamp sample gossip * revamp view chain synchronize gossip * revamp view LeydenJar gossip. remove RingCommunications and RingIterator finally

2024-07-05

super-majority (#214) * use sampling for msg limits * max * no sampling of accusations * no sampling of observations * better shuffling, etc. * d'oh. observers now concurrent skip list. reject joins if not observer. better logging * enjoin. replicated join to observers in the cluster * interim * fixins. Reimplement ReservoirSampler. 5x5 * fixins. Reimplement ReservoirSampler. 5x5 * handle unknown accusations, initiate view change under lock * serialize via single threaded executor * cleanup * handle error already sent * multiple fixin's ensure valid bft set for reconfigured validators. clean up observations replication - ugh fix bootstrap view change logic fix vote process allow observations to propagate post vote info log accusations correct PID assignment * Max length BLOBs for all binary values. reuse/close scheduled executors. Pretty massive refactoring of the stereotomy and thoth schemas. :: sigh :: * consolidate executors - just fork threads - and schedulers. better concurrency control/management * consolidate executors - just fork threads - and schedulers. better concurrency control/management * firm up join * don't use parallel stream on join. suppress metric reporting, gated with system prop * decouple * better observation attempt lifecycle. use executor for linearizing * moar observation lifecycle fixie. don't "start/end" ViewAssembly * subtle * oops * refactor bft subset. simplify TxDataSource * require 11; moar TxDataSource tweak * D'oh. Fix maxEpochs geebus * use b64 encoded rather than bytes KerlSpace still effed * combine hashes for view id, simplify bft subset.mtls server/client vthread default * lol. roll back MTLS vthreads * consumption

2024-06-02

BFT alignment (#204) * better logging. bootstrap sampling via successor cuts of ring 0 iteration. checkpoint assembly gossip of reconfiguration committee members only. Some general tidying up * better logging * set maven opts * set surefire jvm mem opts * fix logging, back out some exec/sched changes in routing * cleanup * anchor at first checkpoint correctly, filter unknown members * allow providing the scheduler * ignore self in gossip, better logging * no aux populating * ignore * rename * use super majority of observers. some clean up, crisp fpr defaults, etc Fails on churn test @30 members due to knocking out > 1/4 of the observers. * fix BFT logic, plus clean up * double T/O * use bftSlice for gorgoneion/maat successors * use SliceIterator for Ethereal.simplify RingCommunications. * squelch observer logging * clean up, slice iterator traversal now copy on write array, cleaner Bootstrapper

2024-05-19

Networking (#199) * Change endpoints to opaque string encoding * cleanup + 4 forks

2024-04-28

vsync 3 (#194) * v sync 3 * Properly handle committee members who did not supply their view signing key in the CHOAM view reconfiguration. Some clean up on builders, adding missing clone() * try 2 forks again * 4 forks * interim. Save progress before refactoring Join protocol. * interim. Invert join protocol. Future contacts past. Revive ASSEMBLE block. Dear lord. * interim. moar * interim. moar * hallelujah! build runs green View change protocol now stable. * 2 forks * 4 * info logging for CHOAM * no forks * fix test params * trace instead of info * prioritize joins * more efficient joins.

2024-04-18

v sync 2 (#193) * interim * delete * revert dat and clean * Simplify genesis and view reconfiguration remove unnecessary signatures n' such. * Further simplification of View Reassembly. Remove ASSEMBLE block type. Further clean up of view assembly * correct signing key * correct signing key. active at genesis, not (missing) assemble * prepare for view synchronization; moar refactoring around view change. * refactor to ordered list of pending views current context determined by the advance of the pending views the current Context of the CHOAM (params.context) is defined as the head view in the pending views of the CHOAM, thus teh pending views are never empty. reassembly currently uses the last pending view of the CHOAM * View agreement on reassembly. F*ck yea

2024-04-12

v sync (#191) * move scheduler for iterator into final inst var of iterator * create compact from existing context, adapt pred/succ testing to CompactContext * extract Tracked, start read only context work * extract BaseContext * static/dynamic context refactoring phase 1 * static/dynamic context refactoring phase 2 * cleanup * move to context package * move to context package. working memberships testint * copyright * remove Ring access from Context * consolidate. add tests for static context. * Mirror tests for StaticContext passing * pull up * dynamic -> static conversion * remove CompactContext * adapt builder * moar refactoring. implement missed methods on StaticContext. Add new view change notification using StaticContext. Add pendingView to CHOAM in prep for virtual sync * move to inner * fix StaticContext.isMember. Use StaticContext for CHOAM Committee views * moar fixes to StaticContext to match existing contracts. Replace all the CHOAM Context use with StaticContext * clean up PendingView logic. Partial view synchronization implementation * delegate next view logic to the current Committee * reconfiguration logic uses pending view for virtual view synchrony * number o' epochs * allow unbounded number of epochs * allow unbounded number of epochs * Complete Ethereal controller upon view assembly. Complete Driven context upon entering the Complete state. * no reuse of forks. handle new epochs after produce. * reuse forks * 4 epochs * do not reuse forks * remove old style view change listener * refactor fernet token handling * better logging, fix npe's * Advance pending view on Assembly block. Virtually synchronize RBC context in Domain. Some minor, but important fixin's revealed during POC uptake testing * separate configuration for genesis gen. use supplier for diadem check in assembly * squelch * update mvnw * java 22 * java 22 * revert * enforcer * java 22 * mtls virtual thread exec * minor cleanup * minor logging * whole lotta stabilizing, clarifying, etc * interim. too many sins and saints to count * interim. a few more sins and saints to count * interim. serialize tests until I figure out the lifecycle lossage causing hangs in parallel testing * clarify * clarify * genesis/reassembly unblocking. use phaser instead of countdownlatch * disable build parallelism * restore build parallelism. cleanup. do not restore on genii * optimize gather, parallelism * amp up logging for these 2 problem children * prevent view replay on ViewMember in GenesisAssembly and ViewAssembly. Remove second thread from CI build - just 4 forks of testing now * use Client as committee if fail quorum check. better logging. If quorum too small for consensus, default to client. 2 threads, 4 forks * just 4 forks * anchor only if cardinality >= majority. correct use of whenComplete for critical completable futures. clear observations * some cleanup. don't fail but complete on view assembly * enough with the forks * amp logging * perhaps...

2024-02-19

Fernet token auth (#189) * Fernet token auth * delete unused * add api for additional server interceptors, api for fernet token evaluation, E2E fernet token testing * Consolidate duplication * Consolidate duplication * Fernet token AuthNZ available on all Apollo services. Token call credentials supplier is optionally available in the creation of the ServerConnectionCache. Clients now call the supplied ManagedServerChannel to wrap interceptors and call credentials appropriately. * Provide optional furnet token validation for services. Router level validation, or individual service level validation provided. * Provide optional furnet token validation on router creation * npe protection * debug log and fail with unauthenticated when null token

2024-02-04

Room with a view (#184) * synchronous processing of list of view lifecycle listeners. * Add HexBloom diadem to the mix. Some cleanup The diadem is the standin for the view context. This maps to the Fireflies View diadem that succinctly defines the membership of a Context * moar rationalization of diadem value * validate agreement on diadem between joining and current validator * validate agreement on diadem between joining and current validator * Hide context in view change listener, add diadem * Use the compact digest of the view diadem - do'h fix up view listener * Use checked in ./mvnw

2024-01-18

B 5 (#181) * Moar cleanup Mostly lambda cleanup, but a few minor bugs as well. * split out verifiers from Parameters to allow de/serialization * missed * Configure DHT URL template, rather than hard-wire * Configure DHT URL template, rather than hard-wire * add encryption caps, expand keystore to alias storage :( moar cleanup * add Key Encapsulation Method to EncryptionAlgorithm moar cleanup * forgot about the algo diff. Weird asymmetry * error messages * add context interceptor and provide bootstrapping * further cleanup of notes/certs/validations Fallout from the addition of event sequence to signature. No longer need event coordinates, just the identifier. * better error handling/logging for KerlDHT. * bootstrap optimization for context majority * far better logging. add support for getKeyState(id, seq) * ugh. forgot this * implement getKS(id, seq) - wow. Gate with empty check on reconcile. * bootstrapping accommodations. use diadem's cardinality, rather than context bft normalized cardinality. Ring* handle no members * moar bootstrapping accommodations. "up to 4"

2024-01-01

bootstrapping (#176) * thoth grpc service. an unfortunate reformatting event, also, too * clean up dep declarations * Provide action upon Thoth inception. clean up parameters/etc dep clean up for kq/ep madness, yet again * Inline action upon Thoth inception. clean up close errors with KerlSpace * decent amount of logging/clean up on thoth dynamic membership For the moment, just disable the rebalancing test until I can pay some much needed attention to it. * Refactor Seal to have Any message type value() Allows Seals of arbitrary self describing protobuf messages * Fix mispelling of proto packages - d'oh Add leyden module, new proto services for such. * basic skeleton of Binder and Reconciliation service infra * MVStore Finally a use for the ProtobufDatatype - lol * Further flesh on the bone * Further flesh on the bone * The general idea. Flesh out the remaining comms, validation strategy, etc. Replication pattern established. * Flesh out get/unbind API * Woohoo! First successful replication. * correct reconciliation logic. track digests of bound. * General cleanup. Add README.md's button up Leyden for now * Moar cleanup. Expose dht in proc domain * test comms cleanup * refactor out subdomain behaviour from top level ProcessDomain with ProcessContainerDomain * correctly attach sources * moar cleanup * remove * cache? * process domain uses the event validation of the DHT's Ani * properly parameterize ProcessDomain Geebus :) * use active majority bootstrapping * revert RingIterator bootstrap majority * logging * use controlled identifier directly for seed from Node * override additional * amp logging * amp logging * targeted loggin * better config. just model for now * arg * again? * moar logging * decouple context * only unique * add BootstrapVerifiers to provide key state resolution via delegation to successors in joined group * fix compilation error * use stereotomy msg, add BootstrapVerifiersTest * Rename to Bootstrapper, add EventValidation functionality. Basically Ani for bootstrapping the view member's join * interim cannot seem to connect to entrance server for get key state. Weirdly intermittent connect (and subsequent fail on the retrieval) if debug/single stepping is happening. * Handle view join failure. Reseed if we cannot join the view. * logging * Bootstrapping Fireflies. Revert back to olden daze where the EstablishmentEvent was used in ye Note. Use this as the verifier for the Participant who's Note this is. This locks the View into the current Note of the Participant, as there's only that event as the Verifier. Validation is used in the Gateway admission, to ensure only valid member IDs join. * Cleanup and extend SwarmTest timeout * moar clean up * Use non virtual threads on inbound servers * halve swarm test for the poor build hamsters. Moar view cleanup * Moar view stuff Spawn a ViewManagement.populate using the supplied initial seed set to help ensure population coverage and prevent successor loops with other members who just happened to joined in the same view. * The problem child * This seems to make everything all right again. Fix/revert a lot of threading changes made that effed things up. Scheduled execs do not work well with VThreads, so always fork the running. * Reinstitute delegated event validation and verification to Fireflies. Segregate append from read only KERL/KEL Revert back to EventCoordinates for Fireflies note. * fix isolates * Work around the Verifiers * Minimize sig verification in replication. Re-enable sig verification in Firefly tests. Whoooo boy. * add keyState api for bootstrap verification * readd Bootstrapper * Use delegated Verifiers and EventValidation in the View * Moar cleanup wrt scheduling Also remove delegated verifier/validation as I don't believe that will be necessary

2023-11-27

Rename (#171) * rename Also add Digest.toUUID() * consistent naming * Digest.toLong() * reduce test iteratios

2023-11-25

aging bloom filter (#170) * aging bloom filter Use proto enumeratio for BloomFilter type * BloomWindow replaces DigestWindow. Remove DigestWindow. * add doc

2023-11-23

Kerl verifier (#165) * kerl-verifier * add StereotomyVerifier, add KERL.getKeyState(Identifier, ULong) api to leverage JohnHancock.sequenceNumber for verification * kerl based verifier, not integrated into main member processing * fix memkerl, kerl based verification used for sterotomy identifier verifiers

2023-11-20

Add sequenceNumber to signatures (#164)

2023-11-19

Hex check (#162) * no need to pass prioritized executor. Hides this. less chance for errors. * clean up * rename utils.proto * HexBloom for CheckpointState crown * Validate crown upon checkpoint assembly * use HexBloom for checkpoint validation, no hash list. Removed the list of segment hashes for the checkpoint. Use the HexBloom crown for the validation of the segments. * use HexBloom for checkpoint validation, no hash list. Removed the list of segment hashes for the checkpoint. Use the HexBloom crown for the validation of the segments. * add compactWrapped() calculation to Accumulator

2023-11-15

Do not use Any for RBC content (#160)

2023-11-11

nu (#155) * don't force update snapshots * proactive transaction cancellation instead of timeouts

2023-06-25

Bootstrapping Thoth (#145) * minor * interim - isolate ftesting fixed * handle logging configuration * fix logback stuff? no idea why. * Further work on demesne. Test outer context registration. Various refactoring cleanups. * succumb and use non routing server for demesne kerl/outer ctx. Clean build through isolates functional testing! Work on cleaning up ProcessDomain's spawning of Demesnes SubDomains. * SMH: shutdown process domains gracefully/correctly * Add Thoth jni api for delegated identifier operations * expand event loop group factory methods * add delegator to membership schema, witness now explicit Basic ID * Use Delegated Keys for Demesnes, using Thoth to manage lifecycle Hehe. Clever, eh? * Incorporate key event delegation in ProcessDomain.spawn Falcon A * moar work on process domain spawn * Ha! Yes! Ha Ha! Yes! Thoth delegated bootstrapping of Demesne Isolate Very cool. * clean up * Interim: Add identifier delegation mapping replication for subdomains * update GraalVM trace info for MVStore use in SubDomain * Interim. flesh out state replication * moar refactoring for interfaces * finish refactoring for interfaces * Cleanup Router * router cleanup * add outer routing agent id to header. general cleanup. * Use agent for identity in delegation gossip * interim * Interim * update to jdk 20, with attendant GraalVM stuff * update maven-shade-plugin

2023-03-07

Demesne Isolates (#144) * better redirect * Update to GraalVM 22.3.1 * squelch for CI * Progress demesne isolates Fix JNI parameters - D'oh. Use protobuf ViewChange, etc. Eliminate graalvm sdk "work around" Now failing because of my bozo-ness ;) * Finally get the demesne isolate JNI model working ;) lordy. I am an idiot. Key was just getting the JNI Env stuff in to call the functions to get to the byte[] content when inside the isolate. Geebus. ;) * JUL logging, dynamic config. update grpc/proto * moar progress on demesne isolate * Seems to work? Falcon A.

2023-02-01

Mostly monty (#134) * remove "root" event validation twas a silly idea * remove threshold, just check witnessed * Add certificate validator to Ani, general cleanup * split out GraalVM isolate logic for Demesnes, add it back to mainline * Shift requirement for GraalVM to isolates specific functionality Now only require Java 19 * Refactor to clean up dependencies. No need for linkage that isn't there * normalize pom naming/descriptions * Refactoring - move GorgoneionClient to own module, distinct from server side dependency - static set verifiers - misc cleanups * fix isolates module n testing * Some refactoring, renaming to Demesnes (from DemesnesImpl) Original Demesnes renamed to JniBridge * refactor demenses module into model module. remove model membership "state" * interim. add services infra to support Demesne * moar refactoring n laying down track * Use event coordinates for joining members * add kerl dht to process domain. rename to view lifecycle listener * change lifecycle listener contract, add updates of note coordinates When note's key event coordinates update, notify listeners. * Update for use with "act" local GitHub Action runner * Add ring_digests table for tracking replication * update ring hashes on append of identifier * No rebalancing. Just use a max ring count, rather than either deleting or adding ring hashes on context membership changes. * replicate full event + attach + validations Need only 1 location hash - no per ring hash for identifier location * interim work on thoth replication/repair dht protocol * Do not insert when no events to insert * add update support to PENDING_* in the thoth schema * add have digests from pending_events as well, per interval Commit pending_events after update, if possible <- stubbed, todo Change sequence number from BIGINT to NUMERIC(20) * transfer from pending_* to destination unikerl in reconcilliation * Simple KerlSpace replication smoke testing for KerlDHT * Implement and test updating logic in KerlSpace * fix validations upsert. make gossip 10 ms for exercising

2022-11-17

Gorgoneion prep (#132) * long test tweaking * cleanup test * goroneion a dependency of thoth * Rename DelegatedKERL, readd as real delegate. Add wrapping in uni kerl create to provide validation * EventObserver implementation Maat and smoke test for same. Remove unneeded KERL in Fireflies Registration. :: sigh :: Still trying to pin this into a corner where I can box it up. lol. * Clean up Gogoneion. Perform all validations/verifications/checks Pretty darn slick * First cut at documenting the Gorgoneion service protocol * correct the protocol documentation :) heh

2022-11-12

CHOAM combine RBC opti (#131) * use ids for logging. handle rate limiting in the ch-rbc gossip * interim * Prep for more efficient RBC * Fix race condition in test Not sure why this suddenly showed up... * Fix async error introduced with vthreads Geebus ;) executing the transaction asynchronously is - obviously - a very bad idea. Modify to pass the executor in for async completion of locally sourced transation results. * Use Any, rather than simply Message and ByteString. Lol. Full circle back to Any * Optimize CHOAM combine RBC published certified blocks are now congruent across the membership, rather than unique Sig/Hash per publisher. RBC message hash is the xor sum of the block body hash with the certifier's sigs.

2022-11-08

HexBloom (#130) * Add HexBloom support * Add api for wrapping crowns with another hash transform * moar waxing * validation api, added constraint checking on functions. * only t+1 reads required not "majority" as in 2/3 case, that's only required for writes * log choam state when not activated * state logging for domain * print retries * Convert to Project Loom virtual threads * increase vthread count in testing * some more thread tweaking. surefire "fork always" the fix? * remove model's override of surefire * dunno why this is needed for the CI * redirect * tweakin' * tweak * ? * Turn off virtual thread use for most cases * pass exception through observer * reenable vthreads * moar * finally * tweakin' * ? * revert to platform threads * Revert swarm test to platform threads * use fork join pools * use Executors.newVirtualThreadPerTaskExecutor() for most everything * extract to inst var * ::) * tuning for large_tests * remove indirection except for modules boned by Eclipse J19 issue * missed this one * Integrate HexBloom into Fireflies Replace "cheesy" version previously implemented with a "real" implementation. * update readme

2022-10-31

Demesnes (#128) * Add Demesnes: Apollo enclave isolate model for GraalVM Seperate profile for now, as I can't figure out how to integrate it normally into the build. For some reason, when properly parented to the apollo.app of the build, the graalvm sdk classes are not included. Very odd. * test GraalVM native build * Enable GraalVM native image build of demesne default in CI Holy flurkin' snit! * move metadata repo dep to native image plugin * add invocation of Fireflies scenario to pull in closure * interim checkpoint. Lotta work on graalVM native image. Still stuck on javax.sql.rowset.RowSetProvider missing for some inexplicable reason. Clean otherwise (so far). * missed this * Seem to have resolved issues with JOOQ and GraalVM native image. Got further, moving on to the next failure in the sql state machine * Working full stack in Graal Isolate. F*ck Yea. * refactor to split enclave from isolate creation. update readmes modules demesnes and isolates excluded by default, added with profile "isolates" * Get static imports + friends to support model.SubDomain Got past the BouncyCastle Great Filter it seems. * breakout * v1 * on workflow_call * :: sigh :: * ? * remove isolates for now. have to figure all that out Seem to be creating isolate just fine. * Break out Smokin.main() for testing. Put meat on the Demesnes skeleton * split out domain socket support into seperate plat dependent modules Had a heck of a time dealing with that when creating the isolate. Isolate creation now does not use fallback * prep for conversion to shared library isolate. SBOM gen support. Some general clean up. * remove Pal and PAL-D modules, as they can be replicated in simpler ways * Move to GraalVM 22.3.0, JDK 19 * bump actual version to 19. add "--enable-preview" to compile/surefire * :: sigh :: Mostly cosmetic cleanup of poms. * use actions/cache * update version * finally get something of a compromise with Eclipse/Graalvm.sdk :: geebus :: Need to specify property -Declipse.graalvm.jdk.workaround=true when building with maven command line * First smoke test of non-isolate Demesne * Break out demesnes module (again) into the "isolates" profile. Update readme with build instructions for using the isolates profile * prep for demesne isolate functional testing. * Refactor to split out isolate from host interface Demesne is now the wrapper around the native methods for the shared isolate. DemesneIsolate is the aptly named isolate implementation. * add isolate modules to CI build * the iterating :) * trace down epoll.Native * lol. backing out isolates modules as dat is going to take a bit No need to do that on the CI for now anyway.l * :: sigh :: Dead in the water due to: https://github.com/oracle/graal/issues/5321

2022-10-17

rpc context cleanup (#127) * cleanup rbc * cleanup fireflies * clean up CHOAM * clean up ethereal * clean up Gorgoneion * Clean up Stereotomy Services and Thoth * remaining * missed this

2022-09-26

Transitions (#121) * Generalize join credentials with attestation and predicate test * surefire specialization for model testing * Use proto Any for attestation no need to shirk on structured metatadata * maven caching * Provide root and kerl validation modes Root validation requires validation of configured roots meeting supplied threshold. KERL validation uses validated known member KERLs meeting supplied threshold. * Pass Event coordinate in Certificate (b54 encoded protobuf) * root event validator, kerl event verifiers access Some minor cleanup * remove attestation - that will be done elsewhere. perform 1 phase of gossip with redirected members in rumors wrap remaining formal join with stable() * Attested admission service * Bare bones Gorgoneion attested admission service Add replication service, set up structure/comms/routing basics * moar flesh on the gossip bone * moar * Fireflies View can now handle 2/3+1 BFT Contexts, not just f+1. Further refine Gorgoneion protocol * extract bias to constant, set back to 2f+1 normal FF testing * better unauth handling in interceptor ?? * trace * pass in common cluster map for local router. For some reason swarm now fails without bias 3. curious, as churn does not. * fix * fix * better delegated identifier logic * Moar Gorgoneion. Streamline with BloomWindow for RBC * rw locking * moar flesh on the protocol skeleton * refactor for clarity * Add some metrics interfaces, moar waxing before I plunge into the pit * Use suppliers for dynamic Ani root identifiers and signing thresholds * use event coordinates instead of identifier for validations * Validation is {EventCoordinate, JohnHancock} Gorgoneion testing - simple construction, at the moment * further flesh on the smoke test. update readme * Integrate Ani into KerlDHT. Moar fleshing out Gorgoneion Further testing of singluar member (bootstrap) context. * refactor Gorgoneion into own module simplifies Thoth * rename * add gorgoneion dep in dep management * sql schema for gorgoneion empty at the moment. * Full round trip for Gorgoneion bootstrap * moar * remove associated state on commit, add to processed window * Add event observer, test KERL is published * validate * Add CompactContext, remove cruft, some refactoring * Minor tweaks on Context. Fix & smoke test CompactContext. * interim Refactored ViewManagement and Binding out of View. :: sigh :: Just became unwieldy * recreate current behavior with refactored state * clean up :: geebus :: private encapsulated state, package visible interactions * minor remaining cleanup * minor doc update * Add signed nonce generation, validation of registration * wire in verification of credentials * interim 3 phase join/attest/validate working. Now running into ye olde fast path consensus. Joins not stable, my clever scheme to GC old joins failing spectacularly. lol. * interim * Interim * take gogoneion attestation out of join protocol :: sigh :: move to seperate service, recover stability. * remove * refactor again. * use multiple sigs for nonce This will allow a BFT sig set for the gorgoneion protocol * add generalized grpc domain socket support in protocols, test. Some minor gorgoneion work * further waxing * further fleshing out * little bit o' extra paranoia to validate Just ensure that, after stopping, everyone is on the same block before comparing state. Will fail if not on the same block, but at least the state difference will be clear, rather than a protocol failure. * JDK 19 * oracle jdk 19 * enable preview for Project Loom * revert back to GraalVM 17 * Basic testing * Further flesh out Endorsement rpc Basic skeleton for apply endorsement logic * Full metal protocol test Smoke testing server side protocol with 1 context member * Smoke test client + server full metal protocol Nice

2022-08-21

Scarab (#119) * use declared set of validators for Ani. clean up cruft * resurrect hmac. reintegrate so hmac can be dynamic Simple Supplier/Function api base to provide and validate hmacs * Refactor for clarity * minor generic

2022-08-18

remove hmac. (#118) not a good idea

papyrus-2 (#117) * rational thread resourcing * Provide a java.security.Keystore implementation of StereotomyKeyStore File based and generic * initial Gorgoneion two-persion rule de/encryption server * infinite cert duration * whoops ;) * further flesh out Gorgoneion. Convert en/decrypted Secrets to proto Any. As a treat. * moar fleshing out. Add locking around JksKeyStore mutations * bare naked minimum smoke test manually validated create user - lol. nice * a rough sketch for now. on to other things * rate limit rationality. another round of hardening on FF. Drove it to madness with 1900 joining members. Stable at 1K on my machine, with minimal cpu/threads. Working on 2K * Add hmac authentication for seed/join * interim * separate out seed/join as an optional router endpoint Fix a lot of rate limiting nigglers with resource exhaused Fix rate limiting in join with resource exhausted labels around router service registration less verbose logging by eliding member ids all over the place. need to figure out better observability scalability around membership ;) * elide file logging * Break out join from visible join with full membership. RingCommunications returns member when empty ring * don't cast votes until formally joined * aggregate seeds n' members

2022-08-10

Lyre (#114) * interim package specialization ;) * remove format it is silly * initial sketch of PAL Permissive Action Link for providing secrets Daemon side sketch using domain sockets * fix profile activation * d'oh. fix test os specific client config * PeerCredentials for GRPC over Domain Sockets. :) * no idea * Seems like the bloom filter seed was to blame - smdh * cleanup, elevate membership test to warn * multithreaded hardening * use concurrent map for timer tracking. fix NPEs in ring communications Fix NPE in router * Fix ring communications to accomidate Fireflies filling in of members Membership is now determined per iteration * PAL Daemon working smoke test * PAL client smoked ;)

2022-08-04

Owo (#112) * upgrade grpc/proto * big plug o' tech debt * on.. and on... refactoring the base is tedius ;) * everything but the impl - lol * last test * stereotomy tests green * tech debt: complete ;) * some moar cleanup * Fix RingIterator (again, lol). Use full write for KerlDHT. Final iteration calc was off and wrap around call to link occurred. * KERL wrapper around KerlDHT now tested, tanned and ready * Bootstrap protocol. ftw. Continuation based interface in Thoth for bootstrap inception and delegated (forced) rotation from controller. * Fill out remaining validation api * cleanup * move validations to KERL clean up and refactor the fallout * clean up schema * refactor getKeyStateWithEndorsementsAndAttachments to KERL Geebus. that was a lot of refactoring. Much cleaner now. Refactor Ani to pure KEL interface dependency * remove params * Interim * still have an issue with the KerlDHT with validations - lol * interim * Solve various entertainment from sharding the UniKerl across the DHT * Sweeeeet. Ani validations using KerlDHT partitioning working. Woo * dependency convergence * lol. debug timeouts ;) back down to mere practical values

2022-07-24

d'oh. successors rather than predecessors (#109)

Aani (#106) * fix format params * tweak * tweak * no idea * baffled * Update READMEs * update readme * update * Use HEX-BLOOM for authenticated membership acquisition. Disable FF test in domain until that is figured out. * Holy flurkin snit. Optimized Join protocol using HEX-BLOOM. looking good Added return of SignedNotes of joining cohorts with gateway. Reenabled FF test in model. * whoops. back to 100 ;) * accomidate lack of CPUs in CI * try not using FJ * :: sigh :: * revert. amp up trace logging * whoops * use cached thread pool for grpc. my suspicion is the handling of the async join callbacks is being stalled * Scheduler? * don't tie up the scheduler. seem to run fine with a single scheduler across everything now * now with 10 scheduler threads * deamp logging, decrease gossip freq * :: sigh :: * up timeout * moar txfr * Finally. New join protocol working. use crown = members' hashes xord. view identity is crown.rehash() Stable and fast. * full build * update readme

2022-07-18

Seshat (#100) * Seshat Charting the stars, the rhythms and the causality of such. * lol. continue, not break you idiot, you moron, you aqua maroon * whoops ;) * fix a couple of issues - Parameters for exp delay policies explicitly defaulted in params - Allocate ye PRNG once in the sql state machine - Fix delay logic duration calc in the DS * metrics for reassemblies. Fix metric sampling site * minor cleanup * my nemesis * amp logging on lifecycle testing some cleanup on View * just inc the chkpt blocksize until I get this fixed * trace session * refactor fireflies to eliminate identity, shift info to Note Note now contains: - host - port - event coordinates Signing the note is the key indicated by the coordinates of the note. This folds in the functionality previously supplied by identity, simplifying fireflies back to the original formulation. Key rotation is accomplished by new event coordinates, which requires a note of a new epoch. * fix activate/offline logic in context. take churn test to 0. remove on/offline metrics in FF * use histograms for comm structure metrics. use 5ms gossip for faster testing. * some cleanup. don't use netty-shaded as it has no source * use gossip of 10 ms * 50 ms gossip on restart. don't warn when no active successors on ring * don't be silly * 50 ms it is, then * better ring hash management n' such RoundScheduler now uses a full "round o ticks" for every delayRound indicated in the timer. RoundScheduler now represents # of ticks, not number of rounds. * refactor to remove multiple regenerate nonsense in the updateAndGet * use histograms for ethereal comm structs metrics * use histograms for rbc comm struct metrics * SignedViewChange observations for consensus view membership * shun the non believer, Charlie. SHUUUUUUUUUUUUUUUUN! * moar * first bare bones view election! Does not install the view, but d*mn, it actually works! Only fast path paxos with 3/4 + 1 votes implemented. Fallback consensus using ethereal still tbd * moar work on fast path view membership change consensus * catch NPE * shutdown outbound connection cache before server stop * down to 40, high frequency gossip * ignore until flapper fixed. * don't get fancy. use digest list for leaving set. * whoopsie * rebalance on cardinality change * reset round scheduler. use total member count for consensus cardinality * interim. seems like a lot more thought is needed here * fix REALLY annoying grpc NPE. Fix SC graph testing in Swarm. * interim. fix some whoopsies. slow down gossip frequency. Mostly behaving * simplify, turn down logging * integrate pending rebutals as proxy for stability general clean up and improved logging * spelling ;) * revert. lol. 40 it is. * integrate view change id as epoch to Note and logic. view change trigger new note from all surviving members. notes are only exchanged between equivalent views. cleaner shunning. metric tracking of join/leaving rate * recalculate metrics for note/accusation placement. don't count filtered in metrics. * Filter notes for view, shunned. Use 1 TTL for scheduling retries. Dare I believe the CI can run churn down to 20??? * first bones of the join protocol * Flesh out the basic join protocol. No timeouts, no retries, no errors, just happy path. * validity checks on join, add member to joins data. * fix mask generation lol * parallel seed/join from the client side connections * Use FJPool.commonPool() for all pools in swarm testing. 10 thread common scheduler. Yes. * Lock view stability during gossip. 2 common sched threads in swarming Holy sweet mother of Moses. This is beautiful. * parallelize, schedule, functionalize the Binding protocol Now uses simple, parallel gossip with the joining member's predecessors to fill out the membership notes. * Add BFT predicate on gateway for synchronization * add logging * tweak * Don't use BC for JSSE, etc. Upgrade GRPC The new BC complete effs up JCA compliance, which is sad, and seems to be required to keep MTLS working with GRPC. * upgrade BC * fix * fix * Iterim Somewhat working. needs a lot of refactoring. * Further work. Calmed bootstrapping so it's not swamping the system. Will need rated limiting there. Now have a stuck handful of joins that don't seem to join nor be gc'd * better. incremental stability progress. * remove all view change locking logic. seems like the remaining problem is simply view change under sync * cleaner logging * much better logging * push to orig * add back locking, better error signaling Only use blocking version, use strategically so outbound comms doesn't tie up the lock. Switch to fail after join failure for clean tracking * Leverage gateway to provide at least one batch of members. * Interim. Shiny new strategy. Join is now the seeding of the member digest list, and a seed list of SignedNotes which is the union of the joining member's predecessors and successors. Gossip takes care of the rest of the fill * improve redirect. epoch == -1 for seeds * moar redirecting boxing * finally nix silly bugz :: sigh :: swarm working again, churn working through ramp up, not deleting now * refactor validation to centralize policy. clear rebuttal timers on view installation * swarm churn now working :: sigh :: messed up accusation mask filtering when I refactored that. lol * cleanup * Interim - general cleanup getting mtls test working * fixed mtls. now using shunned, rather than offline Reintroduced shunned set to allow old behavior of filling in notes by discovery. This in effect is a 2 seeding process. The join provides the true seeding set that allows new members to discover and replicate notes from known members, while still maininting the communication overlay correctly. nice. * whoopsie * Fix FF test in model * minor tweaks * parameterize all the things. * add listeners for view change events * initial hookup of model ProcessDomain using view change listener Woo Hoo ;) Just works. Still need to do real coordination, 'natch. But still, quite encouraging * clean up no need for view on accusation. clean up metrics. add txfr limit to params * fix test. d'oh totalCount -> activeCount hold over from previous scheme * Update readmes * Restore accusation view logic, raise unknown when contacted by shunned Fix churn test logic. Allow accusations for null note members. Update Fireflies README Throw unknown for shunned communication as well as incorrect view, or comms while still joining. * Split tests because reasons catch exception around pending trigger in install * edits * Update README.md Co-authored-by: Hal Hildebrand <hal.hildebrand@salesforce.com>

2022-06-26

Ibis (#98) * use completable future composition for retrieving keystates * better form * first cut of Sakshi integration. - modify to use KeyPair for witness, not Identifier abstraction - incorporate Sakshi into Ani for witnessing and validation - ViewAssembly2 for future view change work * refactor constructor to formalized parameter record * think it's the stragler from the low quality bloom filter strategy. increase by another order o mag. * also increase timeout, for more gossip to happen for stragler * squelch choam logging now that things seem sane. better member logging * introduce alert and shunning into the fireflies model This is the idea from Rapid. Allow View nodes to issue Alerts indicating booting from the membership. * Convert View to RingCommunications Generisize RingCommunications appropriately * handle npe * npe fix, add thread * integrate monitoring into gossip * remove ping as redundant * Refactor View to RingCommunication - Remove ping, as it is redundant. gossip service same function. - Crystalize accusation/offline functionality. - Simplify and enhance RingCommunication. - Leverage new RingCommunication.Destination for monitoring in FF * rip out alerts, as we can use accusations for the same. allow accusation testing for passed majority. only restore when existing accusations are <= majority Thoth now a dep of Fireflies * remove accusation testing scheme kinder test params * Interim. New view assembly regime * interim * Next gen view assembly/reconfiguration smoke test integration with full Ethereal sim * Integration of new ViewAssembly. First cut. Still have some work to do, so we'll see. * revamp tx data source * tighten up testing The good fight * double timeout * was not taking advantage of retries. d'oh * increase timeout * Encapsulate state maps in the context interface. Been meaning to do this for quite a while * Better logging of this quantum fluctuation * use atomic reference * don't take it to zero * logging * use cardinality 5 * interim * rip out concurrency over kill Rationalize remaining critical sections pt 1 * fix npe when candidate for self not set yet * moar surgery Trying to hone in where the timing round det goes wrong. * liddle moar The seems to be 2 issues: 1) same CRP on all procs, but 3 (quorum) deciding a different from the 4th. Suspect these 3 have different futures than the 1 and thus make different consensus votes. Have debugged down that line, but haven't made much progress 2) despite same consensus decision, emitted proto blocks are different. Have not been able to track this down. Firmed up the config and such. Still no love * button up before more spelunking * disable * remove * remove lock. idiot proof EtherealTest.lots() - d'oh * remove exec service until needed * back to regular trace logging * executor to decouple pre block processing by client. Needed this, but now where it was being used ;) * think I need 2 threads for ethereal 1 for decoupling the consumption client of the blocks and 1 for production of new units, which blocks on datasource * interim. restore topological sort to update dag. Now we're getting to the heart of the matter. Livelocks. Need to debug rbc again - lol. * whole lotta work revamp adder -> timing -> creator and back again. * add back timing round consumption loop * current * closer * closer * think this is licked * fix checkpoint bootstrapping test * moar churn :: sigh :: * Finally. Fix permutation in Extender. Let's see if this fixes ye things Geebus. The permutation mechanism I was using was causing the split decisions. The previous mechanism broke down when there wasn't the same number of units on the level to be decided. I am an idiot. Fixed a bunch of other stuff. Tightened up quite a bit. Still have an issue with the last epoch not finishing due to timing round stuff. Will work on that. Disable Model functional testing until I get txn retry logic in the sharded oracle ;) * wait for at least 1 reconfig block before considering one's self active * enable tests. up logging to debug genesis bootstrapping. Need to revamp genesis assembly anyways. * amp up logging in model * add drain delay for tx data source. * exponential delay during drain * genesis assembly loggin * exponential delay for data source, txn submission backoff View assembly attempt extended throughout last epoch. * Fix batch queuing. Fix exponential decay defaults. Session submission uses submit timeout for retry duration. some test tightening and clean up refactor abstract lifecycle testing for commonality amp logging during flapping genesis bootstrap test in sql-state * hold on tight to your dreams * add timer metrics where they are literally required ;) Back off FPR, as - lol - yea, that wasn't the problem ;) * moar * amp logging * current height accessor in CHOAM * moar amp logging * set height before countdown * fix batching q logic * not sure if this helps, but worth a try * force election when it's time * interim debuggin * simplify fiber * Add back in epochDone logic. FPR back to 0.0125 * Narrowed it down to a bizarre update error on height fibers in DAG Can see the update. Read again at a later date - WITH NO INTERVENING UPDATE - returns null. Kill me now. * Add all members of the view context to the ethereal set Offline members are still members and need to be accounted for. Add checks to ensure valid quorum set. Should, in theory, run green * Issue log warning, don't NPE when Assemble block isn't processed. * long testing

2022-05-30

KERI Witness/Validation (#95) * add append for attachment events, get keystate with attachments * Fill out Dht with validations and attachments apis * Finish fleshing out KerlDHT api implementations * KeyStateWithEndorsementsAndValidations * Flesh out Ani moar Minimal smoke test for Ani Squelch limits metrics * moar thread resource management - lol * Async Ani Sweet * Saksi and Validator * use countdown latch * crisp definition of "active". increase sync cycles in tests to 2 * Clean up logging to remove member type. Further crisping of CHOAM.active * inc to 2 * npe protection and up periodic checks * try with process reuse just for laughs * backout, use adopt open jdk jdk lol * use atomic ref * Segregate txn exec/sched for domain shard use D'oh. Was undoubtedly starving the shared thread pool with blocking txn requests. * use only one txnConfig for all members, as only 1 emits txns * use default of 10 sync cycles for bootstrap :: sigh :: * use better sync cycles Use more than 1 thread for router in thoth testing :: sigh :: * I am in h*ll * see if this works with Java 19 EA * Success! Back out so I can merge dis

2022-05-28

Running up that hill (#94) * fix potential npe * fix last epoch logic in ethereal. General tightening of screws. Various fixes and cleanup * fix liquibase bug that prevented service override. ruined deterministic migrations - d'oh. * long test params * intermediate refactor event notification from sql state machine update readme's intermediate refactoring on domain/context bridge * First cut of Kairos GRPC communications simulation framework * first cut builders * remove jmock * Start at 0, not -infinity * Tighten up state testing by waiting for all inflight txns to complete * interim commit * interim thoth * upgrade to H2 2.1.212 * h2 deterministic DB Seperate out into module for shaded rewrite to avoid collisions with usages of the non deterministic H2 DB * Finish moving deterministic H2 into shaded module Uses package renaming to avoid any collisions with non deterministic H2 usage. * skeleton deterministic liquibase. excluded for now * update readme * stop routers and choams before consistency check * clean up * deterministic liquibase * no need for shading * remove optional * small sleep to ensure shutdown * Refactor to use completable future for all services apis * interim to capture for reference * delete capture * use executor instead of FJ * First cut of Thoth general clean up of stereotomy services. fix last optional on KERL * move entropy operations into Entropy for pooling. Stage 1 * Entropy pooling fini * back out pooled entropy. :: sigh :: not worth the hassle. Will need to properly perf and check space overhead in VirtualThreads when I have propper perf tooling for Project Loom * initial skeleton for reconciliation * Reconciliation flesh on the bone * async the rest of the things * Fill out reconciliation client/server. Async Thoth server * missed this * Add KERLSpace and Thoth schema KERLSpace now responsible for state replication. Provides pending storage for incomplete KERLs during replication. * fix thoth initialize resource issue * take out of build until project loom * Add pooled version of uni kerl. Convert thoth to jdbc pooling * simple smoke test passing Simple testing for creating an inception event, appending, then rotating and appending. * add simple logging for now * Rename * redirect liquibase console output to log. Add simple output stream -> slf4j log clean up logging chatter for test fix warning in thoth pom * large_tests usage on cardinality to protect the CI hamsters Shuffle initial traversal ordering in ring communications * Significant clean up n bug fixin' Fix ring communication and iteration. Randomize first traversal. Fix successor/predecessor selection using new traversal predicates that accomidate a "fail" or null successor. Fix ring iteration, ensure next iteration occurs after current. Fix bootstrapping/etc. Use Context.majority for calculations. Add correct member predicate on View.update Use fixed size thread pools in testing. Use all K rings for Thoth DHT, as there is no validation possible on writes :( remove useless ListenableCompletableFuture fix Context.diameter calculation to use bias. add findPredecessor/Sucessor traversal to Ring

2022-04-28

chRBC gossip for Ethereal (#89) * first sketch of RBC gossip * Interim. Model after Adder. Pull out nodes, generisize via Node interface. * okay * Better refactoring * basic skeleton * unit test dealing round (round 0) for RbcAdder * test dealing for all members, full round 1 * fix produce logic, test to 3 rounds * The traditional pre Easter code reformating * doggone it Missed this * units produced by the node don't wait * fix test logic for adder. test gossip + adder for 4 members Woo hoo! * Renaming and cleanup * Refactoring phase 1 Move replicated state to the adder, in preparation for epoch refactoring. Remove adder test, as this is now reasonably complicated 3 phase gossip, that is quite tedious to test - sue me; it was scafolding to get to this point. * refactor to include epoch * Interim Lotsa work * interim * squash that bug. ultimately, the timing round was not decided correctly. The previous method just took whatever units were on the level, and used the prefix size, which is 2. For quorums of 4, this is still less than 2f+1 (3). Worse, sometimes these samples - because of deterministic mixing of order on rounds - would be size 1 instead of size 2. The fix is to always use the first 2f+1 of the permutation, and require 2f+1 units in that permutation. This was "caused" by the logic not checking for 2f+1 exact. This was a random failure because most of the time it still works, when in close synchronization and so would only show up when the nodes' views are out of synch, due to natural variation. Which is hard to produce in the current, single process testing, as things are very uniform for the most part. lol. * removed unused config, general sweeping up. increase chrbc test to 13 members. * use fixed number of random bloom filters per adder Don't recreate the BFs each and every phase. only care about updating >= current epoch * interim * finally figured out the problem. Lots of clean up, adding logging, adding testing. Problem was I was passing the parent, and not the child when the parent is output and the child now has no waiting or missing parents. lol * Behold the new regime * update README

2022-04-08

Txn rate limiting revamp (#88) * cosmetic * Simplify txn submit rpc/spi part 1 Use enum for submission results * revamp transaction submission * interim * fini

2022-03-21

Next things (#83) * remove unneeded param * EventValidationService * test event validation service * test resolver, binder. refactor KERLClient to use async * refactoring * decrease txn timeout * don't use completeAsync * cleanup * don't reuse forks for testing * use cached thread pool geebus * reorganize. Add EventObserver service. * Add event observer test modify checkpoint block size for membership tests * fix resource * tighten up lifecycle testing * Node -> Domain * SubDomain, ProcessDomain and RootDomain * timeout? * Binding stores map of String -> URI doc domain classes, add structure. * simply use host, port remembering the log4j ldap nightmare * trigger after checkpoint * statically allocate and reuse executors in tests * clean up context model * revamp certificate provisioning and validation * better refactoring of provisioning clean up remaining turds from context refactoring * fix test * I just can't understand this flapper * don't use ForkJoinPool.commonPool * explicity configure backoff * set default backoff configuration * trigger * set cycles to 1 * modify params * Cleanup View * update readme * tweak params * really becoming the joker at this point * use zulu * don't expose service * bow to the gods * D'oh * lol. wait for longer than 10 milliseconds for initial txn * Pre refactor remove View references to params.tolerance/ringcount/cardinality * refactor false positive rate from params * moar refactoring of params. slowly, slowly * remove explicit backoff config in tests * cert validator passed in View constructor * format, ordering update readme * ensure the bastiches are at the same block for predicate checking * update readme * don't use timeout * alt formulation * minor formatting, readmes added * lengthen timeout, remove delay * encapsulate Node and Participant. Fix nasty FF mtls bug - D'oh. That was a killer. remove thread.sleeps. include initial inserts on choam test genesis. * move the test proto into main. it's innocuous and simplifies integration/workflow because there's no "generate-test-sources" built in to the maven/eclipse ide. * further cleansing * Interim. SuccessorTest working ;) * SwarmTest working holy flurkin' snit * MtlsTest now working * fix

2022-03-06

stereotomy-services module (#79) * stereotomy-services module First cut of ResolveService implementation * Binder service * fixes * timeout? * bump * just 1 txn for genesis bootstrap testing silly. don't need multiple txns, which only create flappers at this point because for the life of me I can't seem to make things sync yet for this test. * Okay, got the initial comms smoked out for ye test * Refactor the stereotomy services Close, but not perfect. Much better seperation, though * heh. wrong object * bow to reality * test kerl provider api * decrease submit timeout * use azul 17.0.2+8, reuse test fork apparently I've been a very good boy on my test cleanup. * lol. use correct distro * :: sigh :: * okay then * Fix current block logic in sql state machine Extend TransactionExecutor with endBlock. Fix logic setting current block in CHOAM, and ending such. Externally visible current block represents current committed block, not executing block. * print out duration. netflix ;) * apparently a little slop is in order for this test. * refactoring of proto names, flesh out remaining grpc apis interim * Refactor to KERL base * record after block ;)

2022-02-21

Refactoring of metrics (#73) * fix error handling logic around status - oy. Back off rbc and producer gossip to 50 ms in ff testing in model. Still have to figure out why high frequency polling does not deal with the disco. Need to winnow thread usage in all testing - I suspect this is causing issues and needs to be cleaned up anyway * single thread tests for squashing the rare flappers. * scoping * Looking good for smoke testing, but... Need to really focus on txn submission logic :( But, I knew that already ;) Suprised that it works quite well without such ;) * massive refactoring of metrics * add warning of expected exceptions in testing * fix ring test * Prefix metrics with context digest * firefly metrics prefixing with context id

2022-02-14

fireflies and secure network overlay work (#71) * add default exception handler to show any nasty ones in the background * Merge branch 'master' into strapping * Refactor Fireflies for signing sanity ;) Create a NULL_SIGNATURE for dealing with... null signatures used in fireflies. * reenable mtls testing for FF. seggregate the cert/key from member's * interim * dynamic context ring support * Some fireflies clean up. provide constructor for member implementations add a fireflies test skeleton for further integration work * clean up certification provisioning/validation in stereotomy Fireflies view booted from stereotomy provisioned certs. * linkage of Context membership states As "Bob" foretold * extract Context interface from implementation Geebus. Not like it's the foundation of everything... * moar clean up on context * Clean up Fireflies long due * logging levels * other cases * link handling

2022-01-22

Remodeling - Work on service/app model (#63) * fix long tests in sql state bootstrapping. Logic around rendezvous after transactions have been completed was causing the tests to wait the full timeout :( * Use a separate limit for each client connection, rather than sharing. moar rational thread pooling for local servers. moar rational params for default limits. * test fixins n' stablizers * Clean up tech debt on Context impl. 3 failing tests only. interim commit * refactor to manage context membership back off some changes * fix assembly test * fix sql state bootstrapping testing genesis wasn't filtering out the test subject in the transactioneers and so was flapping pretty wildly, as the test subject was banging until bootstrapped, and then the only transactioneer - lol. So would take forever. * refactor liquibase schemas to a common module. This is a work around until JOOQ is capable of working with current H2 version. Clean up Stereotomy Resolver. Clean up liquibase change log organization for modular reuse. probably will need to go into full blown package space naming. Clean up model schema, link with stereotomy * Favor completion over throughput for lifecycle bootstrapping tests * mea culpa * update * make KEL.append() async. Clean up some cruft in stereotomy * use supplier for genesis data * flesh out grpc Resolver/Binder. Rename stereotomy_kerl schema * misc * Move to overlap of view reconfiguration. Fix age batching q limit * stabilize * better error msg * fix batch limit in producer. Apparently I have it correct now. No dropped batches when starting assembly. * update grpc version * Refactor Genesis Assembly Genesis block now assembled in a rational manner, no queries for vm, as there is no ability to validate them and we know the initial set. Add Foundation and FoundationSeal for trust establishment of genesis. Joins now contain the KEL of the founding members of the genesis committee. * generic prefix for genesis reContext * Tighten up Stereotomy API * moar cleanup, honing in on delegated rotation/inception in direct mode * Clean up AttachmentEvent & Co. Overhaul stereotomy schema for cascade delete constraints, etc. * remove unused columns * still continuing * Revamp delegated establishment, phase 1 Delegation linkage to delegator event is now handled by attachements Phase 1 is the revamp of the delegated establishment events and validation * Direct mode delegation working for MVLog KERL version * Append attachment for UniKERL * don't parallelize sql-state testing * Bulk append. Refactor out delegated direct mode logic in incept and rot * commentary * Refactor to use ULong instead of Long for block height Needed to be done sooner, rather than later. * uint64 lol. like it matters ;) Still... * Use ULong for event sequence number Sooner > later ;) * fix type issue. accepted by ide, rejected by command line compiler. lol * minor waxing. Seems clear my failures for these two are caused by random interactions of eclipse with these during a command line maven build ;) Close all the editors, and don't touch the IDE during the CLI build. * Update to H2 2.206 Mods ported * first smoke test of mutator testing * Update to JOOQ 3.16.0 Use only one version of H2 for gen and runtime. * back off JOOQ 3.16.0 lol * add batched transaction testing * Fix Sharded Kerl, using batch * update schemas to use identity, as is the fashion * Simple Emulation for a SQL State Machine Shard. Sutable for development, testing, etc. Should have done this years ago - lol * lotta cleanup. normalize default H2 UPPER_CASE default behavior. Eliminate YML as a liquibase format. Geebus. Add changelog resources from defined {path -> URL} mappings. Fix a lot of stupid stuff in Mutator around Call, BatchedTransactions (yay!). Model now depends on everything (lol). Intialize model genesis transaction (i.e. the combined schemas liquibase migrations), test ShardedKERL for basic functionality. * Async Delphinius API First cut at at CHOAM mutation for Delphinius Oracle in the ShardedOracle implementation (tl;dr: that test currently fails) * import cleanup * Update README.md * fix sharded oracle test * sync * Remove MVLog based KERL, just use in memory maps. Serialization never worked anyway. * Stereotomy cleanup. Get attachments for event, get KERL of identifier. * Optional api conformance * forgot these * Spring cleaning * Access KERL on controlled identifier * Paramaterize the MvStore for CHOAM * Work on defining the bootstrapping of the Node more work on Stereotomy, fleshing out the rest. * Flesh out some remaining impls, etc. * Add type variables for identifier construction, etc. Lotta hair on that ;) Integrate fake provisioning of the member. * typing * moar cleanup. Generisize identifier creation in Sterotomy - prolly needs a bit more work, though. Remove unneeded publicKey() from Member - lot of fallout from that * Node simple smoke test. Bootstrapping with KERL. Some fixins Test simply boots and waits. * remove unneeded interface * Update to H2 2.1.210 * Update to JOOQ 3.16.2 Remove need to use a seperate version of H2 for generation * update bouncy castle 1.70, Dropwizard 2.0.28, Jackson 2.13.1 * First cut at non multi-tenant demo of RBAC service * flesh out implementations * combine. implement check * use DELETE for remove/delete * Refactor to seperate runtime parameters from constructable params * Moar split of runtime/configtime. Can now use for YAML dropwizard config * Further fleshing out the demo API lol. Proto over HTTP for the KERI resolver

2021-12-25

Liquibase, Stereotomy and Delphinius (#60) * fix up event publishing model for sql state * initial resurection and clean up of the model module * Initial liquibase and jooq integration Enabled and generating correct schemas ;) * Add Liquibase Migration txn type. Fix protoc platform dependency. update readmes. update liquibase and jooq versions. rename cdc.proto * Fix multi thread safety issues with Liquibase Scope and manager. * Add prefix to local routing. Fix SmartMap concurrency. Use only 1 fork in sql-state, as I like green builds rather than random flappers due to happenstance. * simple liquibase update test Refactored jdbc connection wrapper from model to util. added liquibase version to prevent closing connection. * differentiate between the two timers * update/tag/rollback * flesh out the Mutator DSL support for migrations * Integrate migrations into genesis transactions for testing. * update readme * Stereotomy integration with choam and sql state mangement Clean up stuff not needed * some clean up for Migration mergin * First cut of KERL schema * upgrade H2 unleash glorious snipe hunt for deterministic overrides. Seem to have created a plug in the async choam logic, so investigating that. Interim commit so I capture good work * lot of fixes. fixed ethereal to only choose parents at previous levels. fixed issue with the capacity batching queue to respect total allocation. refactored view change to be linear with the last epoch - may refactor back to overlapped ensure less unknown view, etc by waiting until the producer node sees the assembly block. Assembly block now emitted at the beginning of the 2nd epoch of the producer. Currently no txns accepted in DRAIN state of producer - may refactor * use full capacity (all epochs) no op out drain mode rejection. warn if buffered txns are dropped. * Revert "use full capacity (all epochs)" This reverts commit 4ca0e1d9eb9209cdc1d20b4303311bfeada488c5. * :: sigh :: Now seem to be limited by H2 SQL parse overhead ;) Which isn't horrible. * switch back to windowed/gradient rate limiting. Basically boosted sql state txn rate by a factor of 2, with no tuning ;) * fix comp errors * tweaking * lotta debt in stereotomy. Unified KERL DB implementation start. * moar prototyping of UniKERL. Fix stereotomy as required. * Key rotation in UniKerl. Associated fixes in generic stereotomy. :: SMH :: * 2 rotations * multiple identifier testing * Add retrieval by coordinates testing * add check on coordinates.ilk to validate * refactor, cleanup * UniKerl now active for non attachement events * test SQL Append API * waxing the test * Unified KERL in Stereotomy testing * fix name case * Clean up KEL api for append(event) only. KeyEventProcessor now part of the KEL, by convention. Fix up liquibase logging in stereotomy. * Support multi signature identifiers * Clean up specification apis * test renaming * Refactor to consolodate interface through the ControllableIdentifier Provide an implementation of CI that uses the current KeyState of the KEL, as well as a Bound variant, that is bound to a particular KeyState for a given Identifier. Pull out nested types from Stereotomy * Extract to interface. * Tighten up interfaces. Controllable identifier now tracking key state changes, rather than looking up. Use Optional to indicate the many failure results. Generate all signatures configured for a key when event signing. * binding accessor * Clean up basic identifier and Verifier api * Remove eroneous AID. cleanup * comments n' cert provisioning using basic identifiers * Provisioning certificate creation/encoding/decoding for an identifier * Use KeyCoordinates for certificate provisioning and bootstrapping add a bunch of java doc * add verifier access to the KEL, more comments * Remove PublicKey accessor for Verifier * Refactor to sane multi-signature support. Geebus. Good stuff, though. Big plug because Signer and Verifier sit at the bottom and are used everywhere, for obvious reasons. SigningThreshold is enforced, and signing/verifying in general should now be multisig transparent. This simplifies chain validation and this significantly clarified and simplified the event receipt and witness structure for the KERL * cleanup. work on stereotomy cert validation plugin * fix * wax * Add Delphius Datalog -> SQL module. Stereotomy certificate validator. * Simple syntax exploration smokin' test * interim add diff datalog grammar. txfm'd from python version (lol). baseline before refactoring * interim * moar work on sql stuff * fix add logic * remove edge.resource * Optimized edges and insert logic * Add domain types for parent/child * Reify the model. Map(a -> b) - create edge for Object, Relation, Subject domains Add(a) - create Object, Relation, Subject or Tuple Delete(a) - create Object, Relation, Subject or Tuple * Add check(Tuple) api. Update read me Everything is namespaced, natural API * Fix Subject to use namespacing * update readme * Add delete()s. Add remove()s for mappings * pull out static constants * generalize current block to current {block, txn} * fix n wax * refactor to include Relation in Subject and Object. * minor wax * Switch to a more RDF/Zanzibar tuple and assertion model * renaming * rename * wax * Fix subject resolution. flesh out test some more * forgot these * Read API read(Object) - all subjects that can reach the Object read(Relation, Object) - all subjects with predicate that can reach the Object * Oracle Read and Expand APIs * format * cascade deletes for everything except ye edge lords * Edge table constraint maintainence * cleanup * wax * correct non null constraints * some comments * first step * Refactor to seperate read/write actions. Refactor to static form for access from stored proceedures and functions * Add read and expand for subject Expand is currently not working for subject * break out static analogs for SQL Function integration * Delphinius stored procedure API. Test both direct and call Oracles Nice * update readme's * update

2021-11-28

Liquibase migrations (#61) * fix up event publishing model for sql state * initial resurection and clean up of the model module * Initial liquibase and jooq integration Enabled and generating correct schemas ;) * Add Liquibase Migration txn type. Fix protoc platform dependency. update readmes. update liquibase and jooq versions. rename cdc.proto * Fix multi thread safety issues with Liquibase Scope and manager. * Add prefix to local routing. Fix SmartMap concurrency. Use only 1 fork in sql-state, as I like green builds rather than random flappers due to happenstance. * simple liquibase update test Refactored jdbc connection wrapper from model to util. added liquibase version to prevent closing connection. * differentiate between the two timers * update/tag/rollback * flesh out the Mutator DSL support for migrations * Integrate migrations into genesis transactions for testing. * update readme * Stereotomy integration with choam and sql state mangement Clean up stuff not needed * some clean up for Migration mergin

2021-11-21

Opti (#56) * Remove use of exceptions to signal txn back pressure D'oh. * some temp testing * Update maven deps. Revert to grpc netty shaded for simplicity * interim * mostly gone * die est flappers

2021-11-18

Clean up (#55) * Remove all thread pools. Use seperate scheduler for txn submission yi * garbage collect the old * remove ghost * missed this

2021-11-17

CHOAM (#53) * Unify WeakThresholdKey ABFT Random becon logic with mocked Really hated the hacky way. Now Ethereal respects 2F+1 votes to advance the epoch, but doesn't require the WTK shares to be real.' This serves as a placeholder for any further work on the ABFTRB * fix epoch generation and preblock output. testing 5x5 now * tighten test * lol * clean up controller * Reliable Broadcasting broken out as a primitive messaging service. Should have done this long ago - lol -HSH * Initial CHOAM module * Revert to the more KERI style of wrapping witnesses around the block * Clean up. Replace with consumers and queues where appropriate * remove unneeded flag * Ethereal bugs fixed, tested with reliable broadcast local sim. 50 fully emulated nodes, 3 epochs, 10ms gossip. Geebus, this thing is a monster. * fix async race condition * add guard condition, remove test * union routing metrics * add gossip duration metrics to the reliable broadcast * Use protobuf ByteString for unit data in Ethereal * interim moar CHOAM work. FSM, etc * Interim Strategy is next view reconfiguration initiated and generated by current view. * Moar work Add synchronous endpoint service/client for CHOAM Terminal RPC * moar flesh * Remove miracl, jcsp from repo. * slog through just setting things up to instantiate correctly. lol * moar * Interim. View regenerated, accepted genenesis Now comes the easier part * Full genesis regen through reconfigure * moar work * paramaterize for f+1 consensus in CHOAM * Ethereal configured for F+1 consensus, rather than 2F+1. * Tone down ye logging * further waxing * Get midieval on concurrency in ethereal Fix a bunch o' bugs in CHOAM. Now working with 101 nodes, but with 1/3 byzantine (haven't tested 1/2 yet) * Use 50% byzantine for CHOAM Producers Hehe * fix wtk threshold - ugh. revert back to 1/3, as I don't know where the bug is * clean up * Refactor protocol to broadcast produced blocks and witnesses seperately Much better ;) * modify old consortium txn exec to simply a consumer of blocks Up ReliableBroadcaster buffer max age to 2 * TTL + 1, which I belive is ye correct answer according to the FF paper, but I keep refusing to believe it. ;) * revert protocol change. ugh. * moar concurrency midievalness. :: sigh :: Too many queues, so I'll eventually refactor the lot of it. Semantics are way different than go's green threads... * minor cleanup, prep for continual view regeneration * interim * A long strange trip to view reconfiguration View reconfiguration refactored to its own FSM. Use Ethereal instance to come to consensus on the next view, emitting reconfigure block. Tried really hard to mock for single node testing, but comms is complicated to mock. Will have to work on that. * add finally to close local comms * Replace Genesis regeneration with ViewReconfiguration * View regeneration now refactored to ViewReconfiguration FSM Woo hoo! Eliminated some Q's from Ethereal. More to come * remove trigger q * remove linear q * moar better * Bootstrapping subsystem * Checkpoint assembly subsystem * integrate bootstrap/restore framework * cleaner shutdown of producer * integrate bootstrap/restore * cleanup * interim * interim * moar integration of tx data source and Session * Bit of refactoring * update grpc/proto * Successful round trip smoke testing of transaction submission * require signing and nonce. Use Sig -> Digest txfm for Txn Hash * cache sig algorithm * debug tracing. adjust timeout.remove class for name overhead from grpc * disable the huge class for name time sink in grpc channel creation Man. * correctly tune data source capacity to configured ethereal epochs * multiple clients per node txn processing testing * CHOAM now with integrated view change. Simplified things drastically by using the txn blocks for consensus on joins from next view. Pretty smooth, as there's now no transition latency around view reconfiguration - it happens " in parallel" with the normal transaction processing. * Don't use shutdownNow() This was causing the sporadict NPEs and other grpc failures * minor * Green build again ;) lol * moar work on checkpointing, move to synhronizing combine with fsm state Checkpointing is triggered by a countdown. Reconfiguration blocks have a count which is decremented each reconfiguration. When this reaches 0, the first block generated by this reconfigured view is the checkpoint block. * temp * Nice first real cut of CHOAM. View changes are now performed inline, using the transactions of the view to come to consensus on the next view ;) Fixed a lot of issues, streamlined a lot of locks and executions - more remaining, no doubt. No rate limiting, etc, on txn submission - that's next on the plate. Cleand up state machines in CHOAM and simplified the leaves. Can likely do more. View reconfiguration ala Genesis is broken out and is planned for future recovery regeneration of the view in "we're all dead, Jim" scenarios n' such. Still need to implement the equivalent of SYNCH from Consortium, to force rebroadcast of each member's state for liveness. That will be tricky. * remove unused rpc * import the netflix grpc limiter library. ugh. but very good. not available on maven repo, of course, gradle build, not updated to modern... anyways, now is incorporated and working. Really, really, really improved the behavior of CHOAM under high load (>5K clients for each of 51 nodes in a cluster, can do 7K TPS without breaking a sweat, which was nice to see) * up the client count while digging up dirt * unrefactor * add rate limiting on submit * refactor golang legacy * Add SYNC to Ethereal consensus usage in CHOAM. If we haven't produced a block within the TTL of the underlying messaging, rebroadcast the individual state generated by the receiver. Remove resilience for J stuff. Cool, but weirdly excessive * Green build remove flappers. * Integrate checkpointing Firm up some state distribution stuff * expand buffer * Interim, befor Ch-RBC * interim * introduce generalized CH-RBC message * interim to figure out stuff * refactor to class * pass down ch-rbc messaging * interim. introduce ch-rbc logic into ye adder * fix compile * tracking data structures, etc for CH-RBC * use waiting PU for tracking any pre/commit votes * moar cleanup, prep for ch-rbc * Remove unneeded pipelining * cleanup * Falcon A' CH-RBC smoke testing ;) * Fixed ch-rbc protocol errors, cleaned up tests, etc created units subject to same ch-rbc as units from other processes. Still have 2 flappers: a) preblocks differ in data at some mid point b) stall in closing. b appears to be due (in the use of the rbc module) to just random crap, and the chain produced by the stalled unit matches, but it has not transitioned the epoch. Annoying. a is undoubtedly related, and is likely due to some logic bomb I haven't defused as of yet. * test fix * Revert "test fix" This reverts commit f0253bd2c4562a30dddb313141bd76665cce6c44. * Revert "Fixed ch-rbc protocol errors, cleaned up tests, etc" This reverts commit 070d93d0641ea9a371f0fab789ae47831b5abfc0. * Revert "Falcon A' CH-RBC smoke testing ;)" This reverts commit 425f414c1f5fb83b8c3ab54fcb1e80c961427592. * Revert ch-RBC Cannot make it work without massive failure. Pity. Reverted to previous partially synchronous version, preparing to use strict gossip for missed proposed units, etc. Will implement alert protocol, as well, to prevent fork bomb * interim. Fixed some major problems, to do - of course - with calculations of super majority and majority use cases. Also tagged the nasty bug where parent selection (and thus Creator.ready()) was not ensuring 2f+1 parents for levels 1->N. Ugh. Next up gossip to fill in missing units in the DAG for large groups. * moar interim clean up, some opti * use signature for hash of Unit * Rectify use of tolerance level Geebus. how did any of this work at all ;) * Refactor Orderer.add for topological insert * Exclusively use gossip for DAG replication. In retrospect, the rbc was highly inappropriate * Remove bi-directional gossip in RBC. Concluded the 3rd leg is dicey. It's not an rpc initiated by the entity being updated, so I don't think it really works out from a security/risk pov ;) * Use gossip for underlying DAG state replication Add memberships communications for ring based gossip. Communication mechanism now orthogical to operation. * interim Consolodated strategy - use Ethereal for all linear ops - is working for the view reconfiguration. Moving on to Producer * ViewRecnfiguration special case implemented with new regime * interim * update, move to unshaded netty bunch of clean up. Unfortunately, hard coding arch in grpcc temporarily * missed * interim * TestChoam.regenerateGenesis() now "working" Flapping with mysterious validation errors. 2f+1 to the rescue to cloak all dat - lol. * interim * Debugged the view change/assembly fiasco. Back to normals, green CHOAM, async all the way baby. * huzzah! CHOAM is back, and better. - asynchronous producer/view assembly - super solid txn metering/rate limiting - single gossip for Ethereal consensus Still uses RBC layer to distribute finalized blocks to group. CHOAM group and committee should now self heal from partition events. * fix random variability in bif testing, remove scratch test * harden * debug stuff. membership test proto passing in CHOAM Have a problem with the GossiperContext check for valid member, disabled for now. * fix flapper * track timeouts * interim * interim * CHOAM genesis bootstrap test GREEN! * clean up logging * deprecated * interim * Add ExponentialBackoff utility for clients * Add support for async calls * Lotta work on txn submission conditioning. Backoffs, retries... oh my. Again :( Still a lot of work to do, but it is at least tolerable :: sigh :: * Trim and slim and bim bam bim Candidate for merge to main Avalanche -> Ethereal Consortium -> CHOAM SQL state now on CHOAM Async and binky dinky dink * clean up ;) * readme update * emphasis on tiny and annoying

2021-07-22

Ethereal phase I (#52) * Simplify causal system using polymorphic proto Generalize Ghost to any type of Stamped clock (e.g. TimeStamped - nano level - uint32 and uint64, for the current time). * Firm up ghost security by passing partner identity Move CausalMessage to utils. Integrate CausalMessage with Ghost. Add ghosting(GhostChat) rpc to the service to facilitate the epidemic spreading of the Ghost system causal clock. * Aleph interim * Moar eye bleeding * add test data * moar eye bleeding translation This time, with active golang debugging ;) lol * First smoke testing of Dag construction/function Geebus ;) lot of work tracking this down. * moar fixin' multi-level dag and new testing * moar work. Transitivity of Above testing green * fill out remaining DAG tests. Still have 2 tests disabled, pending future functionality and decisions * finish remaining. all working 5x5 but the tests dealing with forks * moar work. starting on the common random permutation * Fix problem with pids iteration in permutation generation - d'oh Steady progress * CommonRandomPermutation now 5x5 * TimingRound fleshed out * UnanimousVoter and SuperMajorityVoter imlemented * Extender implemented * Extender tested. Cooking with gas * Add MIRACL Java library for BLS, etc. * Add tests fror BLS24-479 * Breakout into seperate module, rename. * Interim * Moar on Creator, Adder, Orderer * buff a bit * firm up construction of orderer * fleshed out Ethereal. Still need WeakThresholdKey, Beacon and Coin impls * Further flesh out creator * reformat comments * move test resources, add logback * first Creator test passing ;) * refactor to use arrays, rather than lists, as is correct. * bite the bullet and add JCSP to deal correctly with channels. :: sigh :: * Stage 1 generic * revert these * Moar work. Think Ima stop this for now ;) * clean up tests * Clean up, order, organize imports, format * Use a simple channel abstraction rather than flow api * Finish the testing of Creator * Finish out the Orderer * ExtenderService fleshed out * Fini. Now on to functional testing with multiple nodes. * Remove obselete module for pregen population * doc and waxing. removed membership dependency as not needed * Refactor Signer into an interface. D'oh. Should have been this way from the start. * format, cleanup * :: sigh :: * Initialize with mock signer for now * Break out Verifier interface double :: sigh :: should have been done long ago * cleanup, finish up Adder * first bare bones e2e testing of the ethereal pipeline * closer, but still not connecting the pre block sink for some reason * fix epoch changes, e2e test with 4 processes ;) nice

2021-07-08

Causality (#49) * fix use of witnesses in public identifier * Add support for Autonomic and Legitimized Human Meaningful Identifiers That's a mouthful. Can now create identifiers from other identifiers, chain, etc. * remove system property setting now that BC provider is setup * refactor kel/kerl, integrate event processor, fix interaction bug * delete old keystores * Interim work * clean up event coordinates * Refactor seals to be self identifying. add moar seal types * clean up event coordinates, seals, etc. match KERI spec for delegated events * some renaming and refactoring * serialization of key coords to ByteString * Bite the bullet and refactor out common gossiping behavior on ff rings * wow. lots of stuff fix messaging hash generation. consolidate executor usage across the board, eliminate FJPool.getCommon() Simplify consortium testing. Plug liveness holes in membership testing. improve logging - lol normalize context handling of digests/members normalize ring handling of digests/members add timeouts to submitted transactions in consortium * rename gossiper to ringcommunications generalize to handle digest as well as member cuts * Further ring communication enhancements, Ghost work * Ghost is back. Need this for KERI resolver piece. Nice to see it fixed again ;) * reengage intervals gossip on Ghost * clean up dag test so compiles * clean up sql state module * fixed tests. Now back to a green build. woo * vastly simplify Bootstrapper with RingCommunications. for the win * convert checkpoint assembler implementation to RingCommunications * add local loopback for dealing with policies around self calls * fix test * fix successors test * Butt waxing * retire slush * green through consortium * Fix remaining issues. green build. * execute each iteration of the ring through executor to decouple * properly randomize the traversal order of the rings * revive Ghost. fix termination detection in ring comm iterate * Fix diameter calculation. Fix TTL calc. Fix purgeTheAged Use stream instead of parallelStream for testing, etc. diameter calc was a nasty bug. geebus. TTL was another for messaging. * Fix client member id passing in local communications simulation * Fix Ghost * remove func test as it is obsolete and a pita * Interim * Ghost with the most Break out RingIterator from RingCommunications. Handle iteration state wrt failed and achieved majority to prevent multiple invocations on full iterations. * waxing * Refactor Validator,Verifier to pure functional interfaces * restore identifier in inception * include format in Version. * revamp message signing. signature of message is from whole message, not hash. id of message is the hash of the signature. * Cleanup * Add eventually consistent mappings to Ghost API Clean up local loopbacks * update doc * Use digest of key to store mutable value * Add eventually consistent mutable bindings to Ghost * update readme * hmmm * Fix ring iterator terminal iteration bug. Move to JDK 16. Update mockito to modern version - lol * update readme * fix termination. whoopsie * clean up * Add initial version invertible bloom filter and strata estimator need to refactor strata estimator further * add benchmark test for IBF + strata * Fix twister hasher * Update Hash to new regime based on IBF work * Unify hashing strategies. Simply hashing strategies. geebus ;) * Clean up strata estimator, tighten strata est benchmarking * remove obselete * docs, copyright * thorough clean up/organizing/formating Yea. that's right. * ha. have now figured out how to reuse proto across modules. No more ByteString encoding! lol Messaging first * better calc in H(k) processing. Better encoding of Biff * overall proto files using newly found knowledge of xmodule includes Almost everything is now type safe at the proto definition level * Clean up the rest of context free bytes ;0 Types, f*ck yea * seperate services from core * minor waxing, kinder tests (getting to be a long build again) * buff the hashing of all things. Also, too, solve some concurrency flaps only check filled locations (do'h). move function invocation out of the generation loop, back into caller (double d'oh). Don't keep allocating instance of Hasher as there is only single threaded access. Much improved. Probably close to optimal now. Fixed flapper bug in bootstrapping, where the logic of view synchronization got flubbed - the initial view can be non null and checkpoint height 0 (genesis bootstrapping), so the logic would incorrectly try to assemble and complete the non existent view chain. Hopefully nail some of the other nasty flappers by server.awaitTermination() in the routers. Can likely add yet more scoping to the local server id required to make it more idiot proof for testing. * D/Encoding of IBF * reorg into new package. Fix hash processing for integers * disable profile test for strata * further waxing * fix digest last to be -1L, as unsigned long. * accomidate for package move * Add Bloom Clock * De/serialize BloomClocks * Fix Bloom clock. Realistically simulate the Bloom Clock paper scenario Did a lot of work around the prefix rolling of the BC. Needs another revision to pack nibbles into bytes * fix DigestAlgorithm.orginLast() initialization. d'oh * Bloom Clock generalized to values that can be passed around and compared * Refactor Ghost in prep for shift to Bloom Clock causal processing * fix compilation errors * Moar work * Fill out Bloom Filter types, refactor testing package * revert to build() * first draft of byzantine causal broadcast * moar flesh * moar work * fix * successful bare bones smoke testing of causal messaging * moar work, progressing nicely * optimiso * fix tests * Moar excellent work on byzantine causal broadcast. Have now integrated the basic bcb with the bounded buffer. Optimize reconcilliation using youngest age messages first. Message age is merged when receiving messages and is incremented every gossip round of the messenger. The max age of a message is the Time To Live of the Context's fireflies rings (diameter * 2 * k + 1). * refactor and wax * buffersize big enough for test * moar refactoring, waxing and simplification/consolidation * revert to kinder parameters for build * f * Fix Hash generation for small M, M is even, and even for common factors Geebus. * Add stamped clock abstraction, allowing for more than just timestamp * interim Add analysis and testing for hash distribution, avalanche effect, etc. * minor fixin's * Purge the aged on tick update of age * tick in background * fixi * kinder params for causal ordering test, disable tests in progress green build. * fix occasional flapper * buz hash for ensuring seed is teh good * causal processing as a stream * back to green. Geebus. That was a bit of refactoring. Cleaned up the bloom clock stuff considerably Cleaned up causl processing. - don't send messages sent by a node back to itself - node filter out received messages sent by the node - calculate size dynamically - an inc/dec is always a mistake - clean up causal processing - remove locking, use locking queue per stream - event hash is now the signature txfm into digest (xor halves) - clock is now part of signed causal message - break out envelope from causal message * update readme

2021-06-15

Stereotomy (#47) * initial creation * much refactoring * generic * interim refactoring towards a better metaphore * breakout * interim * better exception handling on batchedTransactions * tests use new Mutator * use refactored statics * convert to Mutator * update to Mutator * Biting the bullet on support for multiple digests/signing/etc Initial support * add tests and license * Clarify design Digests and signatures are enumerations * Renames Because why not * tastes and colors * waxing * refactor a bit * d'oh * use formal getters in prep for proxying to protobuffs * grpc implemention of key event interfaces * refactor a bit * refactor base crypto to utils, seggregate from DID/SSI * seggregate into base and Identifier Qualified functions * moar ruminations on MVStore o' event state * flesh out state storage * moar fleshing out Add state store basics, add basic specifications, basic operations on the controller * buff * reify EventCoordinates * readme for stereotomy and fix readme for sql-state * i hate markdown * Add Verifier, Validator Clean up * Buff. Trim api, refactor to consolidate processing. * cleanup * clean, elegant. Works like a charm * update base readme * wax * add environmental awareness * fix miscategorization * Inception event lifecycle (local) working and tested. Now for my rotation act... * Inception of identifier debugged and tested. Still failing on rotations, etc. Tests and cleanup for crypto, encoding, identifiers, etc. * add self signing/signed certificate generation * update grpc * refactor to util module in prep for conversion to Stereotomy * moar refactoring. get ED25519 keys working with TLS. * moar work on sterotomizing basic membership * Whole lotta wrangling Nailed Bouncycastle provider configuration - finally. All algos using BC provider, including MTLS. MTLS over Ed25519 working, refactoring to these certs, etc. Refactored membership module to use Digest as IDs. Still a lot of work to wind through that change. Memssaging test working, but MemberOrdering test hanging. Need to fix that. Man, this was a lot of work. * Refactor bloom fliter to utils. break out Biff proto. Lots o damage ;) Add rudimentary testing for Bloomfilter Digest * Fix bloom filter hashing logic. fix logic for Context ring K value Add better constructors to Context to create FF bysantine tolerant rings from parameters. Work on MO test. * Interim * fix member order implementatiion. * Finish refactoring Fireflies. MTLS test doesn't work, because of course. Refactored FF protocol to own proto module in FF. Abstracted FF node and member information to wrap members, rather than implementing the interfaces. Moved Member, SigningMember to interfaces with corrresponding impls. Lotta work * switch back to SecRandom * revert logging * Avalanche converted to Stereotomy * Refactor Digest to array of longs. ByteString efficient encoding. Prep for refactoring encoding of digest with bytestring efficient encoding rather than QB64 encoding. Revamped equality and hashcode. Refactor Messaging proto into the messaging module * messaging, fireflies converted to new bytestring encoding Avalanche proto moved to avalanche proto. Ghost updated to Any, rather than DagEntry * fogot to remvoe * refactor avalanche to Digest bytestring encoding. optimize key for MVMap in working set - lol Back to 5K TPS. * whoops * convert Stereotomy to Digest bytestring encoding * Signature ByteString encoding * Messaging using bytestring encoding for signatures * fire flies converted to signature bytestring encoding * Interim on refactoring Consortium to Stereotomy framework * refactored consorium to stereotomy, interim * Interim. Flapping on membership tests. Flapping because not all txns are processed. Ugh. * Fix consortium membership testing. It was the confusion of cardinality all along. LOL. Cleaned up quite a bit in messagng n' such Amped up basic messaging test for a 10,000 messages total (100 members x 100 messages) with a buffer size of 500. That'll do, pig * SQL State now converted to Stereotomy and Tested. Yes, this deserves Caps. * ByteString encoding of Identifier types * Stereotomy now using ByteString encodings for sigs, identifiers. * remove modules frm build. Not the right model, anyway. no longer useful Need to stop porting this until I'm ready for the web access exposure model. * Trim system to eliminate modules that are no longer working. They will change quite a bit with the work currently going on. Work on ghost, but add it to the remove lists. More important things to do. * buffing * key rotation now implemented and tested. * Stereotomy implementing KERI direct mode Revamped protobuff encodings to match signed specifications for each of the three events. Good enough to for orthogonal iteration on the controller and delegated mode. * clean up * Add testing for new public identifier * cleanup. add to readme

2021-05-21

Yes bootstrapping (#45) * make time functions deterministic in H2 Add beginnings of deterministic JVM use for functions. Smoke test for teasing out construction/setup/use of djvm * additional * integrate more of djvm setup into Functions * moar djvm work. bit frustrating until I crack the wormhole I need * Interim * interim * do da work to translate to sandbox.* classes for java.sql and h2 Trigger * use methods for reflect * start bridge work on connectioin * Interim * interim * interim work on bridge So's I donna lose it * moar wrappin' * moar * moar * moar * moar * First cut at jdbc from djvm lol * Seems to be working, finally. Need to go through casts to sandbox.java.lang.Object[] and remove those * clean up Object[] * full e2e smoke test ;) * readme updates. wrap aggregate/function to round out api * initial import (unchanged) * Forgot about these wrappers * Functions, triggers, scripts all run in Sandbox * initial import * initial import * initial import * Wire in deterministic functions, stored procs, and triggers. Woo hoo! Even had a volunteer test in the event publishing ;) * Fix Publishing function for deterministic * ¯\_(ツ)_/¯ DJVM is gplv2, so removed. :: sigh :: * for da future * fix annoying concurrent mod ex on shutdown * store ye certified blocks instead of just blocks in current state distinguish between consensus state and proposed state * Rename * interim to txf envs * Bump bc.version from 1.65.01 to 1.68 Bumps `bc.version` from 1.65.01 to 1.68. Updates `bcprov-jdk15on` from 1.65.01 to 1.68 - [Release notes](https://github.com/bcgit/bc-java/releases) - [Changelog](https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) Updates `bctls-jdk15on` from 1.65.01 to 1.68 - [Release notes](https://github.com/bcgit/bc-java/releases) - [Changelog](https://github.com/bcgit/bc-java/blob/master/docs/releasenotes.html) - [Commits](https://github.com/bcgit/bc-java/commits) Signed-off-by: dependabot[bot] <support@github.com> * Interim * fix whoopsie * bootstrap by super consensus in Ava (or whatever). Assume Consensus is correct, and bootstrap by retrieving the last checkpoint, view, genesis, and N blocks in the view chain to ye genesis Trust is in the parent consensus, so... * Interim * Bootstrapper smoke testing working. Refactored block and body constructor methods to statics. Next is to build out the testing to shore up the design * moar work on bootstrapper * fix view chain iteration * Long test chain checks, multiple reconfigs and checkpoints * simplify test chain building. thus making test more understandable * bootstrapper working test genesis/checkpoint/anchor synchronization. Still needs a lot of work wrt validation and error handling, but good enough to progress. * additional conditions * add validation of the anchor chain * Validate view chain. Damn. Nice. * fix change made to block construction * Exception handling for composed. Dunno why the exception isn't passed through, though * Combine bootstrap service into the main consortium service * Pin to genesis and > checkpoint when restoring from persistent state * break out into interface long have I resisted. * Recover consortium block state from store, if available * moar work on bootstrapping * interim * interim * synchronizations * f * Interim Testing scenarios for bootstrapping * Woo hoo! Successful smoke testing of bootstrapping from genesis. No checkpoint testcase * update base readme * update readme * add sql state * Smoke test for restore from checkpoint nice * complete checkpointing * redundant * clean up genesis view establishment * sql-state genesis bootstrap had some funky monkey going on due to hashedcertifiedblock not using hash(block), which, of course messing things up dramatically. Cleaned that up by only having constructor for block, no additional. cleaned up bootstrap via genesis only, no checkpoinnt * genesis bootstrap sql state smokin' needs to complete on block height, then validate sql state * genesis test bootstrap buffing * Bootstrapping from genesis and from checkpoint tested and working Woo hoo * turn off trace logging * fix test * update readmes Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

2021-03-23

Bootstrapping (#44) * replace all uses of thread pools in tests with fj pools for local comms * remove * better threading/testing jvm args * fix the flapping bootstrap test Only use one FJPool for the local sim in config * fix flapping bootstrap test. The issue is the seed discovery protocol for the initial server - the first - which I'll look at later. For now, just expand the rings, so each seed is hit * interim refactoring * fix segmentation error * Generalize Bloomfilters Simple, efficient, extensible design. Use GRPC Biff for serialization. Randomize seeds as foretold by ancient prophecy. Successfull whiff testing of Bootstrap'in * some more lock fiddlin' * Finish the SQL state interface Parameter conversion for serialization. Add callable and prepared statement cache. * Allow dynamic context of the FSM Provide mechanism to explicitly set context of FSM. Provide mechanism to manage the current context of the FSM via push/pop * Fix processing issues with sql-state. Still have checkpointing woes * Checkpointing fixed. Checkpointing is a synchronous action of the consortium chain, across the membership. * synchronizing checkpoints * fix npe * fix next block looping. D'oh Did not think that through ;) * Add onSubmit consumer to distinguish from committed result completion Tests now gate on completion, not submission * Refactor the local router identity provision, add exception handling Gracefully handle the asynchrony involved when shutting down. This involves handling a null in the remote identity over the channel. * Fix deadlock. Fix npe * Refactor to extract View, stage 1 * refactor out Regency state n' some behavior * fix stuff fix message processing - ignore self. thought that was in, but apparently not in all cases. Fix consortium tests to trigger on txn completion, not submit. Heh. Fix 1st timeout logic. Timed out txns are batched, not just the first 99. Ensure that all txn's timers in the set are cancelled and rescheduled for 2nd timeout * up txn count * haha. yes. of course Various and sundry. Fixes, stability, concurrency, thread cardinality. * back off txn count for tests. Don't reuse forks * require JDK 15+. Add Java Scripting txn for anonymous blocks * don't require javaparser as of now * VM experimentation * Start fleshing out VM * first cut at migrating quantum-db. delombok - ugh. upgrade to Junit5 * First cut at import of quantum-db delombok, upgrade to JUnit5 * add qdb cli, fix tests * clean up, etc * cleanup + work on porting to H2 land Generation of schemas - information and quantumdb - in Jooq * moar progress on porting to H2. Slow migration to JOOQ. Generate Jooq for information_schema and quantumdb schemas in core. First cut of H2 synch trigger - should be rather efficient. * Move quantum-db to a seperate repository No need to pile that in * remove H2 mods * fix flapper * remove tramp from consortium, start event work in sql-state * remove djvm for now * Make H2 deterministic random stuff is deterministic (but still random!) across the nodes * remove * fix tests

2021-02-11

Databasing (#41) * fix block generation to eliminate tiny blocks Driven off block gen timeout now. * commit transactions Do'h * Two sql-state txn types: statement and batch. Consortium leaves txn batching to passed txn implementation * All tables have a new column _BLOCK_HEIGHT_ Modify the H2 parser to inject this new column into create table, insert, update, merge. Session modified to define the __BLOCK_HEIGHT_ value to be inserted/updated. * Pass relevant info to txn exector * Set blockheight on session * Fix txn processing in nodes which did not submit txn - d'oh. Fix parser to use column from table so updates/inserts/etc get processed correctly. * Processing * synchronize block processing through FSM Rudimentary prep for checkpointing * interim checkpoint lol heh * uniform * Add new BLOCKSCRIPT command for creating snapshots at given block height * first pass at checkpointing logic in consortium * fix ava test. fix checkpoint block prev to point to correct block. At least system detected this as a protocol violation. * checkpointing for sql-state woo hoo! this effin' rocks. * rename * fix naming * Use MVStore for block storage in consortium * fix checkpoint temp file in state * finish deflation. d'oh * Interim. Work on replication gossip for Checkpoint blocks/segments * impl * reorg for clarity * moar reorg * reorg * genesis view id, genesis data as Any Prep for next stage * Use initial genesis data to seed state Sql State uses this to establish initial schema, data, stored procs, functions, users, etc. * refactoring store to include certifications of blocks * parameterize geebus, what a lot o params. Almost like you need a DB to keep track of 'em. lol ;) * flush() to synch() * Store checkpoints in MVStore * work on checkpoint assembler work on recovery of view/checkpoint clean up block header, add hashes for above :( May remove the ref by height, though * refactorings, simplifications moar work on assembly * no idea why commit is needed for this * normalize last check point/view change use hash in header. balancing recovery over block overhead :( * think the fnf issue is ye fixed * clean up, renaming * remove Persist. May add later for opti v persist * cache cert -> member id cert equality comparison is pretty lame (array.equals) and so will have to do better, but this is cheaper than decoding * Move to MVStore for all our persistence needs. H2 is built on top of MVStore, so "for the win" * remove calcite * interim refactoring to 2 phase query because I can - seems legit. Round trip is trivail compared to set up and the bandwidth "bet" is a good one. * End of experiment * revert experiment * don't reuse FjPool.commonPool(). * bite the bullet with atomic refs * fix noop txn description - do'h * rethink locking in working set. use mark for traversing nodes * fix livelock issues in Ava * fix for noop purge. opt finalize minor perf fix for tests * reduce lock contention SecureRandom per thread. remove ye FJPool - yi. Block processing in Consortium no longer mediated through FSM. * fix test * fix test failures For whatever reason, the jvm settings (from ye olden thymes) were causing teh effing up. * intermediate save * Finish massive parallelism work Refactor locking and operations to maximize parallelism. Now much cleaner and useful

2020-12-15

Simulation (#36) * Normalize DagNode Grpc all the things * no copy hashing * test fix * working in a proto mine, goin' down, down * cleanup n' work on integration with Consortium * Revamp cdc Unfortunately modify a couple more H2 classes :: sigh :: The change is simply to call to the session to record the CDC action event with the supplied values, so tiny and easily maintained. * clean up and alignment Clean up consortium validator contract * minimal POC. Validation (CDC txn gen) plus application. * fix testing * add initial skeleton of consortium integration testing * commit to command execution, rather than CDC via txn sim/em/ulation :: sigh :: Model is basically serial execution of commands. Commands would be SQL statements, etc, executed serially or (hopefully) parallel. Still, big disapointment, but now vastly simplifies my DDL "issues" ;) lol * continue ye descent * clean up * realistic test sql, info/warn for exception logging * larger batches, calc theoretical TPS

2020-12-04

Cohesion (#30) * Remove forked execution in messaging Queue up incoming messages and deliver them once a round rather than on demand. Alt thread safe priority q * Moar clear logging. * interim. Lotta work on SMR, latency optimal version * better stack trace on errors/etc * view formation, regency change * moar logging tweaks. fix problems pinning on current state changes If one calls another transition in a transition or in an entry action, the current state can get changed due to pushes/pops/etc. Pin the current state before the transition, transition to next state only if the current state is == the pinned state. * no need to log for loopback transition * add member for logging * cleanup. add synchronizing leader state to prevent multiple SYNCs * cleanup * provide simple mech to throw invalid transition, copying a useful msg Clean up the stack trace, eliding one now useless cause * clean up * break out EstablishView submap this encapsulates the machine to establish a missing view, genesis or otherwise * establish genesis view finally ;) * provide ability to forward and transform transitions to pushed state. * through genesis trim down test case to 5 nodes, view size: 3, tolerance 1 debugged message passing, all fine. need more look/see around genesis view change with resetting of the message seq #, but I believe it is correct. Handle cur/next regent. Clean up state machines. * cleanup logging n' such * green consortium smoke test. genesis generation -> 3 blocks 11 transactions. woo hoo. ;) Flaps, though. am tracking that down. * clean up poms * attempting to get > 3 committee to agree ;) * better ordering in TO. Found that - of course - validations could accumulate independent of the block ordering proposed ;) * moar work generate new view id for genesis block. solves issues with old messages from genesis creation. Will work ye same for view rotation. Mark txns as "completed" when processing blocks in group. Eliminated TO message, seems unneeded. Messenger now executes messages inline with updates. MemberOrder consumer processes batches of messages. Filter delivered messages sent from self. Flapping test on last txn processing. Stall. Weird. * fix pom * eliminated flapping. Runs green, afict * Fixed bug in batch creation. Was causing the stall ;). Upped test to 1000 txns. max 10 batches. Pipeline working! * clean up logging * pom upgrades Fix surefire settings. add parallelism except for avalanche module. Much faster ;) * clean up state machines. Prepare for testing new view formation. Messenger now randomizes first delay to prevent synchronization. * Transitions cleanup, ConsortiumContext cleanup * clean up n fixin's Change stop order in pause. Finally crushed the bug where old view messages were still being delivered after the view change and thus different messenger group. Geebus. But combined with a new view id in genesis reconfigure section, this seems to have eliminated that entire nightmare. Smooth transitions now. * ha. did not crush it. Further tightening of ye screws. Modify messenger to pass context id when delivering messages. Same for member order. In consortium, validate that the current view is the same for which the msgs originated. Validated in testing (but very hard to catch). Tighten up the timing of the view change. Currently there are only 2 guards against stale messages. One is stopping the messenger from the previous view, but concurrency being what it is, in flight processing needs to ensure that it lands in the correct processing state. So the other mechanism is the currentRegency checks. I've been tracking this with invalid transactions, and that has narrowed the gap significantly. The currentRegency checks seem to be batting away any stale messages that slip through from the old view. Note, this will only happen around view change, and only from the previous view. We generate genesis with current regency -1 and thus next regency of 0 for the view establishment and genesis block creation. Upon processing the genesis block, the current regency is set to 2, with next regency -1. This should catch any that make it through the view change gap, and so far the tightening of the view change timing and the messenger changes seem to have plugged the hole tight enough. Should be a significantly less problem with higher gossip durations - this test runs @ 100ms, which is pretty tight. * tone down the test until I can figure out the mem stuff Dying in maven from CLI for some reason I can't SO * Add sync/stopdata transitions to leader/follower should have enough handling to ensure correct delivery. Still need transition logic for handling real regency change events. * Lotta concurrency and state debugging. Works, except for buffer flush on overflow ;) Purposefully made the batch timeout duration huge to provoke the constant timeout * do not use fair sync * use fair sync for view change * moar tightening. with test parameters, current state machine will form genesis block and then never process another txn, cycling through multiple regency changes, successfully. * cleanup'n * general increments * holy shit snacks Green run on consortium test again! Major fix was submitting a proof for a zero length log! Excellent. Thought I was doing this already - lol ;) * expand buffer. Fixes issue with flapping by dropping some transactions. Suspect the transaction simulation/batch processing logic of course ;) Off by 1-3 when flapping, consistently. Likely ye post or is it the hole? * clean up. warn for test * A valid path Reify protocol states in ye FSM. Delay messages targeted at future regency changes a node has not seen yet. Massive consolidation of FSM logic, moving to Transitions default methods. Still fails testing, but this is largely because of non terminating regency changes due to timeouts, logic and - well - idiocy on my part. Still, this seems like the correct direction * interim. Consolidate behavior on transitions - eliminate useless dupe's. Yeild to the stack, pushing regency changes accordingly. Filter out and delay synchronized messaging around regency change for "future" regency changes not yet processed. Brings us back to txn processing again ;) yay. Seems to operate through the regency change with producing blocks and doesn't seem to miss a step until the tail of the test. * holy flurkin' snit. Green smoke test. Did 10K within timeouts. Lol. Tighted up state rules and reimplemented Timers - use Duration, etc, rather than round ticks - lol. * clean up. good params/balance for now. * fix dep issue

2020-11-17

Consorting (#29) * interim. Move buffered messaging logic to membership Refactoring to make messaging a distinct service, independent of fire flies - building on top of it, instead. * Finally produce seperate testing module for pregen certificate groups Took me long enough * Finished refactoring messaging out of base FF * Further consortium work Peel more off of View to Context Ring is dependent only on hashing function :) duh. * moar fleshing out of SMR with FF consortiums * Starting to fill out the State machine interface * Total ordering of intra consortium messaging * fix FF * refactor to reuse pregenerated test population certificate/key pairs * interim commit on Consortium work. Further fleshing out of protocol. Building smoke test to drive further dev. * Iterim. Refactor communications to correct routing * Finish up Router refactoring. Revert back to using Apollo's pregen population, as the test-population does not have the correct key store :( May fix later, but prolly not as the expected keystore has seeds predetermined, which isn't useful. May do some transform, though, so can be reused. * Add correct start/stop logic for de/reg of routed service. Correct Consortium messaging view logic for the 3 cases - follower, leader, client. Correct pause/resume and state logic for comms + messaging/TO * Move TotalOrder to messaging. Add sequence # to Msg, remove channel * Refactor TotalOrder to messaging * interim * Fix garbage collection of messaging buffer. hehe. nice. * update readmes * streamlining * fix stubbed ordering service. forgot the onComplete. D'oh * discard on merge, if too old * parallelize * further state consolidationing * Client submitted txns to consortium * fix diameter calc on context. Cleanup * initial import of Tron * Simplify with the FsmExecutor * minor * remove useless lock syncing * Convert to Tron driven FSM for Consortium * clean up tests * use less rings. moar state expansion * remove unused * restore sync. not so useless after all ;) * moar tuning of gc logic * block certification and validation logic Add TickScheduler for timers based on gossip pulse of messenger. Clean up PendingTransactions, make doubly linked list in prep * Heavy refactoring for clarity and responsibility. Verification/signing. Will need to consolodate crypto utilities, but not now ;) * moar refactoring for compactness n' clarity * interim kind of a mess. lot of refactoring. got sidetracked into revamping MessageOrder (nee' TotalOrder) because I need at most once ordering, not necessarily exactly once and in fact, exactly is bad because of skew ;) * Transition to generate genesis Group formed, joined, consensus keys validated and join transactions submitted * genesis block generation and validation Don't deliver messages from self to message handlers in gossip exchange * interim * Formation, generation, validation and publish of genesis block. finally. * generate genesis and process from consensus ordering. fix signature validation stuff * cleanup * revert loglevel * fix pred/succ logic around dynamic membership * tighten up committee selection. Tighten up testing. * Clean up transition logic, start/stop, etc * Split out generation of genesis into submap FSM Sweeeeeet * Shutdown on PROTOCOL_ERROR * Allow loopback transitions in pop * Refine endstate of members of genesis creation to JOINING_MEMBER * Use next view consensus key for validator when generating genesis * Use compressed Body contents for Blocks * decrease gossip duration to speed up test cycle * refactor test for read-ability * handle volatile cases correctly * Fully genercize Refactor out Enum<?> * moar work on volatile consistency - lol. work on txn submission -> ordering -> validation -> block production pipeline. * add name to logging * cancel timers, as push does not execute exit actions push isn't a leaving of the state, rather a nesting, so FSM does not execute the exit action to kill the timer. * fix inversion in recovering D'oh * handling of pending/unreplicated * better signing, no copying * Full round trip of genesis -> submit -> user block -> complete txn However, much remains in error as user txn blocks seem to have multiple copies of the txn ;) * interim Fix deadlock issues * Fix validation generation on leader for non genesis blocks * Clean generation of 1st block, txn completion lifecycle. * manage unreplicated better * remove from unreplicated when previously seen * tightening up * Bite the bullet and use grpc Any to enable polymorphic messaging. Should ahve done this a while ago, frankly. * Generate additional 10 txns, wait for block processing of them. Woo hoo

2020-10-23

Fix hash key comparison - unsigned. Fix proto for ghost (#27)

use long[] instead of byte[] for HashKey (#26)

2020-10-21

Consortium (#23) * Remove extraneous ports * Make local comm sim work like every com, rather than a singleton * add large (10K) local simulation test. Reduce thread use in FF * further reduction. Ghost still failing, though * disable ghost testing until I can devote attention to the poor thing * reservoir sampling * Use random ring to sample * Clean up * moar metrics. * Optimizations from profiling. 2x preview TPS rate ;) 3.1K TPS in 31 node local sim testing on 8 core MBP. Removed sleeps (ugh) and usage of Stack (double ugh). * enable compression * Use FJPool.common for finalization * Use FJPool for query completion * Clean up top level round. Fix ASC for NPE. Change HashKey.hashcode * Fix concurrency bug in WS. Fix read only buffer bug in grpc inbound call Really wondering about the latter bug. Each call would use a list of read only byte buffers. Each BB would be accessed with BB.array(), which throws ROE because of GRPC's getReadOnlyBB - like, duh. So, why did this EVER work? Should be throwing the ROE ALWAYS, instead of once in a blue moon. Weird. * Don't fork prefer/finalize. It was a silly idea,frankly. Need to weave this in with serializing access to DagWood, so perhaps locking overhead can be optimized, removing ye horrific serialized() predicates in dat * Abstract out view in prep for consortium * Saner hashCode for HashKey * less aggressive gossip. "less" - lol. 1 round/sec is still pretty "agressive" ;) survives well under pressure, I must say. You've done well, Pig. * Factor out the Processor of the transactions from Avalanche. * interim * Basic consortium foundation stuff * Ironing out block structure * Interim wrap up.

2020-10-12

Chain scopes (#22) * Versions upgrade * Move to latest released * latest + JUnit 5 update * Basic api * Yi. Calcite CSV example + correct deps * clean up * copy over jdbc pass through from calcite In prep for AOP stylin' * move out * original * original * Initial sketch of CDC based materialized table state * Add WASM -> JVM * Fix deps * By Jove, I think this will work. Use Calcite SQL rewriting to handle translation for DDL into Apollo realm metadata accounting. Use GraalVM for WASM and language integration for Lambdas (WASM, LLVM). CDC from single writer (atomic) observer, mediated by consensus (block based). F(ck yea! * compiles. still getting plsql removed * better clean up sketches. flesh out current path * settling in * update readme, add polyglot deps * Remove Graal sdk dep. Not sure why this was necessary on my other IDE (identically configured). Removing this for now. Will triage later * Revert "Remove Graal sdk dep." This reverts commit ed1918beb1a1a626d1c982a575820444252d87ed. * More work. Fleshing out CDC engine, membership n' rings * Cleanup * Ring set ops on Context * more flesh. get testing for calcite and ddl to drive transform development. * pg/sql antlr4 For der WASM And I need anonymous functions (lambda) for schema *grade * add back dep * Starting to fill out calcite model for Apollo chains * interim * Initial kernel schema sketch. My thought is to reify the metadata schema, then transform DDL into predicates which do the DML on the metadata schema, then perform the corresponding H2 DDL commands Implementation of DDL transactions (rather than single statement DDL transactions) would be through Calcite "caching", implicitly making the trade offs of expensive ops (build index, drop column, drop table, schema, etc). Any DDL in batches would be simulated through Calcite, and DML mixed into all that would operate against this simulation. The implicit trade off is one cannot do massive bulk ops through loops or whatever, unless teh ops are state replicable without breaking bandwidth. * Generate information schema Update liquibase to 4.0.0 * interim * refactor out seeds, so not instance state * shore up membership abstraction * moar better HashKey impl * bite bullet, use HashKey instead of UUID for member id. It is the right thing to do * Generisize membership * use hash for probed key based on id, ring, input key * clean up * Interim refactoring commit. FF refactored to use new membership Context. Some additional cleanup * update the rest to membership context. HashKey[32] is used for ids, rather than UUIDs. * Use new sampling * first cut o grpc/proto protocol * use plugin correctly for grpc code gen * Refactoring to prepare for GRPC transition. Working MTLS client/server GRPC with access to ssl session on server side, tested. * Remove Avro artifacts, adapt to GRPC * Refactor Fireflies to grpc regime. Interim * merge pre View merge I think this is all "safe" - i.e. correct. * merge tests * Working GRPC FF local comm sim Geebus * first pass conversin of Avalanche to GRPC * Avalanche local sim working :: Sigh :: GRPC. Amirite? * first pass on grpc conversion of Ghost * Further cleanup. Prep for MTLS conversion * fix override auth * Revamp local communications sim. Create generalized communications abstraction for the multiple services of Apollo. Validate with the LocalCommSim implementation for Fireflies * More work on communications model. Request to open connection is now never denied, as this does cause problems. Clean up. Ghost, Avalanche, FF working with LocalCommSim * up to 30 members for testing * Mtls grpc Communications implementation Tested for fireflies * Avalanche working with MTLS * 5x5 * Remove experiments * remove posgresql lexer/parser Will add back when appropriate * upgrade liquibase version. allow tls avalanche test to set test cardinality lower, due to limitations on lesser machines. * Scope by context ID * refine a bit for reuse pattern * Context scoped Avalanche * Scope all rpcs. Simplify implementations. * whoops. Not Member, but by HashKey id ;) * Merkle patricia trie for moar efficient anti entropy Also, too, work on SMR over committee * rm cruft. minimal BF * Add hashes to acc and notes. BFs to gossip messages * add BF for Gossip digests * Remove MessageDigest * Cert gossip now BloomFiltered * Reimplemented BF. Cert antientropy via BF Geebus ;) * notes replicated via BF * Accusations replicated via BF * use .25 FPR, everything 5x5 Slightly longer time to reach eventual consistency, far, far less network traffic. Good tradeoff. Uses randomly seeded BloomFilters in gossip to determine replication between Firefly pairs * doc * use active members for accs replication, total view for certs, notes * metrics stuff * clean up * First cut of cleaning deps * pull in dropwizard deps * Clean through FF * clean through avalanche * clean through bootstrap-service * bootstrap-client clean * clean through apollo-web. still have test failure in BootstrapTest. Will fix later, as at least it's not barfing with verify errors now ;) * generate * clean deps, through and through * fix BootstrapTest * Really fix BootstrapTest. Problem was that seeds are now passed into the view.start, rather than the constructor. D'oh. * fix dep convergence * update README.md Co-authored-by: Hal <hal.hildebrand@salesforce.com>