Proto commits in google/santa

These 32 commits are when the Protocol Buffers files have changed:

2024-12-10

Add ancestry annotator

2024-09-08

Fix: Change uint64 fields in syncv1.proto to uint32 for backwards compatibility (#1422) Change the uint64 fields in the syncv1.proto to uint32 to ensure backwards compatibility. This also updates the SNTSyncEventUpload code to use the uint32 values and updates sync protocol docs.

The documentation is generated from this commit.

2024-08-12

sync: Drop rules_* fields in postflight to uint32 (#1415) * sync: Drop rules_* fields in postflight to uint32 This lets the protobuf json serializer to send the values as ints (like NSJSONSerialization did) instead of strings. This will cause problems if someone has 4B rules but that's probably a sign of bigger problems

2024-07-29

sync: Fix serial_num field name (#1404) Disable the preserve_proto_field_names option when marshalling JSON requests as this prevents the json_name attribute on fields from working properly. Add that attribute to all fields so that they marshal as expected. Stop setting the always_print_enums_as_ints field as the value we're setting to is the default anyway. Also add a test that preflight request data looks as expected.

2024-07-16

Improve handling of sync response default values (#1395) * Improve default value handling for sync proto processing * Fix capitalization of new enum values in comments * Fix/add tests, update some docs * Update more docs * Lint * Remove comment. Add LEGACY_NAMES tag for the linter

2024-07-09

sync: Add machine_id field to facilitate a GRPC version of the protocol (#1390)

sync: Handle parse errors, add UNKNOWN_CLIENT_MODE to enum (#1389)

2024-07-02

Login/logout events (#1371) * WIP Basic new enriched types, hooked up serializers * WIP Expanded enriched types, finished basic string logging * WIP Standardize instigator and event user strings. * WIP Remove sudo event for now. Fix proto types. * Update proto field names. Fix builds on older SDKs. * Fix more issues with builds on older SDKs. * Even more build fixes for older SDKs * Fix basic string test build on older sdks * More fixes for older SDKs * WIP Started on proto encoding and tests * WIP expanded proto support for new events * Lint. Fix recorder tests for missing event types * WIP continued expanding proto support for new events * WIP finished proto support for all new event types * WIP Comment all new messages and fields in santa.proto * WIP Use different impl to set strings to sidestep internal absl issues * Temporarily removing serializer impls and tests to reduce PR size * Lint fixes * PR feedback

2024-05-29

sync: Add a protobuf for the existing sync protocol (#1359) This PR is intended to have no impact on existing sync servers. The fields and enum values in the protobuf have been named such that their JSON equivalents match the existing constants we have in the codebase. Adding this provides a few benefits: 1. The protobuf serves as canonical documentation of the protocol in a form that's much easier to read than the existing code. 2. Protobuf parsing of JSON is likely to be better than our hand-written version. 3. We can (in a later PR) add a configuration option to use binary encoding instead of JSON, saving network during syncs. 4. Servers written in other languages are easier to write and update as time goes on, especially as we extend the protocol.

2024-04-11

ProcessTree: add the first annotation, originator (4/4) (#1296)

2024-03-14

ProcessTree: integrate process tree throughout the event processing lifecycle (3/4) (#1281) * process annotations: thread the tree through santa * Update enricher to read annotations from the ProcessTree * rebase changes * add configuration for annotations, disabling the tree entirely if none are enabled * lingering build dep * use tree factory constructor * fix configurator * build fixes * rebase fixes * fix tests * review comments * lint * english hard * record metrics even when event only used for process tree

2024-03-05

Add support for CDHash rule types (#1301) * Support CDHash rules * Ensure hardened runtime for cdhash eval. Update docs. * minor fixups * Clarify docs

2024-02-05

ProcessTree: add core process tree logic (1/4) (#1236) * ProcessTree: add core process tree logic * make Step implicitly called by Handle* methods * lint * naming convention * widen pidversion to be generic * move os specific backfill to os specific impl * simplify ts checking * retain/release a whole vec of pids * document processtoken * lint * namespace * add process tree to project-wide unit test target * case change annotations * case change annotations * remove stray comment * default initialize seen_timestamps * fix missing initialization of refcnt and tombstoned * reshuffle pb namespace * pr review * move annotation registration to tree construction * use factory function for tree construction

2023-11-13

Entitlements logging config options (#1233) * WIP add config support to filter logged entitlements * Add EntitlementInfo proto message to store if entitlements were filtered * Log cleanup * Address PR feedback * Address PR feedback

2023-11-09

Fix internal build issues, minor cleanup. (#1231)

Add support for logging entitlements in EXEC events (#1225) * Add support for logging entitlements in EXEC events * Standardize entitlement dictionary formatting

2023-11-02

Add Support for CS_INVALIDATED events (#1210) Add support for logging when codesigning has become invalidated for a process. This adds support to the Recorder to log when codesigning is invalidated as reported by the Endpoint Security Framework's ES_EVENT_TYPE_NOTIFY_CS_INVALIDATED event.

2023-08-17

Add mount from information to disk appear events (#1153)

2023-05-12

Add new rule type for Signing IDs (#1090) * WIP: Signing ID rules * WIP: More work supporting signing ID rules * Expanded exec controller tests for signing ID and team ID * wip all current tests now pass * Added integration tests * Branch cleanup * Update protobuf tests for signing id reason types * Remove old commented out code --------- Co-authored-by: Russell Hancox <russell@hancox.us>

2023-03-31

Change types of repeated args and envs fields (#1063) * Change types of repeated args and envs fields * Update args and env testdata strings to base64 * Remove whitespace

2023-01-25

Reduce proto warning severity (#1012)

2023-01-06

More event type support (#992) * Add truncate and create support * Add metrics support

2022-12-14

Serialize File Access events (#964) * WIP skeleton code for file access event serialization * Added basic string serializer for file access event * Added proto string serializer for file access event

2022-12-07

Draft proto for new FileAccess log (#952) * Draft proto for new FileAccess log * Update Source/common/santa.proto Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com> Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com>

2022-10-21

Proto minimization (#909) * Create Light variants of File and ProcessInfo messages to reduce disk/wire byte counts * Updated golden test data

2022-10-19

Machine id proto (#907) * Add MachineID to all BasicString serialized log messages * machine_id now a top level proto field * Remove commented code

2022-10-14

Import fixes (#902) * Apply clang-format to cc files * Modify binaryproto namespace * Add more required includes * Add proto includes * Assert message parsing succeeds in test * Add optional keyword to proto fields to track presence. TESTS BROKEN. * Update golden test data

Fsspool adopt (#900) * Added fsspool library, tests * Cleanup * Remove extra visibility from BUILD file * Import foundation so the linter doesn't complain

2022-10-13

Proto serializer (#897) * Initial proto serializer with close event * Define move ctors for enriched types, delete copy ctors * More event proto serialization. Commonized proto test code. * Started work serializing exec event. Added serializer utilities. * More progress serializing exec event * Add mroe test data. Test restructure to permit fine grained mocking. * Env/FD ES types now wrapped in EndpointSecurityAPI. Added calls to proto serializer. * Add fd type names to proto * Version compat. Script and Working Dir encoding. * Add process start time * Serialize Link event * Add null check, mainly to fix tests * Handle versioned expectations * Each test now build msg in callbacks to set better expectations * Serialize rename event and tests * Serialize unlink event and tests * Serialize allowlist and bundle events. Add utilities tests. * Formatting * Disk event proto serialization and tests * Fix test only issues * Rename santa_new.proto to santa.proto * Change fd type int and string to an enum * Proto namespace now versioned * Added comments to proto schema * Add proto support to indicate if fd list truncated

2022-09-22

ES and Logging Interfaces Redesign (#888) * Initial structure for ES wrappers, enriched types, logging * Basic working ES and logging functionality * Add in oneTBB and thread-safe-lru deps * Added a bunch of enriched types * Auto-mute self when establishing ES client * Basic auth, tamper client. Syslog of all events. Basic compiler tracking. * Update copyright header blobs, convert some tabs to spaces * Auth result cache. Fix getting translocation path. * Added remaining cache methods * Add AuthResultCache to Recorder client. Cache now operates on es_file_t. * Hooked up SNTPrefixTree * Fix CompilerController for RENAME. Fix AllowList logging missing path. * Block loading Santa kext * Added device manager client * Properly log DiskAppear events * Fix build to adopt new adhoc build * Handle clearing cache on UNMOUNT events * Ignore other ES clients if configured * Remove SNTAllowlistInfo. Rename AllowList to Allowlist. Minor cleanup. * Recorder now logs asynchronously. Enricher now returns shared_ptrs. * Added File writer. Added timestamps to BasicStream serializer. * Skip calling stat in SNTFileInfo when path given by ES. * Fix build issue * Address draft PR feedback * santactl integrated, XPC works, fix file writer bug * Integrate syncservice. Start observing some config changes. * Add metrics service wrapper * Add metrics config observers and metrics interval reset. * Start better dependency control. Add Null logger support. * Added more deps * Added more deps * Fix issue where metric service wasn't starting * Add missing variant include * Fix missing parent proc name * Added googletest and new unit test macro * Started expanding AuthResultCacheTest * Properly mock EndpointSecurityAPI * Finished AuthResultCacheTest * bazelrc now builds all C++ as C++17. Added LoggerTest. * Add FileTest. Abstract some File constants to Logger. * Added Empty serializer test * Started work on BasicStringTest. Fixed some BasicString serialization bugs. * Added Unlink BasicString serialization test * Added some more tests. Commonized some test code * Finished BasicStringTest. Converted to XCTest. * Standardize esapi variable naming * Bubble up gTest expect failures to XCTest failures * AuthResultCacheTest now uses XCTest. Added common TestUtils.h * EmptyTest now uses XCTest. * FileTest now uses XCTest * LoggerTest now uses XCTest. Removed santa_unit_gtest bazel macro. * Added ClientTest * Add basic Enricher tests * Add MessageTest. Make more TestUtils. * Rename metrics to Metrics * Add MetricsTest. * Apply template pattern to Serializer * Add SNTDecisionCacheTest. * Add SNTCachedDecisionTest. * Testing with coveralls debug mode * Allow manual CI runs * Remove unused property * Started work on SNTEndpointSecurityClientTest. * WIP SNTEndpointSecurityClientTest, fix test run issue * Added more base ES client tests * Add more base ES client tests * Base ES client tests done. Added serializer utils/tests. Expanded basic string tests. * Add utils test to test suite * Add copy ctor. Add test output to bazel coverage. * Single thread bazel coverage * Updaload coverage file * Updaload coverage file * Old gen cov test * Restructure message handlers to enable better testability * Added enable tests for all ES clients * Made a single MockEndpointSecurityAPI class to share everywhere * Added most of SNTCompilerControllerTest * Cleanup SNTCompilerControllerTest * Started expanding Auth client test * Finished up the Authorizer tests * Move to using enum class for notify/auth instead of bool * WIP for tamper resistance test. ASAN issues. * Add OCMock patch to fix test issue on ARM Macs * Changed patches directory name to external_patches * Update WORKSPACE path * Finished up Tamper Resistance tests * Finished up Recorder tests. * Move SNTExecutionControllerTest to ObjC++ * Initial work to port SNTExecutionControllerTest * Finished porting SNTExecutionControllerTest. * Added SNTExecutionControllerTest to list of unit tests * Ported SNTEndpointSecurityDeviceManager. * Test cleanup, use MockESAPI expectation helpers * Verify SNTEndpointSecurityDeviceManager expectations differently * Test cleanup, omit gTest param list where unused * Log message cleanup * Rename SNTApplicationTest to santad_test.mm * Finished porting santad_test, formerly SNTApplicationTest * Fix SNTEndpointSecurityDeviceManager issues * Pulled in missed fixes. Updated tests. * Renamed lowercase filenames to match rest of codebase * Fix non-static dispatch_once_t, and noisy watching compiler log message * WIP Started process of removing components no longer used * WIP Continued process of removing components no longer used * BUILD file cleanup. Proto warning. Removed unused global * Rename SNTEventProvider to SNTEndpointSecurityEventHandler * Rename SNTEndpointSecurityEventHandler protocol * Remove EnableSysxCache option. Remove --quick flag used during dev. * Ran testing/fix.sh * Addmissing param to fix.sh that was omitting .mm files. * clang-format * Fix linter: find cmd missing .mm ext, git grep exclude patch files. * Use MakeESProcess default params in tests * Move variables to camelCase in objc classes * More case changes * Sanitize strings * Change dispatch queue priorities and standardize daemon queue naming * Exclude patch files in markdown check * Ensure string log messages end with newline * Fix BasicStringTest * Disable clang-format in code producing different results in local/remote versions * Moved to using date ranges in copyright notices as per current guidelines * Update Source/common/SNTConfigurator.h Suggestion adding whitespace in comment to fix clang-format mangling Co-authored-by: Russell Hancox <russellhancox@users.noreply.github.com> * Removed santa_panic macro used in one place * Updated comment about ES cachability * Pin oneTBB to specific commit * Address outstanding WORKSPACE 'canonical reproducible form' messages * Use string append instead of ostringstream due to benchmark results * Remove use of freind classes in EnrichedTypes.h * Added SNTKVOManager, removed observers from SNTConfigurator. * Fixed SNTEndpointSecurityRecorderTest class name * Reduce usage of the auto keyword * Each SNTKVOManager instance now adds its own observer * Replaced more auto keywords with real types. * Remove leftover code coverage debugging from ci.yml * Updated comment * Memoize SNTFileInfo sha256. Reduce some cache sizes. * Fix issue checking for translocated paths * Use more performant NSURL creation method * Fix lint issue * Address PR feedback * Use an array literal for kvo objects * Fix some clang tidy and import issues * Replace third party LRU cache with SantaCache for now * Fix clang tidy issues * Address PR feedback * Fix comment typo Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com> * Added todo for when we adopt macOS 13 Co-authored-by: Russell Hancox <russellhancox@users.noreply.github.com> Co-authored-by: Pete Markowsky <pmarkowsky@users.noreply.github.com>

2022-07-15

santad: Log team ID in execution logs, where available (#850)

2022-03-14

Protobuf support, maildir format logging (#731) * Initial protobuf support, maildir logging Fix build issues in the integration test Deduped some test code Formatting Address feedback from draft PR Removed legacy labels. Updated docs. Add in metrics. Fix protobuf logging test. * Now use the Any proto for the LogBatch wrapper * Changes based on PR feedback * Added gauge metrics for spool dir * Formatting * Add event time to proto * Fix build issue after rebase * Update BUILD rules * Updated language around protobuf logging to mark as beta