Proto commits in dreadl0ck/netcap

These commits are when the Protocol Buffers files have changed: (only the last 100 relevant commits are shown)

2026-05-01

refactor: rename IPProfile audit record to Host Rename audit record type IPProfile -> Host across the codebase. To free up the Host name (previously used for HTTP Hostname Maltego entity), also rename the existing netcap.Host entity to netcap.HTTPHost. IPProfile -> Host: - proto: NC_IPProfile -> NC_Host, message IPProfile -> Host - types/ip_profile.go -> types/host.go (type, vars, methods) - decoder/packet/ip_profile.go -> decoder/packet/host.go (atomicIPProfileMap -> atomicHostMap, ipProfiles -> hosts, ipProfile wrapper -> hostEntry, ResetIPProfiles -> ResetHosts, updateIPProfile -> updateHost, etc.) - on-disk filename IPProfile.ncap.gz -> Host.ncap.gz - io/elastic.go switch case IPProfile -> Host - maltego/ip_profile.go -> maltego/host.go (LoadIPProfiles -> LoadHosts, IPProfileTransform -> HostTransform, IPProfileTransformationFunc -> HostTransformationFunc, ipProfileCountFunc -> hostCountFunc) - Maltego entity netcap.IPProfile -> netcap.Host (InternalIPProfile -> InternalHost, ExternalIPProfile -> ExternalHost, IPProfileAuditRecords -> HostAuditRecords) - Maltego transforms ToIPProfiles* -> ToHosts* - Webui: IPProfileSummary -> HostSummary, readIPProfiles -> readHosts, getIPProfileGeolocations -> getHostGeolocations, getIPProfileScatter3DData -> getHostScatter3DData, readJA4FromIPProfile -> readJA4FromHost - Frontend: value="IPProfile" -> value="Host" in MenuItem, chart map keys, dashboard text Conflict resolution (existing netcap.Host -> netcap.HTTPHost): - maltego/entity_generation_test.go: "Host" -> "HTTPHost" - maltego/transform_generation_test.go: netcap.Host -> netcap.HTTPHost - cmd/transform/ToHTTPHostNames.go: netcap.Host -> netcap.HTTPHost - cmd/transform/ToHTTPHostsFiltered.go: netcap.Host -> netcap.HTTPHost - New maltego/netcap/Entities/netcap.HTTPHost.entity for the renamed entity (parent type netcap.IPAddr, icon web_black) Also rename existing toHosts (POP3-specific filter) -> toPOP3Hosts to free up the toHosts function name for the new Hosts list transform. Breaking changes: - old IPProfile.ncap.gz files cannot be read by the new code - existing Maltego installations need .mtz archives regenerated via 'zeus gen-maltego-config' (env-dependent, not run here) - proto bindings for cpp/csharp/swift/js were not regenerated; only Go binding was updated via 'protoc --gogofaster_out=types/. netcap.proto'

refactor: rename Credentials audit record to Secret Rename audit record type Credentials -> Secret across the codebase: - proto: NC_Credentials -> NC_Secret, message Credentials -> Secret - types/credentials.go -> types/secret.go (type, vars, methods) - decoder/stream/credentials/ -> decoder/stream/secret/ (package + dir) - on-disk filename Credentials.ncap.gz -> Secret.ncap.gz - WriteCredentials -> WriteSecret, ResetCredStore -> ResetSecretStore - Maltego entity netcap.Credentials -> netcap.Secret - Maltego entity netcap.CredentialsAuditRecords -> netcap.SecretAuditRecords - HTTP API /api/credentials/* -> /api/secrets/* - Frontend route /credentials -> /secrets - Frontend page CredentialsPage -> SecretsPage - Test fixture renamed (existing data needs regeneration) - Maltego icons regenerated separately via 'zeus gen-icons' Breaking changes: - old Credentials.ncap.gz files cannot be read by the new code - existing Maltego installations need .mtz archives regenerated via 'zeus gen-maltego-config' (env-dependent, not run here) - proto bindings for cpp/csharp/swift/js were not regenerated; only Go binding was updated via 'protoc --gogofaster_out=types/. netcap.proto'

2026-04-09

protobuf support in ui

Add generic protobuf wire format stream decoder Detect and decode arbitrary Protocol Buffer messages on the wire without requiring a schema. The decoder uses heuristic detection (wire type distribution, varint patterns, Shannon entropy) and parses all six wire types including recursive nested message support. - New stream decoder registered on ports 9090 and 50051 (gRPC) - Fields keyed as type_fieldnum (e.g. varint_1, string_2) with wire order preserved in FieldOrder - Entropy calculation respects CalculateEntropy config flag - Proto type NC_Protobuf = 145 with map<string,string> Fields - Unit tests (11 suites) and integration tests (6 suites, 3 benchmarks) - No false positives on HTTP, CIP, S7Comm test PCAPs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2026-04-07

yara and magika support

support for additional protocols

device enrichment with discovery fields, refined credential harvesters

2026-04-02

fix: ensure CommunityID is set on all fingerprint records (JA4T, JA4, JA4S) Three issues prevented CommunityID from appearing on fingerprint records: 1. CommunityID computation in the worker was gated behind the AddContext config flag — moved it outside so it's always calculated. 2. Packet decoders (TLS, TCP) didn't receive PacketContext. Extended DecoderAPI.Decode to accept *PacketContext and apply it via SetPacketContext, matching what GoPacketDecoder already does. Removed redundant per-packet CalcCommunityID calls from TLS decoders. 3. Added CommunityID field to TCP proto message and updated SetPacketContext, CSV, and Encode methods. Fixed the webui JA4T/JA4TS fingerprint handler to populate communityIDs on the aggregator. Includes integration test that processes https.pcap with AddContext=false and verifies all TCP/JA4T, TLSClientHello, and TLSServerHello records have non-empty CommunityIDs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

2025-12-01

ui improvements

2025-11-30

file extraction engine, dedup and decompress

feat: corelight community ids

various ui improvements

feat: quic support and ja4

2025-11-29

sip improvements

improve ethernetctp, ui improvements

feat: more industrial protocols

2025-11-28

more credential harvesters

2025-11-27

feat: more protos, more fields

2025-11-25

feat: new file extraction framework and additional audit record types

feat: TLSCertificate audit records and new pages

2025-11-24

ui fixes

2025-11-23

extending credential harvesters

feat: ui view raw conversation streams and download pcap, services page

2025-11-08

wip: webUI and rules + alerting engine

2025-11-02

fix: various issues, roadmap, upgrade gopacket, enable cip,enip again, default decode opt datagrams in capture mode

2025-10-24

feat: dpi results for additional audit record types

The documentation is generated from this commit.

chore: switch to new releases of deps ja3,go-dpi and tlsx that use gopacket/gopacket, disable modbus,cip,enip,diameter decoders until they are available in gopacket/gopacket

2025-07-01

Changes from background composer bc-dfb12fc6-3a7c-4e86-9fef-32ba6a758f76

2021-07-06

connection proto updates

2021-06-13

update proto defs for connection flow volume tracking

2021-06-09

modeled alerts, field type updates

2021-06-06

formatting, renamed field for Ja3 hashmap on profile audit records

2021-01-05

move port stats into dedicated structure to avoid alignment issues on ARM

2020-10-04

pop3: save command sequence

bootstrapped SMTP stream parser

2020-09-27

removed flow audit records, fixed lookup DHCP fingerprints transforms

2020-09-23

added to contacted ports transform

2020-09-19

add open in disassembler transform, fixed to cookie and to params transforms on host entities, added support to show all http headers and values for a given host

2020-09-16

use cookie for frontend framework detection, cleanup

2020-09-12

improved build process, added option to compile without dynamically linked dpi libs

2020-09-09

fmt

2020-08-30

added a service port to name mapping for service probe matching, to make an educated guess which signatures to try first

2020-08-27

improved port modeling on ipprofiles

2020-08-25

POP3: move mails into dedicated audit record file and cross reference via IDs

write IP profile into dedicated audit record file

added command to format the protobuf definitions

removed Context from structures, added fields directly

made all timestamps numeric

2020-08-22

changed dhcp type for option data field to string

2020-08-21

store service banners as strings

dns audit records: change name fields to type string

2020-07-12

added support to capture payloads for http requests and responses, return an error for ReadHeader(), added packets per second to progress log, added profiling with fgprof, added zeus commands for profiling, move membuffer before the compression step, since pgzip acts as a buffer as well, graphics update, allow to ignore custom encoder init errors in config, rename NoDefrag config Option to DefragIPv4

2020-05-25

refactored service probe matching and added unit tests

2020-05-24

added exploit audit records and refactored queries for bleve databases

2020-05-23

enrich software and vuln maltego entities

2020-05-20

maltego iteration, added support for vulnerability audit records

added vulnerability audit records, go fmt project

prepare dhcp fingerprinting, add option to resolve service ips to hostnames via dns lookup with -reverse-dns flag

2020-05-17

deduplicate services and store all flows towards a service

2020-05-16

bootstrapped SSH audit records

2020-05-11

service audit records: save num bytes seen for server and client

2020-05-06

bootstrapped credentials audit records and use it to capture http basic auth and pop3 credentials

2020-05-04

testing and fixes for the service banner regexes, added support for using a different RE engine that supports backtracking

2020-05-03

software audit records: collect flows when updating a record

updated software fields, preserve full parsed client info for debugging

2020-05-02

extended service, cleanup and roadmap

added service audit record for banner grabbing

2020-04-30

testing, notes and fixed race in tcp stream reader

cleanup

2020-04-29

bootstrapped software audit records

2020-04-27

removed layer flows

2020-04-26

testing and cleanup

2020-04-10

added TLSServerHello audit records and renamed TLS records to TLSClientHello

2020-04-04

cleanup

2020-03-25

pop3 attachments, version bump

HTTP POST parameter extraction, add host to file info, added URL reference to file source field, match file types on the part before the semicolon, to avoid exotic encodings breaking the matching

2020-03-22

stream reassembly refactoring: switched to using on assembler per worker with a shared stream pool

2020-03-21

implemented GetApplicationCategories and GetApplicationsForCategory

2020-03-19

added POP3 support and first experimental mail extraction plugin for maltego

added experimental POP3 parsing support

2020-03-18

implemented diameter decoder

added SMTP decoding support

2020-03-16

modeled http cookies and addeed maltego plugin

cleanup

implement CSV generation for DeviceProfile, added File audit record type

http: added fields for detected content type, added file extraction for http POST, fix files being saved outside of files folder when generating paths for existing targets, improved mapping of content types to file extensions

2020-03-10

implemented service lookups for ports

2020-03-09

added more stats and bootstrapped maltego integration

2020-03-05

added nDPI

added cached ja3 lookups, added caching for DNS lookups, added timestamps to profiles

2020-03-01

added descriptive comments about protocols to proto defs

added type definitions for DeviceProfile and IPProfile

2020-01-14

implemented ENIP

implemented support for Common Industrial Protocol

2020-01-13

updated modbus

2019-11-12

implemented adding context information to audit records

2019-05-08

added comment to http audit record enhanced fields

2019-05-02

added new HTTP fields

2019-01-18

added optional payloads for modbusTCP

2019-01-17

added gogo proto gode generator for faster protobuf serialization, renamed Size field from several audit records to TotalSize to resolve conflict with generated Size() func