Solana AccountsDB Plugin for Kafka

Kafka publisher for use with Solana's plugin framework.

What This Plugin Does

Streams three types of Solana data to Kafka:

Quick Start

Want to see data flowing immediately? Use this minimal config:

{
  "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
  "kafka": {
    "bootstrap.servers": "localhost:9092"
  },
  "filters": [{
    "update_account_topic": "solana.testnet.account_updates",
    "slot_status_topic": "solana.testnet.slot_status",
    "transaction_topic": "solana.testnet.transactions",
    "publish_all_accounts": true
  }]
}

This will publish all account updates, transactions, and slot status to Kafka. Perfect for testing and development.

Installation

Binary releases

Find binary releases on GitHub.

Building from source

Prerequisites

Note that as of this writing, Ubuntu 22.04 still has an obsolete of protoc.

Previously, you could use protocbuf-compiiler from Debian trixie, but trixie's libc6 is no longer compatible with Ubuntu 22.04.

You will need to use Ubuntu 24.04 or later, or Debian 12 or later.

Build

cargo build --release

Important: Solana's plugin interface requires the build environment of the Solana validator and this plugin to be identical.

This includes the Solana version and Rust compiler version. Loading a plugin targeting wrong versions will result in memory corruption and crashes.

Config

Config is specified via the plugin's JSON config file.

Example Config

⚠️ WARNING: This example config will NOT publish most data by default!

The following config is a minimal example that demonstrates the structure, but with publish_all_accounts: false and no program_filters, you'll only see slot status updates. For testing, consider using the scenarios below.

{
  "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
  "kafka": {
    "bootstrap.servers": "localhost:9092",
    "request.required.acks": "1",
    "message.timeout.ms": "30000",
    "compression.type": "lz4",
    "partitioner": "murmur2_random",
    "statistics.interval.ms": "1000"
  },
  "shutdown_timeout_ms": 30000,
  "block_events_topic": {
      "topic": "solana.testnet.block_events",
      "wrap_messages": true
  },
  "filters": [{
    "update_account_topic": "solana.testnet.account_updates",
    "slot_status_topic": "solana.testnet.slot_status",
    "transaction_topic": "solana.testnet.transactions",
    "program_ignores": [
      "Sysvar1111111111111111111111111111111111111",
      "Vote111111111111111111111111111111111111111"
    ],
    "publish_all_accounts": false,
    "wrap_messages": false
  }]
}

For Testing/Development (Recommended):

{
  "libpath": "target/release/libsolana_accountsdb_plugin_kafka.so",
  "kafka": {
    "bootstrap.servers": "localhost:9092"
  },
  "filters": [{
    "update_account_topic": "solana.testnet.account_updates",
    "slot_status_topic": "solana.testnet.slot_status",
    "transaction_topic": "solana.testnet.transactions",
    "publish_all_accounts": true
  }]
}

Reference

Message Keys

The message types are keyed as follows:

Filtering

⚠️ IMPORTANT: Understanding how filtering works is crucial for getting data flowing to Kafka!

The plugin uses a whitelist approach for filtering. By default, most events are filtered out unless you explicitly configure what you want to see.

How Filtering Works

  1. Account Updates: Only published if the account's owner program is in program_filters OR the account address is in account_filters
  2. Transactions: Only published if they involve accounts from programs in program_filters OR specific accounts in account_filters
  3. Slot Status: Always published (not affected by filters)
  4. Program Ignores: Blacklist of programs/accounts to exclude (applied after whitelist filtering)

Common Filtering Scenarios

Scenario 1: See Everything (Recommended for testing)

{
  "publish_all_accounts": true,
  "program_filters": [],
  "account_filters": []
}

Scenario 2: See Specific Programs Only

{
  "publish_all_accounts": false,
  "program_filters": [
    "11111111111111111111111111111111",  // System Program
    "TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"  // Token Program
  ]
}

Scenario 3: See Specific Accounts Only

{
  "publish_all_accounts": false,
  "account_filters": [
    "YourAccountAddressHere111111111111111111111111"
  ]
}

Troubleshooting: No Data in Kafka?

If you're not seeing messages in Kafka despite successful slot processing:

  1. Check your filters: Make sure you have either publish_all_accounts: true or specific program_filters/account_filters
  2. Verify topics: Ensure your topic names are correct and Kafka is running
  3. Check program ignores: Make sure you're not accidentally filtering out everything with overly restrictive program_ignores
  4. Test with minimal config: Start with publish_all_accounts: true to verify the plugin is working

Message Wrapping

In some cases it may be desirable to send multiple types of messages to the same topic, for instance to preserve relative order. In this case it is helpful if all messages conform to a single schema. Setting wrap_messages to true will wrap all three message types in a uniform wrapper object so that they conform to a single schema.

Note that if wrap_messages is true, in order to avoid key collision, the message keys are prefixed with a single byte, which is dependent on the type of the message being wrapped. Account update message keys are prefixed with 65 (A), slot status keys with 83 (S), and transaction keys with 84 (T).

Buffering

The Kafka producer acts strictly non-blocking to allow the Solana validator to sync without much induced lag. This means incoming events from the Solana validator will get buffered and published asynchronously.

When the publishing buffer is exhausted any additional events will get dropped. This can happen when Kafka brokers are too slow or the connection to Kafka fails. Therefore it is crucial to choose a sufficiently large buffer.

The buffer size can be controlled using librdkafka config options, including:

Enhanced Analytics Support

The plugin now provides significantly richer analytics data for blockchain monitoring and analysis, enhancing all supported Solana transaction formats.

Transaction Types

The plugin supports multiple Solana transaction formats:

All transaction types are enhanced with comprehensive analytics metadata.

Enhanced Analytics Features

All transactions provide additional analytics data that can be used for:

Performance Monitoring

Error Analysis & Debugging

Address Intelligence

Slot & Network Analytics

Message Schema Enhancements

The protobuf schema has been enhanced with analytics fields:

UpdateAccountEvent

SlotStatusEvent

TransactionEvent

LoadedAddresses

Configuration for Analytics Features

Analytics features are enabled by default and require no additional configuration. The plugin automatically:

Use Cases

Analytics enhancements enable new use cases:

Troubleshooting

Common Issues

1. No Data in Kafka Topics

Symptoms: Solana validator shows slot processing but no messages appear in Kafka topics.

Causes & Solutions:

Quick Test: Use the Quick Start config above to verify the plugin works.

2. Only Slot Status Messages Appear

Cause: This is expected behavior with the default example config! Slot status is always published, but account updates and transactions require explicit filter configuration.

Solution: Add publish_all_accounts: true or configure program_filters.

3. Plugin Fails to Load

Common Causes:

4. High Memory Usage

Cause: Large Kafka producer buffers can consume significant memory.

Solution: Adjust buffer settings:

{
  "kafka": {
    "queue.buffering.max.messages": "10000",
    "queue.buffering.max.kbytes": "1048576"
  }
}

Debugging Tips

  1. Start Simple: Begin with publish_all_accounts: true to verify basic functionality
  2. Check Topics: Use Kafdrop or kafka-console-consumer to verify topics exist
  3. Monitor Metrics: Enable Prometheus metrics to see message counts and errors
  4. Verify Filters: Double-check your filter configuration matches your expectations

Getting Help

If you're still having issues:

  1. Check this troubleshooting section
  2. Review the filtering documentation above
  3. Try the Quick Start configuration
  4. Open an issue with your config and error details