The Problem: Where Did That Packet Go?

Every Linux networking engineer has faced the question: "Where did this packet actually go?" The answer is rarely simple. A packet can traverse network namespaces, routing decisions, Netfilter hooks, traffic control, XDP programs, tunnels, clones, copies, and drop paths. A single tcpdump capture at one interface is correct for that point, but shows only a fraction of the journey.

skbx, a new Rust and CO-RE eBPF tool, aims to answer that question by acting as a "flight recorder" for packets inside the Linux kernel. It observes kernel networking functions, writes an append-only JSONL evidence stream, and allows replay without root.

What skbx Observes

skbx can observe:

  • Kernel functions handling an sk_buff
  • Packet clones, copies, copy-on-write, and XDP-to-SKB transitions
  • TC and XDP program entry and exit
  • Tunnels and inner packet tuples
  • Kernel-reported drop reasons
  • Selected BPF helper and map activity
  • Capture loss, decoding failures, and output failures

This is not a replacement for tcpdump or Wireshark. Those tools are excellent for packet bytes and protocols at a single point. skbx is for questions about the path through the local kernel.

How It Works

The live tracer uses CO-RE eBPF to attach probes to kernel functions. Observations are written to a bounded JSONL file called traceq. Each event gets a stable handle like event:111111111111111111111111. Replay groups ordered events into route patterns, each with its own handle: route:1c0201e74424e253f8363577.

The footer records stop reason and reliability counters (kernel reservation failures, tracer recursion misses, read failures, etc.). If the footer is absent, the artifact is incomplete.

Installation

skbx requires Rust 1.85+, Clang/LLVM with BPF backend, bpftool, libelf, libpcap, and a kernel exposing /sys/kernel/btf/vmlinux. On Ubuntu:

sudo apt-get install "linux-tools-$(uname -r)" clang llvm libelf-dev libpcap-dev pkg-config

Then install from GitHub:

cargo install --git https://github.com/copyleftdev/skbx --locked skbx-cli

Before attaching, run skbx doctor --json to check the host. skbx plan --filter-func 'ip.*' --json shows which functions would be attached without actually attaching.

Capturing a Packet

Here's a simple ICMP capture:

sudo skbx capture --probe ip_rcv --duration 10 --output trace.jsonl icmp

While it runs, ping:

ping -c 3 1.1.1.1

Replay without root:

skbx replay trace.jsonl

To see a specific event with surrounding evidence:

skbx explain trace.jsonl event:

Use Cases

Debugging a Website Request That Times Out

If curl starts an HTTPS request and times out, a capture on the client can answer:

  • Did the request enter the local TCP and IP output paths?
  • Which interfaces and namespaces were associated?
  • Was its mark changed?
  • Did a TC or XDP program handle it?
  • Was it tunneled?
  • Did the local kernel report a drop?

This evidence clears or implicates the client host. The absence of a local drop is not proof the remote host received it, but it narrows the search area.

Tracking a Packet That Disappears Inside the Host

"The network dropped it" can hide many failures. skbx can show if a TC classifier, XDP program, or normal kernel path rejected the packet. For focused TCP drop questions, a small bpftrace program may be faster. For packet contents, use tcpdump. skbx is for when you don't know which subsystem owns the failure.

Following Packet Transformations

Linux doesn't guarantee one struct sk_buff address per logical packet. Packets can be cloned, copied, or copy-on-write. XDP frames become SKBs. Tunnels introduce outer/inner identities. skbx uses capture-local lineage state to connect transitions. For example, to follow a marked packet:

sudo skbx capture --filter-mark 0x2a --filter-track-skb --output trace.jsonl

Lineage is capture-local, not a distributed trace ID.

Capture Once, Investigate Without Root

Loading eBPF programs requires root. Reading JSONL does not. This enables a handoff: an operator performs a bounded capture, the artifact is copied to a normal environment, and someone replays it. Useful for incident handoffs and learning.

Giving an AI Evidence

skbx supports machine-readable output: skbx describe --format json, skbx schema, skbx doctor --json, skbx plan --json. An AI can summarize a route or explain an event, but the event must already exist in the trace. This gives the explanation something concrete to cite.

Engineering Highlights

The author enjoyed:

  • Discovering valid struct sk_buff * arguments from the target kernel's BTF instead of guessing
  • Keeping kernel-side state bounded
  • Moving JSON encoding, symbolization, and filesystem work out of the eBPF hot path
  • Making probe planning inspectable before attachment
  • Separating raw observation from later explanation
  • Treating partial output as incomplete
  • Making replay deterministic for tests

What It Doesn't Know

skbx only observes what the running kernel exposes. It doesn't see inside an ISP, a remote host (unless it runs there), or replace Wireshark for packet contents. Kernel config, BTF availability, permissions, and capture loss can limit evidence. Those limitations are part of the interface.

Getting Started

Install skbx, run skbx doctor, then skbx plan to see what would be traced. Capture a bounded trace, replay it, and explore. The project is on GitHub at copyleftdev/skbx. The author welcomes bug reports with kernel version, exact command, doctor --json output, and reliability footer.