Stenographer - A Parcel Capture Solution Which Aims To Apace Spool All Packets To Disk, In Addition To Hence Render Simple, Fast Access To Subsets Of Those Packets

Stenographer is a full-packet-capture utility for buffering packets to disk for intrusion detection as well as incident response purposes. It provides a high-performance implementation of NIC-to-disk packet writing, handles deleting those files equally disk fills up, as well as provides methods for reading dorsum specific sets of packets speedily as well as easily.

It is designed to:

• Write packets to disk, really speedily ( 10Gbps on multi-core, multi-disk machines)
• Store equally much history equally it tin (managing disk usage, storing longer durations when traffic slows, as well as thus deleting the oldest packets when it hits disk limits)
• Read a really pocket-sized per centum (<1%) of packets from disk based on analyst needs

It is NOT designed for:

• Complex packet processing (TCP flow reassembly, etc)
  
• It’s fast because it doesn’t practice this.  Even amongst the really minimal, single-pass processing of packets nosotros do, processing 1Gbps for indexing lonely tin receive got >75% of a unmarried core.
  
• Processing the information yesteryear reading it dorsum from disk equally good doesn’t work:  see adjacent bullet point.
  
• Reading dorsum large amounts of packets (> 1% of packets written)
  
• The cardinal concept hither is that disk reads compete amongst disk writes… you lot tin write at 90% of disk speed, but that alone gives you lot 10% of your disk’s fourth dimension for reading.  Also, we’re writing highly sequential data, which disks are really skilful at doing quickly, as well as by as well as large reading dorsum lean information amongst lots of seeks, which disks practice slowly.
  

For farther reading, cheque out DESIGN.md for a give-and-take of stenographer's design, or read INSTALL.md for how to install stenographer on a machine.

Querying

Query Language
Influenza A virus subtype H5N1 user requests packets from stenographer yesteryear specifying them amongst a really uncomplicated inquiry language. This linguistic communication is a uncomplicated subset of BPF, as well as includes the primitives:

host 8.8.8.8          # Single IP address (hostnames non allowed) cyberspace 1.0.0.0/8         # Network amongst CIDR cyberspace 1.0.0.0 mask 255.255.255.0  # Network amongst mask port fourscore               # Port pose out (UDP or TCP) ip proto vi            # IP protocol pose out vi icmp                  # equivalent to 'ip proto 1' tcp                   # equivalent to 'ip proto 6' udp                   # equivalent to 'ip proto 17'  # Stenographer-specific fourth dimension additions: earlier 2012-11-03T11:05:00Z      # Packets earlier a specific fourth dimension (UTC) later on 2012-11-03T11:05:00-07:00  # Packets later on a specific fourth dimension (with TZ) earlier 45m agone        # Packets earlier a relative fourth dimension earlier 3h agone         # Packets later on a relative time

NOTE: Relative times must move measured inward integer values of hours or minutes equally demonstrated above.
Primitives tin move combined amongst and/&& as well as amongst or/||, which receive got equal precendence as well as evaluate left-to-right. Parens tin equally good move used to group.

(udp as well as port 514) or (tcp as well as port 8080)

Stenoread CLI
The stenoread command line script automates pulling packets from Stenographer as well as presenting them inward a usable format to analysts. It requests raw packets from stenographer, as well as thus runs them through tcpdump to render a to a greater extent than full-featured formatting/filtering experience. The source declaration to stenoread is a stenographer inquiry (see 'Query Language' above). All other arguments are passed to tcpdump. For example:

# Request all packets from IP 1.2.3.4 port 6543, as well as thus practice extra filtering yesteryear # TCP flag, which typical stenographer does non support. $ stenoread 'host 1.2.3.4 as well as port 6543' 'tcp[tcpflags] & tcp-push != 0'  # Request packets on port 8765, disabling IP resolution (-n) as well as showing # link-level headers (-e) when printing them out. $ stenoread 'port 8765' -n -e  # Request packets for whatever IPs inward the arrive at 1.1.1.0-1.1.1.255, writing them # out to a local PCAP file thus they tin move opened inward Wireshark. $ stenoread 'net 1.1.1.0/24' -w /tmp/output_for_wireshark.pcap

Downloading
To download the source code, install Go locally, as well as thus run:

$ larn start github.com/google/stenographer

Go volition grip downloading as well as installing all Go libraries that stenographer depends on. To construct stenotype, larn into the stenotype directory as well as run make. You may demand to install the next Ubuntu packages (or their equivalents on other Linux distros):

• libaio-dev
• libleveldb-dev
• libsnappy-dev
• g++
• libcap2-bin
• libseccomp-dev

Obligatory Fine Print
This is non an official Google production (experimental or otherwise), it is but code that happens to move owned yesteryear Google.
This code is non intended (or used) to scout Google's users. Its run is to growth safety on our networks yesteryear augmenting our internal monitoring capabilities.

Download Stenographer