Digger is a web application that offers a graphical user interface to explore Suricata Extensible Event Format (EVE) outputs. Its primary focus is to help Capture-the-Flag players analyse network flows during stressful and time-limited attack-defense games, such as FAUSTCTF, ENOWARS, CyberChallengeIT or ECSC. Digger is a fork of Shovel made by Pwnzer0tt1.
You might also want to have a look at these other awesome traffic analyser tools:
- https://github.com/secgroup/flower (first commit in 2018)
- https://github.com/eciavatta/caronte (first commit in 2020)
- https://github.com/OpenAttackDefenseTools/tulip (fork from flower in May 2022)
- https://github.com/FCSC-FR/shovel (repository on which Digger is based)
Compared to these traffic analyser tools, excluding Shovel, Digger only relies on Suricata while making opinionated choices for the frontend. This has a few nice implications:
- dissection of all application protocols supported by Suricata (HTTP2, modbus, SMB, DNS, etc),
- flows payloads and dissections are stored inside PostgreSQL database for fast queries,
- ingest can be a folder of pcaps for non-root CTF, or a live capture (less delay),
- tags are defined using Suricata rules (regex, libmagic match, HTTP header, etc),
- no heavy build tools needed, Digger is easy to tweak.
Moreover, Digger is batteries-included with some Suricata alert rules.
┌───►stats.log
│
│ ┌────►suricata.log ctf_config.json
│ │ ┌───────────────
│ │ ┌────►pcaps/* │
│ │ │ │
┌──┴─────┴─────┴────────────────┐ ┌──────────▼ ────────┐
device │ │ │ │
or pcap │ Suricata with: │ │ │
──────────►│ - Eve PostgreSQL plugin │ │ SvelteKit webapp │
│ - TCP & UDP payloads plugin │ │ │
│ │ │ │
└──────▲────────┬───────────────┘ └──────────┬─────────┘
│ │ │
│ │ ┌──────────────────┐ │
suricata.rules │ │ │ │ │
───────────────┘ │ │ │ │
└───────►│ PostgreSQL │◄─────┘
│ │
│ │
└──────────────────┘
- The frontend is made with SvelteKit
- The UI has a fresh style with some new features: hexdump viewer, code highlighting, stats page
- CTF related settings (e.g. tick length, datetime start) are stored using a JSON file that can be edited from the webapp.
- A
start.pythat automatically starts Digger - A single PostgreSQL container is used in place of two separated SQLite files
- Digger is distributed with a GPL-3.0 license.
Most of the original code of Shovel has been completly changed.
The webapp now inside the folder frontend shares some similiarities but none of the original files remained, everything is new.
The part related to Suricata is what remain in common with Shovel, except for the use of PostgreSQL as database and Diesel ORM as SQL driver. Some minor changes were also made like: we use crossbeam instead of mspc, remove regex_lite and use serde_json.
An important part that remained the same is the template used for registering the plugins. Thanks to FCSC-FR, in particular erdnaxe, we figured how to use Suricata 8.0.0 in Digger without breaking stuff.
The files that are currently in common with Shovel are all found inside suricata, if a file was modified by us you will find a comment that specify that.
CTF related infos are stored in a JSON file and can be edited directly from the webapp:
- tick length
- available services
- start datetime
- end datetime
Add the flag format in suricata/rules/suricata.rules if needed.
If you modify this file after starting Suricata, you may reload rules using
pkill -USR2 suricata.
Digger currently implements 3 capture modes:
- Mode A: pcap replay (slower, for archives replay or rootless CTF).
- Mode B: capture interface (fast, requires root on vulnbox and in Docker).
- Mode C: PCAP-over-IP (fast, requires root on vulnbox).
Please prefer mode B or C to get the best latency between the game network and Suricata. Use mode A only if you are not root on the vulnbox and have access to pcap files indirectly.
Place pcap files in a folder such as input_pcaps/.
If you are continuously adding new pcap, add --pcap-file-continuous to
Suricata command line.
A sample configuration can be found in docker-compose-a.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
./suricata/entrypoint.sh -r input_pcaps
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that restarting Suricata will cause all network capture files to be loaded again. It might add some delay before observing new flows.
Tip
For a Microsoft Windows system, you may capture network traffic using the following command (3389 is RDP) inside a PowerShell console:
&'C:\Program Files\Wireshark\tshark.exe' -i game -w Z:\ -f "tcp port not 3389" -b duration:60This mode requires to have direct access to the game network interface.
This can be achieved by mirroring vulnbox traffic through a tunnel,
see FAQ for more details.
Here this device is named tun5.
A sample configuration can be found in docker-compose-b.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
sudo ./suricata/entrypoint.sh -i tun5
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that stopping Suricata will stop network capture.
You may also run sudo tcpdump -n -i tun5 -G 30 -w trace-%Y-%m-%d_%H-%M-%S.pcap
for archiving purposes.
This mode requires to have access to a TCP listener exposing PCAP-over-IP. Such server can be easily spawned using:
tcpdump -U --immediate-mode -ni game -s 65535 -w - not tcp port 22 | nc -l 57012If you need to route PCAP-over-IP to multiple clients, you should consider using
pcap-broker.
A sample configuration is given in docker-compose-c.yml.
If you don't want to use Docker, you may manually launch Suricata and the web application using the two following commands:
PCAP_OVER_IP=pcap-broker:4242 ./suricata/entrypoint.sh -r /dev/stdin
(cd webapp && uvicorn --host 127.0.0.1 main:app)Warning
Please note that stopping Suricata will stop network capture.
Digger provides a convenient ./start.py script to manage containers and configuration. The script supports multiple commands with a user-friendly interface.
If you run ./start.py without arguments, it will enter interactive mode where you can select actions and configure parameters through guided prompts.
Here you can find a list of available commands to use with ./start.py:
start- Build and start containersstop- Stop running containersclear- Clear data (Suricata output, config files, PCAP files) and stop containersstatus- Show container statuslogs- Follow container logshelp- Show help information
To start Digger, use the start command with one of the following capture modes:
--mode-a- Mode A, for pcap replay mode--mode-b- Mode B, for live capture interface mode--mode-c- Mode C, for live capture using PCAP-over-IP (default if not specified)
When using Mode C, you can specify additional parameters:
--target-ip TARGET_IP(or-ip): IP address of the vulnbox (MANDATORY for Mode C)--date START_DATE: CTF start date in ISO formatYYYY-MM-DDThh:mm+ZZ:zz--tick-length LENGTH(or-t): Tick length in seconds (default: 120)--refresh-rate RATE(or-r): Auto-refresh rate in seconds (default: 30)--key ALGORITHM(or-k): SSH key algorithm for connection (default: ed25519)
Here you can find an example with the full configuration:
./start.py start --mode-c \
--target-ip 10.60.0.1 \
--date "1970-01-01T10:00+02:00" \
--tick-length 120 \
--refresh-rate 30 \
--key ed25519--no-build: Skip building Docker images (use existing images)--no-clean: Skip cleaning environment (keep existing data)
To stop running containers:
./start.py stopThe clear command provides granular control over data cleanup. You can use the interactive clear by running:
./start.py clearAlternatively, you can specify what to clean:
./start.py clear --config # (or -c): Clear .env and services_config.json
./start.py clear --suricata # (or -s): Clear Suricata output and stop containers
./start.py clear --pcap # (or -p): Clear PCAP filesTo rapidly delete everything listed above you can use the flag --all (or -A).
To avoid the cleanup, you can add the flag --no-clean to the startup command.
Check container status:
./start.py statusFollow container logs:
# Follow all container logs
./start.py logs
# Follow logs with tail limit
./start.py logs --tail 100
# Follow specific service logs
./start.py logs webapp --tail 50For a complete list of commands and options, run:
./start.py helpCommon command examples:
# Quick start Mode C with target IP
./start.py start --mode-c --target-ip 10.60.2.1
# Start without rebuilding images
./start.py start --mode-c --no-build
# Start preserving existing data
./start.py start --mode-c --no-clean
# Stop everything
./start.py stop
# Clean everything and stop
./start.py clear --all
# Monitor containers
./start.py status
./start.py logsYou can customize the configuration directly from the web interface, all the changes are saved in file called ctf_config.json.
Click on the settings icon in the top left corner to open the Service Manager modal. Here you can add:
- service name,
- ip:port used (we support services with multiple IPs and ports),
- colour (helps to identify the service in the flow list).
Warning
Note that if Digger is restarted without the --no-clean flag, the configuration in the web interface will be lost. Use ./start.py start --no-clean to preserve existing configuration.
In order to work properly and display correct informations Digger requires specific configuration about the CTF you are playing:
- tick length (default to 120 seconds)
- start datetime (default to present datetime)
- end datetime (default to present datetime + 8 hours)
Digger includes an auto-refresh feature that automatically refreshes the flow list with an interval specified in the configuration. After updating, the interface will scroll to the previously selected flow (if any), by default is every 60 seconds. The auto-refresh is client specific, that means every user can decide the refresh interval.
The auto-refresh feature can be toggled by clicking the Auto-Update button in the top left corner of the web interface.
You can update this values at runtime from Settings in the web interface.
flow_id is derived from timestamp (ms scale) and current flow parameters (such
as source and destination ports and addresses). See source code:
https://github.com/OISF/suricata/blob/suricata-6.0.13/src/flow.h#L680.
Most CTF uses OpenVPN or Wireguard for the "game" network interface on the vulnbox,
which means you can duplicate the traffic to an OpenSSH tun tunnel.
Using this method, Digger can run on another machine in live capture mode.
Warning
If you need to clone a physical Ethernet interface such as eth0,
you will need to use -o Tunnel=ethernet -w 5:5 in the SSH command line to create a tap.
To achieve traffic mirroring, you may use these steps as reference:
-
Enable SSH tunneling in vulnbox OpenSSH server:
echo -e 'PermitTunnel yes' | sudo tee -a /etc/ssh/sshd_config systemctl restart ssh
-
Create
tun5tunnel from the local machine to the vulnbox and uptun5on vulnbox:sudo ip tuntap add tun5 mode tun user $USER ssh -w 5:5 [email protected] ip link set tun5 up
-
Up
tun5on the local machine and starttcpdumpto create pcap files:sudo ip link set tun5 up sudo tcpdump -n -i tun5 -G 30 -Z root -w trace-%Y-%m-%d_%H-%M-%S.pcap -
Mirror
gametraffic totun5on the vulnbox. This can be done using Nftables netdevdupoption oningressandegress.
You can edit suricata rules in suricata/rules/suricata.rules, then reload the rules
using:
pkill -USR2 suricataThis project is a fork of Shovel, which included code under the following licenses:
In accordance with the "or later" clause of the GPL-2.0-or-later license, and the GPL compatibility of CC0 and MIT, this fork is distributed under the terms of the GNU General Public License, version 3.0 (GPL-3.0).
All future contributions to this repository will be licensed under GPL-3.0.