Turbinia - Automation Too Scaling Of Digital Forensics Tools
Turbinia is an open-source framework for deploying, managing, as well as running distributed forensic workloads. It is intended to automate running of mutual forensic processing tools (i.e. Plaso, TSK, strings, etc) to attention amongst processing evidence inwards the Cloud, scaling the processing of large amounts of evidence, as well as decreasing reply fourth dimension yesteryear parallelizing processing where possible.
How it works
Turbinia is composed of dissimilar components for the client, server as well as the workers. These components tin dismiss live on run inwards the Cloud, on local machines, or as a hybrid of both. The Turbinia customer makes requests to procedure evidence to the Turbinia server. The Turbinia server creates logical jobs from these incoming user requests, which creates as well as schedules forensic processing tasks to live on run yesteryear the workers. The evidence to live on processed volition live on dissever upwards yesteryear the jobs when possible, as well as many tasks tin dismiss live on created inwards social club to procedure the evidence inwards parallel. One or to a greater extent than workers run continuously to procedure tasks from the server. Any novel evidence created or discovered yesteryear the tasks volition live on fed dorsum into Turbinia for farther processing.
Communication from the customer to the server is currently done amongst either Google Cloud PubSub or Kombu messaging. The worker implementation tin dismiss role either PSQ (a Google Cloud PubSub Task Queue) or Celery for task scheduling.
More information on Turbinia as well as how it plant tin dismiss live on found here.
Status
Turbinia is currently inwards Alpha release.
Installation
There is an rough installation guide here.
Usage
The basic steps to teach things running afterwards the initial installation as well as configuration are:
- Start Turbinia server element amongst
turbiniactl servercommand - Start ane or to a greater extent than Turbinia workers amongst
turbiniactl psqworker - Send evidence to live on processed from the turbinia customer amongst
turbiniactl ${evidencetype} - Check status of running tasks amongst
turbiniactl status
$ turbiniactl --help usage: turbiniactl [-h] [-q] [-v] [-d] [-a] [-f] [-o OUTPUT_DIR] [-L LOG_FILE] [-r REQUEST_ID] [-R] [-S] [-C] [-V] [-D] [-F FILTER_PATTERNS_FILE] [-j JOBS_WHITELIST] [-J JOBS_BLACKLIST] [-p POLL_INTERVAL] [-t TASK] [-w] ... optional arguments: -h, --help present this attention message as well as move out -q, --quiet Show minimal output -v, --verbose Show verbose output -d, --debug Show debug output -a, --all_fields Show all task status fields inwards output -f, --force_evidence Force evidence processing asking inwards potentially dangerous weather -o OUTPUT_DIR, --output_dir OUTPUT_DIR Directory path for output -L LOG_FILE, --log_file LOG_FILE Log file -r REQUEST_ID, --request_id REQUEST_ID Create novel requests amongst this Request ID -R, --run_local Run completely locally without whatsoever server or other infrastructure. This tin dismiss live on used to run one-off Tasks to procedure information locally. -S, --server Run Turbinia Server indefinitely -C, --use_celery Pass this flag when using Celery/Kombu for task queuing as well as messaging (instead of Google PSQ/pubsub) -V, --version Show the version -D, --dump_json Dump JSON output of Turbinia Request instead of sending it -F FILTER_PATTERNS_FILE, --filter_patterns_file FILTER_PATTERNS_FILE H5N1 file containing newline separated string patterns to filter text based evidence files amongst (in extended grep regex format). This filtered output volition live on inwards improver to the consummate output -j JOBS_WHITELIST, --jobs_whitelist JOBS_WHITELIST H5N1 whitelist for Jobs that nosotros volition allow to run (note that it volition non forcefulness them to run). -J JOBS_BLACKLIST, --jobs_blacklist JOBS_BLACKLIST H5N1 blacklist for Jobs nosotros volition non allow to run -p POLL_INTERVAL, --poll_interval POLL_INTERVAL Number of seconds to expect betwixt polling for task country information -t TASK, --task TASK The mention of a unmarried Task to run locally (must live on used amongst --run_local. -w, --wait Wait to move out until all tasks for the given asking accept completed Commands: rawdisk Process RawDisk as Evidence googleclouddisk Process Google Cloud Persistent Disk as Evidence googleclouddiskembedded Process Google Cloud Persistent Disk amongst an embedded raw disk picture as Evidence directory Process a directory as Evidence listjobs List all available jobs psqworker Run PSQ worker celeryworker Run Celery worker status Get Turbinia Task status server Run Turbinia Server --server is specified, it volition offset upwards its ain Turbinia server process. Here's the turbiniactl usage for adding a raw disk type of evidence to live on processed yesteryear Turbinia:$ ./turbiniactl rawdisk -h usage: turbiniactl rawdisk [-h] -l LOCAL_PATH [-s SOURCE] [-n NAME] optional arguments: -h, --help present this attention message as well as move out -l LOCAL_PATH, --local_path LOCAL_PATH Local path to the evidence -s SOURCE, --source SOURCE Description of the source of the evidence -n NAME, --name NAME Descriptive mention of the evidenceOther documentation
- Installation
- How it works
- Contributing to Turbinia
- Developing novel Tasks
- FAQ
- Debugging as well as Common Errors
Notes
- Turbinia currently assumes that Evidence is every bit available to all worker nodes (e.g. through locally mapped storage, or through attachable persistent Google Cloud Disks, etc).
- Not all evidence types are supported yet
- Still solely a small-scale position out of processing task types supported, but to a greater extent than are beingness developed.
Obligatory Fine Print
This is non an official Google production (experimental or otherwise), it is simply code that happens to live on owned yesteryear Google.
