Ail Framework - Framework For Analysis Of Data Leaks

AIL is a modular framework to analyse potential information leaks from unstructured information sources similar pastes from Pastebin or similar services or unstructured information streams. AIL framework is flexible together with tin endure extended to back upward other functionalities to mine or procedure sensitive information (e.g. information leak prevention).

Features

Modular architecture to handgrip streams of unstructured or structured information
Default back upward for external ZMQ feeds, such equally provided yesteryear CIRCL or other providers
Multiple feed support
Each module tin procedure together with reprocess the information already processed yesteryear AIL
Detecting together with extracting URLs including their geographical place (e.g. IP address location)
Extracting together with validating potential leak of credit cards numbers, credentials, ...
Extracting together with validating e-mail addresses leaked including DNS MX validation
Module for extracting Tor .onion addresses (to endure farther processed for analysis)
Keep tracks of duplicates (and diffing betwixt each duplicate found)
Extracting together with validating potential hostnames (e.g. to feed Passive DNS systems)
A full-text indexer module to index unstructured information
Statistics on modules together with web
Real-time modules managing director inwards terminal
Global stance analysis for each providers based on nltk vader module
Terms, Set of damage together with Regex tracking together with occurrence
Many to a greater extent than modules for extracting call numbers, credentials together with others
Alerting to MISP to part constitute leaks inside a threat tidings platform using MISP standard
Detect together with decode encoded file (Base64, hex encoded or your ain decoding scheme) together with shop files
Detect Amazon AWS together with Google API keys
Detect Bitcoin address together with Bitcoin person keys
Detect person keys, certificate, keys (including SSH, OpenVPN)
Detect IBAN banking concern accounts
Tagging scheme alongside MISP Galaxy together with MISP Taxonomies tags
UI glue submission
Create events on MISP together with cases on The Hive
Automatic glue export at detection on MISP (events) together with The Hive (alerts) on selected tags
Extracted together with decoded files tin endure searched yesteryear engagement range, type of file (mime-type) together with encoding discovered
Graph relationships betwixt decoded file (hashes), similar PGP UIDs together with addresses of cryptocurrencies
Tor hidden services crawler to crawl together with parse output
Tor onion availability is monitored to discovery upward together with downwardly of hidden services
Browser hidden services are screenshot together with integrated inwards the analysed output including a blurring screenshot interface (to avoid "burning the eyes" of the safety analysis alongside specific content)
Tor hidden services is constituent of the measure framework, all the AIL modules are available to the crawled hidden services
Generic spider web crawler to trigger crawling on take or at regular interval URL or Tor hidden services

Installation
Type these dominance lines for a fully automated installation together with get-go AIL framework:

git clone https://github.com/CIRCL/AIL-framework.git cd AIL-framework ./installing_deps.sh  cd  /AIL-framework/ cd bin/ ./LAUNCH.sh -l

The default installing_deps.sh is for Debian together with Ubuntu based distributions.
There is besides a Travis file used for automating the installation that tin endure used to reach together with install AIL on other systems.
Requirement:

Python 3.5+

Installation Notes
In club to utilisation AIL combined alongside ZFS or unprivileged LXC it's necessary to disable Direct I/O inwards $AIL_HOME/configs/6382.conf yesteryear changing the value of the directive use_direct_io_for_flush_and_compaction to false.

Starting AIL

cd bin/ ./LAUNCH -l

Eventually you lot tin browse the condition of the AIL framework website at the next URL:

https://localhost:7000/

The default credentials for the spider web interface are located inwards DEFAULT_PASSWORD. This file is removed when you lot alter your password.

Training
CIRCL organises preparation on how to utilisation or extend the AIL framework. AIL preparation materials are available at https://www.circl.lu/services/ail-training-materials/.

HOWTO
HOWTO are available inwards HOWTO.md

Privacy together with GDPR
AIL information leaks analysis together with the GDPR inwards the context of collection, analysis together with sharing information leaks document provides an overview how to utilisation AIL inwards a lawfulness context particularly inwards the compass of General Data Protection Regulation.

Research using AIL
If you lot write academic paper, relying or using AIL, it tin endure cited alongside the next BibTeX:

@inproceedings{mokaddem2018ail,   title={AIL-The pattern together with implementation of an Analysis Information Leak framework},   author={Mokaddem, Sami together with Wagener, G{\'e}rard together with Dulaunoy, Alexandre},   booktitle={2018 IEEE International Conference on Big Data (Big Data)},   pages={5049--5057},   year={2018},   organization={IEEE} }

Screenshots

Tor hidden service crawler