Helk - The Hunting Elk

The Hunting ELK or precisely the HELK is i of the showtime opened upward source hunt platforms amongst advanced analytics capabilities such equally SQL declarative language, graphing, structured streaming, together with fifty-fifty machine learning via Jupyter notebooks together with Apache Spark over an ELK stack. This projection was developed primarily for research, but due to its flexible blueprint together with nub components, it tin live deployed inwards larger environments amongst the correct configurations together with scalable infrastructure.

Goals

• Provide an opened upward source hunting platform to the community together with portion the basics of Threat Hunting.
• Expedite the fourth dimension it takes to deploy a hunt platform.
• Improve the testing together with evolution of hunting utilization cases inwards an easier together with to a greater extent than affordable way.
• Enable Data Science capabilities piece analyzing information via Apache Spark, GraphFrames & Jupyter Notebooks.

Current Status: Alpha
The projection is currently inwards an alpha stage, which agency that the code together with the functionality are all the same changing. We haven't yet tested the organisation amongst large information sources together with inwards many scenarios. We invite you lot to endeavor it together with welcome whatsoever feedback.

HELK Features

• Kafka: Influenza A virus subtype H5N1 distributed publish-subscribe messaging organisation that is designed to live fast, scalable, fault-tolerant, together with durable.
• Elasticsearch: Influenza A virus subtype H5N1 highly scalable open-source full-text search together with analytics engine.
• Logstash: Influenza A virus subtype H5N1 information collection engine amongst real-time pipelining capabilities.
• Kibana: An opened upward source analytics together with visualization platform designed to piece of work amongst Elasticsearch.
• ES-Hadoop: An open-source, stand-alone, self-contained, small-scale library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such equally Hive, Pig or Cascading or novel upcoming libraries similar Apache Spark ) to interact amongst Elasticsearch.
• Spark: Influenza A virus subtype H5N1 fast together with general-purpose cluster computing system. It provides high-level APIs inwards Java, Scala, Python together with R, together with an optimized engine that supports full general execution graphs.
• GraphFrames: Influenza A virus subtype H5N1 bundle for Apache Spark which provides DataFrame-based Graphs.
• Jupyter Notebook: An open-source spider web application that allows you lot to practice together with portion documents that incorporate alive code, equations, visualizations together with narrative text.
• KSQL: Confluent KSQL is the opened upward source, streaming SQL engine that enables real-time information processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for current processing on Kafka, without the need to write code inwards a programming linguistic communication such equally Java or Python
• Elastalert: ElastAlert is a unproblematic framework for alerting on anomalies, spikes, or other patterns of involvement from information inwards Elasticsearch
  • Sigma: Sigma is a generic together with opened upward signature format that allows you lot to depict relevant log events inwards a straightforward manner.

Getting Started

WIKI

• Introduction
• Architecture Overview
  • Kafka
  • Logstash
  • Elasticsearch
  • Kibana
  • Spark
• Installation

(Docker) Accessing the HELK's Images
By default, the HELK's containers are run inwards the background (Detached). You tin run across all your docker containers past times running the next command:

sudo docker ps  CONTAINER ID        IMAGE                                  COMMAND                  CREATED             STATUS              PORTS                                            NAMES a97bd895a2b3        cyb3rward0g/helk-spark-worker:2.3.0    "./spark-worker-entr…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:8082->8082/tcp                           helk-spark-worker2 cbb31f688e0a        cyb3rward0g/helk-spark-worker:2.3.0    "./spark-worker-entr…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:8081->8081/tcp                           helk-spark-worker 5d58068aa7e3        cyb3rward0g/helk-kafka-broker:1.1.0    "./kafka-entrypoint.…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:9092->9092/tcp                           helk-kafka-broker bdb303b09878        cyb3rward0g/helk-kafka-broker:1.1.0    "./kafka-entrypoint.…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:9093->9093/tcp                              helk-kafka-broker2 7761d1e43d37        cyb3rward0g/helk-nginx:0.0.2           "./nginx-entrypoint.…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:80->80/tcp                               helk-nginx ede2a2503030        cyb3rward0g/helk-jupyter:0.32.1        "./jupyter-entrypoin…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:4040->4040/tcp, 0.0.0.0:8880->8880/tcp   helk-jupyter ede19510e959        cyb3rward0g/helk-logstash:6.2.4        "/usr/local/bin/dock…"   About an lx minutes agone   Up About an lx minutes    5044/tcp, 9600/tcp                               helk-logstash e92823b24b2d        cyb3rward0g/helk-spark-master:2.3.0    "./spark-master-entr…"   About an lx minutes agone   Up About an lx minutes    0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp   helk-spark-master 6125921b310d        cyb3rward0g/helk-kibana:6.2.4          "./kibana-entrypoint…"   About an lx minutes agone   Up About an lx minutes    5601/tcp                                            helk-kibana 4321d609ae07        cyb3rward0g/helk-zookeeper:3.4.10      "./zookeeper-entrypo…"   About an lx minutes agone   Up About an lx minutes    2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp       helk-zookeeper 9cbca145fb3e        cyb3rward0g/helk-elasticsearch:6.2.4   "/usr/local/bin/dock…"   About an lx minutes agone   Up About an lx minutes    9200/tcp, 9300/tcp                               helk-elasticsearch

Then, you lot volition precisely guide hold to alternative which container you lot desire to access together with run the next following commands:

sudo docker exec -ti  bash root@ede2a2503030:/opt/helk/scripts#

Resources

• Welcome to HELK! : Enabling Advanced Analytics Capabilities
• Spark
• Spark Standalone Mode
• Setting upward a Pentesting.. I mean, a Threat Hunting Lab - Part 5
• An Integrated API for Mixing Graph together with Relational Queries
• Graph queries inwards Spark SQL
• Graphframes Overview
• Elastic Producs
• Elastic Subscriptions
• Elasticsearch Guide
• spujadas elk-docker
• deviantony docker-elk

Author

• Roberto Rodriguez @Cyb3rWard0g @THE_HELK

Contributors

• Jose Luis Rodriguez @Cyb3rPandaH
• Robby Winchester @robwinchester3
• Jared Atkinson @jaredatkinson
• Nate Guagenti @neu5ron
• Lee Christensen @tifkin_

Contributing
There are a few things that I would similar to make amongst the HELK equally shown inwards the To-Do listing below. I would dearest to brand the HELK a stable laid upward for everyone inwards the community. If you lot are interested on making this laid upward a to a greater extent than robust i together with adding approximately cool features to it, PLEASE experience gratis to submit a line request. #SharingIsCaring

Download HELK