mirror of https://github.com/infosecn1nja/HELK.git
d623246f4c
Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available |
||
|---|---|---|
| cerebro | ||
| elasticsearch | ||
| enrichments | ||
| kafka | ||
| kibana | ||
| logstash | ||
| nginx | ||
| resources/images | ||
| scripts | ||
| spark | ||
| training/jupyter_notebooks/getting_started | ||
| winlogbeat | ||
| .gitignore | ||
| Dockerfile | ||
| LICENSE | ||
| README.md | ||
| helk_install.sh | ||
README.md
HELK [Alpha]
A Hunting ELK (Elasticsearch, Logstash, Kibana) with advanced analytic capabilities.
Goals
- Provide a free hunting platform to the community and share the basics of Threat Hunting.
- Make sense of a large amount of event logs and add more context to suspicious events during hunting.
- Expedite the time it takes to deploy an ELK stack.
- Improve the testing of hunting use cases in an easier and more affordable way.
- Enable Data Science via Apache Spark, GraphFrames & Jupyter Notebooks.
Current Status: Alpha
The project is currently in an alpha stage, which means that the code and the functionality are still changing. We haven't yet tested the system with large data sources and in many scenarios. We invite you to try it and welcome any feedback.
HELK Features
- Kafka: A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable.
- Elasticsearch: A highly scalable open-source full-text search and analytics engine.
- Logstash: A data collection engine with real-time pipelining capabilities.
- Kibana: An open source analytics and visualization platform designed to work with Elasticsearch.
- ES-Hadoop: An open-source, stand-alone, self-contained, small library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such as Hive, Pig or Cascading or new upcoming libraries like Apache Spark ) to interact with Elasticsearch.
- Spark: A fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
- GraphFrames: A package for Apache Spark which provides DataFrame-based Graphs.
- Jupyter Notebook: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.
Resources
- Setting up a Pentesting.. I mean, a Threat Hunting Lab - Part 5
- An Integrated API for Mixing Graph and Relational Queries
- Graph queries in Spark SQL
- Graphframes Overview
- Elastic Producs
- Elasticsearch Guide
- spujadas elk-docker
- deviantony docker-elk
Getting Started
WIKI
(Docker) Accessing the HELK's container
By default, the HELK's container is run in the background. Therefore, you will have to access your docker container by running the following commands:
sudo docker exec -ti helk bash
root@7a9d6443a4bf:/opt/helk/scripts#
Author
- Roberto Rodriguez @Cyb3rWard0g @THE_HELK
Contributors
- Robby Winchester @robwinchester3
- Jared Atkinson @jaredatkinson
- Nate Guagenti @neu5ron
- Jordan Potti @ok_bye_now
- esebese esebese
- Lee Christensen @tifkin_
Contributing
There are a few things that I would like to accomplish with the HELK as shown in the To-Do list below. I would love to make the HELK a stable build for everyone in the community. If you are interested on making this build a more robust one and adding some cool features to it, PLEASE feel free to submit a pull request. #SharingIsCaring
TO-Do
- Upload basic Kibana Dashboards
- Integrate Spark & Graphframes
- Add Jupyter Notebook on the top of Spark
- Kafka Integration
- Create Jupyter Notebooks showing how to use Spark & GraphFrames
- Enhance elasticsearch configuration to make it more scalable
- MITRE ATT&CK mapping to logs or dashboards
- Cypher for Apache Spark Integration (Might have to switch from Jupyter to Zeppelin Notebook)
- Somehow integrate neo4j spark connectors with build
- Install Elastalert
- Create Elastalert rules
- Nxlog parsers (Logstash Filters)
- Add more network data sources (i.e Bro)
More coming soon...