infosecn1nja
/
HELK
mirror of https://github.com/infosecn1nja/HELK.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
HELK/README.md

133 lines
8.6 KiB
Markdown
Raw Normal View History

HELK - Alpha ELK 6.1.3 + ELK 6.1.3 version (Jun 30,2018 release) + Kafka Integration -- Bash, DockerFile & Docker Image + Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control) + Logstash: JVM Heap 2GB default + ELK (Init Files created) -- More control over service start + Left Linux DEB install bash script (deprecating it in next release) + ELK .yml files are not available to adjust deployment in an easier way. + Fixed Docker Run environment parameters to be call before pointing to the HELK image. + Edited every single file to have the right headers: -- ELK version 6.1.3 -- Aplha Version
2018-01-31 22:52:50 +00:00
# HELK [Alpha]
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
v0.1.6-alpha12132018 HELK base image + Updated to 0.0.3 HELK ELK Version + Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released) helk_install + Users can now select between two deployments: ++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT) ++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER) + Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option. helk-elastalert + Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g helk-elasticsearch + consolidated main configs in one + added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK. helk-logstash + updated to 6.5.3 + simplified pipeline to have only one folder + logstash-entrypoint script can now enable elastic password on all logstash output conf files. + New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT) helk-nginx + split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base) helk-kibana + Updated to version 6.5.3 + Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic helk-jupyter + updated Jupyterlab to 0.35.4 + updated jupyterhub to 0.9.4 + updated jupyterlab hub extension to 0.12.0 + updated ES_HADOOP to 6.5.3 + updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 + Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time helk-kafka-base + reduced docker container size + updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper) helk-kafka-broker + User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own. helk-zookeeper + reduced size of container + updated build to kafka 2.1.0 helk-KSQL + initial integration of KSQL + KSQL Server and KSQL CLI are available + Blog post coming soon ;)
2018-12-13 21:27:17 +00:00
![version](https://img.shields.io/badge/version-0.1.4-blue.svg?maxAge=2592000)
[![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
[![GitHub issues-closed](https://img.shields.io/github/issues-closed/Cyb3rward0g/HELK.svg)](https://GitHub.com/Cyb3rWard0g/HELK/issues?q=is%3Aissue+is%3Aclosed)
[![Twitter](https://img.shields.io/twitter/follow/THE_HELK.svg?style=social&label=Follow)](https://twitter.com/THE_HELK)
[![Open Source Love svg1](https://badges.frapsoft.com/os/v1/open-source.svg?v=103)](https://github.com/ellerbrock/open-source-badges/)
Updated Readme
2018-01-06 22:14:43 +00:00
A Hunting ELK (Elasticsearch, Logstash, Kibana) with advanced analytic capabilities.
Update README.md
2018-01-16 01:11:13 +00:00
Updated HELK's Design & README - Added ES-Hadoop connector to Design to show how Spark interacts with Elasticsearch - Updated README to-do list to add Kafka to the build
2018-01-16 01:07:44 +00:00
![alt text](resources/images/HELK_Design.png "HELK Infrastructure")
Update README.md
2017-04-14 05:29:04 +00:00
Update README.md
2017-06-29 15:21:59 +00:00
# Goals
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
Update README.md
2017-06-29 15:21:59 +00:00
* Provide a free hunting platform to the community and share the basics of Threat Hunting.
* Make sense of a large amount of event logs and add more context to suspicious events during hunting.
* Expedite the time it takes to deploy an ELK stack.
Updated Readme
2018-01-06 22:14:43 +00:00
* Improve the testing of hunting use cases in an easier and more affordable way.
Updated README Feedback taken. Changed Learn to Enable
2018-01-08 23:26:44 +00:00
* Enable Data Science via Apache Spark, GraphFrames & Jupyter Notebooks.
Update README.md
2017-06-29 15:21:59 +00:00
HELK - Alpha ELK 6.1.3 + ELK 6.1.3 version (Jun 30,2018 release) + Kafka Integration -- Bash, DockerFile & Docker Image + Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control) + Logstash: JVM Heap 2GB default + ELK (Init Files created) -- More control over service start + Left Linux DEB install bash script (deprecating it in next release) + ELK .yml files are not available to adjust deployment in an easier way. + Fixed Docker Run environment parameters to be call before pointing to the HELK image. + Edited every single file to have the right headers: -- ELK version 6.1.3 -- Aplha Version
2018-01-31 22:52:50 +00:00
# Current Status: Alpha
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
HELK - Alpha ELK 6.1.3 + ELK 6.1.3 version (Jun 30,2018 release) + Kafka Integration -- Bash, DockerFile & Docker Image + Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control) + Logstash: JVM Heap 2GB default + ELK (Init Files created) -- More control over service start + Left Linux DEB install bash script (deprecating it in next release) + ELK .yml files are not available to adjust deployment in an easier way. + Fixed Docker Run environment parameters to be call before pointing to the HELK image. + Edited every single file to have the right headers: -- ELK version 6.1.3 -- Aplha Version
2018-01-31 22:52:50 +00:00
The project is currently in an alpha stage, which means that the code and the functionality are still changing. We haven't yet tested the system with large data sources and in many scenarios. We invite you to try it and welcome any feedback.
# HELK Features
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
HELK - Alpha ELK 6.1.3 + ELK 6.1.3 version (Jun 30,2018 release) + Kafka Integration -- Bash, DockerFile & Docker Image + Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control) + Logstash: JVM Heap 2GB default + ELK (Init Files created) -- More control over service start + Left Linux DEB install bash script (deprecating it in next release) + ELK .yml files are not available to adjust deployment in an easier way. + Fixed Docker Run environment parameters to be call before pointing to the HELK image. + Edited every single file to have the right headers: -- ELK version 6.1.3 -- Aplha Version
2018-01-31 22:52:50 +00:00
* **Kafka:** A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable.
* **Elasticsearch:** A highly scalable open-source full-text search and analytics engine.
* **Logstash:** A data collection engine with real-time pipelining capabilities.
* **Kibana:** An open source analytics and visualization platform designed to work with Elasticsearch.
* **ES-Hadoop:** An open-source, stand-alone, self-contained, small library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such as Hive, Pig or Cascading or new upcoming libraries like Apache Spark ) to interact with Elasticsearch.
* **Spark:** A fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
* **GraphFrames:** A package for Apache Spark which provides DataFrame-based Graphs.
* **Jupyter Notebook:** An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.
Update README.md
2017-06-29 15:21:59 +00:00
# Resources
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
Updsted README
2018-04-10 07:03:19 +00:00
* [Welcome to HELK! : Enabling Advanced Analytics Capabilities](https://cyberwardog.blogspot.com/2018/04/welcome-to-helk-enabling-advanced_9.html)
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
* [Spark](https://spark.apache.org/docs/latest/index.html)
* [Spark Standalone Mode](https://spark.apache.org/docs/latest/spark-standalone.html)
Update README.md
2017-06-29 15:21:59 +00:00
* [Setting up a Pentesting.. I mean, a Threat Hunting Lab - Part 5](https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_98.html)
Updated README
2018-01-08 22:58:42 +00:00
* [An Integrated API for Mixing Graph and Relational Queries](https://cs.stanford.edu/~matei/papers/2016/grades_graphframes.pdf)
* [Graph queries in Spark SQL](https://www.slideshare.net/SparkSummit/graphframes-graph-queries-in-spark-sql)
* [Graphframes Overview](http://graphframes.github.io/index.html)
Update README.md
2017-06-29 15:21:59 +00:00
* [Elastic Producs](https://www.elastic.co/products)
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
* [Elastic Subscriptions](https://www.elastic.co/subscriptions)
Updated README
2018-01-08 22:58:42 +00:00
* [Elasticsearch Guide](https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html)
* [spujadas elk-docker](https://github.com/spujadas/elk-docker)
* [deviantony docker-elk](https://github.com/deviantony/docker-elk)
Update README.md
2017-06-29 15:21:59 +00:00
Update README.md
2017-05-26 06:11:09 +00:00
# Getting Started
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
## WIKI
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
* [Introduction](https://github.com/Cyb3rWard0g/HELK/wiki)
* [Architecture Overview](https://github.com/Cyb3rWard0g/HELK/wiki/Architecture-Overview)
* [Kafka](https://github.com/Cyb3rWard0g/HELK/wiki/Kafka)
* [Logstash](https://github.com/Cyb3rWard0g/HELK/wiki/Logstash)
* [Elasticsearch](https://github.com/Cyb3rWard0g/HELK/wiki/Elasticsearch)
* [Kibana](https://github.com/Cyb3rWard0g/HELK/wiki/Kibana)
* [Spark](https://github.com/Cyb3rWard0g/HELK/wiki/Spark)
* [Installation](https://github.com/Cyb3rWard0g/HELK/wiki/Installation)
Updated README & Added Images Added Dashboard and Discovery images Updated To-Do List
2018-01-08 23:20:50 +00:00
HELK 6.2.2 - 022518 HELK Design + moved everything to docker-compose approach for a more modular design. + separated the HELK in 3 services: ++helk-elk, helk-kafka, helk-analytics + Updated Design picture to show WEF ideas and also show Jupyter Lab integrations. HELK Docker-Compose + Added ESDATA volume to keep logs after contaners get stopped + Services restart automatically after reboot + created blank env file for Kafka service. This allows the host to pass its own local IP to Kafka. This is needed for advertised listener configs on each broker. HELK-ELK Version - Updated to 6.2.2 ELasticsearch - Added local docker network as part of the network.host option. This allows the HELK-ELK service to publish its docker local IP to other services/images in the docker compose environment. Logstash + minimal updates to certain configs (Mainly renaming files and replacing certain strings) Kibana + enableExternalUrls set to true for Vega visualization that need external libraries. Spark - Analytics + Renamed service to Analytics + Integrated Apache Toree to allow Scala kernel in Jupyter + Pyspark, Scala and SQL are now available in Jupyter Jupyter + Jupyter LAB has been enabled
2018-02-25 07:59:44 +00:00
## (Docker) Accessing the HELK's Images
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
By default, the HELK's containers are run in the background (Detached). You can see all your docker containers by running the following command:
```
sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a97bd895a2b3 cyb3rward0g/helk-spark-worker:2.3.0 "./spark-worker-entr…" About an hour ago Up About an hour 0.0.0.0:8082->8082/tcp helk-spark-worker2
cbb31f688e0a cyb3rward0g/helk-spark-worker:2.3.0 "./spark-worker-entr…" About an hour ago Up About an hour 0.0.0.0:8081->8081/tcp helk-spark-worker
5d58068aa7e3 cyb3rward0g/helk-kafka-broker:1.1.0 "./kafka-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:9092->9092/tcp helk-kafka-broker
bdb303b09878 cyb3rward0g/helk-kafka-broker:1.1.0 "./kafka-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:9093->9093/tcp helk-kafka-broker2
7761d1e43d37 cyb3rward0g/helk-nginx:0.0.2 "./nginx-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:80->80/tcp helk-nginx
ede2a2503030 cyb3rward0g/helk-jupyter:0.32.1 "./jupyter-entrypoin…" About an hour ago Up About an hour 0.0.0.0:4040->4040/tcp, 0.0.0.0:8880->8880/tcp helk-jupyter
ede19510e959 cyb3rward0g/helk-logstash:6.2.4 "/usr/local/bin/dock…" About an hour ago Up About an hour 5044/tcp, 9600/tcp helk-logstash
e92823b24b2d cyb3rward0g/helk-spark-master:2.3.0 "./spark-master-entr…" About an hour ago Up About an hour 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp helk-spark-master
6125921b310d cyb3rward0g/helk-kibana:6.2.4 "./kibana-entrypoint…" About an hour ago Up About an hour 5601/tcp helk-kibana
4321d609ae07 cyb3rward0g/helk-zookeeper:3.4.10 "./zookeeper-entrypo…" About an hour ago Up About an hour 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp helk-zookeeper
9cbca145fb3e cyb3rward0g/helk-elasticsearch:6.2.4 "/usr/local/bin/dock…" About an hour ago Up About an hour 9200/tcp, 9300/tcp helk-elasticsearch
```
Then, you will just have to pick which container you want to access and run the following following commands:
Updated Readme
2018-01-06 22:14:43 +00:00
```
HELK 6.2.2 - 022518 HELK Design + moved everything to docker-compose approach for a more modular design. + separated the HELK in 3 services: ++helk-elk, helk-kafka, helk-analytics + Updated Design picture to show WEF ideas and also show Jupyter Lab integrations. HELK Docker-Compose + Added ESDATA volume to keep logs after contaners get stopped + Services restart automatically after reboot + created blank env file for Kafka service. This allows the host to pass its own local IP to Kafka. This is needed for advertised listener configs on each broker. HELK-ELK Version - Updated to 6.2.2 ELasticsearch - Added local docker network as part of the network.host option. This allows the HELK-ELK service to publish its docker local IP to other services/images in the docker compose environment. Logstash + minimal updates to certain configs (Mainly renaming files and replacing certain strings) Kibana + enableExternalUrls set to true for Vega visualization that need external libraries. Spark - Analytics + Renamed service to Analytics + Integrated Apache Toree to allow Scala kernel in Jupyter + Pyspark, Scala and SQL are now available in Jupyter Jupyter + Jupyter LAB has been enabled
2018-02-25 07:59:44 +00:00
sudo docker exec -ti <image-name> bash
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
root@ede2a2503030:/opt/helk/scripts#
Updated Readme
2018-01-06 22:14:43 +00:00
```
updated Readme, created enrichments folder, and organized logstash configs
2017-08-12 04:50:56 +00:00
Update README.md
2017-06-29 15:21:59 +00:00
# Author
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
Updated README
2018-01-08 22:58:42 +00:00
* Roberto Rodriguez [@Cyb3rWard0g](https://twitter.com/Cyb3rWard0g) [@THE_HELK](https://twitter.com/THE_HELK)
Update README.md
2017-05-26 06:11:09 +00:00
Update README.md
2017-06-29 15:21:59 +00:00
# Contributors
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
* Jose Luis Rodriguez [@Cyb3rPandaH](https://twitter.com/Cyb3rPandaH)
Update ACE enrichment and add docker volumes Added docker volume for elasticsearch data to persist Added documentation for increasing memory of elasticsearch Updated ACE logstash input for durable queue
2017-09-07 23:21:12 +00:00
* Robby Winchester [@robwinchester3](https://twitter.com/robwinchester3)
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
* Jared Atkinson [@jaredatkinson](https://twitter.com/jaredcatkinson)
Updated README
2018-01-08 22:58:42 +00:00
* Nate Guagenti [@neu5ron](https://twitter.com/neu5ron)
Contributors & Alpha Versions + Added Lee Christensen to contributors list + Updated Main install script to reflect Alpha version and latest ELK version (6.1.3)
2018-01-31 23:36:46 +00:00
* Lee Christensen [@tifkin_](https://twitter.com/tifkin_)
Update README.md
2017-06-29 15:21:59 +00:00
# Contributing
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
Updated README
2018-01-08 22:58:42 +00:00
There are a few things that I would like to accomplish with the HELK as shown in the To-Do list below. I would love to make the HELK a stable build for everyone in the community. If you are interested on making this build a more robust one and adding some cool features to it, PLEASE feel free to submit a pull request. #SharingIsCaring
Update README.md
2017-06-29 15:21:59 +00:00
HELK-07122018 License: GPL-3.0 Update ++ Updated all the local documents ++ Docker images in Dockerhub in progreess Docker-Compose ++ Created two options: basic and trial ELK Stack Docker Files ++ Created Trial Folders to make sure the configurations are set properly for when the user selects trial version of HELK. ++++ HELK trial = x-pack + trial license + security enabled ++ Deprecating the HELKs Platinum's Branch. Merging that branch with the HELKs master to allow user to select the type of license during the install process. Jupyter ++ Getting ready for Jupyterhub ++ Created two folders: basic and trial to allow elasticsearch interaciton with username and password hardcoded in the spark session. trial license requires any interaction with elasticsearch to be authenticated. Kibana ++ Added trial folder with scripts that set up security configs for the trial version of HELK. It creates users and roles to test the security features of x-pack Logstash ++ Created trial folder with another pipeline folder in it. The pipeline in trial has output configs with elasticsearch's username and password hardcoded. Ready for when the user sets the build with trial license and wants to send logs to elasticsearch. The logstash configs are the same as the ones from the defailt pipeline. They only have username and password configs on all the output configs. Nginx ++ set trial folder with the right config to allow Kibana handle the authentication process when user builds and installs HELK with a trial license. No need for nginx to handle the authentication. helk_install bash script ++ Updated script to handle license choice : basic or trial ++ basic license is selected by default. If user selects trial, it runs the specific docker-compose file needed to build and install HELK with the right trial configs. ++ Updated also the CLI options. User now will have to specify the license for HELK. Example: sudo ./helk_install.sh -i 192.168.64.131 -l basic
2018-07-12 04:29:09 +00:00
# License: GPL-3.0
[ HELK's GNU General Public License](https://github.com/Cyb3rWard0g/HELK/blob/master/LICENSE)
Update README.md
2017-06-29 15:21:59 +00:00
# TO-Do
Minor updates - Docker compose file order of volumes - Kibana Dockerfile: removed comment - README: Added Contributor
2018-06-12 05:28:26 +00:00
Updated README
2018-01-08 22:58:42 +00:00
- [X] Upload basic Kibana Dashboards
- [X] Integrate Spark & Graphframes
- [X] Add Jupyter Notebook on the top of Spark
HELK - Alpha ELK 6.1.3 + ELK 6.1.3 version (Jun 30,2018 release) + Kafka Integration -- Bash, DockerFile & Docker Image + Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control) + Logstash: JVM Heap 2GB default + ELK (Init Files created) -- More control over service start + Left Linux DEB install bash script (deprecating it in next release) + ELK .yml files are not available to adjust deployment in an easier way. + Fixed Docker Run environment parameters to be call before pointing to the HELK image. + Edited every single file to have the right headers: -- ELK version 6.1.3 -- Aplha Version
2018-01-31 22:52:50 +00:00
- [X] Kafka Integration
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
- [X] Default X-Pack Basic - Free License Build for ELKStack
- [X] Spark Standalone Cluster Manager integration
- [X] Apache Arrow Integration for Pandas Dataframes
- [ ] Zepplin Notebook Docker option
v0.1.6-alpha12132018 HELK base image + Updated to 0.0.3 HELK ELK Version + Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released) helk_install + Users can now select between two deployments: ++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT) ++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER) + Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option. helk-elastalert + Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g helk-elasticsearch + consolidated main configs in one + added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK. helk-logstash + updated to 6.5.3 + simplified pipeline to have only one folder + logstash-entrypoint script can now enable elastic password on all logstash output conf files. + New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT) helk-nginx + split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base) helk-kibana + Updated to version 6.5.3 + Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic helk-jupyter + updated Jupyterlab to 0.35.4 + updated jupyterhub to 0.9.4 + updated jupyterlab hub extension to 0.12.0 + updated ES_HADOOP to 6.5.3 + updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 + Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time helk-kafka-base + reduced docker container size + updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper) helk-kafka-broker + User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own. helk-zookeeper + reduced size of container + updated build to kafka 2.1.0 helk-KSQL + initial integration of KSQL + KSQL Server and KSQL CLI are available + Blog post coming soon ;)
2018-12-13 21:27:17 +00:00
- [X] KSQL Client & Server Deployment (Waiting for v5.0)
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
- [ ] Kubernetes Cluster Migration
- [ ] OSQuery Data Ingestion
v0.1.6-alpha12132018 HELK base image + Updated to 0.0.3 HELK ELK Version + Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released) helk_install + Users can now select between two deployments: ++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT) ++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER) + Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option. helk-elastalert + Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g helk-elasticsearch + consolidated main configs in one + added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK. helk-logstash + updated to 6.5.3 + simplified pipeline to have only one folder + logstash-entrypoint script can now enable elastic password on all logstash output conf files. + New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT) helk-nginx + split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base) helk-kibana + Updated to version 6.5.3 + Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic helk-jupyter + updated Jupyterlab to 0.35.4 + updated jupyterhub to 0.9.4 + updated jupyterlab hub extension to 0.12.0 + updated ES_HADOOP to 6.5.3 + updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 + Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time helk-kafka-base + reduced docker container size + updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper) helk-kafka-broker + User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own. helk-zookeeper + reduced size of container + updated build to kafka 2.1.0 helk-KSQL + initial integration of KSQL + KSQL Server and KSQL CLI are available + Blog post coming soon ;)
2018-12-13 21:27:17 +00:00
- [X] Create Jupyter Notebooks showing how to use Spark & GraphFrames
Updated README & Added Images Added Dashboard and Discovery images Updated To-Do List
2018-01-08 23:20:50 +00:00
- [ ] MITRE ATT&CK mapping to logs or dashboards
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
- [ ] Cypher for Apache Spark Integration (Might have to switch from Jupyter to Zeppelin Notebook)
Updated README
2018-01-08 22:58:42 +00:00
- [ ] Somehow integrate neo4j spark connectors with build
v0.1.6-alpha12132018 HELK base image + Updated to 0.0.3 HELK ELK Version + Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released) helk_install + Users can now select between two deployments: ++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT) ++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER) + Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option. helk-elastalert + Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g helk-elasticsearch + consolidated main configs in one + added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK. helk-logstash + updated to 6.5.3 + simplified pipeline to have only one folder + logstash-entrypoint script can now enable elastic password on all logstash output conf files. + New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT) helk-nginx + split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base) helk-kibana + Updated to version 6.5.3 + Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic helk-jupyter + updated Jupyterlab to 0.35.4 + updated jupyterhub to 0.9.4 + updated jupyterlab hub extension to 0.12.0 + updated ES_HADOOP to 6.5.3 + updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0 + Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time helk-kafka-base + reduced docker container size + updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper) helk-kafka-broker + User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own. helk-zookeeper + reduced size of container + updated build to kafka 2.1.0 helk-KSQL + initial integration of KSQL + KSQL Server and KSQL CLI are available + Blog post coming soon ;)
2018-12-13 21:27:17 +00:00
- [X] Nxlog parsers (Logstash Filters)
Updated README
2018-01-08 22:58:42 +00:00
- [ ] Add more network data sources (i.e Bro)
Updated Apache Spark Kafka jar 2.3.0
2018-03-04 04:44:09 +00:00
- [ ] Research & integrate spark structured direct streaming
Update README.md fixed markdown code and subtitles
2017-05-26 06:31:12 +00:00
More coming soon...