Docker Compose
++ Updated Spark images to 2.3.1
ELK Stack
++ Docker Images updated to 6.3.1
helk-jupyter
++ Preparing Jupyter for Jupyterhub
++ Spark base image now comes with a sparkuser user
++ Updated es-hadoop package to 6.3.1
helk-logstash
++ Updated Sysmon parser to transform new datafield from Sysmon V8.0. RuleName
helk-spark-base
++ Images updated to 2.3.1
helk_install
++ fixed https://github.com/Cyb3rWard0g/HELK/issues/81
++ Updated banner to show right version
HELK Version
+ ELK update tp 6.3.0
Logstash
+ Integrated ATT&CK CTI to the build. Created from https://github.com/Cyb3rWard0g/ATTACK-Python-Client/tree/master/integrations/helk_cti
+ Added the mitre_attack file to the build which contains information from Enterprise, PRE and Mobile Matrices
+ Enabled x-pack monitoring (new feature)
Kibana
+ Added Dashboards for the ATT&CK Integration
helk_install script
+ reduced docker-compose build and run steps to one
scripts
+ Added script export_attack.py to export the file used for logstash and kibana.
Updated several errors that I got after testing a few configurations.
Logstash
- Updated Field Names for Sysmon and Security Logs
- Updated Logstash Templates to maintain consistency
- Updated Dockerfile to directly use official Centos Elastic Docker Image
- Updated Logstash main config file to update pipeline batch value and the Kafka input config to fix https://github.com/Cyb3rWard0g/HELK/issues/73
- Updated Logstash Security parser to fix:
-- https://github.com/Cyb3rWard0g/HELK/issues/71
-- https://github.com/Cyb3rWard0g/HELK/issues/72
Kibana & Elasticsearch
- Updated Docker files to directly use Official Centos Elastic Docker images with only a few updates.
Docker Compose File
- Updated file to mount Logstash, Kibana and Elasticsearch config files, dashboards and output_templates. This will simplify utilization of custom pipes, visualizations and dashboards. User can now just replace those the local folder and HELK will use those configurations.
- ES_JAVA_OPTS is now set to 6GB by default. No more MEM string. It confuses the user.
helk_install.sh script
- Updated the string that it replaces when setting ES JAVA OPTS. It went from looking for the string "MEME" to "6GB".
Addressed https://github.com/Cyb3rWard0g/HELK/issues/67 where I was grabbing Disk size in human readable format which was breaking the calculations when the HELK was being built on a computer with TB space.
Logstash
- Added Local Pipeline to the build to allow custom local configurations
- updated sysmon config to fix https://github.com/Cyb3rWard0g/HELK/issues/63
- removed port exposed in local logstash Dockerfile. It will be pushed to official docker image in the next update
- removed logstash init file (not being used anymore)
Zeppelin
- not available yet
- initial draft dockerfile
- created spark-defaults file for future zeppelin dockerfile
Install Script
- incrased minimum memory size required
Todo:
* Structure (scripts in directory)
* Recognition when Kibana and index config of it gets available (polling)
* Cron job for auto update
* Integration in compose file
## Overall
+ Removed the Init files dependencies on all containers
+ Added more resources to the resources folder (papers and presentations)
+ Updated to-do list on main README
+ Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43)
+ Updated WIki and added new images to it
+ Started documenting potential error messages or bugs with a few quick fixes
## Helk Install Script
+ Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK)
## ELK Stack
+ Started using Elastic Docker Images as a base
+ Updated ELK stack to 6.2.4 version
+ X-Pack Basic Free License attached to build automatically
+ Monitoring capabilities are now enabled in the build (Reason why Cerebro went away)
## Spark
+ Integrated Spark Standalone Cluster Manager
+ Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code
+ Added Spark Master and Worker Docker Images
+ Build runs now with 2 Workers and 1 Master by default.
+ Apache Arrow is enabled for Pandas Dataframe optimization
+ Created Spark-Base Docker Image (Applied to the Jupyter Image)
## Kafka
+ Kafka Container was split in Kafka Brokers and one Zookeeper
+ Helk runs with 2 Kafka Brokers and 1 Zookeeper by default
## Jupyter Container
+ Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess
+ New packages were added:
++ nxviz
++ hiveplot
++ pyarrow
+ Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
Removed OTX Enrichment for now to reduce the load on logstash and keep it clean for now. It will be added in the future. Implementation is already developed.
Docker-Compose File
+ Split helk-elk service in 3 (Logstash, Kibana, Logstash)
HELK-base
+ New Docker Base image applied to all HELK's Docker images
HELK-analytics
+ updated file due to new helk-base image
HELK-elk
+ Removed Helk-elk folder
HELK-kafka
+ Updated it to version 1.1.0
HELK-Logstash
+ Updated all files to point to helk-kafka and helk-elasticsearch (New image after splitting helk-elk)
New Docker Images
+ helk-elasticsearch
+ helk-logstash
+ helk-kibana
+ helk-nginx
HELK-nginx
+ Removed route to elasticsearch:8082. Cerebro now can point to 172.18.0.2 (Internal Docker IP)
HELK-Install
+ organized script a little better by creating install_dockerl and install_docker_compose functions
HELK-kibana
+ updated Kibana configuration to set Kibana server to the name of the service helk-kibana. It allows remote connections to it (internally among docer images)
+ Updated elasticsearch url to new docker image (helk-elasticsearch:9200)
HELK-kafka
+ updated internal listeners on each broker to helk-kafka
Docker-Compose file
+ Updated Image versions
++ helk-elk:6.2.3
++ helk-kafka:1.0.1
++ helk-analytics:0.0.2
HELK-ANALYTICS
+ Upgraded spark to version 2.3.0
++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html
+ Upgraded Jupyter Lab to 0.31.12
+ Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3.
+ Upgraded ES-Hadoop package to version 6.2.3
++ Check release notes:
https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html
HELK-ELK
+ Upgraded elastic components to 6.2.3
++ Check elasticsearch release notes:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html
++ No changes for Kibana
++ Check Logstash release notes:
https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html
+ Logstash kafka input now adds metadata from kafka. Topic name, etc.
+ Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field.
+ logstash config 11-winevent-sysmon-filter.conf
++ removed field "user". This was causing issues when parsing events with Spark.
HELK-KAFKA
+ Upgraded Kafka to version 2.11-1.0.1
++ Check kafka release notes:
https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html
+ Removed sleep time for kafka init file
+ updated kafka entrypoint updating version values
HELK helk_install main script
+ Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33
HELK Winlogbeat install script
+ Updated beat version to 6.2.3
Dev Dua
Roberto Rodriguez
Robert Simplicio
Thomas Patzke