All
+ Moved all to docker folder. Getting ready to start sharing other ways to deploy helk (terraform & Packer maybe)
Compose-files
+ Basic & Trial Elastic Subscriptions available now and can be automatically managed via the helk_install script
ELK Version : 6.3.2
Elasticsearch
+ Set 4GB for ES_JAVA_OPTS by default allowing the modification of it via docker-compose and calculating half of the host memory if it is not set
+ Added Entrypoint script and using docker-entrypoint to start ES
Logstash
+ Big Pipeline Update by Nate Guagenti (@neu5ron)
++better cli & file name searching
++”dst_ip_public:true” filter out all rfc1918/non-routable
++Geo ASName
++Identification of 16+ windows IP fields
++Arrayed IPs support
++IPv6&IPv4 differentiation
++removing “-“ values and MORE!!!
++ THANK YOU SO MUCH NATE!!!
++ PR: https://github.com/Cyb3rWard0g/HELK/pull/93
+ Added entrypoint script to push new output_templates straight to Elasticsearch per Nate's recommendation
+ Starting Logstash now with docker-entrypoint
+ "event_data" is now taken out of winlogbeat logs to allow integration with nxlog (sauce added by Nate Guagenti (@neu5ron)
Kibana
+ Kibana yml file updated to allow a longer time for timeout
Nginx:
+ it handles communications to Kibana and Jupyterhub via port 443 SSL
+ certificate and key get created at build time
+ Nate added several settings to improve the way how nginx operates
Jupyterhub
+ Multiple users and mulitple notebooks open at the same time are possible now
+ Jupytehub now has 3 users hunter1,hunter2.hunter3 and password patterh is <user>P@ssw0rd!
+ Every notebook created is also JupyterLab
+ Updated ES-Hadoop 6.3.2
Kafka Update
+ 1.1.1 Update
Spark Master + Brokers
+ reduce memory for brokers by default to 512m
Resources:
+ Added new images for Wiki
License: GPL-3.0 Update
++ Updated all the local documents
++ Docker images in Dockerhub in progreess
Docker-Compose
++ Created two options: basic and trial
ELK Stack Docker Files
++ Created Trial Folders to make sure the configurations are set properly for when the user selects trial version of HELK.
++++ HELK trial = x-pack + trial license + security enabled
++ Deprecating the HELKs Platinum's Branch. Merging that branch with the HELKs master to allow user to select the type of license during the install process.
Jupyter
++ Getting ready for Jupyterhub
++ Created two folders: basic and trial to allow elasticsearch interaciton with username and password hardcoded in the spark session. trial license requires any interaction with elasticsearch to be authenticated.
Kibana
++ Added trial folder with scripts that set up security configs for the trial version of HELK. It creates users and roles to test the security features of x-pack
Logstash
++ Created trial folder with another pipeline folder in it. The pipeline in trial has output configs with elasticsearch's username and password hardcoded. Ready for when the user sets the build with trial license and wants to send logs to elasticsearch. The logstash configs are the same as the ones from the defailt pipeline. They only have username and password configs on all the output configs.
Nginx
++ set trial folder with the right config to allow Kibana handle the authentication process when user builds and installs HELK with a trial license. No need for nginx to handle the authentication.
helk_install bash script
++ Updated script to handle license choice : basic or trial
++ basic license is selected by default. If user selects trial, it runs the specific docker-compose file needed to build and install HELK with the right trial configs.
++ Updated also the CLI options. User now will have to specify the license for HELK. Example: sudo ./helk_install.sh -i 192.168.64.131 -l basic
++Added Image property to docker-compose file for ELK images.
++Updated Docker files to remove extra configs that were being already specified in docker-compose file
++ Kibana scripts are now added in the docker-compose file. Move them to /usr/share/kibana/scripts folder in the Kibana container
++Updated ELK config files to follow basic subscription templates (default settings)
++ Updated Winlogbeat template to only point to two Kafka Brokers
Docker Compose
++ Updated Spark images to 2.3.1
ELK Stack
++ Docker Images updated to 6.3.1
helk-jupyter
++ Preparing Jupyter for Jupyterhub
++ Spark base image now comes with a sparkuser user
++ Updated es-hadoop package to 6.3.1
helk-logstash
++ Updated Sysmon parser to transform new datafield from Sysmon V8.0. RuleName
helk-spark-base
++ Images updated to 2.3.1
helk_install
++ fixed https://github.com/Cyb3rWard0g/HELK/issues/81
++ Updated banner to show right version
HELK Version
+ ELK update tp 6.3.0
Logstash
+ Integrated ATT&CK CTI to the build. Created from https://github.com/Cyb3rWard0g/ATTACK-Python-Client/tree/master/integrations/helk_cti
+ Added the mitre_attack file to the build which contains information from Enterprise, PRE and Mobile Matrices
+ Enabled x-pack monitoring (new feature)
Kibana
+ Added Dashboards for the ATT&CK Integration
helk_install script
+ reduced docker-compose build and run steps to one
scripts
+ Added script export_attack.py to export the file used for logstash and kibana.
Updated several errors that I got after testing a few configurations.
Logstash
- Updated Field Names for Sysmon and Security Logs
- Updated Logstash Templates to maintain consistency
- Updated Dockerfile to directly use official Centos Elastic Docker Image
- Updated Logstash main config file to update pipeline batch value and the Kafka input config to fix https://github.com/Cyb3rWard0g/HELK/issues/73
- Updated Logstash Security parser to fix:
-- https://github.com/Cyb3rWard0g/HELK/issues/71
-- https://github.com/Cyb3rWard0g/HELK/issues/72
Kibana & Elasticsearch
- Updated Docker files to directly use Official Centos Elastic Docker images with only a few updates.
Docker Compose File
- Updated file to mount Logstash, Kibana and Elasticsearch config files, dashboards and output_templates. This will simplify utilization of custom pipes, visualizations and dashboards. User can now just replace those the local folder and HELK will use those configurations.
- ES_JAVA_OPTS is now set to 6GB by default. No more MEM string. It confuses the user.
helk_install.sh script
- Updated the string that it replaces when setting ES JAVA OPTS. It went from looking for the string "MEME" to "6GB".
Addressed https://github.com/Cyb3rWard0g/HELK/issues/67 where I was grabbing Disk size in human readable format which was breaking the calculations when the HELK was being built on a computer with TB space.
Logstash
- Added Local Pipeline to the build to allow custom local configurations
- updated sysmon config to fix https://github.com/Cyb3rWard0g/HELK/issues/63
- removed port exposed in local logstash Dockerfile. It will be pushed to official docker image in the next update
- removed logstash init file (not being used anymore)
Zeppelin
- not available yet
- initial draft dockerfile
- created spark-defaults file for future zeppelin dockerfile
Install Script
- incrased minimum memory size required
Todo:
* Structure (scripts in directory)
* Recognition when Kibana and index config of it gets available (polling)
* Cron job for auto update
* Integration in compose file
Roberto Rodriguez
Nate Guagenti
neutron
Dev Dua
Robert Simplicio
Thomas Patzke