HELK-JUPYTER
+ Miniconda3 to handle python packages
+ Python 3.7
+ Container not running as root
+ new entrypoint and cmd scripts
+ postgres not running as root and under the same container
+ Spark Jar and Python dependencies provided offline (not downloading from maven directly - Sometimes this fails)
+ Jupyter PySpark kernel using conda to run ipykernel module
+ PYSPARK_PYTHON Python 3.7
HELK-LOGSTASH
+ Fix https://github.com/Cyb3rWard0g/HELK/issues/217
HELK base image
+ Updated to 0.0.3
HELK ELK Version
+ Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released)
helk_install
+ Users can now select between two deployments:
++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT)
++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER)
+ Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option.
helk-elastalert
+ Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g
helk-elasticsearch
+ consolidated main configs in one
+ added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK.
helk-logstash
+ updated to 6.5.3
+ simplified pipeline to have only one folder
+ logstash-entrypoint script can now enable elastic password on all logstash output conf files.
+ New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT)
helk-nginx
+ split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base)
helk-kibana
+ Updated to version 6.5.3
+ Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic
helk-jupyter
+ updated Jupyterlab to 0.35.4
+ updated jupyterhub to 0.9.4
+ updated jupyterlab hub extension to 0.12.0
+ updated ES_HADOOP to 6.5.3
+ updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0
+ Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time
helk-kafka-base
+ reduced docker container size
+ updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper)
helk-kafka-broker
+ User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own.
helk-zookeeper
+ reduced size of container
+ updated build to kafka 2.1.0
helk-KSQL
+ initial integration of KSQL
+ KSQL Server and KSQL CLI are available
+ Blog post coming soon ;)
helk-spark-worker
+ set SPARK_WORKER_MEMORY to 1g
+ Enabled spark shuffle service to safely remove executors from apps
helk-jupyter
+ Upgraded ES-Hadoop to 6.4.0
+ Added Postgresql JAR and installed postgresql to manage the usage of multiple notebooks
+ Added entrypoint script to create hive user, set a password and create a hive_metastore database
+ Set Spark dynamic allocation settings to avoid Spark workers getting sucked on one application only
Docker-compose Files Version
+ Updated version to 3.5
Base Docker Ubuntu Image
+ Updated to phusion/baseimage version 0.11 (https://github.com/phusion/baseimage-docker/releases/tag/0.11)
HELK base image
+ Updated to 0.0.2 due to Ubuntu upgrade
HELK ELK Version
+ Now using 6.4.0 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-4-0-released?blade=tw&hulk=social)
helk_install
+ Fixed https://github.com/Cyb3rWard0g/HELK/issues/99
helk-elasticsearch
+ Updated main yml config to set most of the settings via environment variables via docker-compose
+ Trial docker-compose file now has ELASTICSEARCH_PASSWORD environment variable set/available. Trial Dockerfile was deleted since the elasticsearch_password update is now taken care of by the internal elasticsearch docker script that is comes with the official elasticsearch docker image.
+ reduced the memory requirements from 4GB to 2GB
helk-logstash
+ entrypoint scripts remove kafka output plugin 7.1.2 and installs version 7.1.1 due to https://github.com/logstash-plugins/logstash-output-kafka/pull/198
++ this error happens right after upgrading ELK built from 6.3.2 to 6.4.0
helk-jupyter
+ Added Altair python package
+ updated Jupyterlab to 0.34.1
+ updated jupyterhub to 0.9.2
+ updated jupyterlab hub extension to 0.11.0
+ updated Spark config to use Graphframes 0.6.0 (https://graphframes.github.io/user-guide.html)
+ updated spark-kafka library to spark-sql-kafka-0-10_2.11:2.3.1
helk-kafka-base
+ updated Kafka to 2.0.0 (this affects Kafka brokers and zookeeper)
+ Created user kafkauser to run kafka containers as non-root
helk-kafka-broker
+ split entrypoint script to have topics creation separate
++ auomated the way how the container checks for the kafka broker port availability. If the port is open, then it attempts to create kafka topics
+ No need to tail kafka logs to keep the container alive after running the kafka start script. It now just starts the broker via Dockerfile CMD command and stays alive.
helk-zookeeper
+ updated entrypoint to only set the main server config
+ zookeeper is now started via Dockerfile CMD command
All
+ Moved all to docker folder. Getting ready to start sharing other ways to deploy helk (terraform & Packer maybe)
Compose-files
+ Basic & Trial Elastic Subscriptions available now and can be automatically managed via the helk_install script
ELK Version : 6.3.2
Elasticsearch
+ Set 4GB for ES_JAVA_OPTS by default allowing the modification of it via docker-compose and calculating half of the host memory if it is not set
+ Added Entrypoint script and using docker-entrypoint to start ES
Logstash
+ Big Pipeline Update by Nate Guagenti (@neu5ron)
++better cli & file name searching
++”dst_ip_public:true” filter out all rfc1918/non-routable
++Geo ASName
++Identification of 16+ windows IP fields
++Arrayed IPs support
++IPv6&IPv4 differentiation
++removing “-“ values and MORE!!!
++ THANK YOU SO MUCH NATE!!!
++ PR: https://github.com/Cyb3rWard0g/HELK/pull/93
+ Added entrypoint script to push new output_templates straight to Elasticsearch per Nate's recommendation
+ Starting Logstash now with docker-entrypoint
+ "event_data" is now taken out of winlogbeat logs to allow integration with nxlog (sauce added by Nate Guagenti (@neu5ron)
Kibana
+ Kibana yml file updated to allow a longer time for timeout
Nginx:
+ it handles communications to Kibana and Jupyterhub via port 443 SSL
+ certificate and key get created at build time
+ Nate added several settings to improve the way how nginx operates
Jupyterhub
+ Multiple users and mulitple notebooks open at the same time are possible now
+ Jupytehub now has 3 users hunter1,hunter2.hunter3 and password patterh is <user>P@ssw0rd!
+ Every notebook created is also JupyterLab
+ Updated ES-Hadoop 6.3.2
Kafka Update
+ 1.1.1 Update
Spark Master + Brokers
+ reduce memory for brokers by default to 512m
Resources:
+ Added new images for Wiki
neu5ron
Roberto Rodriguez