helk-spark-worker
+ set SPARK_WORKER_MEMORY to 1g
+ Enabled spark shuffle service to safely remove executors from apps
helk-jupyter
+ Upgraded ES-Hadoop to 6.4.0
+ Added Postgresql JAR and installed postgresql to manage the usage of multiple notebooks
+ Added entrypoint script to create hive user, set a password and create a hive_metastore database
+ Set Spark dynamic allocation settings to avoid Spark workers getting sucked on one application only
All
+ Moved all to docker folder. Getting ready to start sharing other ways to deploy helk (terraform & Packer maybe)
Compose-files
+ Basic & Trial Elastic Subscriptions available now and can be automatically managed via the helk_install script
ELK Version : 6.3.2
Elasticsearch
+ Set 4GB for ES_JAVA_OPTS by default allowing the modification of it via docker-compose and calculating half of the host memory if it is not set
+ Added Entrypoint script and using docker-entrypoint to start ES
Logstash
+ Big Pipeline Update by Nate Guagenti (@neu5ron)
++better cli & file name searching
++”dst_ip_public:true” filter out all rfc1918/non-routable
++Geo ASName
++Identification of 16+ windows IP fields
++Arrayed IPs support
++IPv6&IPv4 differentiation
++removing “-“ values and MORE!!!
++ THANK YOU SO MUCH NATE!!!
++ PR: https://github.com/Cyb3rWard0g/HELK/pull/93
+ Added entrypoint script to push new output_templates straight to Elasticsearch per Nate's recommendation
+ Starting Logstash now with docker-entrypoint
+ "event_data" is now taken out of winlogbeat logs to allow integration with nxlog (sauce added by Nate Guagenti (@neu5ron)
Kibana
+ Kibana yml file updated to allow a longer time for timeout
Nginx:
+ it handles communications to Kibana and Jupyterhub via port 443 SSL
+ certificate and key get created at build time
+ Nate added several settings to improve the way how nginx operates
Jupyterhub
+ Multiple users and mulitple notebooks open at the same time are possible now
+ Jupytehub now has 3 users hunter1,hunter2.hunter3 and password patterh is <user>P@ssw0rd!
+ Every notebook created is also JupyterLab
+ Updated ES-Hadoop 6.3.2
Kafka Update
+ 1.1.1 Update
Spark Master + Brokers
+ reduce memory for brokers by default to 512m
Resources:
+ Added new images for Wiki
Roberto Rodriguez