HELK base image
+ Updated to 0.0.3
HELK ELK Version
+ Now using 6.5.3 official ELK Docker Images (https://www.elastic.co/blog/elastic-stack-6-5-3-released)
helk_install
+ Users can now select between two deployments:
++ helk-kibana-analysis (KAFKA + KSQL + ELK + NGNIX + ELASTALERT)
++ helk-kibana-notebooks (KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER)
+ Fixed https://github.com/Cyb3rWard0g/HELK/issues/131 . Users can now set up the Kibana UI User password during installation. Also, user can set the Elasticsearch elastic account password when using the Trial license option.
helk-elastalert
+ Elastalert deployed and ready to use with SIGMA integration. Blog available at https://medium.com/@Cyb3rWard0g
helk-elasticsearch
+ consolidated main configs in one
+ added more environment variables for ELASTIC_PASSWORD and default values in case it is not used to be compatible with the default values applied to HELK.
helk-logstash
+ updated to 6.5.3
+ simplified pipeline to have only one folder
+ logstash-entrypoint script can now enable elastic password on all logstash output conf files.
+ New environment variables (ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT)
helk-nginx
+ split the default config for the two deployment options (helk-kibana-analysis (trial/base) and helk-kibana-notebook-analysis (trial/base)
helk-kibana
+ Updated to version 6.5.3
+ Added new environment variables (ELASTICSEARCH_URL, SERVER_HOST, SERVER_PORT, ELASTIC_PASSWORD, ELASTIC_HOST, ELASTIC_PORT, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD, KIBANA_UI_PASSWORD) and logic to make the build more dynamic
helk-jupyter
+ updated Jupyterlab to 0.35.4
+ updated jupyterhub to 0.9.4
+ updated jupyterlab hub extension to 0.12.0
+ updated ES_HADOOP to 6.5.3
+ updated org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.0
+ Added extra notebooks to test deployment and provide more information for analyst experiencing Jupyter for the first time
helk-kafka-base
+ reduced docker container size
+ updated Kafka to 2.1.0 (this affects Kafka brokers and zookeeper)
helk-kafka-broker
+ User can now define a list of topics to be created via the new environment variable KAFKA_CREATE_TOPICS. That needs to be defined either in the docker-compose file or while running the docker container on its own.
helk-zookeeper
+ reduced size of container
+ updated build to kafka 2.1.0
helk-KSQL
+ initial integration of KSQL
+ KSQL Server and KSQL CLI are available
+ Blog post coming soon ;)
All
+ Moved all to docker folder. Getting ready to start sharing other ways to deploy helk (terraform & Packer maybe)
Compose-files
+ Basic & Trial Elastic Subscriptions available now and can be automatically managed via the helk_install script
ELK Version : 6.3.2
Elasticsearch
+ Set 4GB for ES_JAVA_OPTS by default allowing the modification of it via docker-compose and calculating half of the host memory if it is not set
+ Added Entrypoint script and using docker-entrypoint to start ES
Logstash
+ Big Pipeline Update by Nate Guagenti (@neu5ron)
++better cli & file name searching
++”dst_ip_public:true” filter out all rfc1918/non-routable
++Geo ASName
++Identification of 16+ windows IP fields
++Arrayed IPs support
++IPv6&IPv4 differentiation
++removing “-“ values and MORE!!!
++ THANK YOU SO MUCH NATE!!!
++ PR: https://github.com/Cyb3rWard0g/HELK/pull/93
+ Added entrypoint script to push new output_templates straight to Elasticsearch per Nate's recommendation
+ Starting Logstash now with docker-entrypoint
+ "event_data" is now taken out of winlogbeat logs to allow integration with nxlog (sauce added by Nate Guagenti (@neu5ron)
Kibana
+ Kibana yml file updated to allow a longer time for timeout
Nginx:
+ it handles communications to Kibana and Jupyterhub via port 443 SSL
+ certificate and key get created at build time
+ Nate added several settings to improve the way how nginx operates
Jupyterhub
+ Multiple users and mulitple notebooks open at the same time are possible now
+ Jupytehub now has 3 users hunter1,hunter2.hunter3 and password patterh is <user>P@ssw0rd!
+ Every notebook created is also JupyterLab
+ Updated ES-Hadoop 6.3.2
Kafka Update
+ 1.1.1 Update
Spark Master + Brokers
+ reduce memory for brokers by default to 512m
Resources:
+ Added new images for Wiki
## Overall
+ Removed the Init files dependencies on all containers
+ Added more resources to the resources folder (papers and presentations)
+ Updated to-do list on main README
+ Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43)
+ Updated WIki and added new images to it
+ Started documenting potential error messages or bugs with a few quick fixes
## Helk Install Script
+ Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK)
## ELK Stack
+ Started using Elastic Docker Images as a base
+ Updated ELK stack to 6.2.4 version
+ X-Pack Basic Free License attached to build automatically
+ Monitoring capabilities are now enabled in the build (Reason why Cerebro went away)
## Spark
+ Integrated Spark Standalone Cluster Manager
+ Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code
+ Added Spark Master and Worker Docker Images
+ Build runs now with 2 Workers and 1 Master by default.
+ Apache Arrow is enabled for Pandas Dataframe optimization
+ Created Spark-Base Docker Image (Applied to the Jupyter Image)
## Kafka
+ Kafka Container was split in Kafka Brokers and one Zookeeper
+ Helk runs with 2 Kafka Brokers and 1 Zookeeper by default
## Jupyter Container
+ Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess
+ New packages were added:
++ nxviz
++ hiveplot
++ pyarrow
+ Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
Removed OTX Enrichment for now to reduce the load on logstash and keep it clean for now. It will be added in the future. Implementation is already developed.
Docker-Compose File
+ Split helk-elk service in 3 (Logstash, Kibana, Logstash)
HELK-base
+ New Docker Base image applied to all HELK's Docker images
HELK-analytics
+ updated file due to new helk-base image
HELK-elk
+ Removed Helk-elk folder
HELK-kafka
+ Updated it to version 1.1.0
HELK-Logstash
+ Updated all files to point to helk-kafka and helk-elasticsearch (New image after splitting helk-elk)
New Docker Images
+ helk-elasticsearch
+ helk-logstash
+ helk-kibana
+ helk-nginx
HELK-nginx
+ Removed route to elasticsearch:8082. Cerebro now can point to 172.18.0.2 (Internal Docker IP)
HELK-Install
+ organized script a little better by creating install_dockerl and install_docker_compose functions
HELK-kibana
+ updated Kibana configuration to set Kibana server to the name of the service helk-kibana. It allows remote connections to it (internally among docer images)
+ Updated elasticsearch url to new docker image (helk-elasticsearch:9200)
HELK-kafka
+ updated internal listeners on each broker to helk-kafka
helk-analytics
+ Init file and Dockerfile updated with Spark version 2.3.0
+Jupyter Notebook from getting started folder updated
+ New jupyter notebook with graphframes example presented in BSColumbus 2018
helk-elk
+ Added properties to elasticsearch config file to set it as a standalone cluster. (It helps for when elasticsearch is restarted)
+ Updated Dashboards
+ Updated Kibana timeout to 60000
+ Updated Logstas - elasticsearch mapping templates after renaming fields.
+ Updated logstash filters renaming fields keeping a new flat schema. No more nested fields style.
helk-kafka
+ Updated Log retention hours to 2 hours
Resources:
- Created README to share all the blog posts, documentes and presentations that helped me to work on the HELK
Scripts
+ Deprecated most of the scripts used before to install ELK via TAR and DEB. Also deprecated scripts to updated geoip database.
HELK Design
+ moved everything to docker-compose approach for a more modular design.
+ separated the HELK in 3 services:
++helk-elk, helk-kafka, helk-analytics
+ Updated Design picture to show WEF ideas and also show Jupyter Lab integrations.
HELK Docker-Compose
+ Added ESDATA volume to keep logs after contaners get stopped
+ Services restart automatically after reboot
+ created blank env file for Kafka service. This allows the host to pass its own local IP to Kafka. This is needed for advertised listener configs on each broker.
HELK-ELK Version
- Updated to 6.2.2
ELasticsearch
- Added local docker network as part of the network.host option. This allows the HELK-ELK service to publish its docker local IP to other services/images in the docker compose environment.
Logstash
+ minimal updates to certain configs (Mainly renaming files and replacing certain strings)
Kibana
+ enableExternalUrls set to true for Vega visualization that need external libraries.
Spark - Analytics
+ Renamed service to Analytics
+ Integrated Apache Toree to allow Scala kernel in Jupyter
+ Pyspark, Scala and SQL are now available in Jupyter
Jupyter
+ Jupyter LAB has been enabled
Elasticsearch
+ Deleted Docker elasticsearch config file (Duplicate)
Logstash
+ Adjusted Batch size to 300 (Testing)
+ Renamed scripts to follow a standard naming convention
+ Added a fingerprint filter to all logs to help reduce duplicate logs
+ Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated.
+ Added Document_id to every logstash output config to take the fingerprint value.
Kibana
+ Renamed Index Patterns to standard naming convention.
+ Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it.
+ Updated name of index patterns across all visualizations and dashboards.
Kafka
+ Log retention is now 24 hours and not 268 Hours
+ added auto_offset_reset => "earliest" to beats kafka input config
Spark
+ updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3
+ Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully.
Winlogbeat
+ Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs.
New Features
+ Cerebro
+ Python packages:
-scipy==1.0.0
scikit-learn==0.19.1
nltk==3.2.5
matplotlib==2.1.2
seaborn==0.8.1
datasketch==1.2.5
tensorflow==1.5.0
keras==2.1.3
pyflux==0.4.15
imbalanced-learn==0.3.2
lime==0.1.1.29
Docker Hub
+ New HELK image available
+ ELK 6.1.3 version (Jun 30,2018 release)
+ Kafka Integration
-- Bash, DockerFile & Docker Image
+ Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control)
+ Logstash: JVM Heap 2GB default
+ ELK (Init Files created)
-- More control over service start
+ Left Linux DEB install bash script (deprecating it in next release)
+ ELK .yml files are not available to adjust deployment in an easier way.
+ Fixed Docker Run environment parameters to be call before pointing to the HELK image.
+ Edited every single file to have the right headers:
-- ELK version 6.1.3
-- Aplha Version
-Moved spark folder out of enrichments to root.
- Removed ipython & inotebook deb packages. Jupyter is installed via PIP only.
- Added new contributor to README
Roberto Rodriguez
Jared Atkinson