mirror of https://github.com/infosecn1nja/HELK.git
d623246f4c
Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available |
||
|---|---|---|
| .. | ||
| output_templates | ||
| pipeline | ||
| logstash-init | ||
| logstash.yml | ||