Commit Graph

174 Commits (bb321d985adac8e14b68b7c28bee800c29704045)

Author SHA1 Message Date
Roberto Rodriguez bb321d985a
Merge pull request #56 from thomaspatzke/devel-sigma
Sigma integration
2018-05-30 12:12:14 -04:00
Thomas Patzke ea979912ed Finalized Sigma integration 2018-05-22 23:32:07 +02:00
Thomas Patzke d0477a08db Added Sigma container
Todo:
* Structure (scripts in directory)
* Recognition when Kibana and index config of it gets available (polling)
* Cron job for auto update
* Integration in compose file
2018-05-22 23:32:07 +02:00
Thomas Patzke e70eafce09 Added field mappings
Field mappings required for integration of Windows Sigma rules.
2018-05-22 23:32:07 +02:00
Roberto Rodriguez 0519db370a
Update PULL_REQUEST_TEMPLATE.md 2018-05-17 09:33:56 -04:00
Roberto Rodriguez 714f9daacd
Merge pull request #51 from devdua/add-snap-fallback
Added snap as fallback installation method for Docker
2018-05-15 10:54:08 -04:00
Dev Dua 43feaa7407 Added check to see if snap is installed 2018-05-15 09:59:54 +05:30
Dev Dua f8cbe7e825 Added snap as fallback installation method for Docker 2018-05-15 08:10:10 +05:30
Roberto Rodriguez 93c47ea728
Create PULL_REQUEST_TEMPLATE.md 2018-05-11 16:31:41 -04:00
Roberto Rodriguez 4b1c4fbd65 Update issue templates 2018-05-11 16:22:18 -04:00
Roberto Rodriguez 8c08a5bf66
External ES URL and port was removed 2018-05-07 02:15:17 +02:00
Roberto Rodriguez 3baec9f79c Updated post-install info & Resources 2018-05-04 00:35:45 -04:00
Roberto Rodriguez d8070cc204 Updated post-Install information
+ Added the Cluster Master and Worker URLs.
2018-05-03 16:02:48 -04:00
Roberto Rodriguez 10de1b6b0a HELK 6.2.4-050318
## Overall
+ Removed the Init files dependencies on all containers
+ Added more resources to the resources folder (papers and presentations)
+ Updated to-do list on main README
+ Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43)
+ Updated WIki and added new images to it
+ Started documenting potential error messages or bugs with a few quick fixes

## Helk Install Script
+ Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK)

## ELK Stack
+ Started using Elastic Docker Images as a base
+ Updated ELK stack to 6.2.4 version
+ X-Pack Basic Free License attached to build automatically
+ Monitoring capabilities are now enabled in the build (Reason why Cerebro went away)

## Spark
+ Integrated Spark Standalone Cluster Manager
+ Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code
+ Added Spark Master and Worker Docker Images
+ Build runs now with 2 Workers and 1 Master by default.
+ Apache Arrow is enabled for Pandas Dataframe optimization
+ Created Spark-Base Docker Image (Applied to the Jupyter Image)

## Kafka
+ Kafka Container was split in Kafka Brokers and one Zookeeper
+ Helk runs with 2 Kafka Brokers and 1 Zookeeper by default

## Jupyter Container
+ Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess
+ New packages were added:
++ nxviz
++ hiveplot
 ++ pyarrow
+ Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 15:54:12 -04:00
Roberto Rodriguez 084ff68cb2 Updated HELK Design
Removed OTX Enrichment for now to reduce the load on logstash and keep it clean for now. It will be added in the future. Implementation is already developed.
2018-04-10 03:16:54 -04:00
Roberto Rodriguez 24e82eafad Updsted README 2018-04-10 03:03:19 -04:00
Roberto Rodriguez 6054e2be9a HELK 6.2.3-041018
Docker-Compose File
+ Split helk-elk service in 3 (Logstash, Kibana, Logstash)

HELK-base
+ New Docker Base image applied to all HELK's Docker images

HELK-analytics
+ updated file due to new helk-base image

HELK-elk
+ Removed Helk-elk folder

HELK-kafka
+ Updated it to version 1.1.0

HELK-Logstash
+ Updated all files to point to helk-kafka and helk-elasticsearch (New image after splitting helk-elk)

New Docker Images
+ helk-elasticsearch
+ helk-logstash
+ helk-kibana
+ helk-nginx

HELK-nginx
+ Removed route to elasticsearch:8082. Cerebro now can point to 172.18.0.2 (Internal Docker IP)

HELK-Install
+ organized script a little better by creating install_dockerl and install_docker_compose functions

HELK-kibana
+ updated Kibana configuration to set Kibana server to the name of the service helk-kibana. It allows remote connections to it (internally among docer images)
+ Updated elasticsearch url to new docker image (helk-elasticsearch:9200)

HELK-kafka
+ updated internal listeners on each broker to helk-kafka
2018-04-10 02:56:28 -04:00
Roberto Rodriguez c43eaa08e3 HELK 6.2.3 - 032218
Docker-Compose file
+ Updated Image versions
++ helk-elk:6.2.3
++ helk-kafka:1.0.1
++ helk-analytics:0.0.2

HELK-ANALYTICS
+ Upgraded spark to version 2.3.0
++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html
+ Upgraded Jupyter Lab to 0.31.12
+ Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3.
+ Upgraded ES-Hadoop package to version 6.2.3
++ Check release notes:
https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html

HELK-ELK
+ Upgraded elastic components to 6.2.3
++ Check elasticsearch release notes:
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html
++ No changes for Kibana
++ Check Logstash release notes:
https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html
+ Logstash kafka input now adds metadata from kafka. Topic name, etc.
+ Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field.
+ logstash config 11-winevent-sysmon-filter.conf
++ removed field "user". This was causing issues when parsing events with Spark.

HELK-KAFKA
+ Upgraded Kafka to version 2.11-1.0.1
++ Check kafka release notes:
https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html
+ Removed sleep time for kafka init file
+ updated kafka entrypoint updating version values

HELK helk_install main script
+ Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33

HELK Winlogbeat install script
+ Updated beat version to 6.2.3
2018-03-22 03:32:21 -04:00
Roberto Rodriguez 669758782d
Merge pull request #37 from Cyb3rWard0g/revert-36-devel-sigma
Revert "Sigma integration"
2018-03-20 22:27:40 -04:00
Roberto Rodriguez 492d0bfdac
Revert "Sigma integration" 2018-03-20 22:26:56 -04:00
Roberto Rodriguez d7aad0e15a
Merge pull request #36 from thomaspatzke/devel-sigma
Sigma integration
2018-03-17 21:16:16 -04:00
Roberto Rodriguez 42ac91ea76
Merge pull request #35 from thomaspatzke/devel-dockerref
Changed container references in Docker compose file
2018-03-17 21:13:02 -04:00
Thomas Patzke 53a009673d Sigma integration 2018-03-17 01:27:10 +01:00
Thomas Patzke 255d926246 Docker compose refers to repository directories
Enables building from repository.
2018-03-17 00:25:12 +01:00
Roberto Rodriguez f02cac3280
Update README.md 2018-03-05 09:40:58 -05:00
Roberto Rodriguez 11c8720fe4 Updated Resources Links 2018-03-03 23:58:40 -05:00
Roberto Rodriguez 92d105ce51 Updated Apache Spark Kafka jar 2.3.0 2018-03-03 23:44:09 -05:00
Roberto Rodriguez 5859ba3dab HELK 6.2.2 - 030318
helk-analytics
+ Init file and Dockerfile updated with Spark version 2.3.0
+Jupyter Notebook from getting started folder updated
+ New jupyter notebook with graphframes example presented in BSColumbus 2018

helk-elk
+ Added properties to elasticsearch config file to set it as a standalone cluster. (It helps for when elasticsearch is restarted)
+ Updated Dashboards
+ Updated Kibana timeout to 60000
+ Updated Logstas - elasticsearch mapping templates after renaming fields.
+ Updated logstash filters renaming fields keeping a new flat schema. No more nested fields style.

helk-kafka
+ Updated Log retention hours to 2 hours

Resources:
- Created README to share all the blog posts, documentes and presentations that helped me to work on the HELK

Scripts
+ Deprecated most of the scripts used before to install ELK via TAR and DEB. Also deprecated scripts to updated geoip database.
2018-03-03 21:15:35 -05:00
Roberto Rodriguez aa17e077c0 Merge branch 'master' of https://github.com/Cyb3rWard0g/HELK 2018-02-25 02:59:47 -05:00
Roberto Rodriguez 063e5835ec HELK 6.2.2 - 022518
HELK Design
+ moved everything to docker-compose approach for a more modular design.
+ separated the HELK in 3 services:
++helk-elk, helk-kafka, helk-analytics
+ Updated Design picture to show WEF ideas and also show Jupyter Lab integrations.

HELK Docker-Compose
+ Added ESDATA volume to keep logs after contaners get stopped
+ Services restart automatically after reboot
+ created blank env file for Kafka service. This allows the host to pass its own local IP to Kafka. This is needed for advertised listener configs on each broker.

HELK-ELK Version
- Updated to 6.2.2

ELasticsearch
- Added local docker network as part of the network.host option. This allows the HELK-ELK service to publish its docker local IP to other services/images in the docker compose environment.

Logstash
+ minimal updates to  certain configs (Mainly renaming files and replacing certain strings)

Kibana
+ enableExternalUrls set to true for Vega visualization that need external libraries.

Spark - Analytics
+ Renamed service to Analytics
+ Integrated Apache Toree to allow Scala kernel in Jupyter
+ Pyspark, Scala and SQL are now available in Jupyter

Jupyter
+ Jupyter LAB has been enabled
2018-02-25 02:59:44 -05:00
Roberto Rodriguez db1d594403
Merge pull request #26 from rlarabee/patch-1
Update kafka-init
2018-02-19 23:14:20 -05:00
rlarabee d1d88f85a9
Update kafka-init
The KAFKA_USER variable was listed twice. It seemed to cause an issue with kafka when first starting up. After updating the issue went away.
2018-02-19 22:11:05 -06:00
Roberto Rodriguez d623246f4c HELK ELK 6.2.0 & New features
Elasticsearch
+ Deleted Docker elasticsearch config file (Duplicate)

Logstash
+ Adjusted Batch size to 300 (Testing)
+ Renamed scripts to follow a standard naming convention
+ Added a fingerprint filter to all logs to help reduce duplicate logs
+ Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated.
+ Added Document_id to every logstash output config to take the fingerprint value.

Kibana
+ Renamed Index Patterns to standard naming convention.
+ Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it.
+ Updated name of index patterns across all visualizations and dashboards.

Kafka
+ Log retention is now 24 hours and not 268 Hours
+ added auto_offset_reset => "earliest" to beats kafka input config

Spark
+ updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3
+ Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully.

Winlogbeat
+ Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs.

New Features
+ Cerebro
+ Python packages:
-scipy==1.0.0
  scikit-learn==0.19.1
  nltk==3.2.5
  matplotlib==2.1.2
  seaborn==0.8.1
  datasketch==1.2.5
  tensorflow==1.5.0
  keras==2.1.3
  pyflux==0.4.15
  imbalanced-learn==0.3.2
  lime==0.1.1.29

Docker Hub
+ New HELK image available
2018-02-15 03:28:48 -05:00
Roberto Rodriguez 829ed84d1b
Merge pull request #22 from leechristensen/master
Added WMI event log ingestion, primarily for WMI permanent event subscription detection
2018-02-08 14:32:24 -05:00
Lee Christensen 384b2d3f1e
Added wmiactivity 2018-02-07 20:36:57 -05:00
Lee Christensen c63bcfef95 Add Microsoft-Windows-WMI-Activity ingestion 2018-02-06 16:25:39 -05:00
Roberto Rodriguez 6b4054288b Installation Wiki Images 2018-02-06 07:39:57 -05:00
Roberto Rodriguez 644089c35a Updated ES-Hadoop to 6.1.3
+ Updated Spark Defaults config : elasticsearch-hadoop-6.1.3.jar
+ Updated Shell Script & Dockerfile to download elasticsearch-hadoop-6.1.3.zip
2018-02-04 22:59:52 -05:00
Roberto Rodriguez 0d7075e3cc Added more images for wiki
+ Updated Kafka design
+ Added Elasticsearch, Kibana, Logstash & Spark
2018-02-04 00:14:19 -05:00
Roberto Rodriguez 0df92d3b90 Kafka Images for Wiki 2018-02-01 20:27:11 -05:00
Roberto Rodriguez 3b17da481a Exposed Docker Ports properly
+ @bsisco via Issue #19 let us know that communication between systems and kafka was not working. I forgot to expose the right ports when running the HELK Docker image after being pulled.
2018-02-01 13:30:16 -05:00
Roberto Rodriguez 191275ef18 Contributors & Alpha Versions
+ Added Lee Christensen to contributors list
+ Updated Main install script to reflect Alpha version and latest ELK version (6.1.3)
2018-01-31 18:36:46 -05:00
Roberto Rodriguez 6928f74242 Spark & Kafka Communication
Updatd Design to show potential capability to start using Spark and Kafka
2018-01-31 18:31:46 -05:00
Roberto Rodriguez 25d4aa5996 HELK - Alpha ELK 6.1.3
+ ELK 6.1.3 version (Jun 30,2018 release)
+ Kafka Integration
-- Bash, DockerFile & Docker Image
+ Replaced ELK DEB Install Packages for TAR packages (Easier deployement and more control)
+ Logstash: JVM Heap 2GB default
+ ELK (Init Files created)
-- More control over service start
+ Left Linux DEB install bash script (deprecating it in next release)
+ ELK .yml files are not available to adjust deployment in an easier way.
+ Fixed Docker Run environment parameters to be call before pointing to the HELK image.
+ Edited every single file to have the right headers:
-- ELK version 6.1.3
-- Aplha Version
2018-01-31 17:52:50 -05:00
Roberto Rodriguez 5c5ffafd80 Updated Sysmon user fields for consistency
- Sysmon user (domain, sid, name) were not consistent with security logs.
2018-01-16 21:10:42 -05:00
Roberto Rodriguez 8cbda80112
Merge pull request #14 from jaredcatkinson/master
Cropped that for you ;-)
2018-01-15 22:24:26 -05:00
Jared Atkinson 7c01703f0b Cropped that for you ;-) 2018-01-16 04:17:28 +01:00
Roberto Rodriguez 41c70f29ce Merge branch 'master' of https://github.com/Cyb3rWard0g/HELK 2018-01-15 20:47:32 -05:00
Roberto Rodriguez 4ef706c5a6 Updated HELK Design 2018-01-15 20:42:57 -05:00
Roberto Rodriguez 56550d0c69
Update README.md 2018-01-15 20:11:13 -05:00