HELK/helk_install.sh

262 lines
9.7 KiB
Bash
Raw Normal View History

#!/bin/bash
# HELK script: helk_install.sh
# HELK script description: Start
# HELK build version: 0.9 (Alpha)
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
# HELK ELK version: 6.2.3
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
# *********** Check if user is root ***************
if [[ $EUID -ne 0 ]]; then
echo "[HELK-INSTALLATION-INFO] YOU MUST BE ROOT TO RUN THIS SCRIPT!!!"
exit 1
fi
LOGFILE="/var/log/helk-install.log"
echoerror() {
printf "${RC} * ERROR${EC}: $@\n" 1>&2;
}
# *********** Check System Kernel Name ***************
systemKernel="$(uname -s)"
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
# *********** Getting Jupyter Token ***************
get_jupyter_token(){
echo "[HELK-INSTALLATION-INFO] Waiting for HELK services and Jupyter Server to start.."
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$(docker exec -ti helk-analytics jupyter notebook list | grep -oP '(?<=token=).*(?= ::)' | awk '{$1=$1};1')" >> $LOGFILE 2>&1
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
}
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
# ********** Install Curl ********************
install_curl(){
echo "[HELK-INSTALLATION-INFO] Checking if curl is installed first"
if [ -x "$(command -v curl)" ]; then
echo "[HELK-INSTALLATION-INFO] curl is already installed"
else
echo "[HELK-INSTALLATION-INFO] curl is not installed"
echo "[HELK-INSTALLATION-INFO] Installing curl before installing docker.."
apt-get install -y curl >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install curl (Error Code: $ERROR)."
exit 1
fi
fi
}
# *********** Building and Running HELK Images ***************
install_helk(){
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
echo "[HELK-INSTALLATION-INFO] Building HELK via docker-compose"
echo "ADVERTISED_LISTENER=$host_ip" >> helk.env
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
docker-compose build >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK via docker-compose (Error Code: $ERROR)."
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
echo "get more details in /var/log/helk-install.log locally"
exit 1
fi
echo "[HELK-INSTALLATION-INFO] Running HELK via docker-compose"
docker-compose up -d >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not run HELK via docker-compose (Error Code: $ERROR)."
exit 1
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
fi
}
install_docker(){
# ****** Installing via convenience script ***********
echo "[HELK-INSTALLATION-INFO] Installing docker via convenience script.."
curl -fsSL get.docker.com -o get-docker.sh >> $LOGFILE 2>&1
chmod +x get-docker.sh >> $LOGFILE 2>&1
./get-docker.sh >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker via convenience script (Error Code: $ERROR)."
exit 1
fi
}
install_docker_compose(){
echo "[HELK-INSTALLATION-INFO] Installing docker-compose.."
curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose >> $LOGFILE 2>&1
chmod +x /usr/local/bin/docker-compose >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker-compose (Error Code: $ERROR)."
exit 1
fi
}
get_host_ip(){
# *********** Getting Host IP ***************
# https://github.com/Invoke-IR/ACE/blob/master/ACE-Docker/start.sh
echo "[HELK-INSTALLATION-INFO] Obtaining current host IP.."
case "${systemKernel}" in
Linux*) host_ip=$(ip route get 1 | awk '{print $NF;exit}');;
Darwin*) host_ip=$(ifconfig en0 | grep inet | grep -v inet6 | cut -d ' ' -f2);;
*) host_ip="UNKNOWN:${unameOut}"
esac
}
set_helk_ip(){
# *********** Accepting Defaults or Allowing user to set HELK IP ***************
local ip_choice
local read_input
read -t 30 -p "[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: " -e -i ${host_ip} ip_choice
read_input=$?
ip_choice="${ip_choice:-$host_ip}"
if [ $ip_choice != $host_ip ]; then
host_ip=$ip_choice
fi
if [ $read_input = 142 ]; then
echo -e "\n[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
else
echo "[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
fi
}
prepare_helk(){
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
get_host_ip
set_helk_ip
if [ "$systemKernel" == "Linux" ]; then
# Reference: https://get.docker.com/
echo "[HELK-INSTALLATION-INFO] HELK identified Linux as the system kernel"
echo "[HELK-INSTALLATION-INFO] Checking distribution list and version"
# *********** Check distribution list ***************
lsb_dist="$(. /etc/os-release && echo "$ID")"
lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"
# *********** Check distribution version ***************
case "$lsb_dist" in
ubuntu)
if [ -x "$(command -v lsb_release)" ]; then
dist_version="$(lsb_release --codename | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
fi
;;
debian|raspbian)
dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
case "$dist_version" in
9)
dist_version="stretch"
;;
8)
dist_version="jessie"
;;
7)
dist_version="wheezy"
;;
esac
;;
centos)
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
rhel|ol|sles)
ee_notice "$lsb_dist"
exit 1
;;
*)
if [ -x "$(command -v lsb_release)"]; then
dist_version="$(lsb_release --release | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
esac
echo "[HELK-INSTALLATION-INFO] You're using $lsb_dist version $dist_version"
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not verify distribution or version of the OS (Error Code: $ERROR)."
fi
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-INSTALLATION-INFO] Docker already installed"
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
else
echo "[HELK-INSTALLATION-INFO] Docker is not installed"
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
# ****** Install Curl if it is not installed *********
install_curl
# ****** Installing Docker if it is not installed *********
install_docker
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
fi
# ********** Check if docker-compose is installed *******
if [ -x "$(command -v docker-compose)" ]; then
echo "[HELK-INSTALLATION-INFO] Docker-compose already installed"
else
echo "[HELK-INSTALLATION-INFO] Docker-compose is not installed"
# ****** Install Curl if it is not installed *********
install_curl
# ****** Installing Docker-Compose *******************
install_docker_compose
fi
else
# *********** Check if docker is installed ***************
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
if [ -x "$(command -v docker)" ] && [ -x "$(command -v docker-compose)" ]; then
echo "[HELK-INSTALLATION-INFO] Docker & Docker-compose already installed"
else
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
echo "[HELK-INSTALLATION-INFO] Install Docker & Docker-compose for $systemKernel"
exit 1
fi
fi
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
echo "[HELK-INSTALLATION-INFO] Dockerizing HELK.."
echo "[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 262144"
MAX_MAP_COUNT=262144
if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then
sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not set vm.max_map_count to 262144 (Error Code: $ERROR)."
fi
fi
}
# *********** Showing HELK Docker menu options ***************
echo " "
echo "**********************************************"
echo "** HELK - THE HUNTING ELK **"
echo "** **"
echo "** Author: Roberto Rodriguez (@Cyb3rWard0g) **"
echo "** HELK build version: 0.9 (Alpha) **"
echo "** HELK ELK version: 6.2.3 **"
echo "** License: BSD 3-Clause **"
echo "**********************************************"
echo " "
# *********** Running selected option ***************
prepare_helk
install_helk
get_jupyter_token
sleep 45
echo " "
echo " "
echo "***********************************************************************************"
echo "** [HELK-INSTALLATION-INFO] YOUR HELK IS READY **"
echo "** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **"
echo "***********************************************************************************"
echo " "
echo "HELK KIBANA URL: http://${host_ip}"
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
echo "HELK ELASTICSEARCH EXTERNAL URL: http://${host_ip}:8082"
echo "HELK CEREBRO URL: http://${host_ip}:9000"
echo "HELK KIBANA & ELASTICSEARCH USER: helk"
echo "HELK KIBANA & ELASTICSEARCH PASSWORD: hunting"
echo "HELK JUPYTER CURRENT TOKEN: ${jupyter_token}"
echo "HELK SPARK UI: http://${host_ip}:4040"
echo "HELK JUPYTER LAB URL: http://${host_ip}:8880/lab"
echo " "
echo "IT IS HUNTING SEASON!!!!!"
echo " "
echo " "
echo " "