HELK/helk_install.sh

270 lines
11 KiB
Bash
Raw Normal View History

#!/bin/bash
# HELK script: helk_install.sh
# HELK script description: Start
# HELK build version: 0.9 (Alpha)
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
# HELK ELK version: 6.2.0
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
# *********** Check if user is root ***************
if [[ $EUID -ne 0 ]]; then
echo "[HELK-INSTALLATION-INFO] YOU MUST BE ROOT TO RUN THIS SCRIPT!!!"
exit 1
fi
LOGFILE="/var/log/helk-install.log"
echoerror() {
printf "${RC} * ERROR${EC}: $@\n" 1>&2;
}
# *********** Check System Kernel Name ***************
systemKernel="$(uname -s)"
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
# *********** Getting Jupyter Token ***************
get_token(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Waiting for HELK services and Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$(docker exec -ti helk jupyter notebook list | grep -oP '(?<=token=).*(?= ::)' | awk '{$1=$1};1')" >> $LOGFILE 2>&1
docker_access="HELK DOCKER BASH ACCESS: sudo docker exec -ti helk bash"
}
# *********** Pulling latest HELK image from DockerHub ***************
one(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Pulling the latest HELK image from Dockerhub.."
docker pull cyb3rward0g/helk >> $LOGFILE 2>&1
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk cyb3rward0g/helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
get_token
}
# *********** Building HELK image from local Dockerfile ***************
two(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Building the HELK container from local Dockerfile.."
docker build -t my_helk . >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from local Dockerfile (Error Code: $ERROR)."
exit 1
fi
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk my_helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
get_token
}
# *********** Building the HELK from local bash script ***************
three(){
echo "[HELK-BASH-INSTALLATION-INFO] Installing the HELK from local bash script"
cd scripts/
./helk_debian_tar_install.sh
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from bash script (Error Code: $ERROR)."
exit 1
fi
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
# *********** Getting Jupyter Token ***************
echo "[HELK-BASH-INSTALLATION-INFO] Waiting for Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$( cat /var/log/spark/spark_pyspark.log | grep -oP '(?<=token=).*(?=)' | sort -u)"
}
# *********** Showing HELK Docker menu options ***************
show_menus() {
echo " "
echo "**********************************************"
echo "** HELK - M E N U **"
echo "** **"
echo "** Author: Roberto Rodriguez (@Cyb3rWard0g) **"
echo "** HELK build version: 0.9 (Alpha) **"
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
echo "** HELK ELK version: 6.2.0 **"
echo "** License: BSD 3-Clause **"
echo "**********************************************"
echo " "
echo "1. Pull the latest HELK image from DockerHub"
echo "2. Build the HELK image from local Dockerfile"
echo "3. Install the HELK from local bash script"
echo "4. Exit"
echo " "
}
read_options(){
local choice
read -p "[HELK-INSTALLATION-INFO] Enter choice [ 1 - 4] " choice
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
get_host_ip
if [ $choice = "1" ] || [ $choice = "2" ]; then
if [ "$systemKernel" == "Linux" ]; then
# Reference: https://get.docker.com/
echo "[HELK-DOCKER-INSTALLATION-INFO] HELK identified Linux as the system kernel"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking distribution list and version"
# *********** Check distribution list ***************
lsb_dist="$(. /etc/os-release && echo "$ID")"
lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"
# *********** Check distribution version ***************
case "$lsb_dist" in
ubuntu)
if [ -x "$(command -v lsb_release)" ]; then
dist_version="$(lsb_release --codename | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
fi
;;
debian|raspbian)
dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
case "$dist_version" in
9)
dist_version="stretch"
;;
8)
dist_version="jessie"
;;
7)
dist_version="wheezy"
;;
esac
;;
centos)
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
rhel|ol|sles)
ee_notice "$lsb_dist"
exit 1
;;
*)
if [ -x "$(command -v lsb_release)"]; then
dist_version="$(lsb_release --release | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
esac
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
echo "[HELK-DOCKER-INSTALLATION-INFO] You're using $lsb_dist version $dist_version"
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not verify distribution or version of the OS (Error Code: $ERROR)."
fi
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking if curl is installed first"
if [ -x "$(command -v curl)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Ready to install Docker.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing curl before installing docker.."
2018-01-11 21:16:23 +00:00
apt-get install -y curl >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install curl (Error Code: $ERROR)."
exit 1
fi
fi
# ****** Installing via convenience script ***********
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing docker via convenience script.."
curl -fsSL get.docker.com -o scripts/get-docker.sh >> $LOGFILE 2>&1
chmod +x scripts/get-docker.sh >> $LOGFILE 2>&1
scripts/get-docker.sh >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker via convenience script (Error Code: $ERROR)."
exit 1
fi
fi
else
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Install docker for $systemKernel"
exit 1
fi
fi
fi
echo "[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 262144"
MAX_MAP_COUNT=262144
if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then
sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not set vm.max_map_count to 262144 (Error Code: $ERROR)."
fi
fi
case $choice in
1) one ;;
2) two ;;
3) three ;;
4) exit 0;;
*) echo -e "[HELK-INSTALLATION-INFO] Wrong choice..." && exit 1
esac
}
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
get_host_ip(){
# *********** Getting Host IP ***************
# https://github.com/Invoke-IR/ACE/blob/master/ACE-Docker/start.sh
echo "[HELK-INSTALLATION-INFO] Obtaining current host IP.."
case "${systemKernel}" in
Linux*) host_ip=$(ip route get 1 | awk '{print $NF;exit}');;
Darwin*) host_ip=$(ifconfig en0 | grep inet | grep -v inet6 | cut -d ' ' -f2);;
*) host_ip="UNKNOWN:${unameOut}"
esac
# *********** Accepting Defaults or Allowing user to set HELK IP ***************
local ip_choice
local read_input
read -t 30 -p "[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: " -e -i ${host_ip} ip_choice
read_input=$?
ip_choice="${ip_choice:-$host_ip}"
if [ $ip_choice != $host_ip ]; then
host_ip=$ip_choice
fi
if [ $read_input = 142 ]; then
echo -e "\n[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
else
echo "[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
fi
}
# *********** Running selected option ***************
show_menus
read_options
echo " "
echo " "
echo "***********************************************************************************"
echo "** [HELK-INSTALLATION-INFO] YOUR HELK IS READY **"
echo "** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **"
echo "***********************************************************************************"
echo " "
echo "HELK KIBANA URL: http://${host_ip}"
HELK ELK 6.2.0 & New features Elasticsearch + Deleted Docker elasticsearch config file (Duplicate) Logstash + Adjusted Batch size to 300 (Testing) + Renamed scripts to follow a standard naming convention + Added a fingerprint filter to all logs to help reduce duplicate logs + Removed ELK Version strings from all Logstash configs so that I dont have to update every single script every time ELK gets updated. + Added Document_id to every logstash output config to take the fingerprint value. Kibana + Renamed Index Patterns to standard naming convention. + Added experimental visualization vega setting. Enabling External URLs to use D3 libraries from their repos. This is grayed out in the Kibana config so user will have to enable it. + Updated name of index patterns across all visualizations and dashboards. Kafka + Log retention is now 24 hours and not 268 Hours + added auto_offset_reset => "earliest" to beats kafka input config Spark + updated es-hadoop version to 6.2.0 and added new spark jar packages: org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1 & databricks:spark-sklearn:0.2.3 + Created an init file to run spark and jupyter all together as a service. This will allow us to restart jupyter and pyspark gracefully. Winlogbeat + Updated Winlogbeat config to take PowerShell and Microsoft-Windows-WMI-Activity/Operational logs. New Features + Cerebro + Python packages: -scipy==1.0.0 scikit-learn==0.19.1 nltk==3.2.5 matplotlib==2.1.2 seaborn==0.8.1 datasketch==1.2.5 tensorflow==1.5.0 keras==2.1.3 pyflux==0.4.15 imbalanced-learn==0.3.2 lime==0.1.1.29 Docker Hub + New HELK image available
2018-02-15 08:28:48 +00:00
echo "HELK ELASTICSEARCH EXTERNAL URL: http://${host_ip}:8082"
echo "HELK CEREBRO URL: http://${host_ip}:9000"
echo "HELK KIBANA & ELASTICSEARCH USER: helk"
echo "HELK KIBANA & ELASTICSEARCH PASSWORD: hunting"
echo "HELK JUPYTER CURRENT TOKEN: ${jupyter_token}"
echo "HELK SPARK UI: http://${host_ip}:4040"
echo "HELK JUPYTER NOTEBOOK URI: http://${host_ip}:8880"
echo "${docker_access}"
echo " "
echo "IT IS HUNTING SEASON!!!!!"
echo " "
echo " "
echo " "