infosecn1nja
/
HELK
mirror of https://github.com/infosecn1nja/HELK.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

HELK 6.2.2 - 022518 

HELK Design
+ moved everything to docker-compose approach for a more modular design.
+ separated the HELK in 3 services:
++helk-elk, helk-kafka, helk-analytics
+ Updated Design picture to show WEF ideas and also show Jupyter Lab integrations.

HELK Docker-Compose
+ Added ESDATA volume to keep logs after contaners get stopped
+ Services restart automatically after reboot
+ created blank env file for Kafka service. This allows the host to pass its own local IP to Kafka. This is needed for advertised listener configs on each broker.

HELK-ELK Version
- Updated to 6.2.2

ELasticsearch
- Added local docker network as part of the network.host option. This allows the HELK-ELK service to publish its docker local IP to other services/images in the docker compose environment.

Logstash
+ minimal updates to  certain configs (Mainly renaming files and replacing certain strings)

Kibana
+ enableExternalUrls set to true for Vega visualization that need external libraries.

Spark - Analytics
+ Renamed service to Analytics
+ Integrated Apache Toree to allow Scala kernel in Jupyter
+ Pyspark, Scala and SQL are now available in Jupyter

Jupyter
+ Jupyter LAB has been enabled
keyword-vs-text-changes
Roberto Rodriguez 2018-02-25 02:59:44 -05:00
parent d623246f4c
commit 063e5835ec
74 changed files with 877 additions and 513 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
README.md
View File

@ -44,10 +44,10 @@ The project is currently in an alpha stage, which means that the code and the fu
* [Spark](https://github.com/Cyb3rWard0g/HELK/wiki/Spark)
* [Installation](https://github.com/Cyb3rWard0g/HELK/wiki/Installation)
## (Docker) Accessing the HELK's container
By default, the HELK's container is run in the background. Therefore, you will have to access your docker container by running the following commands:
## (Docker) Accessing the HELK's Images
By default, the HELK's containers are run in the background (Detached). Therefore, you will have to access your docker images by running the following commands:
```
sudo docker exec -ti helk bash
sudo docker exec -ti <image-name> bash
root@7a9d6443a4bf:/opt/helk/scripts#
```
@ -71,12 +71,10 @@ There are a few things that I would like to accomplish with the HELK as shown in
- [X] Add Jupyter Notebook on the top of Spark
- [X] Kafka Integration
- [ ] Create Jupyter Notebooks showing how to use Spark & GraphFrames
- [ ] Enhance elasticsearch configuration to make it more scalable
- [ ] MITRE ATT&CK mapping to logs or dashboards
- [ ] Cypher for Apache Spark Integration (Might have to switch from Jupyter to Zeppelin Notebook)
- [ ] Somehow integrate neo4j spark connectors with build
- [ ] Install Elastalert
- [ ] Create Elastalert rules
- [ ] Install Elastalert & Create Rules
- [ ] Nxlog parsers (Logstash Filters)
- [ ] Add more network data sources (i.e Bro)

68
docker-compose.yml Normal file
View File

@ -0,0 +1,68 @@
version: '3.2'
services:
helk-elk:
image: cyb3rward0g/helk-elk:6.2.2
container_name: helk-elk
volumes:
- esdata:/var/lib/elasticsearch
environment:
- bootstrap.memory_lock=true
ulimits:
memlock:
soft: -1
hard: -1
ports:
- "80:80"
- "5044:5044"
- "9000:9000"
- "8082:8082"
restart: always
networks:
helk:
ipv4_address: 172.18.0.2
aliases:
- helk_elk.hunt.local
helk-kafka:
image: cyb3rward0g/helk-kafka:1.0.0
container_name: helk-kafka
env_file: ./helk.env
ports:
- "2181:2181"
- "9092:9092"
- "9093:9093"
- "9094:9094"
restart: always
depends_on:
- helk-elk
networks:
helk:
ipv4_address: 172.18.0.3
aliases:
- helk_kafka.hunt.local
helk-analytics:
image: cyb3rward0g/helk-analytics:0.0.1
container_name: helk-analytics
ports:
- "8880:8880"
- "4040:4040"
restart: always
depends_on:
- helk-elk
networks:
helk:
ipv4_address: 172.18.0.4
aliases:
- helk_analytics.hunt.local
networks:
helk:
driver: bridge
ipam:
config:
- subnet: 172.18.0.0/16
volumes:
esdata:
driver: local

98
helk-analytics/Dockerfile Normal file
View File

@ -0,0 +1,98 @@
# HELK script: HELK Analytics Dockerfile
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
FROM phusion/baseimage
LABEL maintainer="Roberto Rodriguez @Cyb3rWard0g"
LABEL description="Dockerfile base for HELK Analytics."
ENV DEBIAN_FRONTEND noninteractive
# *********** Installing Prerequisites ***************
# -qq : No output except for errors
RUN echo "[HELK-DOCKER-INSTALLATION-INFO] Updating Ubuntu base image.." \
&& apt-get update -qq \
&& echo "[HELK-DOCKER-INSTALLATION-INFO] Extracting templates from packages.." \
&& apt-get install -qqy \
openjdk-8-jre-headless \
wget \
sudo \
nano \
python3-pip \
python-tk \
unzip \
zip \
locales
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
RUN apt-get -qy clean \
autoremove \
&& rm -rf /var/lib/apt/lists/*
# *********** Upgrading PIP ***************
RUN pip3 install --upgrade pip
# *********** Installing HELK python packages ***************
RUN pip3 install \
pandas==0.22.0 \
jupyter \
jupyterhub==0.8.1 \
jupyterlab==0.31.8 \
https://dist.apache.org/repos/dist/dev/incubator/toree/0.2.0/snapshots/dev1/toree-pip/toree-0.2.0.dev1.tar.gz
RUN pip3 install scipy==1.0.0 \
scikit-learn==0.19.1 \
nltk==3.2.5 \
matplotlib==2.1.2 \
seaborn==0.8.1 \
datasketch==1.2.5 \
keras==2.1.3 \
pyflux==0.4.15 \
imbalanced-learn==0.3.2 \
lime==0.1.1.29 \
bokeh==0.12.14
# *********** Creating the right directories ***************
RUN bash -c 'mkdir -pv /opt/helk/{scripts,training,es-hadoop,spark,packages}'
# *********** Adding HELK scripts and files to Container ***************
ADD scripts/analytics-entrypoint.sh /opt/helk/scripts/
RUN chmod +x /opt/helk/scripts/analytics-entrypoint.sh
ADD training/ /opt/helk/training/
# *********** Install ES-Hadoop ***************
RUN wget https://artifacts.elastic.co/downloads/elasticsearch-hadoop/elasticsearch-hadoop-6.2.2.zip -P /opt/helk/es-hadoop/ \
&& unzip /opt/helk/es-hadoop/*.zip -d /opt/helk/es-hadoop/ \
&& rm /opt/helk/es-hadoop/*.zip
# *********** Install Spark ***************
ENV ANALYTIC_LOGS_PATH=/var/log/analytics
RUN wget -qO- http://mirrors.gigenet.com/apache/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz | sudo tar xvz -C /opt/helk/spark/ \
&& mkdir -v $ANALYTIC_LOGS_PATH
ADD spark/.bashrc ~/.bashrc
ADD spark/log4j.properties /opt/helk/spark/spark-2.2.1-bin-hadoop2.7/conf/
ADD spark/spark-defaults.conf /opt/helk/spark/spark-2.2.1-bin-hadoop2.7/conf/
ADD analytics-init /etc/init.d/analytics
# Adding SPARK environment variables
ENV SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
ENV PATH=$SPARK_HOME/bin:$PATH
#ENV SPARK_OPTS="--driver-java-options=-Xms1024M --driver-java-options=-Xmx2096M --driver-java-options=-Dlog4j.logLevel=info --master=local[4]"
ENV SPARK_OPTS="--master local[*]"
ENV PYSPARK_PYTHON=/usr/bin/python3
ENV PYSPARK_DRIVER_PYTHON=/usr/local/bin/jupyter
ENV PYSPARK_DRIVER_PYTHON_OPTS="lab --no-browser --ip=* --port=8880 --allow-root"
# *********** Attaching Toree Kernel to Jupyter ***************
RUN jupyter toree install --spark_home=$SPARK_HOME --interpreters=Scala,SQL
# *********** Update Jupyter PySpark Kernel *************
ADD jupyter/pyspark_kernel.json /usr/local/share/jupyter/kernels/python3/kernel.json
# *********** RUN HELK ***************
EXPOSE 4040 8880
WORKDIR "/opt/helk/scripts/"
ENTRYPOINT ["./analytics-entrypoint.sh"]

64
spark/spark-init → helk-analytics/analytics-init
View File

@ -1,21 +1,21 @@
#!/bin/bash
# Init script for spark
# /etc/init.d/analytics -- startup script for Analytics
# Maintained by Roberto Rodriguez @Cyb3rWard0g
# Reference:
# https://github.com/elastic/logstash/blob/master/distribution/rpm/src/main/packaging/init.d/logstash
# https://github.com/spujadas/elk-docker/blob/master/logstash-init
### BEGIN INIT INFO
# Provides: spark
# Provides: analytics
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: spark service
# Short-Description: analytic service
### END INIT INFO
PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin
NAME=spark
NAME=analytics
DEFAULT=/etc/default/$NAME
export PATH
@ -32,15 +32,15 @@ if [ -r /etc/default/rcS ]; then
fi
SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
SPARK_CONSOLE_PYSPARK_LOG=/var/log/spark/spark_pyspark.log
SPARK_EXEC=$SPARK_HOME/bin/pyspark
SPARK_CONFIG="2>&1 >> $SPARK_CONSOLE_PYSPARK_LOG 2>&1"
SPARK_USER=root
SPARK_GROUP=root
SPARK_NICE=""
SERVICE_NAME="spark"
SERVICE_DESCRIPTION="spark"
SPARK_PIDFILE=/var/run/spark.pid
ANALYTIC_CONSOLE_LOG=/var/log/analytics/analytics.log
ANALYTIC_EXEC=$SPARK_HOME/bin/pyspark
ANALYTIC_OPTS=">> $ANALYTIC_CONSOLE_LOG 2>&1"
ANALYTIC_USER=root
ANALYTIC_GROUP=root
ANALYTIC_NICE=""
SERVICE_NAME="analytics"
SERVICE_DESCRIPTION="analytics"
ANALYTIC_PIDFILE=/var/run/analytics.pid
# End of variables that can be overwritten in $DEFAULT
@ -49,39 +49,37 @@ if [ -f "$DEFAULT" ]; then
. "$DEFAULT"
fi
# Adding SPARK location
export SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
# Spark Variables
export PATH=$SPARK_HOME/bin:$PATH
# Adding Jupyter Notebook Integration
export SPARK_OPTS="--master local[*]"
export PYSPARK_PYTHON=/usr/bin/python3
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880 --allow-root"
export PYSPARK_PYTHON=/usr/bin/python
export PYSPARK_DRIVER_PYTHON_OPTS="lab --no-browser --ip=* --port=8880 --allow-root"
[ -z "$SPARK_NICE" ] && SPARK_NICE=0
[ -z "$ANALYTIC_NICE" ] && ANALYTIC_NICE=0
if [ ! -x "$SPARK_EXEC" ]; then
echo "The spark startup script does not exists or it is not executable, tried: $SPARK_EXEC"
if [ ! -x "$ANALYTIC_EXEC" ]; then
echo "The analytics startup script does not exists or it is not executable, tried: $ANALYTIC_EXEC"
exit 1
fi
start() {
echo "Starting $NAME"
if [ -n "$SPARK_PIDFILE" ] && [ ! -e "$SPARK_PIDFILE" ]; then
touch "$SPARK_PIDFILE" && chown $SPARK_USER:$SPARK_GROUP "$SPARK_PIDFILE"
if [ -n "$ANALYTIC_PIDFILE" ] && [ ! -e "$ANALYTIC_PIDFILE" ]; then
touch "$ANALYTIC_PIDFILE" && chown $ANALYTIC_USER:$ANALYTIC_GROUP "$ANALYTIC_PIDFILE"
fi
# Start Service
nice -n$SPARK_NICE chroot --userspec $SPARK_USER:$SPARK_GROUP / sh -c "
nice -n$ANALYTIC_NICE chroot --userspec $ANALYTIC_USER:$ANALYTIC_GROUP / sh -c "
cd /opt/helk
exec $SPARK_EXEC $SPARK_CONFIG
exec $ANALYTIC_EXEC $SPARK_OPTS $ANALYTIC_OPTS
" &
# Generate the pidfile from here. If we instead made the forked process
# generate it there will be a race condition between the pidfile writing
# and a process possibly asking for status.
echo $! > $SPARK_PIDFILE
echo $! > $ANALYTIC_PIDFILE
echo "$NAME started."
return 0
@ -90,7 +88,7 @@ start() {
stop() {
# Try a few times to kill TERM the program
if status; then
pid=$(cat "$SPARK_PIDFILE")
pid=$(cat "$ANALYTIC_PIDFILE")
echo "Killing $NAME (pid $pid) with SIGTERM"
kill -TERM $pid
# Wait for it to exit.
@ -103,14 +101,14 @@ stop() {
echo "$NAME stop failed; still running."
else
echo "$NAME stopped."
rm -f $SPARK_PIDFILE
rm -f $ANALYTIC_PIDFILE
fi
fi
}
status() {
if [ -f "$SPARK_PIDFILE" ] ; then
pid=$(cat "$SPARK_PIDFILE")
if [ -f "$ANALYTIC_PIDFILE" ] ; then
pid=$(cat "$ANALYTIC_PIDFILE")
if kill -0 $pid > /dev/null 2> /dev/null; then
# process by this pid is running.
# It may not be our pid, but that's what you get with just pidfiles.
@ -129,8 +127,8 @@ status() {
force_stop() {
if status; then
stop
status && kill -KILL $(cat "$SPARK_PIDFILE")
rm -f $SPARK_PIDFILE
status && kill -KILL $(cat "$ANALYTIC_PIDFILE")
rm -f $ANALYTIC_PIDFILE
fi
}

11
helk-analytics/jupyter/pyspark_kernel.json Normal file
View File

@ -0,0 +1,11 @@
{
"argv": [
"python3",
"-m",
"ipykernel_launcher",
"-f",
"{connection_file}"
],
"display_name": "PySpark",
"language": "python"
}

24
helk-analytics/scripts/analytics-entrypoint.sh Executable file
View File

@ -0,0 +1,24 @@
#!/bin/sh
# HELK script: analytics-entryppoint.sh
# HELK script description: Restart HELK Analytic services
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
# Start graceful termination of HELK services that might be running before running the entrypoint script.
_term() {
echo "Terminating HELK analytics services"
service analytics stop
exit 0
}
trap _term SIGTERM
# Removing PID files just in case the graceful termination fails
rm -f /var/run/analytics.pid
echo "[HELK-DOCKER-INSTALLATION-INFO] Starting analytic services.."
service analytics start
sleep 5
echo "[HELK-DOCKER-INSTALLATION-INFO] Pushing analytic Logs to console.."
tail -f /var/log/analytics/analytics.log

10
spark/.bashrc → helk-analytics/spark/.bashrc
View File

@ -117,10 +117,10 @@ if ! shopt -oq posix; then
fi
# Adding SPARK location
export SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
#export SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
#export PATH=$SPARK_HOME/bin:$PATH
# Adding Jupyter Notebook Integration
export PYSPARK_DRIVER_PYTHON=/usr/local/bin/jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880 --allow-root"
export PYSPARK_PYTHON=/usr/bin/python
#export PYSPARK_DRIVER_PYTHON=/usr/local/bin/jupyter
#export PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880 --allow-root"
#export PYSPARK_PYTHON=/usr/bin/python

0
spark/log4j.properties → helk-analytics/spark/log4j.properties
View File

4
spark/spark-defaults.conf → helk-analytics/spark/spark-defaults.conf
View File

@ -31,6 +31,6 @@
# https://graphframes.github.io/quick-start.html
# https://spark-packages.org/package/graphframes/graphframes
spark.jars /opt/helk/es-hadoop/elasticsearch-hadoop-6.2.0/dist/elasticsearch-hadoop-6.2.0.jar
spark.jars /opt/helk/es-hadoop/elasticsearch-hadoop-6.2.2/dist/elasticsearch-hadoop-6.2.2.jar
spark.jars.packages graphframes:graphframes:0.5.0-spark2.1-s_2.11,org.apache.spark:spark-sql-kafka-0-10_2.11:2.2.1,databricks:spark-sklearn:0.2.3
spark.python.profile true
#spark.python.profile true

249
training/jupyter_notebooks/getting_started/Check_Spark_Graphframes_Integrations.ipynb → helk-analytics/training/jupyter_notebooks/getting_started/Check_Spark_Graphframes_Integrations_PySpark.ipynb
View File

@ -31,7 +31,7 @@
" <div>\n",
" <p><b>SparkContext</b></p>\n",
"\n",
" <p><a href=\"http://192.168.1.243:4040\">Spark UI</a></p>\n",
" <p><a href=\"http://172.18.0.4:4040\">Spark UI</a></p>\n",
"\n",
" <dl>\n",
" <dt>Version</dt>\n",
@ -66,45 +66,49 @@
},
{
"cell_type": "code",
"execution_count": 2,
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(u'EyaM12ABGZB0cH7uy-kS',\n",
" {u'@timestamp': u'2018-01-08T20:19:23.195Z',\n",
"(u'2053487453',\n",
" {u'@meta': {u'log': {u'timestamp': u'2018-02-20T17:16:29.294Z'}},\n",
" u'@timestamp': u'2018-02-20T17:16:29.299Z',\n",
" u'@version': u'1',\n",
" u'beat': {u'hostname': u'WD-HR001',\n",
" u'name': u'WD-HR001',\n",
" u'version': u'6.0.0'},\n",
" u'computer_name': u'WD-HR001.wardog.com',\n",
" u'event': {u'creationtime': {u'utc': u'2018-01-08 20:19:22.978'}},\n",
" u'event_id': 9,\n",
" u'host': u'WD-HR001',\n",
" u'action': u'processaccess',\n",
" u'beat': {u'hostname': u'DESKTOP-29DJI4T',\n",
" u'name': u'DESKTOP-29DJI4T',\n",
" u'version': u'6.1.2'},\n",
" u'computer_name': u'DESKTOP-29DJI4T',\n",
" u'event_id': 10,\n",
" u'level': u'Information',\n",
" u'log_name': u'Microsoft-Windows-Sysmon/Operational',\n",
" u'opcode': u'Info',\n",
" u'process': {u'guid': u'{DBA5A4A0-2F96-5A50-0000-00106D560100}',\n",
" u'id': 1428,\n",
" u'name': u'C:\\\\Windows\\\\System32\\\\svchost.exe'},\n",
" u'process_id': 2216,\n",
" u'process': {u'calltrace': u'C:\\\\WINDOWS\\\\SYSTEM32\\\\ntdll.dll+a0344|C:\\\\WINDOWS\\\\System32\\\\KERNELBASE.dll+3dc5d|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+ec56d|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+ec490|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+6a4fdd|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+6a6b42|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+69e6da|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+69caa0|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+202a6|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+77f76e|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+471b|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+2b5e|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+98d9c|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+11125f|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\Definition Updates\\\\{14FF058F-73CB-47DF-835D-8B578620CD35}\\\\mpengine.dll+110ef6|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.12.17007.18011-0\\\\mpsvc.dll+117f8|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.12.17007.18011-0\\\\mprtp.dll+12113|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.12.17007.18011-0\\\\mprtp.dll+33318|C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.12.17007.18011-0\\\\mpclient.dll+7cb40|C:\\\\WINDOWS\\\\SYSTEM32\\\\ntdll.dll+362a1|C:\\\\WINDOWS\\\\SYSTEM32\\\\ntdll.dll+346fe|C:\\\\WINDOWS\\\\System32\\\\KERNEL32.DLL+11fe4|C:\\\\WINDOWS\\\\SYSTEM32\\\\ntdll.dll+6efc1',\n",
" u'grantedaccess': u'0x1000',\n",
" u'guid': u'{A98268C1-E251-5A83-0000-0010F50A0200}',\n",
" u'id': 2120,\n",
" u'path': u'C:\\\\ProgramData\\\\Microsoft\\\\Windows Defender\\\\platform\\\\4.12.17007.18011-0\\\\MsMpEng.exe',\n",
" u'target': {u'guid': u'{A98268C1-E251-5A83-0000-0010B4120200}',\n",
" u'id': 2152,\n",
" u'path': u'C:\\\\Program Files\\\\winlogbeat\\\\winlogbeat.exe'},\n",
" u'threadid': 3900},\n",
" u'process_id': 1896,\n",
" u'provider_guid': u'{5770385F-C22A-43E0-BF4C-06F5698FFBD9}',\n",
" u'rawaccess': {u'read': {u'device': u'\\\\Device\\\\HarddiskVolume2'}},\n",
" u'record_number': u'1006036',\n",
" u'record_number': u'9876601',\n",
" u'source_name': u'Microsoft-Windows-Sysmon',\n",
" u'subject': {u'user': {u'domain': u'NT AUTHORITY',\n",
" u'name': u'SYSTEM',\n",
" u'sid': u'S-1-5-18'}},\n",
" u'tags': (u'beats_input_codec_plain_applied', u'_grokparsefailure'),\n",
" u'task': u'RawAccessRead detected (rule: RawAccessRead)',\n",
" u'thread_id': 3548,\n",
" u'task': u'Process accessed (rule: ProcessAccess)',\n",
" u'thread_id': 3460,\n",
" u'type': u'wineventlog',\n",
" u'user': {u'type': u'User'},\n",
" u'version': 2})"
" u'version': 3})"
]
},
"execution_count": 2,
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
@ -114,7 +118,10 @@
" inputFormatClass=\"org.elasticsearch.hadoop.mr.EsInputFormat\",\n",
" keyClass=\"org.apache.hadoop.io.NullWritable\",\n",
" valueClass=\"org.elasticsearch.hadoop.mr.LinkedMapWritable\",\n",
" conf={ \"es.resource\" : \"logs-endpoint-winevent-sysmon-*/doc\" })\n",
" conf={ \n",
" \"es.resource\" : \"logs-endpoint-winevent-sysmon-*/doc\",\n",
" \"es.nodes\" : \"172.18.0.2\"\n",
" })\n",
"es_rdd.first()"
]
},
@ -127,49 +134,45 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 19,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(u'tiZ912ABGZB0cH7uLt6P',\n",
" {u'@timestamp': u'2018-01-08T20:31:24.258Z',\n",
"(u'2852068011',\n",
" {u'@timestamp': u'2018-02-21T00:08:12.345Z',\n",
" u'@version': u'1',\n",
" u'beat': {u'hostname': u'WD-DC001',\n",
" u'name': u'WD-DC001',\n",
" u'version': u'6.1.1'},\n",
" u'computer_name': u'WD-DC001.wardog.com',\n",
" u'beat': {u'hostname': u'DESKTOP-29DJI4T',\n",
" u'name': u'DESKTOP-29DJI4T',\n",
" u'version': u'6.1.2'},\n",
" u'computer_name': u'DESKTOP-29DJI4T',\n",
" u'event_data': {},\n",
" u'event_id': 4703,\n",
" u'host': u'WD-DC001',\n",
" u'event_id': 4616,\n",
" u'keywords': (u'Audit Success',),\n",
" u'level': u'Information',\n",
" u'log_name': u'Security',\n",
" u'message': u'A token right was adjusted.\\n\\nSubject:\\n\\tSecurity ID:\\t\\tS-1-5-18\\n\\tAccount Name:\\t\\tWD-DC001$\\n\\tAccount Domain:\\t\\tWARDOG\\n\\tLogon ID:\\t\\t0x3E7\\n\\nTarget Account:\\n\\tSecurity ID:\\t\\tS-1-0-0\\n\\tAccount Name:\\t\\tWD-DC001$\\n\\tAccount Domain:\\t\\tWARDOG\\n\\tLogon ID:\\t\\t0x3E7\\n\\nProcess Information:\\n\\tProcess ID:\\t\\t0xe8\\n\\tProcess Name:\\t\\tC:\\\\Windows\\\\System32\\\\svchost.exe\\n\\nEnabled Privileges:\\n\\t\\t\\tSeAssignPrimaryTokenPrivilege\\n\\t\\t\\tSeIncreaseQuotaPrivilege\\n\\t\\t\\tSeSecurityPrivilege\\n\\t\\t\\tSeTakeOwnershipPrivilege\\n\\t\\t\\tSeLoadDriverPrivilege\\n\\t\\t\\tSeSystemtimePrivilege\\n\\t\\t\\tSeBackupPrivilege\\n\\t\\t\\tSeRestorePrivilege\\n\\t\\t\\tSeShutdownPrivilege\\n\\t\\t\\tSeSystemEnvironmentPrivilege\\n\\t\\t\\tSeUndockPrivilege\\n\\t\\t\\tSeManageVolumePrivilege\\n\\nDisabled Privileges:\\n\\t\\t\\t-',\n",
" u'message': u'The system time was changed.\\n\\nSubject:\\n\\tSecurity ID:\\t\\tS-1-5-18\\n\\tAccount Name:\\t\\tDESKTOP-29DJI4T$\\n\\tAccount Domain:\\t\\tWORKGROUP\\n\\tLogon ID:\\t\\t0x3E7\\n\\nProcess Information:\\n\\tProcess ID:\\t0x834\\n\\tName:\\t\\tC:\\\\Program Files\\\\VMware\\\\VMware Tools\\\\vmtoolsd.exe\\n\\nPrevious Time:\\t\\t\\u200e2018\\u200e-\\u200e02\\u200e-\\u200e20T17:16:32.271066000Z\\nNew Time:\\t\\t\\u200e2018\\u200e-\\u200e02\\u200e-\\u200e21T00:08:12.117000000Z\\n\\nThis event is generated when the system time is changed. It is normal for the Windows Time Service, which runs with System privilege, to change the system time on a regular basis. Other system time changes may be indicative of attempts to tamper with the computer.',\n",
" u'opcode': u'Info',\n",
" u'process': {u'id': 0, u'name': u'C:\\\\Windows\\\\System32\\\\svchost.exe'},\n",
" u'process': {u'id': 0,\n",
" u'path': u'C:\\\\Program Files\\\\VMware\\\\VMware Tools\\\\vmtoolsd.exe'},\n",
" u'process_id': 4,\n",
" u'provider_guid': u'{54849625-5478-4994-A5BA-3E3B0328C30D}',\n",
" u'record_number': u'508597',\n",
" u'record_number': u'11346',\n",
" u'source_name': u'Microsoft-Windows-Security-Auditing',\n",
" u'tags': (u'beats_input_codec_plain_applied',),\n",
" u'task': u'Token Right Adjusted Events',\n",
" u'thread_id': 4800,\n",
" u'subject': {u'logon': {u'id': u'0x3e7'},\n",
" u'user': {u'domain': u'WORKGROUP',\n",
" u'name': u'DESKTOP-29DJI4T$',\n",
" u'sid': u'S-1-5-18'}},\n",
" u'system': {u'newtime': u'2018-02-21T00:08:12.117000000Z',\n",
" u'previoustime': u'2018-02-20T17:16:32.271066000Z'},\n",
" u'task': u'Security State Change',\n",
" u'thread_id': 4300,\n",
" u'type': u'wineventlog',\n",
" u'user': {u'domain': u'WARDOG',\n",
" u'logon': {u'id': u'0x3e7'},\n",
" u'name': u'WD-DC001$',\n",
" u'sid': u'S-1-5-18',\n",
" u'target': {u'disabledprivilegelist': u'-',\n",
" u'domain': u'WARDOG',\n",
" u'enabledprivilegelist': u'SeAssignPrimaryTokenPrivilege\\n\\t\\t\\tSeIncreaseQuotaPrivilege\\n\\t\\t\\tSeSecurityPrivilege\\n\\t\\t\\tSeTakeOwnershipPrivilege\\n\\t\\t\\tSeLoadDriverPrivilege\\n\\t\\t\\tSeSystemtimePrivilege\\n\\t\\t\\tSeBackupPrivilege\\n\\t\\t\\tSeRestorePrivilege\\n\\t\\t\\tSeShutdownPrivilege\\n\\t\\t\\tSeSystemEnvironmentPrivilege\\n\\t\\t\\tSeUndockPrivilege\\n\\t\\t\\tSeManageVolumePrivilege',\n",
" u'logonid': u'0x3e7',\n",
" u'name': u'WD-DC001$',\n",
" u'sid': u'S-1-0-0'}}})"
" u'version': 1})"
]
},
"execution_count": 3,
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
@ -179,7 +182,10 @@
" inputFormatClass=\"org.elasticsearch.hadoop.mr.EsInputFormat\",\n",
" keyClass=\"org.apache.hadoop.io.NullWritable\",\n",
" valueClass=\"org.elasticsearch.hadoop.mr.LinkedMapWritable\",\n",
" conf={ \"es.resource\" : \"logs-endpoint-winevent-security-*/doc\" })\n",
" conf={ \n",
" \"es.resource\" : \"logs-endpoint-winevent-security-*/doc\",\n",
" \"es.nodes\" : \"172.18.0.2\"\n",
" })\n",
"es_rdd.first()"
]
},
@ -192,7 +198,7 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
@ -261,7 +267,7 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
@ -269,6 +275,7 @@
" .builder \\\n",
" .appName(\"HELK\") \\\n",
" .config(\"es.read.field.as.array.include\", \"tags\") \\\n",
" .config(\"es.nodes\",\"172.18.0.2:9200\") \\\n",
" .getOrCreate()"
]
},
@ -281,7 +288,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
@ -290,7 +297,7 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 16,
"metadata": {},
"outputs": [
{
@ -307,43 +314,10 @@
" | |-- version: string (nullable = true)\n",
" |-- computer_name: string (nullable = true)\n",
" |-- destination: struct (nullable = true)\n",
" | |-- hostnameid: string (nullable = true)\n",
" | |-- ip: string (nullable = true)\n",
" | |-- port: struct (nullable = true)\n",
" | | |-- number: integer (nullable = true)\n",
" | |-- userid: string (nullable = true)\n",
" |-- event_data: struct (nullable = true)\n",
" | |-- ActiveProfile: string (nullable = true)\n",
" |-- event_id: long (nullable = true)\n",
" |-- externaldevice: struct (nullable = true)\n",
" | |-- classid: string (nullable = true)\n",
" | |-- classname: string (nullable = true)\n",
" | |-- compatibleids: string (nullable = true)\n",
" | |-- description: string (nullable = true)\n",
" | |-- id: string (nullable = true)\n",
" | |-- locationinformation: string (nullable = true)\n",
" | |-- vendorids: string (nullable = true)\n",
" |-- filtering: struct (nullable = true)\n",
" | |-- action: string (nullable = true)\n",
" | |-- calloutkey: string (nullable = true)\n",
" | |-- calloutname: string (nullable = true)\n",
" | |-- changetype: string (nullable = true)\n",
" | |-- conditions: string (nullable = true)\n",
" | |-- id: string (nullable = true)\n",
" | |-- key: string (nullable = true)\n",
" | |-- layerid: string (nullable = true)\n",
" | |-- layerkey: string (nullable = true)\n",
" | |-- layername: string (nullable = true)\n",
" | |-- name: string (nullable = true)\n",
" | |-- providerkey: string (nullable = true)\n",
" | |-- providername: string (nullable = true)\n",
" | |-- type: string (nullable = true)\n",
" | |-- weight: string (nullable = true)\n",
" |-- firewall: struct (nullable = true)\n",
" | |-- ruleattr: string (nullable = true)\n",
" | |-- ruleid: string (nullable = true)\n",
" | |-- rulename: string (nullable = true)\n",
" |-- host: string (nullable = true)\n",
" |-- impersonationlevel: string (nullable = true)\n",
" |-- keywords: string (nullable = true)\n",
" |-- level: string (nullable = true)\n",
@ -360,66 +334,20 @@
" | |-- type: string (nullable = true)\n",
" | |-- virtualaccount: string (nullable = true)\n",
" |-- message: string (nullable = true)\n",
" |-- network: struct (nullable = true)\n",
" | |-- direction: string (nullable = true)\n",
" | |-- filterrtid: string (nullable = true)\n",
" | |-- layername: string (nullable = true)\n",
" | |-- layerrtid: string (nullable = true)\n",
" | |-- protocol: string (nullable = true)\n",
" |-- object: struct (nullable = true)\n",
" | |-- access: struct (nullable = true)\n",
" | | |-- listrequested: string (nullable = true)\n",
" | | |-- maskrequested: string (nullable = true)\n",
" | | |-- reason: string (nullable = true)\n",
" | | |-- transactionid: string (nullable = true)\n",
" | |-- additionalinfo: string (nullable = true)\n",
" | |-- additionalinfo2: string (nullable = true)\n",
" | |-- handleid: string (nullable = true)\n",
" | |-- name: string (nullable = true)\n",
" | |-- newsddl: string (nullable = true)\n",
" | |-- oldsddl: string (nullable = true)\n",
" | |-- operationtype: string (nullable = true)\n",
" | |-- privilegelist: string (nullable = true)\n",
" | |-- properties: string (nullable = true)\n",
" | |-- resourceattributes: string (nullable = true)\n",
" | |-- restrictedsidcount: string (nullable = true)\n",
" | |-- server: string (nullable = true)\n",
" | |-- type: string (nullable = true)\n",
" |-- opcode: string (nullable = true)\n",
" |-- proces: struct (nullable = true)\n",
" | |-- tokenelevationtype: string (nullable = true)\n",
" |-- process: struct (nullable = true)\n",
" | |-- handleid: string (nullable = true)\n",
" | |-- id: integer (nullable = true)\n",
" | |-- mandatorylevel: string (nullable = true)\n",
" | |-- name: string (nullable = true)\n",
" | |-- parent: struct (nullable = true)\n",
" | | |-- id: integer (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | |-- status: string (nullable = true)\n",
" | |-- path: string (nullable = true)\n",
" | |-- target: struct (nullable = true)\n",
" | | |-- handleid: string (nullable = true)\n",
" | | |-- id: integer (nullable = true)\n",
" | |-- terminalsessionid: integer (nullable = true)\n",
" |-- process_id: long (nullable = true)\n",
" |-- provider_guid: string (nullable = true)\n",
" |-- record_number: string (nullable = true)\n",
" |-- service: struct (nullable = true)\n",
" | |-- name: string (nullable = true)\n",
" | |-- privilegelist: string (nullable = true)\n",
" | |-- ticket: struct (nullable = true)\n",
" | | |-- id: string (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | | |-- preauthtype: string (nullable = true)\n",
" | | |-- requested: string (nullable = true)\n",
" | | |-- status: string (nullable = true)\n",
" |-- share: struct (nullable = true)\n",
" | |-- localpath: string (nullable = true)\n",
" | |-- name: string (nullable = true)\n",
" | |-- relativetargetname: string (nullable = true)\n",
" |-- source: struct (nullable = true)\n",
" | |-- hostname: string (nullable = true)\n",
" | |-- hostnameinfo: string (nullable = true)\n",
" | |-- ip: string (nullable = true)\n",
" | |-- port: struct (nullable = true)\n",
" | | |-- number: integer (nullable = true)\n",
@ -434,25 +362,11 @@
" |-- system: struct (nullable = true)\n",
" | |-- newtime: timestamp (nullable = true)\n",
" | |-- previoustime: timestamp (nullable = true)\n",
" |-- tags: array (nullable = true)\n",
" | |-- element: string (containsNull = true)\n",
" |-- task: string (nullable = true)\n",
" |-- task_name: string (nullable = true)\n",
" |-- task_newcontent: string (nullable = true)\n",
" |-- thread_id: long (nullable = true)\n",
" |-- ticket: struct (nullable = true)\n",
" | |-- encryptiontype: string (nullable = true)\n",
" | |-- options: string (nullable = true)\n",
" |-- type: string (nullable = true)\n",
" |-- user: struct (nullable = true)\n",
" | |-- access: struct (nullable = true)\n",
" | | |-- reason: string (nullable = true)\n",
" | |-- domain: string (nullable = true)\n",
" | |-- explicit: struct (nullable = true)\n",
" | | |-- domain: string (nullable = true)\n",
" | | |-- logonguid: string (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | |-- groupmembership: string (nullable = true)\n",
" | |-- logon: struct (nullable = true)\n",
" | | |-- guid: string (nullable = true)\n",
" | | |-- id: string (nullable = true)\n",
@ -461,20 +375,7 @@
" | |-- networkaccount: struct (nullable = true)\n",
" | | |-- domain: string (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | |-- principal: struct (nullable = true)\n",
" | | |-- domain: string (nullable = true)\n",
" | | |-- id: string (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | | |-- sid: string (nullable = true)\n",
" | |-- sessionid: string (nullable = true)\n",
" | |-- sid: string (nullable = true)\n",
" | |-- target: struct (nullable = true)\n",
" | | |-- disabledprivilegelist: string (nullable = true)\n",
" | | |-- domain: string (nullable = true)\n",
" | | |-- enabledprivilegelist: string (nullable = true)\n",
" | | |-- logonid: string (nullable = true)\n",
" | | |-- name: string (nullable = true)\n",
" | | |-- sid: string (nullable = true)\n",
" |-- version: integer (nullable = true)\n",
"\n"
]
@ -486,7 +387,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 18,
"metadata": {},
"outputs": [
{
@ -496,28 +397,10 @@
"+--------------------+\n",
"| task|\n",
"+--------------------+\n",
"|Token Right Adjus...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Token Right Adjus...|\n",
"|Filtering Platfor...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Filtering Platfor...|\n",
"|Token Right Adjus...|\n",
"|Token Right Adjus...|\n",
"|Security State Ch...|\n",
"| Special Logon|\n",
"| Logon|\n",
"+--------------------+\n",
"only showing top 20 rows\n",
"\n"
]
}

78
Dockerfile → helk-elk/Dockerfile
View File

@ -1,7 +1,6 @@
# HELK script: HELK Dockerfile
# HELK script description: Dockerize the HELK build
# HELK script: HELK ELK Dockerfile
# HELK build version: 0.9 (ALPHA)
# HELK ELK version: 6.2.0
# HELK ELK version: 6.2.2
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
@ -10,8 +9,8 @@
# https://github.com/spujadas/elk-docker/blob/master/Dockerfile
FROM phusion/baseimage
MAINTAINER Roberto Rodriguez @Cyb3rWard0g
LABEL description="Dockerfile base for the HELK."
LABEL maintainer="Roberto Rodriguez @Cyb3rWard0g"
LABEL description="Dockerfile base for the HELK ELK."
ENV DEBIAN_FRONTEND noninteractive
@ -27,8 +26,10 @@ RUN echo "[HELK-DOCKER-INSTALLATION-INFO] Updating Ubuntu base image.." \
nano \
python \
python-pip \
python-tk \
unzip
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
RUN apt-get -qy clean \
autoremove
@ -38,32 +39,21 @@ RUN pip install --upgrade pip
# *********** Installing HELK python packages ***************
RUN pip install \
OTXv2 \
pandas==0.22.0 \
jupyter
RUN pip install scipy==1.0.0 \
scikit-learn==0.19.1 \
nltk==3.2.5 \
matplotlib==2.1.2 \
seaborn==0.8.1 \
datasketch==1.2.5 \
tensorflow==1.5.0 \
keras==2.1.3 \
pyflux==0.4.15 \
imbalanced-learn==0.3.2 \
lime==0.1.1.29
pandas==0.22.0
# *********** Creating the right directories ***************
RUN bash -c 'mkdir -pv /opt/helk/{scripts,training,otx,es-hadoop,spark,output_templates,dashboards,kafka,elasticsearch,logstash,kibana,cerebro,ksql}'
#RUN bash -c 'mkdir -pv /opt/helk/{scripts,training,otx,es-hadoop,spark,output_templates,dashboards,kafka,elasticsearch,logstash,kibana,cerebro,ksql}'
RUN bash -c 'mkdir -pv /opt/helk/{scripts,otx,output_templates,dashboards,elasticsearch,logstash,kibana,cerebro,ksql}'
# *********** Adding HELK scripts and files to Container ***************
ADD scripts/helk_otx.py /opt/helk/scripts/
ADD scripts/helk_kibana_setup.sh /opt/helk/scripts/
ADD scripts/helk_docker_entrypoint.sh /opt/helk/scripts/
ADD training/ /opt/helk/training/
ADD scripts/elk-kibana-setup.sh /opt/helk/scripts/
ADD scripts/elk-entrypoint.sh /opt/helk/scripts/
RUN chmod +x /opt/helk/scripts/elk-kibana-setup.sh
RUN chmod +x /opt/helk/scripts/elk-entrypoint.sh
# *********** ELK Version ***************
ENV ELK_VERSION=6.2.0
ENV ELK_VERSION=6.2.2
# *********** Installing Elasticsearch ***************
ENV ES_HELK_HOME=/opt/helk/elasticsearch
@ -141,31 +131,6 @@ ADD enrichments/otx/ /opt/helk/otx/
RUN cronjob="0 8 * * 1 python /opt/helk/scripts/helk_otx.py" \
&& echo "$cronjob" | crontab
# *********** Install ES-Hadoop ***************
RUN wget https://artifacts.elastic.co/downloads/elasticsearch-hadoop/elasticsearch-hadoop-6.2.0.zip -P /opt/helk/es-hadoop/ \
&& unzip /opt/helk/es-hadoop/*.zip -d /opt/helk/es-hadoop/ \
&& rm /opt/helk/es-hadoop/*.zip
# *********** Install Spark ***************
ENV SPARK_LOGS_PATH=/var/log/spark
RUN wget -qO- http://mirrors.gigenet.com/apache/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz | sudo tar xvz -C /opt/helk/spark/ \
&& mkdir -v $SPARK_LOGS_PATH
ADD spark/.bashrc ~/.bashrc
ADD spark/log4j.properties /opt/helk/spark/spark-2.2.1-bin-hadoop2.7/conf/
ADD spark/spark-defaults.conf /opt/helk/spark/spark-2.2.1-bin-hadoop2.7/conf/
ADD spark/spark-init /etc/init.d/spark
# *********** Install Kafka ***************
ENV KAFKA_LOGS_PATH=/var/log/kafka
RUN wget -qO- http://apache.mirrors.lucidnetworks.net/kafka/1.0.0/kafka_2.11-1.0.0.tgz | sudo tar xvz -C /opt/helk/kafka/ \
&& mkdir -v $KAFKA_LOGS_PATH \
&& mv /opt/helk/kafka/kafka_2.11-1.0.0/config/server.properties /opt/helk/kafka/kafka_2.11-1.0.0/config/backup_server.properties
ADD kafka/*.properties /opt/helk/kafka/kafka_2.11-1.0.0/config/
ADD kafka/kafka-init /etc/init.d/kafka
# *********** Download KSQL (Experiment) ***************
# RUN wget -qO- https://github.com/confluentinc/ksql/archive/v0.4.tar.gz | sudo tar xvz -C /opt/helk/ksql/
# *********** Install Cerebro ***************
ENV CEREBRO_HOME=/opt/helk/cerebro
ENV CEREBRO_LOGS_PATH=/var/log/cerebro
@ -173,16 +138,7 @@ RUN wget -qO- https://github.com/lmenezes/cerebro/releases/download/v0.7.2/cereb
&& mkdir -v $CEREBRO_LOGS_PATH
ADD cerebro/cerebro-init /etc/init.d/cerebro
# Adding SPARK location
ENV SPARK_HOME=/opt/helk/spark/spark-2.2.1-bin-hadoop2.7
ENV PATH=$SPARK_HOME/bin:$PATH
# Adding Jupyter Notebook Integration
ENV PYSPARK_DRIVER_PYTHON=/usr/local/bin/jupyter
ENV PYSPARK_DRIVER_PYTHON_OPTS="notebook --NotebookApp.open_browser=False --NotebookApp.ip='*' --NotebookApp.port=8880 --allow-root"
ENV PYSPARK_PYTHON=/usr/bin/python
# *********** RUN HELK ***************
EXPOSE 80 5044 4040 8880 2181 9092 9093 9094 9000 8082
EXPOSE 80 5044 9000 8082
WORKDIR "/opt/helk/scripts/"
ENTRYPOINT ["./helk_docker_entrypoint.sh"]
ENTRYPOINT ["./elk-entrypoint.sh"]

0
cerebro/cerebro-init → helk-elk/cerebro/cerebro-init
View File

0
elasticsearch/elasticsearch → helk-elk/elasticsearch/elasticsearch
View File

1
elasticsearch/elasticsearch-init → helk-elk/elasticsearch/elasticsearch-init
View File

@ -1,6 +1,7 @@
#!/bin/bash
#
# /etc/init.d/elasticsearch -- startup script for Elasticsearch
# Maintained by Roberto Rodriguez @Cyb3rWard0g
#
### BEGIN INIT INFO
# Provides: elasticsearch

4
elasticsearch/elasticsearch.yml → helk-elk/elasticsearch/elasticsearch.yml
View File

@ -14,7 +14,7 @@
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
cluster.name: helk-elk
#
# ------------------------------------ Node ------------------------------------
#
@ -53,6 +53,7 @@ bootstrap.memory_lock: true
# Set the bind address to a specific IP (IPv4 or IPv6):
#
#network.host: localhost
network.host: ["localhost", "172.18.0.2"]
#
# Set a custom port for HTTP:
#
@ -86,3 +87,4 @@ bootstrap.memory_lock: true
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
discovery.type: single-node

0
enrichments/ACE/logstash/03-ace-rabbitmq-input.conf → helk-elk/enrichments/ACE/logstash/03-ace-rabbitmq-input.conf
View File

0
enrichments/ACE/logstash/55-rabbitmq-elasticsearch-output.conf → helk-elk/enrichments/ACE/logstash/55-rabbitmq-elasticsearch-output.conf
View File

0
enrichments/otx/otx_imphash_.csv → helk-elk/enrichments/otx/otx_imphash_.csv
View File

0
enrichments/otx/otx_ipv4_.csv → helk-elk/enrichments/otx/otx_ipv4_.csv
View File

0
enrichments/otx/otx_md5_.csv → helk-elk/enrichments/otx/otx_md5_.csv
View File

Can't render this file because it is too large.

0
enrichments/otx/otx_sha1_.csv → helk-elk/enrichments/otx/otx_sha1_.csv
View File

0
enrichments/otx/otx_sha256_.csv → helk-elk/enrichments/otx/otx_sha256_.csv
View File

Can't render this file because it is too large.

0
kibana/dashboards/Global_Dashboard.json → helk-elk/kibana/dashboards/Global_Dashboard.json
View File

0
kibana/dashboards/Network_Dashboard.json → helk-elk/kibana/dashboards/Network_Dashboard.json
View File

0
kibana/dashboards/Sysmon_Dashboard.json → helk-elk/kibana/dashboards/Sysmon_Dashboard.json
View File

2
kibana/kibana-init → helk-elk/kibana/kibana-init
View File

@ -1,5 +1,5 @@
#!/bin/sh
# Init script for Kibana
# /etc/init.d/kibana-- startup script for Kibana
# Maintained by Roberto Rodriguez @Cyb3rWard0g
# Reference:
# https://github.com/elastic/logstash/blob/master/distribution/rpm/src/main/packaging/init.d/logstash

2
kibana/kibana.yml → helk-elk/kibana/kibana.yml
View File

@ -99,4 +99,4 @@ server.host: "localhost"
#ops.interval: 5000
# Experimental Visualizations:
# "vega": {"enableExternalUrls": true}
"vega": {"enableExternalUrls": true}

0
logstash/logstash-init → helk-elk/logstash/logstash-init
View File

0
logstash/logstash.yml → helk-elk/logstash/logstash.yml
View File

0
logstash/output_templates/powershell-direct-template.json → helk-elk/logstash/output_templates/powershell-direct-template.json
View File

0
logstash/output_templates/winevent-application-template.json → helk-elk/logstash/output_templates/winevent-application-template.json
View File

0
logstash/output_templates/winevent-security-template.json → helk-elk/logstash/output_templates/winevent-security-template.json
View File

0
logstash/output_templates/winevent-sysmon-template.json → helk-elk/logstash/output_templates/winevent-sysmon-template.json
View File

0
logstash/output_templates/winevent-system-template.json → helk-elk/logstash/output_templates/winevent-system-template.json
View File

0
logstash/output_templates/winevent-wmiactivity-template.json → helk-elk/logstash/output_templates/winevent-wmiactivity-template.json
View File

4
logstash/pipeline/02-beats-input.conf → helk-elk/logstash/pipeline/02-kafka-input.conf
View File

@ -1,4 +1,4 @@
# HELK beats input conf file
# HELK Kafka input conf file
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
@ -6,7 +6,7 @@
input {
kafka
{
bootstrap_servers => "localhost:9092,localhost:9093,localhost:9094"
bootstrap_servers => "172.18.0.3:9092,172.18.0.3:9093,172.18.0.3:9094"
topics => ["winlogbeat"]
codec => "json"
auto_offset_reset => "earliest"

7
logstash/pipeline/03-powershell-direct-input.conf → helk-elk/logstash/pipeline/03-beats-input.conf
View File

@ -1,13 +1,14 @@
# HELK powershell-direct input conf file
# HELK Beats input conf file
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
input {
tcp {
beats {
port => 5044
add_field => { "[@metadata][source]" => "beats"}
codec => "json"
type => "powershell-direct"
ssl => false
}
}

0
logstash/pipeline/09-all-filter.conf → helk-elk/logstash/pipeline/09-all-filter.conf
View File

0
logstash/pipeline/10-winevent-powershell-filter.conf → helk-elk/logstash/pipeline/10-winevent-powershell-filter.conf
View File

28
logstash/pipeline/11-winevent-sysmon-filter.conf → helk-elk/logstash/pipeline/11-winevent-sysmon-filter.conf
View File

@ -5,6 +5,19 @@
filter {
if [log_name] == "Microsoft-Windows-Sysmon/Operational"{
if [event_data][Image] =~ /^(\w*$)|^(\w*\..*$)/ {
mutate {
copy => {"[event_data][Image]" => "[process][name]"}
}
}
else {
grok {
match => {
"[event_data][Image]" => ".*\\%{GREEDYDATA:[process][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
}
mutate {
rename => {
"[user][domain]" => "[subject][user][domain]"
@ -23,7 +36,11 @@ filter {
target => [hash]
}
grok {
match => {"[event_data][User]" => "%{GREEDYDATA:[user][domain]}\\%{GREEDYDATA:[user][name]}"}
match => {
"[event_data][User]" => "%{GREEDYDATA:[user][domain]}\\%{GREEDYDATA:[user][name]}"
"[event_data][ParentImage]" => ".*\\%{GREEDYDATA:[process][parent][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
mutate {
add_field => { "action" => "processcreate" }
@ -147,6 +164,13 @@ filter {
}
}
if [event_id] == 10 {
grok {
match => {
"[event_data][SourceImage]" => ".*\\%{GREEDYDATA:[process][name]}"
"[event_data][TargetImage]" => ".*\\%{GREEDYDATA:[process][target][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
mutate {
add_field => { "action" => "processaccess" }
rename => {
@ -206,7 +230,7 @@ filter {
date {
timezone => "UTC"
match => [ "[event_data][UtcTime]", "YYYY-MM-dd HH:mm:ss.SSS" ]
target => "[@meta][log][timestamp]"
target => "[@meta][sysmon][timestamp]"
remove_field => [ "[event_data][UtcTime]" ]
tag_on_failure => [ "_sysmon_datefailure", "_dateparsefailure" ]
}

27
logstash/pipeline/12-winevent-security-filter.conf → helk-elk/logstash/pipeline/12-winevent-security-filter.conf
View File

@ -5,6 +5,12 @@
filter {
if [log_name] == "Security"{
grok {
match => {
"[event_data][ProcessName]" => ".*\\%{GREEDYDATA:[process][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
if [event_id] == 4611 {
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4611.md
mutate {
@ -73,7 +79,7 @@ filter {
}
}
if [event_id] == 4627 {
# hhttps://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4627.md
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4627.md
mutate {
rename => {
"[event_data][SubjectUserSid]" => "[subject][user][sid]"
@ -273,6 +279,13 @@ filter {
}
if [event_id] == 4688 {
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4688.md
grok {
match => {
"[event_data][NewProcessName]" => ".*\\%{GREEDYDATA:[process][name]}"
"[event_data][ParentProcessName]" => ".*\\%{GREEDYDATA:[process][parent][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
mutate {
rename => {
"[event_data][NewProcessId]" => "[process][id]"
@ -517,6 +530,12 @@ filter {
if [event_id] == 4798 or [event_id] == 4799 {
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4798.md
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-4799.md
grok {
match => {
"[event_data][CallerProcessName]" => ".*\\%{GREEDYDATA:[process][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
mutate {
rename => {
"[event_data][CallerProcessId]" => "[process][id]"
@ -620,6 +639,12 @@ filter {
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-5156.md
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-5157.md
# https://github.com/MicrosoftDocs/windows-itpro-docs/blob/master/windows/security/threat-protection/auditing/event-5158.md
grok {
match => {
"[event_data][Application]" => ".*\\%{GREEDYDATA:[process][name]}"
}
tag_on_failure => [ "_grokparsefailure", "_parsefailure" ]
}
mutate {
rename => {
"[event_data][Application]" => "[process][path]"

0
logstash/pipeline/13-winevent-system-filter.conf → helk-elk/logstash/pipeline/13-winevent-system-filter.conf
View File

0
logstash/pipeline/14-winevent-application-filter.conf → helk-elk/logstash/pipeline/14-winevent-application-filter.conf
View File

0
logstash/pipeline/15-winevent-wmiactivity-filter.conf → helk-elk/logstash/pipeline/15-winevent-wmiactivity-filter.conf
View File

0
logstash/pipeline/50-winevent-sysmon-output.conf → helk-elk/logstash/pipeline/50-winevent-sysmon-output.conf
View File

0
logstash/pipeline/51-winevent-security-output.conf → helk-elk/logstash/pipeline/51-winevent-security-output.conf
View File

0
logstash/pipeline/52-winevent-system-output.conf → helk-elk/logstash/pipeline/52-winevent-system-output.conf
View File

0
logstash/pipeline/53-winevent-application-output.conf → helk-elk/logstash/pipeline/53-winevent-application-output.conf
View File

0
logstash/pipeline/54-winevent-powershell-output.conf → helk-elk/logstash/pipeline/54-winevent-powershell-output.conf
View File

0
logstash/pipeline/55-winevent-wmiactivity-output.conf → helk-elk/logstash/pipeline/55-winevent-wmiactivity-output.conf
View File

13
helk-elk/logstash/pipeline/56-beats-output.conf Normal file
View File

@ -0,0 +1,13 @@
# HELK Beats output conf file
# HELK build version: 0.9 (BETA)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
output {
if [@metadata][source] == "beats"{
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logs-endpoint-beats-%{+YYYY.MM.dd}"
}
}
}

0
nginx/default → helk-elk/nginx/default
View File

0
nginx/htpasswd.users → helk-elk/nginx/htpasswd.users
View File

33
scripts/helk_docker_entrypoint.sh → helk-elk/scripts/elk-entrypoint.sh
View File

@ -1,7 +1,7 @@
#!/bin/bash
#!/bin/sh
# HELK script: helk_docker_entryppoint.sh
# HELK script description: Restart ELK services and runs Spark
# HELK script: elk-entrypoint.sh
# HELK script description: Restarts and runs ELK services
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
@ -13,8 +13,6 @@ _term() {
service logstash stop
service kibana stop
service cerebro stop
service spark stop
service kafka stop
exit 0
}
trap _term SIGTERM
@ -23,12 +21,7 @@ trap _term SIGTERM
rm -f /var/run/elasticsearch/elasticsearch.pid \
/var/run/logstash.pid \
/var/run/kibana.pid \
/var/run/spark.pid \
/var/run/cerebro.pid \
/var/run/kafka_zookeeper.pid \
/var/run/kafka.pid \
/var/run/kafka_1.pid \
/var/run/kafka_2.pid
/var/run/cerebro.pid
# *********** Setting ES Heap Size***************
# https://serverfault.com/questions/881383/automatically-set-java-heap-size-for-elasticsearch-on-linux
@ -55,23 +48,11 @@ service kibana start
service nginx restart
service logstash start
service cerebro start
service spark start
service cron start
# *********** Creating Kibana Dashboards, visualizations and index-patterns ***************
echo "[HELK-DOCKER-INSTALLATION-INFO] Running helk_kibana_setup.sh script..."
./helk_kibana_setup.sh
./elk-kibana-setup.sh
# *********** Start Kafka **************
echo "[HELK-DOCKER-INSTALLATION-INFO] Setting current host IP to brokers server.properties files.."
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9092/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9092/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server.properties
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9093/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9093/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server-1.properties
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9094/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9094/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server-2.properties
echo "[HELK-DOCKER-INSTALLATION-INFO] Starting Kafka.."
service kafka start
sleep 20
echo "[HELK-DOCKER-INSTALLATION-INFO] Creating Kafka Winlogbeat Topic.."
/opt/helk/kafka/kafka_2.11-1.0.0/bin/kafka-topics.sh --create --zookeeper $ADVERTISED_LISTENER:2181 --replication-factor 3 --partitions 1 --topic winlogbeat
echo "[HELK-DOCKER-INSTALLATION-INFO] Pushing Spark Logs to console.."
tail -f /var/log/spark/spark_pyspark.log
echo "[HELK-DOCKER-INSTALLATION-INFO] Pushing logstash Logs to console.."
tail -f /var/log/logstash/*-plain.log

2
scripts/helk_kibana_setup.sh → helk-elk/scripts/elk-kibana-setup.sh
View File

@ -1,6 +1,6 @@
#!/bin/bash
# HELK script: helk_kibana_setup.sh
# HELK script: elk-kibana-setup.sh
# HELK script description: Creates Kibana index patterns, dashboards and visualizations automatically.
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)

0
scripts/helk_otx.py → helk-elk/scripts/helk_otx.py
View File

43
helk-kafka/Dockerfile Normal file
View File

@ -0,0 +1,43 @@
# HELK script: HELK Kafka Dockerfile
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
FROM phusion/baseimage
LABEL maintainer="Roberto Rodriguez @Cyb3rWard0g"
LABEL description="Dockerfile base for the HELK Kafka."
ENV DEBIAN_FRONTEND noninteractive
# *********** Installing Prerequisites ***************
# -qq : No output except for errors
RUN echo "[HELK-DOCKER-INSTALLATION-INFO] Updating Ubuntu base image.." \
&& apt-get update -qq \
&& echo "[HELK-DOCKER-INSTALLATION-INFO] Extracting templates from packages.." \
&& apt-get install -qqy \
openjdk-8-jre-headless \
wget \
sudo \
nano
RUN echo "en_US.UTF-8 UTF-8" > /etc/locale.gen && \
locale-gen
RUN apt-get -qy clean \
autoremove \
&& rm -rf /var/lib/apt/lists/*
# *********** Creating the right directories ***************
RUN bash -c 'mkdir -pv /opt/helk/{scripts,kafka}'
# *********** Install Kafka ***************
ENV KAFKA_LOGS_PATH=/var/log/kafka
RUN wget -qO- http://apache.mirrors.lucidnetworks.net/kafka/1.0.0/kafka_2.11-1.0.0.tgz | sudo tar xvz -C /opt/helk/kafka/ \
&& mkdir -v $KAFKA_LOGS_PATH \
&& mv /opt/helk/kafka/kafka_2.11-1.0.0/config/server.properties /opt/helk/kafka/kafka_2.11-1.0.0/config/backup_server.properties
ADD *.properties /opt/helk/kafka/kafka_2.11-1.0.0/config/
ADD kafka-init /etc/init.d/kafka
ADD scripts/kafka-entrypoint.sh /opt/helk/scripts/
RUN chmod +x /opt/helk/scripts/kafka-entrypoint.sh
EXPOSE 2181 9092 9093 9094
WORKDIR "/opt/helk/scripts/"
ENTRYPOINT ["./kafka-entrypoint.sh"]

4
kafka/kafka-init → helk-kafka/kafka-init
View File

@ -1,5 +1,5 @@
#!/bin/bash
# Init script for logstash
# /etc/init.d/kafka -- startup script for Kafka
# Maintained by Roberto Rodriguez @Cyb3rWard0g
# Reference:
# https://github.com/elastic/logstash/blob/master/distribution/rpm/src/main/packaging/init.d/logstash
@ -34,7 +34,7 @@ fi
KAFKA_HOME=/opt/helk/kafka/kafka_2.11-1.0.0
KAFKA_USER=root
KAFKA_USER=root
KAFKA_GROUP=root
KAFKA_NICE=18
SERVICE_NAME="kafka"
SERVICE_DESCRIPTION="kafka"

35
helk-kafka/scripts/kafka-entrypoint.sh Executable file
View File

@ -0,0 +1,35 @@
#!/bin/sh
# HELK script: kafka-entrypoint.sh
# HELK script description: Restarts and runs Kafka services
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
# Start graceful termination of HELK services that might be running before running the entrypoint script.
_term() {
echo "Terminating HELK-Kafka Service"
service kafka stop
exit 0
}
trap _term SIGTERM
# Removing PID files just in case the graceful termination fails
rm -f /var/run/kafka_zookeeper.pid \
/var/run/kafka.pid \
/var/run/kafka_1.pid \
/var/run/kafka_2.pid
# *********** Start Kafka **************
echo "[HELK-DOCKER-INSTALLATION-INFO] Setting current host IP to brokers server.properties files.."
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9092/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9092/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server.properties
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9093/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9093/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server-1.properties
sed -i "s/advertised\.listeners\=PLAINTEXT:\/\/HELKIP\:9094/advertised\.listeners\=PLAINTEXT\:\/\/${ADVERTISED_LISTENER}\:9094/g" /opt/helk/kafka/kafka_2.11-1.0.0/config/server-2.properties
echo "[HELK-DOCKER-INSTALLATION-INFO] Starting Kafka.."
service kafka start
sleep 30
echo "[HELK-DOCKER-INSTALLATION-INFO] Creating Kafka Winlogbeat Topic.."
/opt/helk/kafka/kafka_2.11-1.0.0/bin/kafka-topics.sh --create --zookeeper $ADVERTISED_LISTENER:2181 --replication-factor 3 --partitions 1 --topic winlogbeat
echo "[HELK-DOCKER-INSTALLATION-INFO] Pushing Spark Logs to console.."
tail -f /var/log/kafka/helk-*.log

0
kafka/server-1.properties → helk-kafka/server-1.properties
View File

0
kafka/server-2.properties → helk-kafka/server-2.properties
View File

0
kafka/server.properties → helk-kafka/server.properties
View File

0
helk.env Normal file
View File

268
helk_install.sh
View File

@ -3,7 +3,7 @@
# HELK script: helk_install.sh
# HELK script description: Start
# HELK build version: 0.9 (Alpha)
# HELK ELK version: 6.2.0
# HELK ELK version: 6.2.2
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
@ -23,180 +23,144 @@ systemKernel="$(uname -s)"
# *********** Getting Jupyter Token ***************
get_token(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Waiting for HELK services and Jupyter Server to start.."
echo "[HELK-INSTALLATION-INFO] Waiting for HELK services and Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$(docker exec -ti helk jupyter notebook list | grep -oP '(?<=token=).*(?= ::)' | awk '{$1=$1};1')" >> $LOGFILE 2>&1
docker_access="HELK DOCKER BASH ACCESS: sudo docker exec -ti helk bash"
jupyter_token="$(docker exec -ti helk-analytics jupyter notebook list | grep -oP '(?<=token=).*(?= ::)' | awk '{$1=$1};1')" >> $LOGFILE 2>&1
}
# *********** Pulling latest HELK image from DockerHub ***************
one(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Pulling the latest HELK image from Dockerhub.."
docker pull cyb3rward0g/helk >> $LOGFILE 2>&1
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk cyb3rward0g/helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
get_token
}
# *********** Building HELK image from local Dockerfile ***************
two(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Building the HELK container from local Dockerfile.."
docker build -t my_helk . >> $LOGFILE 2>&1
ERROR=$?
# *********** Building and Running HELK Images ***************
build_run(){
echo "[HELK-INSTALLATION-INFO] Installing HELK via docker-compose"
echo "ADVERTISED_LISTENER=$host_ip" >> helk.env
docker-compose up -d >> $LOGFILE 2>&1
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from local Dockerfile (Error Code: $ERROR)."
echoerror "Could not build HELK via docker-compose (Error Code: $ERROR)."
exit 1
fi
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk my_helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
get_token
}
# *********** Building the HELK from local bash script ***************
three(){
echo "[HELK-BASH-INSTALLATION-INFO] Installing the HELK from local bash script"
cd scripts/
./helk_debian_tar_install.sh
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from bash script (Error Code: $ERROR)."
exit 1
fi
# *********** Getting Jupyter Token ***************
echo "[HELK-BASH-INSTALLATION-INFO] Waiting for Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$( cat /var/log/spark/spark_pyspark.log | grep -oP '(?<=token=).*(?=)' | sort -u)"
}
# *********** Showing HELK Docker menu options ***************
show_menus() {
show_banner() {
echo " "
echo "**********************************************"
echo "** HELK - M E N U **"
echo "** HELK - THE HUNTING ELK **"
echo "** **"
echo "** Author: Roberto Rodriguez (@Cyb3rWard0g) **"
echo "** HELK build version: 0.9 (Alpha) **"
echo "** HELK ELK version: 6.2.0 **"
echo "** HELK ELK version: 6.2.2 **"
echo "** License: BSD 3-Clause **"
echo "**********************************************"
echo " "
echo "1. Pull the latest HELK image from DockerHub"
echo "2. Build the HELK image from local Dockerfile"
echo "3. Install the HELK from local bash script"
echo "4. Exit"
echo " "
}
read_options(){
local choice
read -p "[HELK-INSTALLATION-INFO] Enter choice [ 1 - 4] " choice
prepare_helk(){
get_host_ip
if [ $choice = "1" ] || [ $choice = "2" ]; then
if [ "$systemKernel" == "Linux" ]; then
# Reference: https://get.docker.com/
echo "[HELK-DOCKER-INSTALLATION-INFO] HELK identified Linux as the system kernel"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking distribution list and version"
# *********** Check distribution list ***************
lsb_dist="$(. /etc/os-release && echo "$ID")"
lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"
if [ "$systemKernel" == "Linux" ]; then
# Reference: https://get.docker.com/
echo "[HELK-INSTALLATION-INFO] HELK identified Linux as the system kernel"
echo "[HELK-INSTALLATION-INFO] Checking distribution list and version"
# *********** Check distribution list ***************
lsb_dist="$(. /etc/os-release && echo "$ID")"
lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"
# *********** Check distribution version ***************
case "$lsb_dist" in
ubuntu)
if [ -x "$(command -v lsb_release)" ]; then
dist_version="$(lsb_release --codename | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
fi
;;
debian|raspbian)
dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
case "$dist_version" in
9)
dist_version="stretch"
;;
8)
dist_version="jessie"
;;
7)
dist_version="wheezy"
;;
esac
;;
centos)
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
rhel|ol|sles)
ee_notice "$lsb_dist"
exit 1
;;
*)
if [ -x "$(command -v lsb_release)"]; then
dist_version="$(lsb_release --release | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
esac
echo "[HELK-DOCKER-INSTALLATION-INFO] You're using $lsb_dist version $dist_version"
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not verify distribution or version of the OS (Error Code: $ERROR)."
fi
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking if curl is installed first"
if [ -x "$(command -v curl)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Ready to install Docker.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing curl before installing docker.."
apt-get install -y curl >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install curl (Error Code: $ERROR)."
exit 1
fi
# *********** Check distribution version ***************
case "$lsb_dist" in
ubuntu)
if [ -x "$(command -v lsb_release)" ]; then
dist_version="$(lsb_release --codename | cut -f2)"
fi
# ****** Installing via convenience script ***********
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing docker via convenience script.."
curl -fsSL get.docker.com -o scripts/get-docker.sh >> $LOGFILE 2>&1
chmod +x scripts/get-docker.sh >> $LOGFILE 2>&1
scripts/get-docker.sh >> $LOGFILE 2>&1
if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
fi
;;
debian|raspbian)
dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
case "$dist_version" in
9)
dist_version="stretch"
;;
8)
dist_version="jessie"
;;
7)
dist_version="wheezy"
;;
esac
;;
centos)
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
rhel|ol|sles)
ee_notice "$lsb_dist"
exit 1
;;
*)
if [ -x "$(command -v lsb_release)"]; then
dist_version="$(lsb_release --release | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
esac
echo "[HELK-INSTALLATION-INFO] You're using $lsb_dist version $dist_version"
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not verify distribution or version of the OS (Error Code: $ERROR)."
fi
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-INSTALLATION-INFO] Docker already installed"
echo "[HELK-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-INSTALLATION-INFO] Docker is not installed"
echo "[HELK-INSTALLATION-INFO] Checking if curl is installed first"
if [ -x "$(command -v curl)" ]; then
echo "[HELK-INSTALLATION-INFO] curl is already installed"
echo "[HELK-INSTALLATION-INFO] Ready to install Docker.."
else
echo "[HELK-INSTALLATION-INFO] curl is not installed"
echo "[HELK-INSTALLATION-INFO] Installing curl before installing docker.."
apt-get install -y curl >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker via convenience script (Error Code: $ERROR)."
echoerror "Could not install curl (Error Code: $ERROR)."
exit 1
fi
fi
else
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Install docker for $systemKernel"
# ****** Installing via convenience script ***********
echo "[HELK-INSTALLATION-INFO] Installing docker via convenience script.."
curl -fsSL get.docker.com -o scripts/get-docker.sh >> $LOGFILE 2>&1
chmod +x scripts/get-docker.sh >> $LOGFILE 2>&1
scripts/get-docker.sh >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker via convenience script (Error Code: $ERROR)."
exit 1
fi
# ****** Installing docker-compose ***********
echo "[HELK-INSTALLATION-INFO] Installing docker-compose .."
curl -L https://github.com/docker/compose/releases/download/1.19.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose >> $LOGFILE 2>&1
chmod +x /usr/local/bin/docker-compose >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker-compose (Error Code: $ERROR)."
exit 1
fi
fi
else
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-INSTALLATION-INFO] Docker already installed"
echo "[HELK-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-INSTALLATION-INFO] Install docker for $systemKernel"
exit 1
fi
fi
echo "[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 262144"
@ -208,13 +172,6 @@ read_options(){
echoerror "Could not set vm.max_map_count to 262144 (Error Code: $ERROR)."
fi
fi
case $choice in
1) one ;;
2) two ;;
3) three ;;
4) exit 0;;
*) echo -e "[HELK-INSTALLATION-INFO] Wrong choice..." && exit 1
esac
}
get_host_ip(){
@ -244,8 +201,10 @@ get_host_ip(){
}
# *********** Running selected option ***************
show_menus
read_options
show_banner
prepare_helk
build_run
get_token
echo " "
echo " "
@ -261,8 +220,7 @@ echo "HELK KIBANA & ELASTICSEARCH USER: helk"
echo "HELK KIBANA & ELASTICSEARCH PASSWORD: hunting"
echo "HELK JUPYTER CURRENT TOKEN: ${jupyter_token}"
echo "HELK SPARK UI: http://${host_ip}:4040"
echo "HELK JUPYTER NOTEBOOK URI: http://${host_ip}:8880"
echo "${docker_access}"
echo "HELK JUPYTER LAB URL: http://${host_ip}:8880/lab"
echo " "
echo "IT IS HUNTING SEASON!!!!!"
echo " "

8
logstash/pipeline/16-powershell-direct-filter.conf
View File

@ -1,8 +0,0 @@
# HELK powershell-direct filter conf file
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
filter {
if [type] == "powershell-direct"{
}
}

17
logstash/pipeline/56-powershell-direct-output.conf
View File

@ -1,17 +0,0 @@
# HELK powershell_direct output conf file
# HELK build version: 0.9 (BETA)
# Author: Lee Christensen (@tifkin_)
# License: BSD 3-Clause
output {
if [type] == "powershell-direct"{
elasticsearch {
hosts => ["127.0.0.1:9200"]
index => "logs-endpoint-powershell-direct-%{+YYYY.MM.dd}"
template => "/opt/helk/output_templates/powershell-direct-template.json"
template_name => "logs-endpoint-powershell-direct"
template_overwrite => true
#document_id => "%{[@metadata][log_hash]}"
}
}
}

BIN
resources/images/HELK_Design.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 615 KiB

After

Width:  |  Height:  |  Size: 615 KiB

BIN
resources/images/JUPYTER-Lab.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 236 KiB

BIN
resources/images/JUPYTER-Token.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 255 KiB

After

Width:  |  Height:  |  Size: 255 KiB

BIN
resources/images/SPARK-Design.png
View File

Binary file not shown.
Side by Side Swipe Overlay

Before

Width:  |  Height:  |  Size: 599 KiB

After

Width:  |  Height:  |  Size: 626 KiB

270
scripts/DEPRECATED_helk_install.sh Executable file
View File

@ -0,0 +1,270 @@
#!/bin/bash
# HELK script: helk_install.sh
# HELK script description: Start
# HELK build version: 0.9 (Alpha)
# HELK ELK version: 6.2.0
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
# *********** Check if user is root ***************
if [[ $EUID -ne 0 ]]; then
echo "[HELK-INSTALLATION-INFO] YOU MUST BE ROOT TO RUN THIS SCRIPT!!!"
exit 1
fi
LOGFILE="/var/log/helk-install.log"
echoerror() {
printf "${RC} * ERROR${EC}: $@\n" 1>&2;
}
# *********** Check System Kernel Name ***************
systemKernel="$(uname -s)"
# *********** Getting Jupyter Token ***************
get_token(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Waiting for HELK services and Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$(docker exec -ti helk jupyter notebook list | grep -oP '(?<=token=).*(?= ::)' | awk '{$1=$1};1')" >> $LOGFILE 2>&1
docker_access="HELK DOCKER BASH ACCESS: sudo docker exec -ti helk bash"
}
# *********** Pulling latest HELK image from DockerHub ***************
one(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Pulling the latest HELK image from Dockerhub.."
docker pull cyb3rward0g/helk >> $LOGFILE 2>&1
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk cyb3rward0g/helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
get_token
}
# *********** Building HELK image from local Dockerfile ***************
two(){
echo "[HELK-DOCKER-INSTALLATION-INFO] Building the HELK container from local Dockerfile.."
docker build -t my_helk . >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from local Dockerfile (Error Code: $ERROR)."
exit 1
fi
echo "[HELK-DOCKER-INSTALLATION-INFO] Running the HELK container in the background.."
docker run -d -p 80:80 -p 5044:5044 -p 8880:8880 -p 4040:4040 -p 2181:2181 -p 9092:9092 -p 9093:9093 -p 9094:9094 -p 9000:9000 -p 8082:8082 -e "bootstrap.memory_lock=true" -e ADVERTISED_LISTENER="${host_ip}" --ulimit memlock=-1:-1 --name helk my_helk >> $LOGFILE 2>&1
# *********** Getting Jupyter Token ***************
get_token
}
# *********** Building the HELK from local bash script ***************
three(){
echo "[HELK-BASH-INSTALLATION-INFO] Installing the HELK from local bash script"
cd scripts/
./helk_debian_tar_install.sh
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not build HELK image from bash script (Error Code: $ERROR)."
exit 1
fi
# *********** Getting Jupyter Token ***************
echo "[HELK-BASH-INSTALLATION-INFO] Waiting for Jupyter Server to start.."
until curl -s localhost:8880 -o /dev/null; do
sleep 1
done
jupyter_token="$( cat /var/log/spark/spark_pyspark.log | grep -oP '(?<=token=).*(?=)' | sort -u)"
}
# *********** Showing HELK Docker menu options ***************
show_menus() {
echo " "
echo "**********************************************"
echo "** HELK - M E N U **"
echo "** **"
echo "** Author: Roberto Rodriguez (@Cyb3rWard0g) **"
echo "** HELK build version: 0.9 (Alpha) **"
echo "** HELK ELK version: 6.2.0 **"
echo "** License: BSD 3-Clause **"
echo "**********************************************"
echo " "
echo "1. Pull the latest HELK image from DockerHub"
echo "2. Build the HELK image from local Dockerfile"
echo "3. Install the HELK from local bash script"
echo "4. Exit"
echo " "
}
read_options(){
local choice
read -p "[HELK-INSTALLATION-INFO] Enter choice [ 1 - 4] " choice
get_host_ip
if [ $choice = "1" ] || [ $choice = "2" ]; then
if [ "$systemKernel" == "Linux" ]; then
# Reference: https://get.docker.com/
echo "[HELK-DOCKER-INSTALLATION-INFO] HELK identified Linux as the system kernel"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking distribution list and version"
# *********** Check distribution list ***************
lsb_dist="$(. /etc/os-release && echo "$ID")"
lsb_dist="$(echo "$lsb_dist" | tr '[:upper:]' '[:lower:]')"
# *********** Check distribution version ***************
case "$lsb_dist" in
ubuntu)
if [ -x "$(command -v lsb_release)" ]; then
dist_version="$(lsb_release --codename | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/lsb-release ]; then
dist_version="$(. /etc/lsb-release && echo "$DISTRIB_CODENAME")"
fi
;;
debian|raspbian)
dist_version="$(sed 's/\/.*//' /etc/debian_version | sed 's/\..*//')"
case "$dist_version" in
9)
dist_version="stretch"
;;
8)
dist_version="jessie"
;;
7)
dist_version="wheezy"
;;
esac
;;
centos)
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
rhel|ol|sles)
ee_notice "$lsb_dist"
exit 1
;;
*)
if [ -x "$(command -v lsb_release)"]; then
dist_version="$(lsb_release --release | cut -f2)"
fi
if [ -z "$dist_version" ] && [ -r /etc/os-release ]; then
dist_version="$(. /etc/os-release && echo "$VERSION_ID")"
fi
;;
esac
echo "[HELK-DOCKER-INSTALLATION-INFO] You're using $lsb_dist version $dist_version"
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not verify distribution or version of the OS (Error Code: $ERROR)."
fi
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Checking if curl is installed first"
if [ -x "$(command -v curl)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Ready to install Docker.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] curl is not installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing curl before installing docker.."
apt-get install -y curl >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install curl (Error Code: $ERROR)."
exit 1
fi
fi
# ****** Installing via convenience script ***********
echo "[HELK-DOCKER-INSTALLATION-INFO] Installing docker via convenience script.."
curl -fsSL get.docker.com -o scripts/get-docker.sh >> $LOGFILE 2>&1
chmod +x scripts/get-docker.sh >> $LOGFILE 2>&1
scripts/get-docker.sh >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not install docker via convenience script (Error Code: $ERROR)."
exit 1
fi
fi
else
# *********** Check if docker is installed ***************
if [ -x "$(command -v docker)" ]; then
echo "[HELK-DOCKER-INSTALLATION-INFO] Docker already installed"
echo "[HELK-DOCKER-INSTALLATION-INFO] Dockerizing HELK.."
else
echo "[HELK-DOCKER-INSTALLATION-INFO] Install docker for $systemKernel"
exit 1
fi
fi
fi
echo "[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 262144"
MAX_MAP_COUNT=262144
if [ -n "$MAX_MAP_COUNT" -a -f /proc/sys/vm/max_map_count ]; then
sysctl -q -w vm.max_map_count=$MAX_MAP_COUNT >> $LOGFILE 2>&1
ERROR=$?
if [ $ERROR -ne 0 ]; then
echoerror "Could not set vm.max_map_count to 262144 (Error Code: $ERROR)."
fi
fi
case $choice in
1) one ;;
2) two ;;
3) three ;;
4) exit 0;;
*) echo -e "[HELK-INSTALLATION-INFO] Wrong choice..." && exit 1
esac
}
get_host_ip(){
# *********** Getting Host IP ***************
# https://github.com/Invoke-IR/ACE/blob/master/ACE-Docker/start.sh
echo "[HELK-INSTALLATION-INFO] Obtaining current host IP.."
case "${systemKernel}" in
Linux*) host_ip=$(ip route get 1 | awk '{print $NF;exit}');;
Darwin*) host_ip=$(ifconfig en0 | grep inet | grep -v inet6 | cut -d ' ' -f2);;
*) host_ip="UNKNOWN:${unameOut}"
esac
# *********** Accepting Defaults or Allowing user to set HELK IP ***************
local ip_choice
local read_input
read -t 30 -p "[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: " -e -i ${host_ip} ip_choice
read_input=$?
ip_choice="${ip_choice:-$host_ip}"
if [ $ip_choice != $host_ip ]; then
host_ip=$ip_choice
fi
if [ $read_input = 142 ]; then
echo -e "\n[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
else
echo "[HELK-INSTALLATION-INFO] HELK IP set to ${host_ip}"
fi
}
# *********** Running selected option ***************
show_menus
read_options
echo " "
echo " "
echo "***********************************************************************************"
echo "** [HELK-INSTALLATION-INFO] YOUR HELK IS READY **"
echo "** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **"
echo "***********************************************************************************"
echo " "
echo "HELK KIBANA URL: http://${host_ip}"
echo "HELK ELASTICSEARCH EXTERNAL URL: http://${host_ip}:8082"
echo "HELK CEREBRO URL: http://${host_ip}:9000"
echo "HELK KIBANA & ELASTICSEARCH USER: helk"
echo "HELK KIBANA & ELASTICSEARCH PASSWORD: hunting"
echo "HELK JUPYTER CURRENT TOKEN: ${jupyter_token}"
echo "HELK SPARK UI: http://${host_ip}:4040"
echo "HELK JUPYTER NOTEBOOK URI: http://${host_ip}:8880"
echo "${docker_access}"
echo " "
echo "IT IS HUNTING SEASON!!!!!"
echo " "
echo " "
echo " "

0
scripts/helk_debian_deb_install.sh → scripts/Deprecated_helk_debian_deb_install.sh
View File

2
scripts/helk_debian_tar_install.sh → scripts/Deprecated_helk_debian_tar_install.sh
View File

@ -105,7 +105,7 @@ ERROR=$?
echo "[HELK-BASH-INSTALLATION-INFO] Creating needed folders for the HELK.."
mkdir -pv /opt/helk/{scripts,training,otx,es-hadoop,spark,output_templates,dashboards,kafka,elasticsearch,logstash,kibana,cerebro,ksql} >> $LOGFILE 2>&1
echo "[HELK-BASH-INSTALLATION-INFO] Copying HELK files over.."
cp -v helk_kibana_setup.sh /opt/helk/scripts/ >> $LOGFILE 2>&1
cp -v helk-elk-kibana-setup.sh /opt/helk/scripts/ >> $LOGFILE 2>&1
cp -v helk_otx.py /opt/helk/scripts/ >> $LOGFILE 2>&1
cp -vr ../training/* /opt/helk/training/ >> $LOGFILE 2>&1
ERROR=$?

0
scripts/helk_geoipdb_update.sh → scripts/Deprecated_helk_geoipdb_update.sh
View File