HELK/helk-kafka-base/Dockerfile

33 lines
1.1 KiB
Docker
Raw Normal View History

HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
# HELK script: HELK Kafka Base Dockerfile
# HELK build version: 0.9 (Alpha)
# Author: Roberto Rodriguez (@Cyb3rWard0g)
# License: BSD 3-Clause
FROM cyb3rward0g/helk-base:0.0.1
LABEL maintainer="Roberto Rodriguez @Cyb3rWard0g"
LABEL description="Dockerfile base for the HELK Kafka."
ENV DEBIAN_FRONTEND noninteractive
# *********** Installing Prerequisites ***************
# -qq : No output except for errors
RUN echo "[HELK-DOCKER-INSTALLATION-INFO] Updating Ubuntu base image.." \
&& apt-get update -qq \
&& echo "[HELK-DOCKER-INSTALLATION-INFO] Extracting templates from packages.." \
&& apt-get install -qqy \
openjdk-8-jre-headless
RUN apt-get -qy clean \
autoremove
# *********** Creating the right directories ***************
RUN bash -c 'mkdir -pv /opt/helk/kafka'
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
# *********** Install Kafka ***************
ENV KAFKA_VERSION=1.1.0
ENV KAFKA_LOGS_PATH=/var/log/kafka
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
ENV KAFKA_CONSOLE_LOG=/var/log/kafka/helk-kafka.log
ENV KAFKA_HOME=/opt/helk/kafka/kafka_2.11-${KAFKA_VERSION}
HELK 6.2.3 - 032218 Docker-Compose file + Updated Image versions ++ helk-elk:6.2.3 ++ helk-kafka:1.0.1 ++ helk-analytics:0.0.2 HELK-ANALYTICS + Upgraded spark to version 2.3.0 ++ Check release notes: https://spark.apache.org/releases/spark-release-2-3-0.html + Upgraded Jupyter Lab to 0.31.12 + Downgraded Tornado to version 4.* This is due to an error in dependencies happening in version 5.0 with python 3. + Upgraded ES-Hadoop package to version 6.2.3 ++ Check release notes: https://www.elastic.co/guide/en/elasticsearch/hadoop/6.2/eshadoop-6.2.3.html HELK-ELK + Upgraded elastic components to 6.2.3 ++ Check elasticsearch release notes: https://www.elastic.co/guide/en/elasticsearch/reference/6.2/release-notes-6.2.3.html ++ No changes for Kibana ++ Check Logstash release notes: https://www.elastic.co/guide/en/logstash/6.2/logstash-6-2-3.html + Logstash kafka input now adds metadata from kafka. Topic name, etc. + Fingerprint plugin in logstash config 09-all-filter.con is applied to only events with the message field. + logstash config 11-winevent-sysmon-filter.conf ++ removed field "user". This was causing issues when parsing events with Spark. HELK-KAFKA + Upgraded Kafka to version 2.11-1.0.1 ++ Check kafka release notes: https://www.apache.org/dist/kafka/1.0.1/RELEASE_NOTES.html + Removed sleep time for kafka init file + updated kafka entrypoint updating version values HELK helk_install main script + Fixed docker & docker-compose installation steps. This fixes issue https://github.com/Cyb3rWard0g/HELK/issues/33 HELK Winlogbeat install script + Updated beat version to 6.2.3
2018-03-22 07:32:21 +00:00
RUN wget -qO- http://mirrors.ocf.berkeley.edu/apache/kafka/1.1.0/kafka_2.11-${KAFKA_VERSION}.tgz | sudo tar xvz -C /opt/helk/kafka/ \
HELK 6.2.4-050318 ## Overall + Removed the Init files dependencies on all containers + Added more resources to the resources folder (papers and presentations) + Updated to-do list on main README + Removed Static Network setting. Addressing overlapping network issues (https://github.com/Cyb3rWard0g/HELK/issues/43) + Updated WIki and added new images to it + Started documenting potential error messages or bugs with a few quick fixes ## Helk Install Script + Script now collects information about Available Memory and Disk size for LINUX host ONLY. it only continues if the box hosting the HELK has at least 12GB of RAM and 50GB of Disk Available. (This can be overwritten manually by just editing the helk_install script before installing the HELK) ## ELK Stack + Started using Elastic Docker Images as a base + Updated ELK stack to 6.2.4 version + X-Pack Basic Free License attached to build automatically + Monitoring capabilities are now enabled in the build (Reason why Cerebro went away) ## Spark + Integrated Spark Standalone Cluster Manager + Spark Node running with Jupyter Notebook now points to the Helk-Spark-Master container for any execution of code + Added Spark Master and Worker Docker Images + Build runs now with 2 Workers and 1 Master by default. + Apache Arrow is enabled for Pandas Dataframe optimization + Created Spark-Base Docker Image (Applied to the Jupyter Image) ## Kafka + Kafka Container was split in Kafka Brokers and one Zookeeper + Helk runs with 2 Kafka Brokers and 1 Zookeeper by default ## Jupyter Container + Preparing to add Zeppelin Notebook. the Analytics container is now named Jupyter. It uses the Spark-Base image to build on the top and install the necessary packagess + New packages were added: ++ nxviz ++ hiveplot ++ pyarrow + Apache Arrow is not enabled on the Jupyter node to be able to optimize the use of Pandas DataFrames
2018-05-03 19:54:12 +00:00
&& mkdir -v $KAFKA_LOGS_PATH