mirror of https://github.com/infosecn1nja/HELK.git
Updating Installation Docs
parent
3f8a006749
commit
0c45a2d621
|
@ -7,7 +7,7 @@ prev_page:
|
|||
next_page:
|
||||
url: /architecture/elasticsearch.html
|
||||
suffix: .md
|
||||
search: helk info docker installation spark cybrwardg mb kibana pulling ksql jupyter server e elasticsearch t o kb kafka tcp xx creating elastalert minutes gib set master done bash script following elastic logstash zookeeper ago p pluginsservice loaded module version access gb elk hour mib url option log nginx worker install co running password logs broker b des ip ngnix sudo default want usr share true using vm run current basic hunting build file confluentinc cp cli useconcmarksweepgc node ubuntu github ce compose sure bit available includes helks helkinstall license ui local monitor container n name data however work centos supported
|
||||
search: helk name cluster info docker server installation kibana ksql node elasticsearch t o type e z kafka timestamp level component message otrf kb pulling minutes mb p pluginsservice loaded module following elastic tcp elastalert zookeeper set script creating gib bash co elk spark password status image des xx access gb jupyter running sudo url digest sha downloaded newer logstash nginx done ago version install ngnix user log broker confluentinc cp cli winlogbeat logs usr share true com cybrwardg ip using vm default want basic mib ubuntu compose sure option setting current run helkinstall sh build local file containers n ce
|
||||
|
||||
comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***"
|
||||
---
|
||||
|
@ -19,193 +19,242 @@ comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /con
|
|||
|
||||
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
|
||||
<div class="text_cell_render border-box-sizing rendered_html">
|
||||
<h2 id="Requirements-(Please-Read-Carefully)">Requirements (Please Read Carefully)<a class="anchor-link" href="#Requirements-(Please-Read-Carefully)"> </a></h2><h3 id="Operating-System-&-Docker:">Operating System & Docker:<a class="anchor-link" href="#Operating-System-&-Docker:"> </a></h3><ul>
|
||||
<li>Ubuntu 18.04 (preferred). However, Ubuntu 16 will work. CentOS is not fully supported but some have been able to get it to work, documentation is yet to come - so use CentOS at your own expense at the moment. However, open a GitHub issue but we cant promise we can help.</li>
|
||||
<li>HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: ubuntu, debian, raspbian, centos, and fedora.</li>
|
||||
<li>You can see the specific distro versions supported in the script here.</li>
|
||||
<li>If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker.</li>
|
||||
<h1 id="Requirements-(Please-Read-Carefully)">Requirements (Please Read Carefully)<a class="anchor-link" href="#Requirements-(Please-Read-Carefully)"> </a></h1><ul>
|
||||
<li><strong>Operating System:</strong><ul>
|
||||
<li>Ubuntu 18.04 (preferred)</li>
|
||||
<li>Ubuntu 16</li>
|
||||
<li>CentOS 7 with or without SELinux in enforcement mode</li>
|
||||
<li>CentOS 8 with or without SELinux in enforcement mode</li>
|
||||
</ul>
|
||||
<h3 id="Processor/OS-Architecture:">Processor/OS Architecture:<a class="anchor-link" href="#Processor/OS-Architecture:"> </a></h3><ul>
|
||||
</li>
|
||||
<li><strong>Docker:</strong><ul>
|
||||
<li>HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: <strong>ubuntu</strong>, <strong>debian</strong>, <strong>raspbian</strong>, <strong>centos</strong>, and <strong>fedora</strong>.</li>
|
||||
<li>You can see the specific distro versions supported in the script <a href="https://get.docker.com/">here</a>.</li>
|
||||
<li>If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker. </li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><strong>Processor/OS Architecture:</strong><ul>
|
||||
<li>64-bit also known as x64, x86_64, AMD64 or Intel 64.</li>
|
||||
<li>FYI: old processors don't support SSE3 instructions to start ML (Machine Learning) on elasticsearch. Since version 6.1 Elastic has been compiling the ML programs on the assumption that SSE4.2 instructions are available (See: <a href="https://github.com/Cyb3rWard0g/HELK/issues/321">https://github.com/Cyb3rWard0g/HELK/issues/321</a> and <a href="https://discuss.elastic.co/t/failed-to-start-machine-learning-on-elasticsearch-7-0-0/178216/7">https://discuss.elastic.co/t/failed-to-start-machine-learning-on-elasticsearch-7-0-0/178216/7</a>)</li>
|
||||
</ul>
|
||||
<h3 id="Cores:">Cores:<a class="anchor-link" href="#Cores:"> </a></h3><p>Minimum of 4 cores (whether logical or physical)</p>
|
||||
<h3 id="Network-Connection:-NAT-or-Bridge">Network Connection: NAT or Bridge<a class="anchor-link" href="#Network-Connection:-NAT-or-Bridge"> </a></h3><ul>
|
||||
</li>
|
||||
<li><strong>Cores:</strong> Minimum of 4 cores (whether logical or physical)</li>
|
||||
<li><strong>Network Connection:</strong> NAT or Bridge<ul>
|
||||
<li>IP version 4 address. IPv6 has not been tested yet.</li>
|
||||
<li>Internet access</li>
|
||||
<li>If using a proxy, documentation is yet to come - so use a proxy at your own expense. However, open a GitHub issue and we will try to help until it is officially documented/supported.</li>
|
||||
<li>If using a VM then NAT or Bridge will work.</li>
|
||||
<li>Internet access<ul>
|
||||
<li>List of required domains/IPs will be listed in future documentation.</li>
|
||||
</ul>
|
||||
<h3 id="RAM:">RAM:<a class="anchor-link" href="#RAM:"> </a></h3><p>There are four options, and the following are minimum requirements (include more if you are able).</p>
|
||||
<ul>
|
||||
<li>Option 1: 5GB includes KAFKA + KSQL + ELK + NGNIX.</li>
|
||||
<li>Option 2: 5GB includes KAFKA + KSQL + ELK + NGNIX + ELASTALERT</li>
|
||||
<li>Option 3: 7GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER.</li>
|
||||
<li>Option 4: 8GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT.</li>
|
||||
</ul>
|
||||
<h3 id="Disk:">Disk:<a class="anchor-link" href="#Disk:"> </a></h3><p>25GB for testing purposes and 100GB+ for production (minimum)</p>
|
||||
<h3 id="Applications:">Applications:<a class="anchor-link" href="#Applications:"> </a></h3><ul>
|
||||
<li>Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)</li>
|
||||
<li>Winlogbeat running on your endpoints or centralized WEF server (that your endpoints are forwarding to).</li>
|
||||
<li>You can install Winlogbeat by following one of @Cyb3rWard0g posts here.</li>
|
||||
<li>Winlogbeat config recommended by the HELK since it uses the Kafka output plugin and it is already pointing to the right ports with recommended options. You will just have to add your HELK's IP address.</li>
|
||||
</ul>
|
||||
<h2 id="HELK-Download">HELK Download<a class="anchor-link" href="#HELK-Download"> </a></h2><p>Run the following commands to clone the HELK repo via git.</p>
|
||||
<div class="highlight"><pre><span></span>git clone https://github.com/Cyb3rWard0g/HELK.git
|
||||
</pre></div>
|
||||
<p>Change your current directory location to the new HELK directory, and run the helk_install.sh bash script as root.</p>
|
||||
<div class="highlight"><pre><span></span><span class="nb">cd</span> HELK/docker
|
||||
sudo ./helk_install.sh
|
||||
</pre></div>
|
||||
<h2 id="HELK-Install">HELK Install<a class="anchor-link" href="#HELK-Install"> </a></h2><p>In order to make the installation of the HELK easy for everyone, the project comes with an install script named helk_install.sh. This script builds and runs everything you for HELK automatically. During the installation process, the script will allow you to set up the following:</p>
|
||||
<ul>
|
||||
<li>Set the HELK's option. For this document we are going to use option 2 (ELK + KSQL + Elastalert + Spark + Jupyter)</li>
|
||||
<li>Set the Kibana User's password. Default user is helk</li>
|
||||
<li>Set the HELK's IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press [Return] or let the script continue on its own (30 Seconds sleep).</li>
|
||||
<li>Set the HELK's License Subscription. By default the HELK has the basic subscription selected. You can set it to trial if you want. If you want to learn more about subscriptions go here<ul>
|
||||
<li>If the license is set to trial, HELK asks you to set the password for the elastic account.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><strong>RAM:</strong> There are four options, and the following are minimum requirements (include more if you are able).<ul>
|
||||
<li><strong>Option 1: 5GB</strong> includes <code>KAFKA + KSQL + ELK + NGNIX.</code></li>
|
||||
<li><strong>Option 2: 5GB</strong> includes <code>KAFKA + KSQL + ELK + NGNIX + ELASTALERT</code></li>
|
||||
<li><strong>Option 3: 7GB</strong> includes <code>KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER</code>.</li>
|
||||
<li><strong>Option 4: 8GB</strong> includes <code>KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT</code>.</li>
|
||||
</ul>
|
||||
</li>
|
||||
<li><strong>Disk:</strong> 20GB for testing purposes and 100GB+ for production (minimum)</li>
|
||||
<li><strong>Applications:</strong><ul>
|
||||
<li>Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)</li>
|
||||
<li><a href="https://www.elastic.co/downloads/beats/winlogbeat">Winlogbeat</a> running on your endpoints or centralized WEF server (that your endpoints are forwarding to).<ul>
|
||||
<li>You can install Winlogbeat by following one of <a href="https://twitter.com/Cyb3rWard0g">@Cyb3rWard0g</a> posts <a href="https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_87.html">here</a>.</li>
|
||||
<li><a href="https://github.com/Cyb3rWard0g/HELK/blob/master/winlogbeat/winlogbeat.yml">Winlogbeat config</a> recommended by the HELK since it uses the <a href="https://www.elastic.co/guide/en/beats/winlogbeat/current/kafka-output.html">Kafka output plugin</a> and it is already pointing to the right ports with recommended options. You will just have to add your HELK's IP address.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<h1 id="HELK-Download">HELK Download<a class="anchor-link" href="#HELK-Download"> </a></h1><p>Run the following commands to clone the HELK repo via git.</p>
|
||||
<div class="highlight"><pre><span></span>git clone https://github.com/Cyb3rWard0g/HELK.git
|
||||
</pre></div>
|
||||
<h1 id="HELK-Install">HELK Install<a class="anchor-link" href="#HELK-Install"> </a></h1><p>In order to make the installation of the HELK easy for everyone, the project comes with an install script named <strong>helk_install.sh</strong>. This script builds and runs everything for HELK automatically. During the installation process, the script will allow you to set up the following:</p>
|
||||
<ul>
|
||||
<li>Set the components/applications for the HELK'</li>
|
||||
<li>Set the Kibana User's password. Default user is <strong>helk</strong></li>
|
||||
<li>Set the HELK's IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press [Return] or let the script continue on its own (90 Seconds sleep).</li>
|
||||
<li>Set the HELK's License Subscription. By default the HELK has the <strong>basic</strong> subscription selected. You can set it to <strong>trial</strong> if you want and will be valid for 30 days. If you want to learn more about subscriptions go <a href="https://www.elastic.co/subscriptions">here</a><ul>
|
||||
<li>If the license is set to <strong>trial</strong>, HELK asks you to set the password for the <strong>elastic</strong> account.</li>
|
||||
</ul>
|
||||
</li>
|
||||
</ul>
|
||||
<p><strong>To install HELK:</strong><br>
|
||||
Change your current directory location to the new HELK directory, and run the <strong>helk_install.sh</strong> bash script as shown:</p>
|
||||
<div class="highlight"><pre><span></span><span class="nb">cd</span> HELK/docker
|
||||
sudo ./helk_install.sh
|
||||
</pre></div>
|
||||
<p><strong>Here is an example output of installing the HELK using Option 2</strong></p>
|
||||
<div class="highlight"><pre><span></span><span class="nb">cd</span> HELK/docker/
|
||||
sudo ./helk_install.sh
|
||||
</pre></div>
|
||||
|
||||
<pre><code>**********************************************
|
||||
** HELK - THE HUNTING ELK **
|
||||
** **
|
||||
** Author: Roberto Rodriguez (@Cyb3rWard0g) **
|
||||
** HELK build version: v0.1.7-alpha02262019 **
|
||||
** HELK ELK version: 6.6.1 **
|
||||
** HELK build version: v0.1.8-alpha01032020 **
|
||||
** HELK ELK version: 7.5.2 **
|
||||
** License: GPL-3.0 **
|
||||
**********************************************
|
||||
|
||||
[HELK-INSTALLATION-INFO] HELK being hosted on a Linux box
|
||||
[HELK-INSTALLATION-INFO] Available Memory: 12463 MBs
|
||||
[HELK-INSTALLATION-INFO] You're using ubuntu version xenial
|
||||
[HELK-INSTALLATION-INFO] HELK hosted on a Linux box
|
||||
[HELK-INSTALLATION-INFO] Available Memory: 8345 MBs
|
||||
[HELK-INSTALLATION-INFO] You're using ubuntu version bionic
|
||||
|
||||
*****************************************************
|
||||
* HELK - Docker Compose Build Choices *
|
||||
*****************************************************
|
||||
|
||||
1. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
|
||||
2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER
|
||||
1. KAFKA + KSQL + ELK + NGNIX
|
||||
2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
|
||||
3. KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER
|
||||
4. KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT
|
||||
|
||||
Enter build choice [ 1 - 2]: 2
|
||||
Enter build choice [ 1 - 4]: 2
|
||||
[HELK-INSTALLATION-INFO] HELK build set to 2
|
||||
[HELK-INSTALLATION-INFO] Set HELK elastic subscription (basic or trial): basic
|
||||
[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 192.168.64.138
|
||||
[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: hunting
|
||||
[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: hunting
|
||||
[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 10.66.6.35
|
||||
|
||||
[HELK-INSTALLATION-INFO] Please make sure to create a custom Kibana password and store it securely for future use.
|
||||
[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: Mmh3QAvQm3535F4f4VZQD
|
||||
[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: Mmh3QAvQm3535F4f4VZQD
|
||||
[HELK-INSTALLATION-INFO] Docker already installed
|
||||
[HELK-INSTALLATION-INFO] Making sure you assigned enough disk space to the current Docker base directory
|
||||
[HELK-INSTALLATION-INFO] Available Docker Disk: 67 GBs
|
||||
[HELK-INSTALLATION-INFO] Installing docker-compose..
|
||||
[HELK-INSTALLATION-INFO] Available Docker Disk: 107 GBs
|
||||
[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 4120294
|
||||
[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-notebook-analysis-basic.yml file..
|
||||
[HELK-INSTALLATION-INFO] Setting local vm.swappiness variable to 25
|
||||
[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-analysis-alert-basic.yml file..
|
||||
[HELK-INSTALLATION-INFO] Waiting for some services to be up .....
|
||||
....
|
||||
......</code></pre>
|
||||
<h2 id="Monitor-HELK-installation-Logs-(Always)">Monitor HELK installation Logs (Always)<a class="anchor-link" href="#Monitor-HELK-installation-Logs-(Always)"> </a></h2><p>Once the installation kicks in, it will start showing you pre-defined messages about the installation, but no all the details of what is actually happening in the background. It is designed that way to keep your main screen clean and let you know where it is in the installation process.</p>
|
||||
<p>What I recommend to do all the time is to open another shell and monitor the HELK installation logs by using the tail command and pointing it to the /var/log/helk-install.log file that gets created by the helk_install script as soon as it is run. This log file is available on your local host even if you are deploying the HELK via Docker (I want to make sure it is clear that it is a local file).</p>
|
||||
|
||||
|
||||
***********************************************************************************
|
||||
** [HELK-INSTALLATION-INFO] HELK WAS INSTALLED SUCCESSFULLY **
|
||||
** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **
|
||||
***********************************************************************************
|
||||
|
||||
HELK KIBANA URL: https://10.66.6.35
|
||||
HELK KIBANA USER: helk
|
||||
HELK KIBANA PASSWORD: Mmh3QAvQm3535F4f4VZQD
|
||||
HELK ZOOKEEPER: 10.66.6.35:2181
|
||||
HELK KSQL SERVER: 10.66.6.35:8088
|
||||
|
||||
IT IS HUNTING SEASON!!!!!
|
||||
|
||||
You can stop all the HELK docker containers by running the following command:
|
||||
[+] sudo docker-compose -f helk-kibana-analysis-alert-basic.yml stop</code></pre>
|
||||
<h1 id="Monitor-HELK-installation-Logs-(Always)">Monitor HELK installation Logs (Always)<a class="anchor-link" href="#Monitor-HELK-installation-Logs-(Always)"> </a></h1><p>Once the installation kicks in, it will start showing you pre-defined messages about the installation, but no all the details of what is actually happening in the background. It is designed that way to keep your main screen clean and let you know where it is in the installation process.</p>
|
||||
<p>What I recommend to do all the time is to open another shell and monitor the HELK installation logs by using the <strong>tail</strong> command and pointing it to the <strong>/var/log/helk-install.log</strong> file that gets created by the <strong>helk_install</strong> script as soon as it is run. This log file is available on your local host even if you are deploying the HELK via Docker (I want to make sure it is clear that it is a local file).</p>
|
||||
<div class="highlight"><pre><span></span>tail -f /var/log/helk-install.log
|
||||
</pre></div>
|
||||
|
||||
<pre><code>Creating network "docker_helk" with driver "bridge"
|
||||
<pre><code>Adding password for user helk
|
||||
Creating network "docker_helk" with driver "bridge"
|
||||
Creating volume "docker_esdata" with local driver
|
||||
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:6.6.1)...
|
||||
6.6.1: Pulling from elasticsearch/elasticsearch
|
||||
Pulling helk-kibana (docker.elastic.co/kibana/kibana:6.6.1)...
|
||||
6.6.1: Pulling from kibana/kibana
|
||||
Pulling helk-logstash (docker.elastic.co/logstash/logstash:6.6.1)...
|
||||
6.6.1: Pulling from logstash/logstash
|
||||
Pulling helk-jupyter (cyb3rward0g/helk-jupyter:0.1.2)...
|
||||
0.1.2: Pulling from cyb3rward0g/helk-jupyter
|
||||
Pulling helk-nginx (cyb3rward0g/helk-nginx:0.0.7)...
|
||||
0.0.7: Pulling from cyb3rward0g/helk-nginx
|
||||
Pulling helk-spark-master (cyb3rward0g/helk-spark-master:2.4.0-a)...
|
||||
2.4.0-a: Pulling from cyb3rward0g/helk-spark-master
|
||||
Pulling helk-spark-worker (cyb3rward0g/helk-spark-worker:2.4.0-a)...
|
||||
2.4.0-a: Pulling from cyb3rward0g/helk-spark-worker
|
||||
Pulling helk-zookeeper (cyb3rward0g/helk-zookeeper:2.1.0)...
|
||||
2.1.0: Pulling from cyb3rward0g/helk-zookeeper
|
||||
Pulling helk-kafka-broker (cyb3rward0g/helk-kafka-broker:2.1.0)...
|
||||
2.1.0: Pulling from cyb3rward0g/helk-kafka-broker
|
||||
Pulling helk-ksql-server (confluentinc/cp-ksql-server:5.1.2)...
|
||||
5.1.2: Pulling from confluentinc/cp-ksql-server
|
||||
Pulling helk-ksql-cli (confluentinc/cp-ksql-cli:5.1.2)...
|
||||
5.1.2: Pulling from confluentinc/cp-ksql-cli
|
||||
Pulling helk-elastalert (cyb3rward0g/helk-elastalert:0.2.1)...
|
||||
0.2.1: Pulling from cyb3rward0g/helk-elastalert
|
||||
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:7.5.2)...
|
||||
7.5.2: Pulling from elasticsearch/elasticsearch
|
||||
Digest: sha256:771240a8e1c76cc6ac6aa740d2b82de94d4b8b7dbcca5ad0cf49d12b88a3b8e7
|
||||
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.5.2
|
||||
Pulling helk-kibana (docker.elastic.co/kibana/kibana:7.5.2)...
|
||||
7.5.2: Pulling from kibana/kibana
|
||||
Digest: sha256:fb0ac36c40de29b321a30805bcbda4cbe486e1c5979780647458ad77b5ee2f98
|
||||
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.5.2
|
||||
Pulling helk-logstash (otrf/helk-logstash:7.5.2)...
|
||||
7.5.2: Pulling from otrf/helk-logstash
|
||||
Digest: sha256:c54057ff1d02d7ebae23e49835060c0b4012844312c674ce2264d8bbaee64f1a
|
||||
Status: Downloaded newer image for otrf/helk-logstash:7.5.2
|
||||
Pulling helk-nginx (otrf/helk-nginx:0.0.8)...
|
||||
0.0.8: Pulling from otrf/helk-nginx
|
||||
Digest: sha256:83e86d3ee3891b8a06173f4278ddc9f85cbba9b2dfceada48fb311411e236341
|
||||
Status: Downloaded newer image for otrf/helk-nginx:0.0.8
|
||||
Pulling helk-zookeeper (otrf/helk-zookeeper:2.3.0)...
|
||||
2.3.0: Pulling from otrf/helk-zookeeper
|
||||
Digest: sha256:3e7a0f3a73bcffeac4f239083618c362017005463dd747392a9b43db99535a68
|
||||
Status: Downloaded newer image for otrf/helk-zookeeper:2.3.0
|
||||
Pulling helk-kafka-broker (otrf/helk-kafka-broker:2.3.0)...
|
||||
2.3.0: Pulling from otrf/helk-kafka-broker
|
||||
Digest: sha256:03569d98c46028715623778b4adf809bf417a055c3c19d21f426db4e1b2d6f55
|
||||
Status: Downloaded newer image for otrf/helk-kafka-broker:2.3.0
|
||||
Pulling helk-ksql-server (confluentinc/cp-ksql-server:5.1.3)...
|
||||
5.1.3: Pulling from confluentinc/cp-ksql-server
|
||||
Digest: sha256:063add111cc93b1a0118f88b577e31303045d4cc08eb1d21458429f05cba4b02
|
||||
Status: Downloaded newer image for confluentinc/cp-ksql-server:5.1.3
|
||||
Pulling helk-ksql-cli (confluentinc/cp-ksql-cli:5.1.3)...
|
||||
5.1.3: Pulling from confluentinc/cp-ksql-cli
|
||||
Digest: sha256:18c0ccb00fbf87679e16e9e0da600548fcb236a2fd173263b09e89b2d3a42cc3
|
||||
Status: Downloaded newer image for confluentinc/cp-ksql-cli:5.1.3
|
||||
Pulling helk-elastalert (otrf/helk-elastalert:0.2.6)...
|
||||
0.2.6: Pulling from otrf/helk-elastalert
|
||||
Digest: sha256:ae1096829aacbadce42bd4024b36da3a9636f1901ef4e9e62a12b881cfc23cf5
|
||||
Status: Downloaded newer image for otrf/helk-elastalert:0.2.6
|
||||
Creating helk-elasticsearch ... done
|
||||
Creating helk-kibana ... done
|
||||
Creating helk-logstash ... done
|
||||
Creating helk-spark-master ... done
|
||||
Creating helk-elastalert ... done
|
||||
Creating helk-zookeeper ... done
|
||||
Creating helk-jupyter ... done
|
||||
Creating helk-spark-worker ... done
|
||||
Creating helk-kafka-broker ... done
|
||||
Creating helk-nginx ... done
|
||||
Creating helk-zookeeper ... done
|
||||
Creating helk-elastalert ... done
|
||||
Creating helk-kafka-broker ... done
|
||||
Creating helk-ksql-server ... done
|
||||
Creating helk-ksql-cli ... done</code></pre>
|
||||
<p>Once you see that the containers have been created you can check all the containers running by executing the following:</p>
|
||||
<div class="highlight"><pre><span></span>sudo docker ps
|
||||
</pre></div>
|
||||
|
||||
<pre><code>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
968576241e9c confluentinc/cp-ksql-server:5.1.2 "/etc/confluent/dock…" 28 minutes ago Up 26 minutes 0.0.0.0:8088->8088/tcp helk-ksql-server
|
||||
154593559d13 cyb3rward0g/helk-kafka-broker:2.1.0 "./kafka-entrypoint.…" 28 minutes ago Up 26 minutes 0.0.0.0:9092->9092/tcp helk-kafka-broker
|
||||
d883541a64f1 cyb3rward0g/helk-nginx:0.0.7 "/opt/helk/scripts/n…" About an hour ago Up 26 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx
|
||||
527ef236543a cyb3rward0g/helk-spark-worker:2.4.0-a "./spark-worker-entr…" About an hour ago Up 26 minutes helk-spark-worker
|
||||
27cfaf7a8e84 cyb3rward0g/helk-jupyter:0.1.2 "./jupyter-entrypoin…" About an hour ago Up 26 minutes 8000/tcp, 8888/tcp helk-jupyter
|
||||
75002248e916 cyb3rward0g/helk-zookeeper:2.1.0 "./zookeeper-entrypo…" About an hour ago Up 26 minutes 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper
|
||||
ee0120167ffa cyb3rward0g/helk-elastalert:0.2.1 "./elastalert-entryp…" About an hour ago Up 26 minutes helk-elastalert
|
||||
4dc2722cdd53 cyb3rward0g/helk-spark-master:2.4.0-a "./spark-master-entr…" About an hour ago Up 26 minutes 7077/tcp, 0.0.0.0:8080->8080/tcp helk-spark-master
|
||||
9c1eb230b0ff docker.elastic.co/logstash/logstash:6.6.1 "/usr/share/logstash…" About an hour ago Up 26 minutes 0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp helk-logstash
|
||||
f018f16d9792 docker.elastic.co/kibana/kibana:6.6.1 "/usr/share/kibana/s…" About an hour ago Up 26 minutes 5601/tcp helk-kibana
|
||||
6ec5779e9e01 docker.elastic.co/elasticsearch/elasticsearch:6.6.1 "/usr/share/elastics…" About an hour ago Up 26 minutes 9200/tcp, 9300/tcp helk-elasticsearch</code></pre>
|
||||
<pre><code>CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
2caa7d86bc9e confluentinc/cp-ksql-cli:5.1.3 "/bin/sh" 5 minutes ago Up 5 minutes helk-ksql-cli
|
||||
1ee3c0d90b2a confluentinc/cp-ksql-server:5.1.3 "/etc/confluent/dock…" 5 minutes ago Up 5 minutes 0.0.0.0:8088->8088/tcp helk-ksql-server
|
||||
e753a811ffd2 otrf/helk-kafka-broker:2.3.0 "./kafka-entrypoint.…" 5 minutes ago Up 5 minutes 0.0.0.0:9092->9092/tcp helk-kafka-broker
|
||||
f93239de7d95 otrf/helk-zookeeper:2.3.0 "./zookeeper-entrypo…" 5 minutes ago Up 5 minutes 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper
|
||||
229ea8467075 otrf/helk-elastalert:0.2.6 "./elastalert-entryp…" 5 minutes ago Up 5 minutes helk-elastalert
|
||||
f6fd290d2a9d otrf/helk-nginx:0.0.8 "/opt/helk/scripts/n…" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx
|
||||
d4f2b6d7d21e otrf/helk-logstash:7.5.2 "/usr/share/logstash…" 5 minutes ago Up 5 minutes 0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp helk-logstash
|
||||
c5ae143741ea docker.elastic.co/kibana/kibana:7.5.2 "/usr/share/kibana/s…" 5 minutes ago Up 5 minutes 5601/tcp helk-kibana
|
||||
1729e3234b91 docker.elastic.co/elasticsearch/elasticsearch:7.5.2 "/usr/share/elastics…" 5 minutes ago Up 5 minutes 9200/tcp, 9300/tcp helk-elasticsearch</code></pre>
|
||||
<p>If you want to monitor the resources being utilized (Memory, CPU, etc), you can run the following:</p>
|
||||
<div class="highlight"><pre><span></span>sudo docker stats --all
|
||||
</pre></div>
|
||||
|
||||
<pre><code>CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
|
||||
ba46d256ee18 helk-ksql-cli 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0
|
||||
968576241e9c helk-ksql-server 1.43% 242MiB / 12.62GiB 1.87% 667kB / 584kB 96.1MB / 73.7kB 29
|
||||
154593559d13 helk-kafka-broker 2.83% 318.7MiB / 12.62GiB 2.47% 1.47MB / 1.6MB 50.7MB / 2.01MB 67
|
||||
d883541a64f1 helk-nginx 0.10% 3.223MiB / 12.62GiB 0.02% 14.7MB / 14.8MB 9.35MB / 12.3kB 5
|
||||
527ef236543a helk-spark-worker 0.43% 177.7MiB / 12.62GiB 1.38% 19.5kB / 147kB 37.1MB / 32.8kB 28
|
||||
27cfaf7a8e84 helk-jupyter 0.12% 45.42MiB / 12.62GiB 0.35% 1.64kB / 0B 66.3MB / 733kB 9
|
||||
75002248e916 helk-zookeeper 0.26% 62.6MiB / 12.62GiB 0.48% 150kB / 118kB 2.75MB / 172kB 23
|
||||
ee0120167ffa helk-elastalert 2.60% 40.97MiB / 12.62GiB 0.32% 12MB / 17.4MB 38.3MB / 8.19kB 1
|
||||
4dc2722cdd53 helk-spark-master 0.50% 187.2MiB / 12.62GiB 1.45% 148kB / 17.8kB 52.3MB / 32.8kB 28
|
||||
9c1eb230b0ff helk-logstash 15.96% 1.807GiB / 12.62GiB 14.32% 871kB / 110MB 165MB / 2.95MB 62
|
||||
f018f16d9792 helk-kibana 2.73% 179.1MiB / 12.62GiB 1.39% 3.71MB / 17.6MB 250MB / 4.1kB 13
|
||||
6ec5779e9e01 helk-elasticsearch 12.56% 2.46GiB / 12.62GiB 19.50% 130MB / 15.8MB 293MB / 226MB 61</code></pre>
|
||||
<pre><code>user@HELK-vm:~$ sudo docker stats --all
|
||||
|
||||
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
|
||||
2caa7d86bc9e helk-ksql-cli 0.00% 840KiB / 8.703GiB 0.01% 26.3kB / 0B 98.3kB / 0B 1
|
||||
1ee3c0d90b2a helk-ksql-server 0.29% 222.6MiB / 8.703GiB 2.50% 177kB / 125kB 147kB / 197kB 31
|
||||
e753a811ffd2 helk-kafka-broker 1.71% 366.4MiB / 8.703GiB 4.11% 381kB / 383kB 823kB / 2.14MB 74
|
||||
f93239de7d95 helk-zookeeper 0.18% 74.24MiB / 8.703GiB 0.83% 109kB / 67.2kB 111kB / 1.39MB 48
|
||||
229ea8467075 helk-elastalert 10.71% 53.78MiB / 8.703GiB 0.60% 2.34MB / 3.39MB 3.62MB / 1.87MB 12
|
||||
f6fd290d2a9d helk-nginx 0.02% 6.562MiB / 8.703GiB 0.07% 28.7kB / 1.54kB 61.4kB / 12.3kB 7
|
||||
d4f2b6d7d21e helk-logstash 10.46% 1.337GiB / 8.703GiB 15.36% 632kB / 154MB 430MB / 31.5MB 81
|
||||
c5ae143741ea helk-kibana 1.10% 359.7MiB / 8.703GiB 4.04% 345kB / 1.18MB 458MB / 12.3kB 13
|
||||
1729e3234b91 helk-elasticsearch 43.62% 3.524GiB / 8.703GiB 40.49% 159MB / 3.14MB 609MB / 600MB 77</code></pre>
|
||||
<p>You should also monitor the logs of each container while they are being initialized:</p>
|
||||
<p>Just run the following:</p>
|
||||
<div class="highlight"><pre><span></span>sudo docker logs --follow helk-elasticsearch
|
||||
</pre></div>
|
||||
|
||||
<pre><code>[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms1200m -Xmx1200m -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC
|
||||
<pre><code>user@HELK-vm:~$ sudo docker logs --follow --tail 20 helk-elasticsearch
|
||||
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms3200m -Xmx3200m from custom HELK "algorithm"
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting Elastic license to basic
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Running docker-entrypoint script..
|
||||
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
|
||||
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
|
||||
[2019-03-16T17:13:58,710][INFO ][o.e.e.NodeEnvironment ] [helk-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [60.7gb], net total_space [72.7gb], types [ext4]
|
||||
[2019-03-16T17:13:58,722][INFO ][o.e.e.NodeEnvironment ] [helk-1] heap size [1.1gb], compressed ordinary object pointers [true]
|
||||
[2019-03-16T17:13:58,728][INFO ][o.e.n.Node ] [helk-1] node name [helk-1], node ID [En7HptZKTNmv4R6-Qb99UA]
|
||||
[2019-03-16T17:13:58,729][INFO ][o.e.n.Node ] [helk-1] version[6.6.1], pid[12], build[default/tar/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/4.4.0-116-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
|
||||
[2019-03-16T17:13:58,734][INFO ][o.e.n.Node ] [helk-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7720073513605769733, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms1200m, -Xmx1200m, -XX:-UseConcMarkSweepGC, -XX:-UseCMSInitiatingOccupancyOnly, -XX:+UseG1GC, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
|
||||
[2019-03-16T17:14:03,510][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [aggs-matrix-stats]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [analysis-common]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [ingest-common]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-expression]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-mustache]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-painless]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [mapper-extras]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [parent-join]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [percolator]
|
||||
[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [rank-eval]
|
||||
[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [reindex]
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,448Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [102.2gb], net total_space [116.6gb], types [ext4]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,451Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "heap size [3gb], compressed ordinary object pointers [true]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,458Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "node name [helk-1], node ID [Ed3L9UydShyLmPCbP3GLxw], cluster name [helk-cluster]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,459Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "version[7.5.2], pid[16], build[default/docker/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/4.15.0-74-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,459Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,460Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Djava.io.tmpdir=/tmp/elasticsearch-3812421782724323797, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xms3200m, -Xmx3200m, -XX:MaxDirectMemorySize=1677721600, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,523Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [aggs-matrix-stats]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,523Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [analysis-common]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [flattened]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [frozen-indices]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-common]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-geoip]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-user-agent]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-expression]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-mustache]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-painless]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [mapper-extras]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [parent-join]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [percolator]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,527Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [rank-eval]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,527Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [reindex]" }
|
||||
|
||||
..
|
||||
....</code></pre>
|
||||
<p>All you need to do now for the other ones is just replace helk-elasticsearch with the specific containers name:</p>
|
||||
|
@ -213,10 +262,9 @@ OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in ve
|
|||
</pre></div>
|
||||
<p>Remember that you can also access your docker images by running the following commands:</p>
|
||||
<div class="highlight"><pre><span></span>sudo docker <span class="nb">exec</span> -ti helk-elasticsearch bash
|
||||
<span class="o">[</span>root@1729e3234b91 elasticsearch<span class="o">]</span><span class="c1">#</span>
|
||||
</pre></div>
|
||||
|
||||
<pre><code>root@7a9d6443a4bf:/opt/helk/scripts#</code></pre>
|
||||
<h2 id="Final-Details">Final Details<a class="anchor-link" href="#Final-Details"> </a></h2><p>Once your HELK installation ends, you will be presented with information that you will need to access the HELK and all its other components.</p>
|
||||
<h1 id="Final-Details">Final Details<a class="anchor-link" href="#Final-Details"> </a></h1><p>Once your HELK installation ends, you will be presented with information that you will need to access the HELK and all its other components.</p>
|
||||
<p>You will get the following information:</p>
|
||||
|
||||
<pre><code>***********************************************************************************
|
||||
|
@ -224,50 +272,50 @@ OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in ve
|
|||
** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **
|
||||
***********************************************************************************
|
||||
|
||||
HELK KIBANA URL: https://192.168.64.138
|
||||
HELK KIBANA URL: https://192.168.1.35
|
||||
HELK KIBANA USER: helk
|
||||
HELK KIBANA PASSWORD: hunting
|
||||
HELK SPARK MASTER UI: http://192.168.64.138:8080
|
||||
HELK JUPYTER SERVER URL: http://192.168.64.138/jupyter
|
||||
HELK JUPYTER CURRENT TOKEN: e8e83f5c9fe93882a970ce352d566adfb032b0975549449c
|
||||
HELK ZOOKEEPER: 192.168.64.138:2181
|
||||
HELK KSQL SERVER: 192.168.64.138:8088
|
||||
HELK KIBANA PASSWORD: Mmh3QAvQm3535F4f4VZQD
|
||||
HELK ZOOKEEPER: 192.168.1.35:2181
|
||||
HELK KSQL SERVER: 192.168.1.35:8088
|
||||
|
||||
IT IS HUNTING SEASON!!!!!</code></pre>
|
||||
IT IS HUNTING SEASON!!!!!
|
||||
|
||||
You can stop all the HELK docker containers by running the following command:
|
||||
[+] sudo docker-compose -f helk-kibana-analysis-alert-trial.yml stop</code></pre>
|
||||
<table>
|
||||
<thead><tr>
|
||||
<th style="text-align:left">Type</th>
|
||||
<th style="text-align:left">Description</th>
|
||||
<th>Type</th>
|
||||
<th>Description</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td style="text-align:left">HELK KIBANA URL</td>
|
||||
<td style="text-align:left">URL to access the Kibana server. You will need to copy that and paste it in your browser to access Kibana. Make sure you use https since Kibana is running behind NGINX via port 443 with a self-signed certificate</td>
|
||||
<td>HELK KIBANA URL</td>
|
||||
<td>URL to access the Kibana server. You will need to copy that and paste it in your browser to access Kibana. Make sure you use <strong>https</strong> since Kibana is running behind NGINX via port 443 with a self-signed certificate</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">HELK KIBANA USER & PASSWORD</td>
|
||||
<td style="text-align:left">Credentials used to access Kibana</td>
|
||||
<td>HELK KIBANA USER & PASSWORD</td>
|
||||
<td>Credentials used to access Kibana</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">HELK SPARK MASTER UI</td>
|
||||
<td style="text-align:left">URL to access the Spark Master server (Spark Standalone). That server manages the Spark Workers used during execution of code by Jupyter Notebooks. Spark Master acts as a proxy to Spark Workers and applications running</td>
|
||||
<td>HELK SPARK MASTER UI</td>
|
||||
<td>URL to access the Spark Master server (Spark Standalone). That server manages the Spark Workers used during execution of code by Jupyter Notebooks. Spark Master acts as a proxy to Spark Workers and applications running</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">HELK JUPYTER SERVER URL</td>
|
||||
<td style="text-align:left">URL to access the Jupyter notebook server.</td>
|
||||
<td>HELK JUPYTER SERVER URL</td>
|
||||
<td>URL to access the Jupyter notebook server.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">HELK JUPYTER CURRENT TOKEN</td>
|
||||
<td style="text-align:left">Jupyter token to log in instead of providing a password</td>
|
||||
<td>HELK JUPYTER CURRENT TOKEN</td>
|
||||
<td>Jupyter token to log in instead of providing a password</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">ZOOKEEPER</td>
|
||||
<td style="text-align:left">URL for the kafka cluster zookeeper</td>
|
||||
<td>ZOOKEEPER</td>
|
||||
<td>URL for the kafka cluster zookeeper</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td style="text-align:left">KSQL SERVER</td>
|
||||
<td style="text-align:left">URL to access the KSQL server and send SQL queries to the data in the kafka brokers</td>
|
||||
<td>KSQL SERVER</td>
|
||||
<td>URL to access the KSQL server and send SQL queries to the data in the kafka brokers</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
|
|
|
@ -1,162 +1,178 @@
|
|||
# Installation
|
||||
|
||||
## Requirements (Please Read Carefully)
|
||||
|
||||
### Operating System & Docker:
|
||||
|
||||
* Ubuntu 18.04 (preferred). However, Ubuntu 16 will work. CentOS is not fully supported but some have been able to get it to work, documentation is yet to come - so use CentOS at your own expense at the moment. However, open a GitHub issue but we cant promise we can help.
|
||||
* HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: ubuntu, debian, raspbian, centos, and fedora.
|
||||
* You can see the specific distro versions supported in the script here.
|
||||
* If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker.
|
||||
|
||||
### Processor/OS Architecture:
|
||||
|
||||
* 64-bit also known as x64, x86_64, AMD64 or Intel 64.
|
||||
* FYI: old processors don't support SSE3 instructions to start ML (Machine Learning) on elasticsearch. Since version 6.1 Elastic has been compiling the ML programs on the assumption that SSE4.2 instructions are available (See: https://github.com/Cyb3rWard0g/HELK/issues/321 and https://discuss.elastic.co/t/failed-to-start-machine-learning-on-elasticsearch-7-0-0/178216/7)
|
||||
|
||||
### Cores:
|
||||
Minimum of 4 cores (whether logical or physical)
|
||||
|
||||
### Network Connection: NAT or Bridge
|
||||
|
||||
* IP version 4 address. IPv6 has not been tested yet.
|
||||
* Internet access
|
||||
* If using a proxy, documentation is yet to come - so use a proxy at your own expense. However, open a GitHub issue and we will try to help until it is officially documented/supported.
|
||||
* If using a VM then NAT or Bridge will work.
|
||||
* List of required domains/IPs will be listed in future documentation.
|
||||
|
||||
### RAM:
|
||||
There are four options, and the following are minimum requirements (include more if you are able).
|
||||
|
||||
* Option 1: 5GB includes KAFKA + KSQL + ELK + NGNIX.
|
||||
* Option 2: 5GB includes KAFKA + KSQL + ELK + NGNIX + ELASTALERT
|
||||
* Option 3: 7GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER.
|
||||
* Option 4: 8GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT.
|
||||
|
||||
### Disk:
|
||||
25GB for testing purposes and 100GB+ for production (minimum)
|
||||
|
||||
### Applications:
|
||||
|
||||
* Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)
|
||||
* Winlogbeat running on your endpoints or centralized WEF server (that your endpoints are forwarding to).
|
||||
* You can install Winlogbeat by following one of @Cyb3rWard0g posts here.
|
||||
* Winlogbeat config recommended by the HELK since it uses the Kafka output plugin and it is already pointing to the right ports with recommended options. You will just have to add your HELK's IP address.
|
||||
|
||||
## HELK Download
|
||||
|
||||
# Requirements (Please Read Carefully)
|
||||
* **Operating System:**
|
||||
* Ubuntu 18.04 (preferred)
|
||||
* Ubuntu 16
|
||||
* CentOS 7 with or without SELinux in enforcement mode
|
||||
* CentOS 8 with or without SELinux in enforcement mode
|
||||
* **Docker:**
|
||||
* HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: **ubuntu**, **debian**, **raspbian**, **centos**, and **fedora**.
|
||||
* You can see the specific distro versions supported in the script [here](https://get.docker.com/).
|
||||
* If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker.
|
||||
* **Processor/OS Architecture:**
|
||||
* 64-bit also known as x64, x86_64, AMD64 or Intel 64.
|
||||
* FYI: old processors don't support SSE3 instructions to start ML (Machine Learning) on elasticsearch. Since version 6.1 Elastic has been compiling the ML programs on the assumption that SSE4.2 instructions are available (See: https://github.com/Cyb3rWard0g/HELK/issues/321 and https://discuss.elastic.co/t/failed-to-start-machine-learning-on-elasticsearch-7-0-0/178216/7)
|
||||
* **Cores:** Minimum of 4 cores (whether logical or physical)
|
||||
* **Network Connection:** NAT or Bridge
|
||||
* IP version 4 address. IPv6 has not been tested yet.
|
||||
* If using a proxy, documentation is yet to come - so use a proxy at your own expense. However, open a GitHub issue and we will try to help until it is officially documented/supported.
|
||||
* If using a VM then NAT or Bridge will work.
|
||||
* Internet access
|
||||
* List of required domains/IPs will be listed in future documentation.
|
||||
* **RAM:** There are four options, and the following are minimum requirements (include more if you are able).
|
||||
* **Option 1: 5GB** includes `KAFKA + KSQL + ELK + NGNIX.`
|
||||
* **Option 2: 5GB** includes `KAFKA + KSQL + ELK + NGNIX + ELASTALERT`
|
||||
* **Option 3: 7GB** includes `KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER`.
|
||||
* **Option 4: 8GB** includes `KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT`.
|
||||
* **Disk:** 20GB for testing purposes and 100GB+ for production (minimum)
|
||||
* **Applications:**
|
||||
* Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)
|
||||
* [Winlogbeat](https://www.elastic.co/downloads/beats/winlogbeat) running on your endpoints or centralized WEF server (that your endpoints are forwarding to).
|
||||
* You can install Winlogbeat by following one of [@Cyb3rWard0g](https://twitter.com/Cyb3rWard0g) posts [here](https://cyberwardog.blogspot.com/2017/02/setting-up-pentesting-i-mean-threat_87.html).
|
||||
* [Winlogbeat config](https://github.com/Cyb3rWard0g/HELK/blob/master/winlogbeat/winlogbeat.yml) recommended by the HELK since it uses the [Kafka output plugin](https://www.elastic.co/guide/en/beats/winlogbeat/current/kafka-output.html) and it is already pointing to the right ports with recommended options. You will just have to add your HELK's IP address.
|
||||
|
||||
# HELK Download
|
||||
Run the following commands to clone the HELK repo via git.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Cyb3rWard0g/HELK.git
|
||||
```
|
||||
|
||||
Change your current directory location to the new HELK directory, and run the helk_install.sh bash script as root.
|
||||
# HELK Install
|
||||
In order to make the installation of the HELK easy for everyone, the project comes with an install script named **helk_install.sh**. This script builds and runs everything for HELK automatically. During the installation process, the script will allow you to set up the following:
|
||||
* Set the components/applications for the HELK'
|
||||
* Set the Kibana User's password. Default user is **helk**
|
||||
* Set the HELK's IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press \[Return\] or let the script continue on its own (90 Seconds sleep).
|
||||
* Set the HELK's License Subscription. By default the HELK has the **basic** subscription selected. You can set it to **trial** if you want and will be valid for 30 days. If you want to learn more about subscriptions go [here](https://www.elastic.co/subscriptions)
|
||||
* If the license is set to **trial**, HELK asks you to set the password for the **elastic** account.
|
||||
|
||||
**To install HELK:**
|
||||
Change your current directory location to the new HELK directory, and run the **helk_install.sh** bash script as shown:
|
||||
|
||||
```bash
|
||||
cd HELK/docker
|
||||
sudo ./helk_install.sh
|
||||
```
|
||||
|
||||
## HELK Install
|
||||
|
||||
In order to make the installation of the HELK easy for everyone, the project comes with an install script named helk_install.sh. This script builds and runs everything you for HELK automatically. During the installation process, the script will allow you to set up the following:
|
||||
|
||||
* Set the HELK's option. For this document we are going to use option 2 (ELK + KSQL + Elastalert + Spark + Jupyter)
|
||||
* Set the Kibana User's password. Default user is helk
|
||||
* Set the HELK's IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press [Return] or let the script continue on its own (30 Seconds sleep).
|
||||
* Set the HELK's License Subscription. By default the HELK has the basic subscription selected. You can set it to trial if you want. If you want to learn more about subscriptions go here
|
||||
* If the license is set to trial, HELK asks you to set the password for the elastic account.
|
||||
**Here is an example output of installing the HELK using Option 2**
|
||||
|
||||
```
|
||||
**********************************************
|
||||
** HELK - THE HUNTING ELK **
|
||||
** **
|
||||
** Author: Roberto Rodriguez (@Cyb3rWard0g) **
|
||||
** HELK build version: v0.1.7-alpha02262019 **
|
||||
** HELK ELK version: 6.6.1 **
|
||||
** HELK build version: v0.1.8-alpha01032020 **
|
||||
** HELK ELK version: 7.5.2 **
|
||||
** License: GPL-3.0 **
|
||||
**********************************************
|
||||
|
||||
[HELK-INSTALLATION-INFO] HELK being hosted on a Linux box
|
||||
[HELK-INSTALLATION-INFO] Available Memory: 12463 MBs
|
||||
[HELK-INSTALLATION-INFO] You're using ubuntu version xenial
|
||||
[HELK-INSTALLATION-INFO] HELK hosted on a Linux box
|
||||
[HELK-INSTALLATION-INFO] Available Memory: 8345 MBs
|
||||
[HELK-INSTALLATION-INFO] You're using ubuntu version bionic
|
||||
|
||||
*****************************************************
|
||||
* HELK - Docker Compose Build Choices *
|
||||
*****************************************************
|
||||
|
||||
1. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
|
||||
2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER
|
||||
1. KAFKA + KSQL + ELK + NGNIX
|
||||
2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
|
||||
3. KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER
|
||||
4. KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT
|
||||
|
||||
Enter build choice [ 1 - 2]: 2
|
||||
Enter build choice [ 1 - 4]: 2
|
||||
[HELK-INSTALLATION-INFO] HELK build set to 2
|
||||
[HELK-INSTALLATION-INFO] Set HELK elastic subscription (basic or trial): basic
|
||||
[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 192.168.64.138
|
||||
[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: hunting
|
||||
[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: hunting
|
||||
[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 10.66.6.35
|
||||
|
||||
[HELK-INSTALLATION-INFO] Please make sure to create a custom Kibana password and store it securely for future use.
|
||||
[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: Mmh3QAvQm3535F4f4VZQD
|
||||
[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: Mmh3QAvQm3535F4f4VZQD
|
||||
[HELK-INSTALLATION-INFO] Docker already installed
|
||||
[HELK-INSTALLATION-INFO] Making sure you assigned enough disk space to the current Docker base directory
|
||||
[HELK-INSTALLATION-INFO] Available Docker Disk: 67 GBs
|
||||
[HELK-INSTALLATION-INFO] Installing docker-compose..
|
||||
[HELK-INSTALLATION-INFO] Available Docker Disk: 107 GBs
|
||||
[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 4120294
|
||||
[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-notebook-analysis-basic.yml file..
|
||||
[HELK-INSTALLATION-INFO] Setting local vm.swappiness variable to 25
|
||||
[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-analysis-alert-basic.yml file..
|
||||
[HELK-INSTALLATION-INFO] Waiting for some services to be up .....
|
||||
....
|
||||
......
|
||||
|
||||
|
||||
***********************************************************************************
|
||||
** [HELK-INSTALLATION-INFO] HELK WAS INSTALLED SUCCESSFULLY **
|
||||
** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **
|
||||
***********************************************************************************
|
||||
|
||||
HELK KIBANA URL: https://10.66.6.35
|
||||
HELK KIBANA USER: helk
|
||||
HELK KIBANA PASSWORD: Mmh3QAvQm3535F4f4VZQD
|
||||
HELK ZOOKEEPER: 10.66.6.35:2181
|
||||
HELK KSQL SERVER: 10.66.6.35:8088
|
||||
|
||||
IT IS HUNTING SEASON!!!!!
|
||||
|
||||
You can stop all the HELK docker containers by running the following command:
|
||||
[+] sudo docker-compose -f helk-kibana-analysis-alert-basic.yml stop
|
||||
|
||||
```
|
||||
|
||||
## Monitor HELK installation Logs (Always)
|
||||
|
||||
# Monitor HELK installation Logs (Always)
|
||||
Once the installation kicks in, it will start showing you pre-defined messages about the installation, but no all the details of what is actually happening in the background. It is designed that way to keep your main screen clean and let you know where it is in the installation process.
|
||||
|
||||
What I recommend to do all the time is to open another shell and monitor the HELK installation logs by using the tail command and pointing it to the /var/log/helk-install.log file that gets created by the helk_install script as soon as it is run. This log file is available on your local host even if you are deploying the HELK via Docker (I want to make sure it is clear that it is a local file).
|
||||
What I recommend to do all the time is to open another shell and monitor the HELK installation logs by using the **tail** command and pointing it to the **/var/log/helk-install.log** file that gets created by the **helk_install** script as soon as it is run. This log file is available on your local host even if you are deploying the HELK via Docker (I want to make sure it is clear that it is a local file).
|
||||
|
||||
```bash
|
||||
tail -f /var/log/helk-install.log
|
||||
tail -f /var/log/helk-install.log
|
||||
```
|
||||
|
||||
```
|
||||
Adding password for user helk
|
||||
Creating network "docker_helk" with driver "bridge"
|
||||
Creating volume "docker_esdata" with local driver
|
||||
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:6.6.1)...
|
||||
6.6.1: Pulling from elasticsearch/elasticsearch
|
||||
Pulling helk-kibana (docker.elastic.co/kibana/kibana:6.6.1)...
|
||||
6.6.1: Pulling from kibana/kibana
|
||||
Pulling helk-logstash (docker.elastic.co/logstash/logstash:6.6.1)...
|
||||
6.6.1: Pulling from logstash/logstash
|
||||
Pulling helk-jupyter (cyb3rward0g/helk-jupyter:0.1.2)...
|
||||
0.1.2: Pulling from cyb3rward0g/helk-jupyter
|
||||
Pulling helk-nginx (cyb3rward0g/helk-nginx:0.0.7)...
|
||||
0.0.7: Pulling from cyb3rward0g/helk-nginx
|
||||
Pulling helk-spark-master (cyb3rward0g/helk-spark-master:2.4.0-a)...
|
||||
2.4.0-a: Pulling from cyb3rward0g/helk-spark-master
|
||||
Pulling helk-spark-worker (cyb3rward0g/helk-spark-worker:2.4.0-a)...
|
||||
2.4.0-a: Pulling from cyb3rward0g/helk-spark-worker
|
||||
Pulling helk-zookeeper (cyb3rward0g/helk-zookeeper:2.1.0)...
|
||||
2.1.0: Pulling from cyb3rward0g/helk-zookeeper
|
||||
Pulling helk-kafka-broker (cyb3rward0g/helk-kafka-broker:2.1.0)...
|
||||
2.1.0: Pulling from cyb3rward0g/helk-kafka-broker
|
||||
Pulling helk-ksql-server (confluentinc/cp-ksql-server:5.1.2)...
|
||||
5.1.2: Pulling from confluentinc/cp-ksql-server
|
||||
Pulling helk-ksql-cli (confluentinc/cp-ksql-cli:5.1.2)...
|
||||
5.1.2: Pulling from confluentinc/cp-ksql-cli
|
||||
Pulling helk-elastalert (cyb3rward0g/helk-elastalert:0.2.1)...
|
||||
0.2.1: Pulling from cyb3rward0g/helk-elastalert
|
||||
Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:7.5.2)...
|
||||
7.5.2: Pulling from elasticsearch/elasticsearch
|
||||
Digest: sha256:771240a8e1c76cc6ac6aa740d2b82de94d4b8b7dbcca5ad0cf49d12b88a3b8e7
|
||||
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.5.2
|
||||
Pulling helk-kibana (docker.elastic.co/kibana/kibana:7.5.2)...
|
||||
7.5.2: Pulling from kibana/kibana
|
||||
Digest: sha256:fb0ac36c40de29b321a30805bcbda4cbe486e1c5979780647458ad77b5ee2f98
|
||||
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.5.2
|
||||
Pulling helk-logstash (otrf/helk-logstash:7.5.2)...
|
||||
7.5.2: Pulling from otrf/helk-logstash
|
||||
Digest: sha256:c54057ff1d02d7ebae23e49835060c0b4012844312c674ce2264d8bbaee64f1a
|
||||
Status: Downloaded newer image for otrf/helk-logstash:7.5.2
|
||||
Pulling helk-nginx (otrf/helk-nginx:0.0.8)...
|
||||
0.0.8: Pulling from otrf/helk-nginx
|
||||
Digest: sha256:83e86d3ee3891b8a06173f4278ddc9f85cbba9b2dfceada48fb311411e236341
|
||||
Status: Downloaded newer image for otrf/helk-nginx:0.0.8
|
||||
Pulling helk-zookeeper (otrf/helk-zookeeper:2.3.0)...
|
||||
2.3.0: Pulling from otrf/helk-zookeeper
|
||||
Digest: sha256:3e7a0f3a73bcffeac4f239083618c362017005463dd747392a9b43db99535a68
|
||||
Status: Downloaded newer image for otrf/helk-zookeeper:2.3.0
|
||||
Pulling helk-kafka-broker (otrf/helk-kafka-broker:2.3.0)...
|
||||
2.3.0: Pulling from otrf/helk-kafka-broker
|
||||
Digest: sha256:03569d98c46028715623778b4adf809bf417a055c3c19d21f426db4e1b2d6f55
|
||||
Status: Downloaded newer image for otrf/helk-kafka-broker:2.3.0
|
||||
Pulling helk-ksql-server (confluentinc/cp-ksql-server:5.1.3)...
|
||||
5.1.3: Pulling from confluentinc/cp-ksql-server
|
||||
Digest: sha256:063add111cc93b1a0118f88b577e31303045d4cc08eb1d21458429f05cba4b02
|
||||
Status: Downloaded newer image for confluentinc/cp-ksql-server:5.1.3
|
||||
Pulling helk-ksql-cli (confluentinc/cp-ksql-cli:5.1.3)...
|
||||
5.1.3: Pulling from confluentinc/cp-ksql-cli
|
||||
Digest: sha256:18c0ccb00fbf87679e16e9e0da600548fcb236a2fd173263b09e89b2d3a42cc3
|
||||
Status: Downloaded newer image for confluentinc/cp-ksql-cli:5.1.3
|
||||
Pulling helk-elastalert (otrf/helk-elastalert:0.2.6)...
|
||||
0.2.6: Pulling from otrf/helk-elastalert
|
||||
Digest: sha256:ae1096829aacbadce42bd4024b36da3a9636f1901ef4e9e62a12b881cfc23cf5
|
||||
Status: Downloaded newer image for otrf/helk-elastalert:0.2.6
|
||||
Creating helk-elasticsearch ... done
|
||||
Creating helk-kibana ... done
|
||||
Creating helk-logstash ... done
|
||||
Creating helk-spark-master ... done
|
||||
Creating helk-elastalert ... done
|
||||
Creating helk-zookeeper ... done
|
||||
Creating helk-jupyter ... done
|
||||
Creating helk-spark-worker ... done
|
||||
Creating helk-kafka-broker ... done
|
||||
Creating helk-nginx ... done
|
||||
Creating helk-zookeeper ... done
|
||||
Creating helk-elastalert ... done
|
||||
Creating helk-kafka-broker ... done
|
||||
Creating helk-ksql-server ... done
|
||||
Creating helk-ksql-cli ... done
|
||||
Creating helk-ksql-cli ... done
|
||||
```
|
||||
|
||||
Once you see that the containers have been created you can check all the containers running by executing the following:
|
||||
|
||||
```bash
|
||||
|
@ -164,71 +180,67 @@ sudo docker ps
|
|||
```
|
||||
|
||||
```
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
968576241e9c confluentinc/cp-ksql-server:5.1.2 "/etc/confluent/dock…" 28 minutes ago Up 26 minutes 0.0.0.0:8088->8088/tcp helk-ksql-server
|
||||
154593559d13 cyb3rward0g/helk-kafka-broker:2.1.0 "./kafka-entrypoint.…" 28 minutes ago Up 26 minutes 0.0.0.0:9092->9092/tcp helk-kafka-broker
|
||||
d883541a64f1 cyb3rward0g/helk-nginx:0.0.7 "/opt/helk/scripts/n…" About an hour ago Up 26 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx
|
||||
527ef236543a cyb3rward0g/helk-spark-worker:2.4.0-a "./spark-worker-entr…" About an hour ago Up 26 minutes helk-spark-worker
|
||||
27cfaf7a8e84 cyb3rward0g/helk-jupyter:0.1.2 "./jupyter-entrypoin…" About an hour ago Up 26 minutes 8000/tcp, 8888/tcp helk-jupyter
|
||||
75002248e916 cyb3rward0g/helk-zookeeper:2.1.0 "./zookeeper-entrypo…" About an hour ago Up 26 minutes 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper
|
||||
ee0120167ffa cyb3rward0g/helk-elastalert:0.2.1 "./elastalert-entryp…" About an hour ago Up 26 minutes helk-elastalert
|
||||
4dc2722cdd53 cyb3rward0g/helk-spark-master:2.4.0-a "./spark-master-entr…" About an hour ago Up 26 minutes 7077/tcp, 0.0.0.0:8080->8080/tcp helk-spark-master
|
||||
9c1eb230b0ff docker.elastic.co/logstash/logstash:6.6.1 "/usr/share/logstash…" About an hour ago Up 26 minutes 0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp helk-logstash
|
||||
f018f16d9792 docker.elastic.co/kibana/kibana:6.6.1 "/usr/share/kibana/s…" About an hour ago Up 26 minutes 5601/tcp helk-kibana
|
||||
6ec5779e9e01 docker.elastic.co/elasticsearch/elasticsearch:6.6.1 "/usr/share/elastics…" About an hour ago Up 26 minutes 9200/tcp, 9300/tcp helk-elasticsearch
|
||||
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
|
||||
2caa7d86bc9e confluentinc/cp-ksql-cli:5.1.3 "/bin/sh" 5 minutes ago Up 5 minutes helk-ksql-cli
|
||||
1ee3c0d90b2a confluentinc/cp-ksql-server:5.1.3 "/etc/confluent/dock…" 5 minutes ago Up 5 minutes 0.0.0.0:8088->8088/tcp helk-ksql-server
|
||||
e753a811ffd2 otrf/helk-kafka-broker:2.3.0 "./kafka-entrypoint.…" 5 minutes ago Up 5 minutes 0.0.0.0:9092->9092/tcp helk-kafka-broker
|
||||
f93239de7d95 otrf/helk-zookeeper:2.3.0 "./zookeeper-entrypo…" 5 minutes ago Up 5 minutes 2181/tcp, 2888/tcp, 3888/tcp helk-zookeeper
|
||||
229ea8467075 otrf/helk-elastalert:0.2.6 "./elastalert-entryp…" 5 minutes ago Up 5 minutes helk-elastalert
|
||||
f6fd290d2a9d otrf/helk-nginx:0.0.8 "/opt/helk/scripts/n…" 5 minutes ago Up 5 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp helk-nginx
|
||||
d4f2b6d7d21e otrf/helk-logstash:7.5.2 "/usr/share/logstash…" 5 minutes ago Up 5 minutes 0.0.0.0:3515->3515/tcp, 0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp helk-logstash
|
||||
c5ae143741ea docker.elastic.co/kibana/kibana:7.5.2 "/usr/share/kibana/s…" 5 minutes ago Up 5 minutes 5601/tcp helk-kibana
|
||||
1729e3234b91 docker.elastic.co/elasticsearch/elasticsearch:7.5.2 "/usr/share/elastics…" 5 minutes ago Up 5 minutes 9200/tcp, 9300/tcp helk-elasticsearch
|
||||
```
|
||||
|
||||
If you want to monitor the resources being utilized (Memory, CPU, etc), you can run the following:
|
||||
|
||||
```bash
|
||||
sudo docker stats --all
|
||||
```
|
||||
user@HELK-vm:~$ sudo docker stats --all
|
||||
|
||||
```
|
||||
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
|
||||
ba46d256ee18 helk-ksql-cli 0.00% 0B / 0B 0.00% 0B / 0B 0B / 0B 0
|
||||
968576241e9c helk-ksql-server 1.43% 242MiB / 12.62GiB 1.87% 667kB / 584kB 96.1MB / 73.7kB 29
|
||||
154593559d13 helk-kafka-broker 2.83% 318.7MiB / 12.62GiB 2.47% 1.47MB / 1.6MB 50.7MB / 2.01MB 67
|
||||
d883541a64f1 helk-nginx 0.10% 3.223MiB / 12.62GiB 0.02% 14.7MB / 14.8MB 9.35MB / 12.3kB 5
|
||||
527ef236543a helk-spark-worker 0.43% 177.7MiB / 12.62GiB 1.38% 19.5kB / 147kB 37.1MB / 32.8kB 28
|
||||
27cfaf7a8e84 helk-jupyter 0.12% 45.42MiB / 12.62GiB 0.35% 1.64kB / 0B 66.3MB / 733kB 9
|
||||
75002248e916 helk-zookeeper 0.26% 62.6MiB / 12.62GiB 0.48% 150kB / 118kB 2.75MB / 172kB 23
|
||||
ee0120167ffa helk-elastalert 2.60% 40.97MiB / 12.62GiB 0.32% 12MB / 17.4MB 38.3MB / 8.19kB 1
|
||||
4dc2722cdd53 helk-spark-master 0.50% 187.2MiB / 12.62GiB 1.45% 148kB / 17.8kB 52.3MB / 32.8kB 28
|
||||
9c1eb230b0ff helk-logstash 15.96% 1.807GiB / 12.62GiB 14.32% 871kB / 110MB 165MB / 2.95MB 62
|
||||
f018f16d9792 helk-kibana 2.73% 179.1MiB / 12.62GiB 1.39% 3.71MB / 17.6MB 250MB / 4.1kB 13
|
||||
6ec5779e9e01 helk-elasticsearch 12.56% 2.46GiB / 12.62GiB 19.50% 130MB / 15.8MB 293MB / 226MB 61
|
||||
2caa7d86bc9e helk-ksql-cli 0.00% 840KiB / 8.703GiB 0.01% 26.3kB / 0B 98.3kB / 0B 1
|
||||
1ee3c0d90b2a helk-ksql-server 0.29% 222.6MiB / 8.703GiB 2.50% 177kB / 125kB 147kB / 197kB 31
|
||||
e753a811ffd2 helk-kafka-broker 1.71% 366.4MiB / 8.703GiB 4.11% 381kB / 383kB 823kB / 2.14MB 74
|
||||
f93239de7d95 helk-zookeeper 0.18% 74.24MiB / 8.703GiB 0.83% 109kB / 67.2kB 111kB / 1.39MB 48
|
||||
229ea8467075 helk-elastalert 10.71% 53.78MiB / 8.703GiB 0.60% 2.34MB / 3.39MB 3.62MB / 1.87MB 12
|
||||
f6fd290d2a9d helk-nginx 0.02% 6.562MiB / 8.703GiB 0.07% 28.7kB / 1.54kB 61.4kB / 12.3kB 7
|
||||
d4f2b6d7d21e helk-logstash 10.46% 1.337GiB / 8.703GiB 15.36% 632kB / 154MB 430MB / 31.5MB 81
|
||||
c5ae143741ea helk-kibana 1.10% 359.7MiB / 8.703GiB 4.04% 345kB / 1.18MB 458MB / 12.3kB 13
|
||||
1729e3234b91 helk-elasticsearch 43.62% 3.524GiB / 8.703GiB 40.49% 159MB / 3.14MB 609MB / 600MB 77
|
||||
```
|
||||
|
||||
You should also monitor the logs of each container while they are being initialized:
|
||||
|
||||
Just run the following:
|
||||
|
||||
```bash
|
||||
sudo docker logs --follow helk-elasticsearch
|
||||
```
|
||||
```
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms1200m -Xmx1200m -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC
|
||||
user@HELK-vm:~$ sudo docker logs --follow --tail 20 helk-elasticsearch
|
||||
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms3200m -Xmx3200m from custom HELK "algorithm"
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting Elastic license to basic
|
||||
[HELK-ES-DOCKER-INSTALLATION-INFO] Running docker-entrypoint script..
|
||||
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
|
||||
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
|
||||
[2019-03-16T17:13:58,710][INFO ][o.e.e.NodeEnvironment ] [helk-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [60.7gb], net total_space [72.7gb], types [ext4]
|
||||
[2019-03-16T17:13:58,722][INFO ][o.e.e.NodeEnvironment ] [helk-1] heap size [1.1gb], compressed ordinary object pointers [true]
|
||||
[2019-03-16T17:13:58,728][INFO ][o.e.n.Node ] [helk-1] node name [helk-1], node ID [En7HptZKTNmv4R6-Qb99UA]
|
||||
[2019-03-16T17:13:58,729][INFO ][o.e.n.Node ] [helk-1] version[6.6.1], pid[12], build[default/tar/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/4.4.0-116-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
|
||||
[2019-03-16T17:13:58,734][INFO ][o.e.n.Node ] [helk-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7720073513605769733, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms1200m, -Xmx1200m, -XX:-UseConcMarkSweepGC, -XX:-UseCMSInitiatingOccupancyOnly, -XX:+UseG1GC, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
|
||||
[2019-03-16T17:14:03,510][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [aggs-matrix-stats]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [analysis-common]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [ingest-common]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-expression]
|
||||
[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-mustache]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [lang-painless]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [mapper-extras]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [parent-join]
|
||||
[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [percolator]
|
||||
[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [rank-eval]
|
||||
[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService ] [helk-1] loaded module [reindex]
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,448Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/mapper/ubuntu--vg-root)]], net usable_space [102.2gb], net total_space [116.6gb], types [ext4]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,451Z", "level": "INFO", "component": "o.e.e.NodeEnvironment", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "heap size [3gb], compressed ordinary object pointers [true]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,458Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "node name [helk-1], node ID [Ed3L9UydShyLmPCbP3GLxw], cluster name [helk-cluster]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,459Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "version[7.5.2], pid[16], build[default/docker/8bec50e1e0ad29dad5653712cf3bb580cd1afcdf/2020-01-15T12:11:52.313576Z], OS[Linux/4.15.0-74-generic/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/13.0.1/13.0.1+9]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,459Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "JVM home [/usr/share/elasticsearch/jdk]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:19,460Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Djava.io.tmpdir=/tmp/elasticsearch-3812421782724323797, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -Des.cgroups.hierarchy.override=/, -Xms3200m, -Xmx3200m, -XX:MaxDirectMemorySize=1677721600, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker, -Des.bundled_jdk=true]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,523Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [aggs-matrix-stats]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,523Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [analysis-common]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [flattened]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [frozen-indices]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-common]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,524Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-geoip]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [ingest-user-agent]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-expression]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-mustache]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [lang-painless]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [mapper-extras]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [parent-join]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,526Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [percolator]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,527Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [rank-eval]" }
|
||||
{"type": "server", "timestamp": "2020-01-25T04:26:21,527Z", "level": "INFO", "component": "o.e.p.PluginsService", "cluster.name": "helk-cluster", "node.name": "helk-1", "message": "loaded module [reindex]" }
|
||||
|
||||
..
|
||||
....
|
||||
```
|
||||
|
@ -243,15 +255,11 @@ Remember that you can also access your docker images by running the following co
|
|||
|
||||
```bash
|
||||
sudo docker exec -ti helk-elasticsearch bash
|
||||
[root@1729e3234b91 elasticsearch]#
|
||||
```
|
||||
|
||||
```
|
||||
root@7a9d6443a4bf:/opt/helk/scripts#
|
||||
```
|
||||
|
||||
## Final Details
|
||||
|
||||
Once your HELK installation ends, you will be presented with information that you will need to access the HELK and all its other components.
|
||||
# Final Details
|
||||
Once your HELK installation ends, you will be presented with information that you will need to access the HELK and all its other components.
|
||||
|
||||
You will get the following information:
|
||||
|
||||
|
@ -261,24 +269,25 @@ You will get the following information:
|
|||
** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **
|
||||
***********************************************************************************
|
||||
|
||||
HELK KIBANA URL: https://192.168.64.138
|
||||
HELK KIBANA URL: https://192.168.1.35
|
||||
HELK KIBANA USER: helk
|
||||
HELK KIBANA PASSWORD: hunting
|
||||
HELK SPARK MASTER UI: http://192.168.64.138:8080
|
||||
HELK JUPYTER SERVER URL: http://192.168.64.138/jupyter
|
||||
HELK JUPYTER CURRENT TOKEN: e8e83f5c9fe93882a970ce352d566adfb032b0975549449c
|
||||
HELK ZOOKEEPER: 192.168.64.138:2181
|
||||
HELK KSQL SERVER: 192.168.64.138:8088
|
||||
HELK KIBANA PASSWORD: Mmh3QAvQm3535F4f4VZQD
|
||||
HELK ZOOKEEPER: 192.168.1.35:2181
|
||||
HELK KSQL SERVER: 192.168.1.35:8088
|
||||
|
||||
IT IS HUNTING SEASON!!!!!
|
||||
|
||||
You can stop all the HELK docker containers by running the following command:
|
||||
[+] sudo docker-compose -f helk-kibana-analysis-alert-trial.yml stop
|
||||
|
||||
```
|
||||
|
||||
| Type| Description|
|
||||
| :---| :---|
|
||||
| HELK KIBANA URL| URL to access the Kibana server. You will need to copy that and paste it in your browser to access Kibana. Make sure you use https since Kibana is running behind NGINX via port 443 with a self-signed certificate |
|
||||
| HELK KIBANA USER & PASSWORD| Credentials used to access Kibana |
|
||||
| HELK SPARK MASTER UI | URL to access the Spark Master server (Spark Standalone). That server manages the Spark Workers used during execution of code by Jupyter Notebooks. Spark Master acts as a proxy to Spark Workers and applications running |
|
||||
| HELK JUPYTER SERVER URL| URL to access the Jupyter notebook server. |
|
||||
| Type | Description |
|
||||
|--------|---------|
|
||||
| HELK KIBANA URL | URL to access the Kibana server. You will need to copy that and paste it in your browser to access Kibana. Make sure you use **https** since Kibana is running behind NGINX via port 443 with a self-signed certificate|
|
||||
| HELK KIBANA USER & PASSWORD | Credentials used to access Kibana |
|
||||
| HELK SPARK MASTER UI | URL to access the Spark Master server (Spark Standalone). That server manages the Spark Workers used during execution of code by Jupyter Notebooks. Spark Master acts as a proxy to Spark Workers and applications running |
|
||||
| HELK JUPYTER SERVER URL | URL to access the Jupyter notebook server. |
|
||||
| HELK JUPYTER CURRENT TOKEN | Jupyter token to log in instead of providing a password |
|
||||
| ZOOKEEPER| URL for the kafka cluster zookeeper |
|
||||
| KSQL SERVER| URL to access the KSQL server and send SQL queries to the data in the kafka brokers |
|
||||
| ZOOKEEPER | URL for the kafka cluster zookeeper |
|
||||
| KSQL SERVER| URL to access the KSQL server and send SQL queries to the data in the kafka brokers|
|
Loading…
Reference in New Issue