mirror of https://github.com/infosecn1nja/HELK.git
80 lines
6.6 KiB
HTML
80 lines
6.6 KiB
HTML
---
|
|
title: |-
|
|
Introduction
|
|
pagenum: 0
|
|
prev_page:
|
|
url:
|
|
next_page:
|
|
url: /installation.html
|
|
suffix: .md
|
|
search: open com source img helk spark data src href svg class left github cybrwardg twitter apache shields io license hunting jupyter notebooks elasticsearch engine thehelk badges analytics capabilities such sql streaming scalable platform features kafka allows general build gnu gpl badge issues follow style v blob master stability div images design elk hunt language structured via research community share basics system designed fast text real hadoop libraries cluster provides java python code ksql processing elastalert sigma neuron adding integration integrate add introduction www org licenses gplv blue q isaissueisaclosedimg closed thehelkimg sociallabel ellerbrock frapsoft os mkenney software guides md alphaimg
|
|
|
|
comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***"
|
|
---
|
|
|
|
<main class="jupyter-page">
|
|
<div id="page-info"><div id="page-title">Introduction</div>
|
|
</div>
|
|
<div class="jb_cell">
|
|
|
|
<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
|
|
<div class="text_cell_render border-box-sizing rendered_html">
|
|
<p><a href="https://www.gnu.org/licenses/gpl-3.0"><img src="https://img.shields.io/badge/License-GPLv3-blue.svg" class="left"></a>
|
|
<a href="https://GitHub.com/Cyb3rWard0g/HELK/issues?q=is%3Aissue+is%3Aclosed"><img src="https://img.shields.io/github/issues-closed/Cyb3rward0g/HELK.svg" class="left"></a>
|
|
<a href="https://twitter.com/THE_HELK"><img src="https://img.shields.io/twitter/follow/THE_HELK.svg?style=social&label=Follow" class="left"></a>
|
|
<a href="https://github.com/ellerbrock/open-source-badges/"><img src="https://badges.frapsoft.com/os/v3/open-source.svg?v=103" class="left"></a>
|
|
<a href="https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#alpha"><img src="https://img.shields.io/badge/stability-alpha-f4d03f.svg" class="left"></a></p>
|
|
<div style="clear:both;"></div><p><img src="images/HELK-Design.png"></p>
|
|
<p>The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.</p>
|
|
<h2 id="Goals">Goals<a class="anchor-link" href="#Goals"> </a></h2><ul>
|
|
<li>Provide an open source hunting platform to the community and share the basics of Threat Hunting.</li>
|
|
<li>Expedite the time it takes to deploy a hunt platform.</li>
|
|
<li>Improve the testing and development of hunting use cases in an easier and more affordable way.</li>
|
|
<li>Enable Data Science capabilities while analyzing data via Apache Spark, GraphFrames & Jupyter Notebooks.</li>
|
|
</ul>
|
|
<h2 id="Main-Features">Main Features<a class="anchor-link" href="#Main-Features"> </a></h2><ul>
|
|
<li><strong>Kafka</strong>: A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable.</li>
|
|
<li><strong>Elasticsearch</strong>: A highly scalable open-source full-text search and analytics engine.</li>
|
|
<li><strong>Logstash</strong>: A data collection engine with real-time pipelining capabilities.</li>
|
|
<li><strong>Kibana</strong>: An open source analytics and visualization platform designed to work with Elasticsearch.</li>
|
|
<li><strong>ES-Hadoop</strong>: An open-source, stand-alone, self-contained, small library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such as Hive, Pig or Cascading or new upcoming libraries like Apache Spark ) to interact with Elasticsearch.</li>
|
|
<li><strong>Spark</strong>: A fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.</li>
|
|
<li><strong>Jupyter Notebooks</strong>: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.</li>
|
|
</ul>
|
|
<h2 id="Optional-Features">Optional Features<a class="anchor-link" href="#Optional-Features"> </a></h2><ul>
|
|
<li><strong>KSQL</strong>: Confluent KSQL is the open source, streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python</li>
|
|
<li><strong>Elastalert</strong>: ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch</li>
|
|
<li><strong>Sigma</strong>: Sigma is a generic and open signature format that allows you to describe relevant log events in a straightforward manner.</li>
|
|
</ul>
|
|
<h2 id="Author">Author<a class="anchor-link" href="#Author"> </a></h2><ul>
|
|
<li>Roberto Rodriguez <a href="https://twitter.com/Cyb3rWard0g">@Cyb3rWard0g</a> <a href="https://twitter.com/THE_HELK">@THE_HELK</a></li>
|
|
</ul>
|
|
<h2 id="Current-Committers">Current Committers<a class="anchor-link" href="#Current-Committers"> </a></h2><ul>
|
|
<li>Nate Guagenti <a href="https://twitter.com/neu5ron">@neu5ron</a></li>
|
|
</ul>
|
|
<h2 id="Contributing">Contributing<a class="anchor-link" href="#Contributing"> </a></h2><p>There are a few things that I would like to accomplish with the HELK as shown in the To-Do list below. I would love to make the HELK a stable build for everyone in the community. If you are interested on making this build a more robust one and adding some cool features to it, PLEASE feel free to submit a pull request. #SharingIsCaring</p>
|
|
<h1 id="TO-Do">TO-Do<a class="anchor-link" href="#TO-Do"> </a></h1><ul>
|
|
<li>[ ] Kubernetes Cluster Migration</li>
|
|
<li>[ ] OSQuery Data Ingestion</li>
|
|
<li>[ ] MITRE ATT&CK mapping to logs or dashboards</li>
|
|
<li>[ ] Cypher for Apache Spark Integration (Adding option for Zeppelin Notebook)</li>
|
|
<li>[ ] Test and integrate neo4j spark connectors with build</li>
|
|
<li>[ ] Add more network data sources (i.e Bro)</li>
|
|
<li>[ ] Research & integrate spark structured direct streaming</li>
|
|
<li>[ ] Packer Images</li>
|
|
<li>[ ] Terraform integration (AWS, Azure, GC)</li>
|
|
<li>[ ] Add more Jupyter Notebooks to teach the basics</li>
|
|
<li>[ ] Auditd beat intergation</li>
|
|
</ul>
|
|
<h2 id="License:-GPL-3.0">License: GPL-3.0<a class="anchor-link" href="#License:-GPL-3.0"> </a></h2><p><a href="https://github.com/Cyb3rWard0g/HELK/blob/master/LICENSE"> HELK's GNU General Public License</a></p>
|
|
|
|
</div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
|
|
|
|
|
|
|
|
</main>
|
|
|