diff --git a/README.md b/README.md index a623569..5e14e80 100644 --- a/README.md +++ b/README.md @@ -1,13 +1,14 @@ -# HELK [Alpha] +# HELK [![License: GPL v3](https://img.shields.io/badge/License-GPLv3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0) [![GitHub issues-closed](https://img.shields.io/github/issues-closed/Cyb3rward0g/HELK.svg)](https://GitHub.com/Cyb3rWard0g/HELK/issues?q=is%3Aissue+is%3Aclosed) [![Twitter](https://img.shields.io/twitter/follow/THE_HELK.svg?style=social&label=Follow)](https://twitter.com/THE_HELK) [![Open Source Love](https://badges.frapsoft.com/os/v1/open-source.png?v=103)](https://github.com/ellerbrock/open-source-badges/) +[![stability-alpha](https://img.shields.io/badge/stability-alpha-f4d03f.svg)](https://github.com/mkenney/software-guides/blob/master/STABILITY-BADGES.md#alpha) The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure. -![alt text](resources/images/HELK_Design.png "HELK Infrastructure") +![alt text](docs/content/images/HELK-Design.png "HELK Infrastructure") # Goals @@ -20,58 +21,11 @@ The Hunting ELK or simply the HELK is one of the first open source hunt platform The project is currently in an alpha stage, which means that the code and the functionality are still changing. We haven't yet tested the system with large data sources and in many scenarios. We invite you to try it and welcome any feedback. -# HELK Features +## Docs: -* **Kafka:** A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable. -* **Elasticsearch:** A highly scalable open-source full-text search and analytics engine. -* **Logstash:** A data collection engine with real-time pipelining capabilities. -* **Kibana:** An open source analytics and visualization platform designed to work with Elasticsearch. -* **ES-Hadoop:** An open-source, stand-alone, self-contained, small library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such as Hive, Pig or Cascading or new upcoming libraries like Apache Spark ) to interact with Elasticsearch. -* **Spark:** A fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. -* **GraphFrames:** A package for Apache Spark which provides DataFrame-based Graphs. -* **Jupyter Notebook:** An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. -* **KSQL:** Confluent KSQL is the open source, streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python -* **Elastalert:** ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch - * **Sigma:** Sigma is a generic and open signature format that allows you to describe relevant log events in a straightforward manner. +* [Introduction](https://thehelk.com/introduction.html) +* [Installation](https://thehelk.com/installation.html) -# Getting Started - -## WIKI - -* [Introduction](https://github.com/Cyb3rWard0g/HELK/wiki) -* [Architecture Overview](https://github.com/Cyb3rWard0g/HELK/wiki/Architecture-Overview) - * [Kafka](https://github.com/Cyb3rWard0g/HELK/wiki/Kafka) - * [Logstash](https://github.com/Cyb3rWard0g/HELK/wiki/Logstash) - * [Elasticsearch](https://github.com/Cyb3rWard0g/HELK/wiki/Elasticsearch) - * [Kibana](https://github.com/Cyb3rWard0g/HELK/wiki/Kibana) - * [Spark](https://github.com/Cyb3rWard0g/HELK/wiki/Spark) -* [Installation](https://github.com/Cyb3rWard0g/HELK/wiki/Installation) - -## (Docker) Accessing the HELK's Images - -By default, the HELK's containers are run in the background (Detached). You can see all your docker containers by running the following command: -``` -sudo docker ps - -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -a97bd895a2b3 cyb3rward0g/helk-spark-worker:2.3.0 "./spark-worker-entr…" About an hour ago Up About an hour 0.0.0.0:8082->8082/tcp helk-spark-worker2 -cbb31f688e0a cyb3rward0g/helk-spark-worker:2.3.0 "./spark-worker-entr…" About an hour ago Up About an hour 0.0.0.0:8081->8081/tcp helk-spark-worker -5d58068aa7e3 cyb3rward0g/helk-kafka-broker:1.1.0 "./kafka-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:9092->9092/tcp helk-kafka-broker -bdb303b09878 cyb3rward0g/helk-kafka-broker:1.1.0 "./kafka-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:9093->9093/tcp helk-kafka-broker2 -7761d1e43d37 cyb3rward0g/helk-nginx:0.0.2 "./nginx-entrypoint.…" About an hour ago Up About an hour 0.0.0.0:80->80/tcp helk-nginx -ede2a2503030 cyb3rward0g/helk-jupyter:0.32.1 "./jupyter-entrypoin…" About an hour ago Up About an hour 0.0.0.0:4040->4040/tcp, 0.0.0.0:8880->8880/tcp helk-jupyter -ede19510e959 cyb3rward0g/helk-logstash:6.2.4 "/usr/local/bin/dock…" About an hour ago Up About an hour 5044/tcp, 9600/tcp helk-logstash -e92823b24b2d cyb3rward0g/helk-spark-master:2.3.0 "./spark-master-entr…" About an hour ago Up About an hour 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp helk-spark-master -6125921b310d cyb3rward0g/helk-kibana:6.2.4 "./kibana-entrypoint…" About an hour ago Up About an hour 5601/tcp helk-kibana -4321d609ae07 cyb3rward0g/helk-zookeeper:3.4.10 "./zookeeper-entrypo…" About an hour ago Up About an hour 2888/tcp, 0.0.0.0:2181->2181/tcp, 3888/tcp helk-zookeeper -9cbca145fb3e cyb3rward0g/helk-elasticsearch:6.2.4 "/usr/local/bin/dock…" About an hour ago Up About an hour 9200/tcp, 9300/tcp helk-elasticsearch -``` - -Then, you will just have to pick which container you want to access and run the following following commands: -``` -sudo docker exec -ti bash -root@ede2a2503030:/opt/helk/scripts# -``` # Resources * [Welcome to HELK! : Enabling Advanced Analytics Capabilities](https://cyberwardog.blogspot.com/2018/04/welcome-to-helk-enabling-advanced_9.html) diff --git a/docs/CNAME b/docs/CNAME new file mode 100644 index 0000000..641d167 --- /dev/null +++ b/docs/CNAME @@ -0,0 +1 @@ +thehelk.com \ No newline at end of file diff --git a/docs/Gemfile b/docs/Gemfile new file mode 100755 index 0000000..5230ce5 --- /dev/null +++ b/docs/Gemfile @@ -0,0 +1,21 @@ +source 'https://rubygems.org' + +group :jekyll_plugins do + gem 'github-pages' + gem 'jekyll-feed', '~> 0.6' + + # Textbook plugins + gem 'jekyll-redirect-from' + gem 'jekyll-scholar' +end + +# Windows does not include zoneinfo files, so bundle the tzinfo-data gem +gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw, :jruby] + +# Performance-booster for watching directories on Windows +gem 'wdm', '~> 0.1.0' if Gem.win_platform? + +# Development tools +gem 'guard', '~> 2.14.2' +gem 'guard-jekyll-plus', '~> 2.0.2' +gem 'guard-livereload', '~> 2.5.2' diff --git a/docs/Gemfile.lock b/docs/Gemfile.lock new file mode 100755 index 0000000..893479d --- /dev/null +++ b/docs/Gemfile.lock @@ -0,0 +1,307 @@ +GEM + remote: https://rubygems.org/ + specs: + activesupport (4.2.11.1) + i18n (~> 0.7) + minitest (~> 5.1) + thread_safe (~> 0.3, >= 0.3.4) + tzinfo (~> 1.1) + addressable (2.7.0) + public_suffix (>= 2.0.2, < 5.0) + bibtex-ruby (4.4.7) + latex-decode (~> 0.0) + citeproc (1.0.9) + namae (~> 1.0) + citeproc-ruby (1.1.10) + citeproc (~> 1.0, >= 1.0.9) + csl (~> 1.5) + coderay (1.1.2) + coffee-script (2.4.1) + coffee-script-source + execjs + coffee-script-source (1.11.1) + colorator (1.1.0) + commonmarker (0.17.13) + ruby-enum (~> 0.5) + concurrent-ruby (1.1.5) + csl (1.5.0) + namae (~> 1.0) + csl-styles (1.0.1.9) + csl (~> 1.0) + dnsruby (1.61.3) + addressable (~> 2.5) + em-websocket (0.5.1) + eventmachine (>= 0.12.9) + http_parser.rb (~> 0.6.0) + ethon (0.12.0) + ffi (>= 1.3.0) + eventmachine (1.2.7) + execjs (2.7.0) + faraday (0.17.0) + multipart-post (>= 1.2, < 3) + ffi (1.11.1) + formatador (0.2.5) + forwardable-extended (2.6.0) + gemoji (3.0.1) + github-pages (202) + activesupport (= 4.2.11.1) + github-pages-health-check (= 1.16.1) + jekyll (= 3.8.5) + jekyll-avatar (= 0.6.0) + jekyll-coffeescript (= 1.1.1) + jekyll-commonmark-ghpages (= 0.1.6) + jekyll-default-layout (= 0.1.4) + jekyll-feed (= 0.11.0) + jekyll-gist (= 1.5.0) + jekyll-github-metadata (= 2.12.1) + jekyll-mentions (= 1.4.1) + jekyll-optional-front-matter (= 0.3.0) + jekyll-paginate (= 1.1.0) + jekyll-readme-index (= 0.2.0) + jekyll-redirect-from (= 0.14.0) + jekyll-relative-links (= 0.6.0) + jekyll-remote-theme (= 0.4.0) + jekyll-sass-converter (= 1.5.2) + jekyll-seo-tag (= 2.5.0) + jekyll-sitemap (= 1.2.0) + jekyll-swiss (= 0.4.0) + jekyll-theme-architect (= 0.1.1) + jekyll-theme-cayman (= 0.1.1) + jekyll-theme-dinky (= 0.1.1) + jekyll-theme-hacker (= 0.1.1) + jekyll-theme-leap-day (= 0.1.1) + jekyll-theme-merlot (= 0.1.1) + jekyll-theme-midnight (= 0.1.1) + jekyll-theme-minimal (= 0.1.1) + jekyll-theme-modernist (= 0.1.1) + jekyll-theme-primer (= 0.5.3) + jekyll-theme-slate (= 0.1.1) + jekyll-theme-tactile (= 0.1.1) + jekyll-theme-time-machine (= 0.1.1) + jekyll-titles-from-headings (= 0.5.1) + jemoji (= 0.10.2) + kramdown (= 1.17.0) + liquid (= 4.0.0) + listen (= 3.1.5) + mercenary (~> 0.3) + minima (= 2.5.0) + nokogiri (>= 1.10.4, < 2.0) + rouge (= 3.11.0) + terminal-table (~> 1.4) + github-pages-health-check (1.16.1) + addressable (~> 2.3) + dnsruby (~> 1.60) + octokit (~> 4.0) + public_suffix (~> 3.0) + typhoeus (~> 1.3) + guard (2.14.2) + formatador (>= 0.2.4) + listen (>= 2.7, < 4.0) + lumberjack (>= 1.0.12, < 2.0) + nenv (~> 0.1) + notiffany (~> 0.0) + pry (>= 0.9.12) + shellany (~> 0.0) + thor (>= 0.18.1) + guard-compat (1.2.1) + guard-jekyll-plus (2.0.2) + guard (~> 2.10, >= 2.10.3) + guard-compat (~> 1.1) + jekyll (>= 1.0.0) + guard-livereload (2.5.2) + em-websocket (~> 0.5) + guard (~> 2.8) + guard-compat (~> 1.0) + multi_json (~> 1.8) + html-pipeline (2.12.0) + activesupport (>= 2) + nokogiri (>= 1.4) + http_parser.rb (0.6.0) + i18n (0.9.5) + concurrent-ruby (~> 1.0) + jekyll (3.8.5) + addressable (~> 2.4) + colorator (~> 1.0) + em-websocket (~> 0.5) + i18n (~> 0.7) + jekyll-sass-converter (~> 1.0) + jekyll-watch (~> 2.0) + kramdown (~> 1.14) + liquid (~> 4.0) + mercenary (~> 0.3.3) + pathutil (~> 0.9) + rouge (>= 1.7, < 4) + safe_yaml (~> 1.0) + jekyll-avatar (0.6.0) + jekyll (~> 3.0) + jekyll-coffeescript (1.1.1) + coffee-script (~> 2.2) + coffee-script-source (~> 1.11.1) + jekyll-commonmark (1.3.1) + commonmarker (~> 0.14) + jekyll (>= 3.7, < 5.0) + jekyll-commonmark-ghpages (0.1.6) + commonmarker (~> 0.17.6) + jekyll-commonmark (~> 1.2) + rouge (>= 2.0, < 4.0) + jekyll-default-layout (0.1.4) + jekyll (~> 3.0) + jekyll-feed (0.11.0) + jekyll (~> 3.3) + jekyll-gist (1.5.0) + octokit (~> 4.2) + jekyll-github-metadata (2.12.1) + jekyll (~> 3.4) + octokit (~> 4.0, != 4.4.0) + jekyll-mentions (1.4.1) + html-pipeline (~> 2.3) + jekyll (~> 3.0) + jekyll-optional-front-matter (0.3.0) + jekyll (~> 3.0) + jekyll-paginate (1.1.0) + jekyll-readme-index (0.2.0) + jekyll (~> 3.0) + jekyll-redirect-from (0.14.0) + jekyll (~> 3.3) + jekyll-relative-links (0.6.0) + jekyll (~> 3.3) + jekyll-remote-theme (0.4.0) + addressable (~> 2.0) + jekyll (~> 3.5) + rubyzip (>= 1.2.1, < 3.0) + jekyll-sass-converter (1.5.2) + sass (~> 3.4) + jekyll-scholar (5.16.0) + bibtex-ruby (~> 4.0, >= 4.0.13) + citeproc-ruby (~> 1.0) + csl-styles (~> 1.0) + jekyll (~> 3.0) + jekyll-seo-tag (2.5.0) + jekyll (~> 3.3) + jekyll-sitemap (1.2.0) + jekyll (~> 3.3) + jekyll-swiss (0.4.0) + jekyll-theme-architect (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-cayman (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-dinky (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-hacker (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-leap-day (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-merlot (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-midnight (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-minimal (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-modernist (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-primer (0.5.3) + jekyll (~> 3.5) + jekyll-github-metadata (~> 2.9) + jekyll-seo-tag (~> 2.0) + jekyll-theme-slate (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-tactile (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-theme-time-machine (0.1.1) + jekyll (~> 3.5) + jekyll-seo-tag (~> 2.0) + jekyll-titles-from-headings (0.5.1) + jekyll (~> 3.3) + jekyll-watch (2.2.1) + listen (~> 3.0) + jemoji (0.10.2) + gemoji (~> 3.0) + html-pipeline (~> 2.2) + jekyll (~> 3.0) + kramdown (1.17.0) + latex-decode (0.3.1) + liquid (4.0.0) + listen (3.1.5) + rb-fsevent (~> 0.9, >= 0.9.4) + rb-inotify (~> 0.9, >= 0.9.7) + ruby_dep (~> 1.2) + lumberjack (1.0.13) + mercenary (0.3.6) + method_source (0.9.2) + mini_portile2 (2.4.0) + minima (2.5.0) + jekyll (~> 3.5) + jekyll-feed (~> 0.9) + jekyll-seo-tag (~> 2.1) + minitest (5.13.0) + multi_json (1.14.1) + multipart-post (2.1.1) + namae (1.0.1) + nenv (0.3.0) + nokogiri (1.10.4) + mini_portile2 (~> 2.4.0) + notiffany (0.1.3) + nenv (~> 0.1) + shellany (~> 0.0) + octokit (4.14.0) + sawyer (~> 0.8.0, >= 0.5.3) + pathutil (0.16.2) + forwardable-extended (~> 2.6) + pry (0.12.2) + coderay (~> 1.1.0) + method_source (~> 0.9.0) + public_suffix (3.1.1) + rb-fsevent (0.10.3) + rb-inotify (0.10.0) + ffi (~> 1.0) + rouge (3.11.0) + ruby-enum (0.7.2) + i18n + ruby_dep (1.5.0) + rubyzip (2.0.0) + safe_yaml (1.0.5) + sass (3.7.4) + sass-listen (~> 4.0.0) + sass-listen (4.0.0) + rb-fsevent (~> 0.9, >= 0.9.4) + rb-inotify (~> 0.9, >= 0.9.7) + sawyer (0.8.2) + addressable (>= 2.3.5) + faraday (> 0.8, < 2.0) + shellany (0.0.1) + terminal-table (1.8.0) + unicode-display_width (~> 1.1, >= 1.1.1) + thor (0.20.3) + thread_safe (0.3.6) + typhoeus (1.3.1) + ethon (>= 0.9.0) + tzinfo (1.2.5) + thread_safe (~> 0.1) + unicode-display_width (1.6.0) + +PLATFORMS + ruby + +DEPENDENCIES + github-pages + guard (~> 2.14.2) + guard-jekyll-plus (~> 2.0.2) + guard-livereload (~> 2.5.2) + jekyll-feed (~> 0.6) + jekyll-redirect-from + jekyll-scholar + tzinfo-data + +BUNDLED WITH + 1.17.2 diff --git a/docs/Guardfile b/docs/Guardfile new file mode 100755 index 0000000..fbf9911 --- /dev/null +++ b/docs/Guardfile @@ -0,0 +1,8 @@ +guard 'jekyll-plus', serve: true do + watch /.*/ + ignore /^_site/ +end + +guard 'livereload' do + watch /.*/ +end diff --git a/docs/Makefile b/docs/Makefile new file mode 100755 index 0000000..cc37ba9 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,34 @@ +.PHONY: help book clean serve + +help: + @echo "Please use 'make ' where is one of:" + @echo " install to install the necessary dependencies for jupyter-book to build" + @echo " book to convert the content/ folder into Jekyll markdown in _build/" + @echo " clean to clean out site build files" + @echo " runall to run all notebooks in-place, capturing outputs with the notebook" + @echo " serve to serve the repository locally with Jekyll" + @echo " build to build the site HTML and store in _site/" + @echo " site to build the site HTML, store in _site/, and serve with Jekyll" + + +install: + jupyter-book install ./ + +book: + jupyter-book build ./ + +runall: + jupyter-book run ./content + +clean: + python scripts/clean.py + +serve: + bundle exec guard + +build: + jupyter-book build ./ --overwrite + +site: build + bundle exec jekyll build + touch _site/.nojekyll diff --git a/docs/_bibliography/references.bib b/docs/_bibliography/references.bib new file mode 100755 index 0000000..cbf9b01 --- /dev/null +++ b/docs/_bibliography/references.bib @@ -0,0 +1,56 @@ +--- +--- + +@inproceedings{holdgraf_evidence_2014, + address = {Brisbane, Australia, Australia}, + title = {Evidence for {Predictive} {Coding} in {Human} {Auditory} {Cortex}}, + booktitle = {International {Conference} on {Cognitive} {Neuroscience}}, + publisher = {Frontiers in Neuroscience}, + author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Knight, Robert T.}, + year = {2014} +} + +@article{holdgraf_rapid_2016, + title = {Rapid tuning shifts in human auditory cortex enhance speech intelligibility}, + volume = {7}, + issn = {2041-1723}, + url = {http://www.nature.com/doifinder/10.1038/ncomms13654}, + doi = {10.1038/ncomms13654}, + number = {May}, + journal = {Nature Communications}, + author = {Holdgraf, Christopher Ramsay and de Heer, Wendy and Pasley, Brian N. and Rieger, Jochem W. and Crone, Nathan and Lin, Jack J. and Knight, Robert T. and Theunissen, Frédéric E.}, + year = {2016}, + pages = {13654}, + file = {Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:C\:\\Users\\chold\\Zotero\\storage\\MDQP3JWE\\Holdgraf et al. - 2016 - Rapid tuning shifts in human auditory cortex enhance speech intelligibility.pdf:application/pdf} +} + +@inproceedings{holdgraf_portable_2017, + title = {Portable learning environments for hands-on computational instruction using container-and cloud-based technology to teach data science}, + volume = {Part F1287}, + isbn = {978-1-4503-5272-7}, + doi = {10.1145/3093338.3093370}, + abstract = {© 2017 ACM. There is an increasing interest in learning outside of the traditional classroom setting. This is especially true for topics covering computational tools and data science, as both are challenging to incorporate in the standard curriculum. These atypical learning environments offer new opportunities for teaching, particularly when it comes to combining conceptual knowledge with hands-on experience/expertise with methods and skills. Advances in cloud computing and containerized environments provide an attractive opportunity to improve the effciency and ease with which students can learn. This manuscript details recent advances towards using commonly-Available cloud computing services and advanced cyberinfrastructure support for improving the learning experience in bootcamp-style events. We cover the benets (and challenges) of using a server hosted remotely instead of relying on student laptops, discuss the technology that was used in order to make this possible, and give suggestions for how others could implement and improve upon this model for pedagogy and reproducibility.}, + booktitle = {{ACM} {International} {Conference} {Proceeding} {Series}}, + author = {Holdgraf, Christopher Ramsay and Culich, A. and Rokem, A. and Deniz, F. and Alegro, M. and Ushizima, D.}, + year = {2017}, + keywords = {Teaching, Bootcamps, Cloud computing, Data science, Docker, Pedagogy} +} + +@article{holdgraf_encoding_2017, + title = {Encoding and decoding models in cognitive electrophysiology}, + volume = {11}, + issn = {16625137}, + doi = {10.3389/fnsys.2017.00061}, + abstract = {© 2017 Holdgraf, Rieger, Micheli, Martin, Knight and Theunissen. Cognitive neuroscience has seen rapid growth in the size and complexity of data recorded from the human brain as well as in the computational tools available to analyze this data. This data explosion has resulted in an increased use of multivariate, model-based methods for asking neuroscience questions, allowing scientists to investigate multiple hypotheses with a single dataset, to use complex, time-varying stimuli, and to study the human brain under more naturalistic conditions. These tools come in the form of “Encoding” models, in which stimulus features are used to model brain activity, and “Decoding” models, in which neural features are used to generated a stimulus output. Here we review the current state of encoding and decoding models in cognitive electrophysiology and provide a practical guide toward conducting experiments and analyses in this emerging field. Our examples focus on using linear models in the study of human language and audition. We show how to calculate auditory receptive fields from natural sounds as well as how to decode neural recordings to predict speech. The paper aims to be a useful tutorial to these approaches, and a practical introduction to using machine learning and applied statistics to build models of neural activity. The data analytic approaches we discuss may also be applied to other sensory modalities, motor systems, and cognitive systems, and we cover some examples in these areas. In addition, a collection of Jupyter notebooks is publicly available as a complement to the material covered in this paper, providing code examples and tutorials for predictive modeling in python. The aimis to provide a practical understanding of predictivemodeling of human brain data and to propose best-practices in conducting these analyses.}, + journal = {Frontiers in Systems Neuroscience}, + author = {Holdgraf, Christopher Ramsay and Rieger, J.W. and Micheli, C. and Martin, S. and Knight, R.T. and Theunissen, F.E.}, + year = {2017}, + keywords = {Decoding models, Encoding models, Electrocorticography (ECoG), Electrophysiology/evoked potentials, Machine learning applied to neuroscience, Natural stimuli, Predictive modeling, Tutorials} +} + +@book{ruby, + title = {The Ruby Programming Language}, + author = {Flanagan, David and Matsumoto, Yukihiro}, + year = {2008}, + publisher = {O'Reilly Media} +} \ No newline at end of file diff --git a/docs/_build/architecture/elasticsearch.html b/docs/_build/architecture/elasticsearch.html new file mode 100644 index 0000000..b0d9cdd --- /dev/null +++ b/docs/_build/architecture/elasticsearch.html @@ -0,0 +1,130 @@ +--- +title: |- + Elasticsearch +pagenum: 2 +prev_page: + url: /installation.html +next_page: + url: /architecture/logstash.html +suffix: .md +search: elasticsearch helk set heap memory docker mbs config available gbs jvm example file not else usr share own using options edit following restart value cluster xmsg xmxg esjavaopts environment yml bash size ram perform functions keep aggregations run amount important variables therefore note max only lines always above add under license basic scripts uses order various list track such data perfect however ways logic below shown gb server getting settings sure min same also restarting temporarily service database coming back online here should wanted services option kibana analysis rebuild container elastic need entrypoint name xpack soft hard compose f build + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Elasticsearch
+
+
+ +
+
+

+

HELK's Elasticsearch Heap Size

Elasticsearch uses heap, which can more specifically be referred to as memory/RAM, in order to perform various functions.
+A list of some of the functions this heap/memory does is as follows (keep in mind this is not an exhaustive list):

+
    +
  • Keep track of indexes
    • +
  • When aggregations are run such as calculating sums, mathematical variations, sub aggregations of aggregations, etc..
    • +
  • When certain searches are
    • +
  • Keep track of offsets of the tokens/terms of indexed values (aka events/logs/data)
    • +
+

As you can see, heap and the amount of it is important in a healthy setup. The HELK installation process uses various functions to try to set the "perfect" amount of heap, however there are thousands of variables in all the different ways people use/install HELK.
+Therefore, we are unable to account for them all and thus our logic will never be perfect and unfortunately may not work best for you. However, we have given you an ability to set your own heap and we have described the logic if you choose to let HELK determine what to set it.

+

Heap can and or is set one of four ways, as detailed below.

+

1) Allow HELK to calculate how much to assign.

This is based on the available memory and variables shown in the code block below.
+It’s very important to note available memory, not the amount of memory the host has.
+An example to show why this is critical to understand.. If you have a 100GB RAM server, but the server is actively using 90GBs of RAM - then you will NOT get the max 31GB heap/memory for elasticsearch. In this example you would actually end up getting roughly 3 GBs for the heap. Because, with only 10 GBs of available/free memory, it could cause drastic issues to lock up all of the remaining memory!

+ +
if available memory >= 1000 MBs and <= 5999 MBs:
+  then set to 2000 MBs
+else if available memory => 6000 MBs and <= 8999 MBs:
+  then set to 3200 MBs
+else if available memory => 9000 MBs and <= 12999 MBs:
+  then set to 5000 MBs
+else if available memory => 13000 MBs and <= 16000 MBs:
+  then set to 7100 MBs
+else:
+  if available memory => 31 GBs:
+    then set to 31 GBs
+  else:
+    set to available memory in GBs
+

2) Set your own heap

In order to define your own heap settings, in the file HELK/docker/helk-elasticsearch/config/jvm.options +edit the following two lines that begin with

+

#-Xms
+#-Xmx

+

Then make sure to restart elasticsearch.
+Always set the min and max JVM heap size to the same value
+Also, you will be restarting elasticsearch. Therefore your cluster will temporarily be down as the elasticsearch service/database is coming back online

+

Here is an example of how to perform the above:

+ +
# Edit the file jvm file
+sudo nano HELK/docker/helk-elasticsearch/config/jvm.options
+# Resulting lines (as mentioned that you should edit from above)
+# should look something like the following if you wanted to set the heap to 16GBs
+-Xms16g
+-Xmx16g
+# Restart elasticsearch
+docker restart helk-elasticsearch
+

3) Add ES_JAVA_OPTS to the docker config file

Which docker config file to use is shown later.
+You will add this value under services.helk-elasticsearch.environment. +Example, if I used the option for ELK + Kafka with no license and no alerting and I wanted to set the heap to 16GBs
+Then I would edit HELK/docker/helk-kibana-analysis-basic.yml and add the following line under the environment seciton:
+- "ES_JAVA_OPTS=-Xms16g -Xmx16g"

+

Then make sure rebuild the elasticsearch docker container.
+Always set the min and max JVM heap size to the same value
+Also, you will be restarting elasticsearch. Therefore your cluster will temporarily be down as the elasticsearch service/database is coming back online
+Note if you are using (elastic) license you will need to set your ELASTIC_PASSWORD and KIBANA_UI_PASSWORD variables (and logstash password if applicable)

+

Here is how to perform the above:

+ +
# Example config (only showing the beginning lines) Note, that these settings may not match your config exactly, but that the important thing is to have the value under the environment section
+version: '3.5'
+
+services:
+  helk-elasticsearch:
+    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.1
+    container_name: helk-elasticsearch
+    secrets:
+      - source: elasticsearch.yml
+        target: /usr/share/elasticsearch/config/elasticsearch.yml
+    volumes:
+      - esdata:/usr/share/elasticsearch/data
+      - ./helk-elasticsearch/scripts:/usr/share/elasticsearch/scripts
+      - ./helk-elasticsearch/config/jvm.options:/usr/share/elasticsearch/config/jvm.options
+    entrypoint: /usr/share/elasticsearch/scripts/elasticsearch-entrypoint.sh
+    environment:
+      - cluster.name=helk-cluster
+      - node.name=helk-1
+      - xpack.license.self_generated.type=basic
+      - xpack.security.enabled=false
+      - "ES_JAVA_OPTS= -Xms16g -Xmx16g"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nproc: 20480
+      nofile:
+        soft: 160000
+        hard: 160000
+    restart: always
+    networks:
+      helk:
+# Rebuild the elasticsearch docker container
+`docker-compose -f HELK/docker/helk-kibana-analysis-basic.yml up --build -d`
+

4) Set at run time using custom bash variable

Example bash variable such as:

+
export ES_JAVA_OPTS="-Xms16g -Xmx16g"
+
+

Then run the following using your own docker config file.

+
docker-compose -f $PlaceDockerConfigFileNameHere up --build -d
+
+

Only use this option if you explicitly need to. Please know what your getting into to ;)

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/architecture/kibana.html b/docs/_build/architecture/kibana.html new file mode 100644 index 0000000..dbad367 --- /dev/null +++ b/docs/_build/architecture/kibana.html @@ -0,0 +1,56 @@ +--- +title: |- + Kibana +pagenum: 4 +prev_page: + url: /architecture/logstash.html +next_page: + url: /how-to/docker/docker.html +suffix: .md +search: logs kibana img src images png endpoint winevent overview helk monitoring sysmon elasticsearch logstash docker security right additionally currently dashboards globaldashboard networkdashboard sysmondashboard tail usr share config kibanalogs log design visualize discover sure being sent least windows events helks ip preferred browser dont away update picker top include farther back window just started sending wait minute check again creates automatically index patterns sets default application system powershell wmiactivity discovery comes views x pack basic free license initial nodes troubleshooting apart running ps follow located example exec f times not working because still starting ran into error + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Kibana
+
+
+ +
+
+

+

Visualize your logs

Discover

Make sure you have logs being sent to your HELK first (At least Windows security and Sysmon events). Then, go to https://<HELK's IP> in your preferred browser. If you don’t see logs right away then update your time picker (in the top right) to include a farther back window. Additionally, if you just started sending logs then wait a minute and check again.

+

Currently, HELK creates automatically 7 index patterns for you and sets logs-endpoint-winevent-sysmon-* as your default one:

+
    +
  • "logs-*"
    • +
  • "logs-endpoint-winevent-sysmon-*"
    • +
  • "logs-endpoint-winevent-security-*"
    • +
  • "logs-endpoint-winevent-application-*"
    • +
  • "logs-endpoint-winevent-system-*"
    • +
  • "logs-endpoint-winevent-powershell-*"
    • +
  • "logs-endpoint-winevent-wmiactivity-*"
    • +
+

+

Dashboards

Currently, the HELK comes with 3 dashboards:

+

Global_Dashboard

+

Network_Dashboard

+

Sysmon_Dashboard

+

Monitoring Views (x-Pack Basic Free License)

Kibana Initial Overview

+

Elasticsearch Overview

+

Logstash Overview

+

+

Troubleshooting

Apart from running docker ps and docker logs --follow --tail 25 helk-kibana, additionally you can look at logs located at /usr/share/kibana/config/kibana_logs.log.

+

Example: docker exec helk-kibana tail -f /usr/share/kibana/config/kibana_logs.log

+

Many times Kibana will not be "working" because elasticsearch is still starting up or has ran into an error.

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/architecture/logstash.html b/docs/_build/architecture/logstash.html new file mode 100644 index 0000000..4a89adb --- /dev/null +++ b/docs/_build/architecture/logstash.html @@ -0,0 +1,33 @@ +--- +title: |- + Logstash +pagenum: 3 +prev_page: + url: /architecture/elasticsearch.html +next_page: + url: /architecture/kibana.html +suffix: .md +search: logstash img src images design png + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Logstash
+
+
+ +
+
+

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/docker/docker-export-images.html b/docs/_build/how-to/docker/docker-export-images.html new file mode 100644 index 0000000..5ea8103 --- /dev/null +++ b/docs/_build/how-to/docker/docker-export-images.html @@ -0,0 +1,109 @@ +--- +title: |- + Export Docker Images locally +pagenum: 6 +prev_page: + url: /how-to/docker/docker.html +next_page: + url: /how-to/docker/docker-load-images.html +suffix: .md +search: helk docker tar ago hours root cybrwardg tcp sudo spark feb ksql save o home rw mb logstash kibana elastic co elasticsearch images jupyter elastalert kafka zookeeper worker master days confluentinc cp server cli broker nginx system months isolated bash export internet files image command usr share locally list non via id created ps entr where planning install run another access built downloaded load those available repository tag size efaeccd gb f bdcebaf efbbee ba cffcbeee fafc weeks befce abbdae bbdb fcde db containers running container status ports names decdcf bin sh ecc etc confluent dock dcc entrypoint edd cadba + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Export Docker Images locally
+
+
+ +
+
+

If the system where you are planning to install HELK is isolated from the Internet, you can run HELK on another system that has access to the Internet and then export the built/downloaded images to .tar files. You can then LOAD Those image files in the system that is isolated from the Internet.

+
    +
  • List all the images available in the non-isolated system via the docker images command
    • +
+
sudo docker images
+
+ +
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
+cyb3rward0g/helk-jupyter                        0.1.1               efa46ecc8d32        2 days ago          2.18GB
+confluentinc/cp-ksql-server                     5.1.2               f57298019757        6 days ago          514MB
+confluentinc/cp-ksql-cli                        5.1.2               bd411ce0ba9f        6 days ago          510MB
+docker.elastic.co/logstash/logstash             6.6.1               3e7fbb7964ee        11 days ago         786MB
+docker.elastic.co/kibana/kibana                 6.6.1               b94222148a00        11 days ago         710MB
+docker.elastic.co/elasticsearch/elasticsearch   6.6.1               c6ffcb0ee97e        11 days ago         842MB
+cyb3rward0g/helk-elastalert                     0.2.1               569f588a22fc        3 weeks ago         758MB
+cyb3rward0g/helk-kafka-broker                   2.1.0               7b3e7f9ce732        2 months ago        388MB
+cyb3rward0g/helk-zookeeper                      2.1.0               abb732da3e50        2 months ago        388MB
+cyb3rward0g/helk-spark-worker                   2.4.0               b1545b0582db        2 months ago        579MB
+cyb3rward0g/helk-spark-master                   2.4.0               70fc61de3445        2 months ago        579MB
+cyb3rward0g/helk-nginx                          0.0.7               280d044b6719        6 months ago        329MB
+
    +
  • List all the containers running in the non-isolated system via the docker ps command
    • +
+
sudo docker ps
+
+ +
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
+de048c88dc7f        confluentinc/cp-ksql-cli:5.1.2                        "/bin/sh"                6 hours ago         Up 6 hours                                                                     helk-ksql-cli
+69e06070c14c        confluentinc/cp-ksql-server:5.1.2                     "/etc/confluent/dock…"   6 hours ago         Up 6 hours          0.0.0.0:8088->8088/tcp                                     helk-ksql-server
+d57967977c9c        cyb3rward0g/helk-kafka-broker:2.1.0                   "./kafka-entrypoint.…"   6 hours ago         Up 6 hours          0.0.0.0:9092->9092/tcp                                     helk-kafka-broker
+4889e917d76d        cyb3rward0g/helk-spark-worker:2.4.0                   "./spark-worker-entr…"   6 hours ago         Up 6 hours                                                                     helk-spark-worker
+c0a29d8b18a7        cyb3rward0g/helk-nginx:0.0.7                          "/opt/helk/scripts/n…"   6 hours ago         Up 6 hours          0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                   helk-nginx
+6a887d693a31        cyb3rward0g/helk-elastalert:0.2.1                     "./elastalert-entryp…"   6 hours ago         Up 6 hours                                                                     helk-elastalert
+a32be7a399c7        cyb3rward0g/helk-zookeeper:2.1.0                      "./zookeeper-entrypo…"   6 hours ago         Up 6 hours          2181/tcp, 2888/tcp, 3888/tcp                               helk-zookeeper
+c636a8a1e8f7        cyb3rward0g/helk-spark-master:2.4.0                   "./spark-master-entr…"   6 hours ago         Up 6 hours          7077/tcp, 0.0.0.0:8080->8080/tcp                           helk-spark-master
+ef1b8d8015ab        cyb3rward0g/helk-jupyter:0.1.1                        "./jupyter-entrypoin…"   6 hours ago         Up 6 hours          8000/tcp                                                   helk-jupyter
+bafeeb1587cf        docker.elastic.co/logstash/logstash:6.6.1             "/usr/share/logstash…"   6 hours ago         Up 6 hours          0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp   helk-logstash
+29b57e5c71e5        docker.elastic.co/kibana/kibana:6.6.1                 "/usr/share/kibana/s…"   6 hours ago         Up 6 hours          5601/tcp                                                   helk-kibana
+48499aa83917        docker.elastic.co/elasticsearch/elasticsearch:6.6.1   "/usr/share/elastics…"   6 hours ago         Up 6 hours          9200/tcp, 9300/tcp                                         helk-elasticsearch
+
    +
  • Export images as tar files:
    • +
+
sudo docker save -o /home/helk/helk-ksql-cli.tar confluentinc/cp-ksql-cli:5.1.2 
+sudo docker save -o /home/helk/helk-ksql-server.tar confluentinc/cp-ksql-server:5.1.2  
+sudo docker save -o /home/helk/helk-kafka-broker.tar cyb3rward0g/helk-kafka-broker:2.1.0
+sudo docker save -o /home/helk/helk-spark-worker.tar cyb3rward0g/helk-spark-worker:2.4.0
+sudo docker save -o /home/helk/helk-nginx.tar cyb3rward0g/helk-nginx:0.0.7
+sudo docker save -o /home/helk/helk-elastalert.tar cyb3rward0g/helk-elastalert:0.2.1
+sudo docker save -o /home/helk/helk-zookeeper.tar cyb3rward0g/helk-zookeeper:2.1.0
+sudo docker save -o /home/helk/helk-spark-master.tar cyb3rward0g/helk-spark-master:2.4.0
+sudo docker save -o /home/helk/helk-logstash.tar docker.elastic.co/logstash/logstash:6.6.1
+sudo docker save -o /home/helk/helk-kibana.tar docker.elastic.co/kibana/kibana:6.6.1
+sudo docker save -o /home/helk/helk-elasticsearch.tar docker.elastic.co/elasticsearch/elasticsearch:6.6.1
+sudo docker save -o /home/helk/helk-jupyter.tar cyb3rward0g/helk-jupyter:0.1.1
+
+
    +
  • check if images exist locally
    • +
+
ls -l
+
+ +
total 10810584
+drwxrwxr-x 9 helk helk       4096 Feb 24 21:01 HELK
+-rw------- 1 root root  778629632 Feb 25 03:07 helk-elastalert.tar
+-rw------- 1 root root  854236160 Feb 25 03:12 helk-elasticsearch.tar
+-rw------- 1 root root 2254629888 Feb 25 03:14 helk-jupyter.tar
+-rw------- 1 root root  395871744 Feb 25 03:04 helk-kafka-broker.tar
+-rw------- 1 root root  767277568 Feb 25 03:11 helk-kibana.tar
+-rw------- 1 root root  521177600 Feb 25 03:00 helk-ksql-cli.tar
+-rw------- 1 root root  525901824 Feb 25 03:02 helk-ksql-server.tar
+-rw------- 1 root root  810578944 Feb 25 03:09 helk-logstash.tar
+-rw------- 1 root root  335945728 Feb 25 03:06 helk-nginx.tar
+-rw------- 1 root root  587616768 Feb 25 03:08 helk-spark-master.tar
+-rw------- 1 root root  587616768 Feb 25 03:05 helk-spark-worker.tar
+-rw------- 1 root root  395854848 Feb 25 03:08 helk-zookeeper.tar
+
+helk@ubuntu:~$
+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/docker/docker-load-images.html b/docs/_build/how-to/docker/docker-load-images.html new file mode 100644 index 0000000..261d3ad --- /dev/null +++ b/docs/_build/how-to/docker/docker-load-images.html @@ -0,0 +1,113 @@ +--- +title: |- + Load Local Docker Images +pagenum: 7 +prev_page: + url: /how-to/docker/docker-export-images.html +next_page: + url: /how-to/logstash/logstash.html +suffix: .md +search: mb kb helk loading layer tar ago docker images cybrwardg load ksql elasticsearch days spark s months bash logstash loaded image elastic co isolated system f tmp worker server cli elastalert jupyter kibana gb done check nginx kafka broker master zookeeper sudo ba confluentinc cp local followed document export locally should ready into where cannot access dockerhub registry copy home scp exist ls commands input fddce fbe ccddaa cdfde ab fffef cffbcf adfdc cce fdecbc bbdae affdeb ccbfe dbd adcf aaff fdbc beced cbe afd eec fbd faacff cbcbe cdaf ecdbbfdc edccf efbcd via command repository tag id created size + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Load Local Docker Images
+
+
+ +
+
+

If you followed this document to export your docker images locally, you should be ready to load them into an isolated system where it cannot access the dockerhub registry.

+
    +
  • Copy images to the isolated (10.0.10.102) system
    • +
+
for f in /home/helk/*.tar; do scp $f helk@10.0.10.102:/tmp/; done
+
+ +
helk-spark-worker.tar  100%  560MB  24.4MB/s   00:23    
+helk-ksql-server.tar   100%  502MB  29.5MB/s   00:17    
+helk-logstash.tar      100%  773MB  28.6MB/s   00:27    
+helk-ksql-cli.tar      100%  497MB  21.6MB/s   00:23    
+helk-elasticsearch.tar 100%  815MB  29.1MB/s   00:28
+
    +
  • Check if images exist in the isolated system
    • +
+
ls /tmp/
+
+ +
helk-elastalert.tar helk-jupyter.tar 
+helk-kibana.tar helk-ksql-server.tar helk-nginx.tar 
+helk-spark-worker.tar helk-elasticsearch.tar
+helk-kafka-broker.tar helk-ksql-cli.tar helk-logstash.tar
+helk-spark-master.tar  helk-zookeeper.tar
+
    +
  • Load images with the docker load commands:
    • +
+
for i in /tmp/*.tar; do sudo docker load --input $i; done
+
+ +
f49017d4d5ce: Loading layer [==================================================>]  85.96MB/85.96MB
+8f2b771487e9: Loading layer [==================================================>]  15.87kB/15.87kB
+ccd4d61916aa: Loading layer [==================================================>]  10.24kB/10.24kB
+c01d74f99de4: Loading layer [==================================================>]  5.632kB/5.632kB
+268a067217b5: Loading layer [==================================================>]  3.072kB/3.072kB
+831fff32e4f2: Loading layer [==================================================>]  65.02kB/65.02kB
+c89f4fbc01f8: Loading layer [==================================================>]  103.4MB/103.4MB
+adfd094c5517: Loading layer [==================================================>]  3.245MB/3.245MB
+c73538215c3e: Loading layer [==================================================>]  567.6MB/567.6MB
+080f01d1ecbc: Loading layer [==================================================>]  13.31kB/13.31kB
+60bbd38a907e: Loading layer [==================================================>]  3.584kB/3.584kB
+9affd17eb100: Loading layer [==================================================>]  5.632kB/5.632kB
+0561c04cbf7e: Loading layer [==================================================>]  7.168kB/7.168kB
+ba0201512417: Loading layer [==================================================>]  18.29MB/18.29MB
+Loaded image: cyb3rward0g/helk-elastalert:0.2.1
+071d8bd76517: Loading layer [==================================================>]  210.2MB/210.2MB
+a175339dcf83: Loading layer [==================================================>]  310.5MB/310.5MB
+9a70a6f483f7: Loading layer [==================================================>]  95.68MB/95.68MB
+f4db77828c81: Loading layer [==================================================>]  311.3kB/311.3kB
+be48c67e9d13: Loading layer [==================================================>]  237.5MB/237.5MB
+432cb712190e: Loading layer [==================================================>]   7.68kB/7.68kB
+a512981fd597: Loading layer [==================================================>]  9.728kB/9.728kB
+Loaded image: docker.elastic.co/elasticsearch/elasticsearch:6.6.1
+49778752e7ec: Loading layer [==================================================>]  394.9MB/394.9MB
+5f3913b1d541: Loading layer [==================================================>]  1.667GB/1.667GB
+77fa3a9c5ff6: Loading layer [==================================================>]  7.168kB/7.168kB
+cbc15b984e03: Loading layer [==================================================>]  10.24kB/10.24kB
+38c44d7a52f6: Loading layer [==================================================>]   5.12kB/5.12kB
+0ec2dbbfd6c7: Loading layer [==================================================>]  3.584kB/3.584kB
+Loaded image: cyb3rward0g/helk-jupyter:0.1.1
+4e31d8c1cf96: Loading layer [==================================================>]  203.1MB/203.1MB
+efb23c49455d: Loading layer [==================================================>]  11.26kB/11.26kB
+
    +
  • check if images are loaded via the docker images command
    • +
+
sudo docker images
+
+ +
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
+cyb3rward0g/helk-jupyter                        0.1.1               efa46ecc8d32        2 days ago          2.18GB
+confluentinc/cp-ksql-server                     5.1.2               f57298019757        6 days ago          514MB
+confluentinc/cp-ksql-cli                        5.1.2               bd411ce0ba9f        6 days ago          510MB
+docker.elastic.co/logstash/logstash             6.6.1               3e7fbb7964ee        11 days ago         786MB
+docker.elastic.co/kibana/kibana                 6.6.1               b94222148a00        11 days ago         710MB
+docker.elastic.co/elasticsearch/elasticsearch   6.6.1               c6ffcb0ee97e        11 days ago         842MB
+cyb3rward0g/helk-elastalert                     0.2.1               569f588a22fc        3 weeks ago         758MB
+cyb3rward0g/helk-kafka-broker                   2.1.0               7b3e7f9ce732        2 months ago        388MB
+cyb3rward0g/helk-zookeeper                      2.1.0               abb732da3e50        2 months ago        388MB
+cyb3rward0g/helk-spark-worker                   2.4.0               b1545b0582db        2 months ago        579MB
+cyb3rward0g/helk-spark-master                   2.4.0               70fc61de3445        2 months ago        579MB
+cyb3rward0g/helk-nginx                          0.0.7               280d044b6719        6 months ago        329MB
+helk@helk:~$
+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/docker/docker.html b/docs/_build/how-to/docker/docker.html new file mode 100644 index 0000000..e8c2373 --- /dev/null +++ b/docs/_build/how-to/docker/docker.html @@ -0,0 +1,22 @@ +--- +title: |- + Docker +pagenum: 5 +prev_page: + url: /architecture/kibana.html +next_page: + url: /how-to/docker/docker-export-images.html +suffix: .md +search: docker + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Docker
+
+ + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/kafka-topic-ingestion.html b/docs/_build/how-to/kafka-topic-ingestion.html new file mode 100644 index 0000000..18e1584 --- /dev/null +++ b/docs/_build/how-to/kafka-topic-ingestion.html @@ -0,0 +1,49 @@ +--- +title: |- + Check Kafka Topic Ingestion +pagenum: 2 +prev_page: + url: /installation.html +next_page: + url: +suffix: .md +search: kafka broker helk kafkacat bash consumer topic container console sh winlogbeat following sudo docker exec ti run script opt bin bootstrap server beginning apache install github com edenhill check ingestion few ways accomplish helks access running command available simply without interactive shell generic non jvm producer think netcat instructions repo b t c references example org quickstart quickstartconsume + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Check Kafka Topic Ingestion
+
+
+ +
+
+

There are a few ways that you can accomplish this

+

HELK's Kafka broker container

Access your kafka broker container by running the following command:

+
sudo docker exec -ti helk-kafka-broker bash
+
+

Run the kafka-console-consumer.sh script available in the container:

+
/opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic winlogbeat --from-beginning
+
+

or simply run the script without an interactive shell

+
sudo docker exec -ti helk-kafka-broker /opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic winlogbeat --from-beginning
+
+

Kafkacat

It is generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can install it by following the instructions from the Kafkacat repo.

+
kafkacat -b 10.0.10.100:9092 -t winlogbeat -C
+
+

References

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/kafka/kafka-topic-ingestion.html b/docs/_build/how-to/kafka/kafka-topic-ingestion.html new file mode 100644 index 0000000..b083716 --- /dev/null +++ b/docs/_build/how-to/kafka/kafka-topic-ingestion.html @@ -0,0 +1,49 @@ +--- +title: |- + Check Kafka Topic Ingestion +pagenum: 11 +prev_page: + url: /how-to/kafka/kafka.html +next_page: + url: /how-to/kafka/kafka-update-ip.html +suffix: .md +search: kafka broker helk kafkacat bash consumer topic container console sh winlogbeat following sudo docker exec ti run script opt bin bootstrap server beginning apache install github com edenhill check ingestion few ways accomplish helks access running command available simply without interactive shell generic non jvm producer think netcat instructions repo b t c references example org quickstart quickstartconsume + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Check Kafka Topic Ingestion
+
+
+ +
+
+

There are a few ways that you can accomplish this

+

HELK's Kafka broker container

Access your kafka broker container by running the following command:

+
sudo docker exec -ti helk-kafka-broker bash
+
+

Run the kafka-console-consumer.sh script available in the container:

+
/opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic winlogbeat --from-beginning
+
+

or simply run the script without an interactive shell

+
sudo docker exec -ti helk-kafka-broker /opt/helk/kafka/bin/kafka-console-consumer.sh --bootstrap-server helk-kafka-broker:9092 --topic winlogbeat --from-beginning
+
+

Kafkacat

It is generic non-JVM producer and consumer for Apache Kafka >=0.8, think of it as a netcat for Kafka. You can install it by following the instructions from the Kafkacat repo.

+
kafkacat -b 10.0.10.100:9092 -t winlogbeat -C
+
+

References

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/kafka/kafka-update-ip.html b/docs/_build/how-to/kafka/kafka-update-ip.html new file mode 100644 index 0000000..2b43e2c --- /dev/null +++ b/docs/_build/how-to/kafka/kafka-update-ip.html @@ -0,0 +1,57 @@ +--- +title: |- + Update Kafka Broker IP +pagenum: 12 +prev_page: + url: /how-to/kafka/kafka-topic-ingestion.html +next_page: + url: /how-to/ksql/ksql.html +suffix: .md +search: not broker kafka warn controller id targetbrokerid connection node established available org apache clients networkclient docker update environment variable advertisedlistener helk re system containers bash just compose create ip deployment hosting entire itself distributed across systems export simply run same used build new value assigned sudo e f kibana notebook analysis basic yml d restart container creating still show messages ones below + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Update Kafka Broker IP
+
+
+ +
+
+

For the docker deployment, you will have to update the environment variable ADVERTISED_LISTENER first. You can do this in your system hosting the entire HELK or the Kafka broker itself if your distributed your docker containers across other systems.

+
export ADVERTISED_LISTENER=10.0.10.104
+
+

Then, you can simply just run docker-compose the same way how it was used to build the HELK. This will re-create the system with the new value assigned to the environment variable ADVERTISED_LISTENER.

+
sudo -E docker-compose -f helk-kibana-notebook-analysis-basic.yml up -d
+
+

If you just restart your containers, it will not update the environment variable in the Kafka broker. You have to re-create the container. Not re-creating the broker would still show you messages like the ones below:

+ +
[2019-01-25 05:35:21,026] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:24,194] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:27,362] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:30,530] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:33,698] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:36,866] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:40,034] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:43,238] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:46,306] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:49,382] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:52,450] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:55,522] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:35:58,594] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:36:01,714] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:36:04,770] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:36:08,450] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+[2019-01-25 05:36:11,650] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (/10.0.10.104:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/kafka/kafka.html b/docs/_build/how-to/kafka/kafka.html new file mode 100644 index 0000000..21c0c48 --- /dev/null +++ b/docs/_build/how-to/kafka/kafka.html @@ -0,0 +1,22 @@ +--- +title: |- + Kafka +pagenum: 10 +prev_page: + url: /how-to/logstash/logstash-create-plugins-offline.html +next_page: + url: /how-to/kafka/kafka-topic-ingestion.html +suffix: .md +search: kafka + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Kafka
+
+ + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/ksql/ksql-deploy-locally.html b/docs/_build/how-to/ksql/ksql-deploy-locally.html new file mode 100644 index 0000000..acb5c28 --- /dev/null +++ b/docs/_build/how-to/ksql/ksql-deploy-locally.html @@ -0,0 +1,184 @@ +--- +title: |- + Deploy KSQL Locally +pagenum: 14 +prev_page: + url: /how-to/ksql/ksql.html +next_page: + url: /how-to/winlogbeat/winlogbeat.html +suffix: .md +search: kafka ksql confluent server stop control center run start tar x class gz bash src consumer options help cli connect console false avro bin schema registry reset producer zookeeper rest query value v download mqtt topics configs file output limit helk self managed software platform format io lib cd ls acls api streams service verifiable security broker replica perf test metrics partitions log robertos mbp wardog config configfile h outputformat row streamedqueryrowlimit timeout streamedquerytimeoutms list available tabular optional maximum streamed queries must fall following range option command line arguments address streaming sql engine apache copyright inc located having trouble type + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Deploy KSQL Locally
+
+
+ +
+
+

You can use KSQL CLI to connect to the HELK's KSQL Server from a different system. You will have to download the self-managed software Confluent platform and then run KSQL

+ +
tar -xvzf confluent-5.1.2-2.11.tar.gz
+
+ +
x confluent-5.1.2/
+x confluent-5.1.2/src/
+x confluent-5.1.2/src/avro-cpp-1.8.0-confluent5.1.2.tar.gz
+x confluent-5.1.2/src/librdkafka-0.11.6-confluent5.1.2.tar.gz
+x confluent-5.1.2/src/confluent-libserdes-5.1.2.tar.gz
+x confluent-5.1.2/src/avro-c-1.8.0-confluent5.1.2.tar.gz
+x confluent-5.1.2/lib/
+
    +
  • Access the KSQL scripts:
    • +
+
cd confluent-5.1.2
+ls
+
+ +
README  bin etc lib logs    share   src
+
cd bin/
+ls
+
+ +
confluent               kafka-acls              kafka-mirror-maker          kafka-server-stop           schema-registry-start
+confluent-hub               kafka-api-start             kafka-mqtt-run-class            kafka-streams-application-reset     schema-registry-stop
+confluent-rebalancer            kafka-avro-console-consumer     kafka-mqtt-start            kafka-topics                schema-registry-stop-service
+connect-distributed         kafka-avro-console-producer     kafka-mqtt-stop             kafka-verifiable-consumer       security-plugins-run-class
+connect-standalone          kafka-broker-api-versions       kafka-preferred-replica-election    kafka-verifiable-producer       sr-acl-cli
+control-center-3_0_0-reset      kafka-configs               kafka-producer-perf-test        ksql                    support-metrics-bundle
+control-center-3_0_1-reset      kafka-console-consumer          kafka-reassign-partitions       ksql-datagen                windows
+control-center-console-consumer     kafka-console-producer          kafka-replica-verification      ksql-print-metrics          zookeeper-security-migration
+control-center-export           kafka-consumer-groups           kafka-rest-run-class            ksql-run-class              zookeeper-server-start
+control-center-reset            kafka-consumer-perf-test        kafka-rest-start            ksql-server-start           zookeeper-server-stop
+control-center-run-class        kafka-delegation-tokens         kafka-rest-stop             ksql-server-stop            zookeeper-shell
+control-center-set-acls         kafka-delete-records            kafka-rest-stop-service         ksql-stop
+control-center-start            kafka-dump-log              kafka-run-class             replicator
+control-center-stop         kafka-log-dirs              kafka-server-start          schema-registry-run-class
+Robertos-MBP:bin wardog$
+
    +
  • Check the options for KSQL:
    • +
+
./ksql --help
+
+ +
NAME
+        ksql - KSQL CLI
+
+SYNOPSIS
+        ksql [ --config-file <configFile> ] [ {-h | --help} ]
+                [ --output <outputFormat> ]
+                [ --query-row-limit <streamedQueryRowLimit> ]
+                [ --query-timeout <streamedQueryTimeoutMs> ] [--] <server>
+
+OPTIONS
+        --config-file <configFile>
+            A file specifying configs for Ksql and its underlying Kafka Streams
+            instance(s). Refer to KSQL documentation for a list of available
+            configs.
+
+        -h, --help
+            Display help information
+
+        --output <outputFormat>
+            The output format to use (either 'JSON' or 'TABULAR'; can be
+            changed during REPL as well; defaults to TABULAR)
+
+        --query-row-limit <streamedQueryRowLimit>
+            An optional maximum number of rows to read from streamed queries
+
+            This options value must fall in the following range: value >= 1
+
+
+        --query-timeout <streamedQueryTimeoutMs>
+            An optional time limit (in milliseconds) for streamed queries
+
+            This options value must fall in the following range: value >= 1
+
+
+        --
+            This option can be used to separate command-line options from the
+            list of arguments (useful when arguments might be mistaken for
+            command-line options)
+
+        <server>
+            The address of the Ksql server to connect to (ex:
+            http://confluent.io:9098)
+
+            This option may occur a maximum of 1 times
+
+Robertos-MBP:bin wardog$
+
    +
  • Connect to the HELK KSQL Server. You will just need to point to the IP address of your HELK Docker environment over port 8088
    • +
+
./ksql http://192.168.64.138:8088
+
+ +
                  ===========================================
+                  =        _  __ _____  ____  _             =
+                  =       | |/ // ____|/ __ \| |            =
+                  =       | ' /| (___ | |  | | |            =
+                  =       |  <  \___ \| |  | | |            =
+                  =       | . \ ____) | |__| | |____        =
+                  =       |_|\_\_____/ \___\_\______|       =
+                  =                                         =
+                  =  Streaming SQL Engine for Apache Kafka® =
+                  ===========================================
+
+Copyright 2017-2018 Confluent Inc.
+
+CLI v5.1.2, Server v5.1.0 located at http://192.168.64.138:8088
+
+Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!
+
+ksql>
+
    +
  • Verify that you can see the topics available in the HELK Kafka broker
    • +
+
./ksql http://192.168.64.138:8088
+
+ +
                  ===========================================
+                  =        _  __ _____  ____  _             =
+                  =       | |/ // ____|/ __ \| |            =
+                  =       | ' /| (___ | |  | | |            =
+                  =       |  <  \___ \| |  | | |            =
+                  =       | . \ ____) | |__| | |____        =
+                  =       |_|\_\_____/ \___\_\______|       =
+                  =                                         =
+                  =  Streaming SQL Engine for Apache Kafka® =
+                  ===========================================
+
+Copyright 2017-2018 Confluent Inc.
+
+CLI v5.1.2, Server v5.1.0 located at http://192.168.64.138:8088
+
+Having trouble? Type 'help' (case-insensitive) for a rundown of how things work!
+
+ksql> SHOW TOPICS;
+
+ Kafka Topic | Registered | Partitions | Partition Replicas | Consumers | ConsumerGroups 
+-----------------------------------------------------------------------------------------
+ filebeat    | false      | 1          | 1                  | 0         | 0              
+ SYSMON_JOIN | false      | 1          | 1                  | 0         | 0              
+ winlogbeat  | false      | 1          | 1                  | 0         | 0              
+ winsecurity | false      | 1          | 1                  | 0         | 0              
+ winsysmon   | false      | 1          | 1                  | 0         | 0              
+-----------------------------------------------------------------------------------------
+ksql>
+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/ksql/ksql.html b/docs/_build/how-to/ksql/ksql.html new file mode 100644 index 0000000..ba55a01 --- /dev/null +++ b/docs/_build/how-to/ksql/ksql.html @@ -0,0 +1,22 @@ +--- +title: |- + KSQL +pagenum: 13 +prev_page: + url: /how-to/kafka/kafka-update-ip.html +next_page: + url: /how-to/ksql/ksql-deploy-locally.html +suffix: .md +search: ksql + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
KSQL
+
+ + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/logstash/logstash-create-plugins-offline.html b/docs/_build/how-to/logstash/logstash-create-plugins-offline.html new file mode 100644 index 0000000..6e68ce8 --- /dev/null +++ b/docs/_build/how-to/logstash/logstash-create-plugins-offline.html @@ -0,0 +1,62 @@ +--- +title: |- + Create Plugins Offline Package +pagenum: 9 +prev_page: + url: /how-to/logstash/logstash.html +next_page: + url: /how-to/kafka/kafka.html +suffix: .md +search: logstash offline filter helk plugins system bash package zip installed docker plugin usr share installing internet export access where already successfully container sudo prepare bin kafka input codec install copy isolated create extra still being following steps zipped loaded does not stuck remember need exec ti using script pack translate dns cidr geoip dissect output alter fingerprint prune gziplines netflow environment dedot wmi clone created command file local cp e bust authorized ssh scp home should able + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Create Plugins Offline Package
+
+
+ +
+
+

If you are installing HELK, and the helk-logstash extra plugins are still being installed over the Internet, you can use the following steps to export them in an zipped offline package to then be loaded to the system that does not have access to the Internet and it is stuck at installing plugins.

+

Remember that you will need to do this in a system where HELK is already installed and the plugins were installed successfully.

+
    +
  • Access your helk-logstash docker container in the system where HELK was successfully installed already:
    • +
+
sudo docker exec -ti helk-logstash bash
+
+ +
bash-4.2$
+
    +
  • Using the logstash-plugin script prepare and export the plugins offline package
    • +
+
bin/logstash-plugin prepare-offline-pack logstash-filter-translate logstash-filter-dns  logstash-filter-cidr  logstash-filter-geoip logstash-filter-dissect  logstash-output-kafka  logstash-input-kafka  logstash-filter-alter  logstash-filter-fingerprint  logstash-filter-prune  logstash-codec-gzip_lines  logstash-codec-netflow  logstash-filter-i18n  logstash-filter-environment  logstash-filter-de_dot  logstash-input-wmi  logstash-filter-clone
+
+ +
Offline package created at: /usr/share/logstash/logstash-offline-plugins-6.6.1.zip
+
+You can install it with this command
+bin/logstash-plugin install file:///usr/share/logstash/logstash-offline-plugins-6.6.1.zip
+
    +
  • Copy the offline package from your helk-logstash container to your local system
    • +
+
sudo docker cp helk-logstash:/usr/share/logstash/logstash-offline-plugins-6.6.1.zip .
+
+
    +
  • Copy the logstash-offline-plugins-6.6.1.zip to the OFFLINE-ISOLATED (i.e. 10.0.10.102) system. You bust be authorized to ssh to it.
    • +
+
scp logstash-offline-plugins-6.6.1.zip helk@10.0.10.102:/home/helk/
+
+

Now you should be able to use it in the offline-isolated HELK system

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/logstash/logstash.html b/docs/_build/how-to/logstash/logstash.html new file mode 100644 index 0000000..45ab427 --- /dev/null +++ b/docs/_build/how-to/logstash/logstash.html @@ -0,0 +1,22 @@ +--- +title: |- + Logstash +pagenum: 8 +prev_page: + url: /how-to/docker/docker-load-images.html +next_page: + url: /how-to/logstash/logstash-create-plugins-offline.html +suffix: .md +search: logstash + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Logstash
+
+ + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/winlogbeat/winlogbeat-shipping.html b/docs/_build/how-to/winlogbeat/winlogbeat-shipping.html new file mode 100644 index 0000000..f563438 --- /dev/null +++ b/docs/_build/how-to/winlogbeat/winlogbeat-shipping.html @@ -0,0 +1,45 @@ +--- +title: |- + Check Winlogbeat Shipping +pagenum: 16 +prev_page: + url: /how-to/winlogbeat/winlogbeat.html +next_page: + url: +suffix: .md +search: winlogbeat logs running shipping contain shown below should installed service within programdata manually notepad powershell output img src images kafka producer png check believe not being sent helk couple things going document stick looking only itself sending properly therefore issue somewhere else thus consult wiki additional resources located executable directory exe path viewing view simply command such get content c tail wait log information verbiage successfully published events similar + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Check Winlogbeat Shipping
+
+
+ +
+
+

If you believe logs are not being sent to HELK from winlogbeat then there are a couple of things that could be going on. For this document we will stick to looking at only winlogbeat itself. If your logs contain what is shown below then winlogbeat is shipping/sending the logs properly and therefore could be an issue somewhere else and thus you should consult the wiki for additional resources.

+

Installed as a Service

If winlogbeat is installed as a service then the logs will be located within:

+ +
%PROGRAMDATA%\winlogbeat\logs\winlogbeat
+

Manually Running the Executable

If you are manually running winlogbeat then the logs will be within the directory you are running winlogbeat.exe at the path

+ +
.\logs\winlogbeat
+

Viewing Logs

To view the logs you can simply use notepad or notepad++ or use a powershell command such as

+
Get-Content C:\ProgramData\winlogbeat\logs\winlogbeat -Tail 10 -Wait
+
+

Log Output

Your logs should contain information with the verbiage successfully published #NUMBER events, similar to the output shown below

+

+

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/how-to/winlogbeat/winlogbeat.html b/docs/_build/how-to/winlogbeat/winlogbeat.html new file mode 100644 index 0000000..d0fef1e --- /dev/null +++ b/docs/_build/how-to/winlogbeat/winlogbeat.html @@ -0,0 +1,22 @@ +--- +title: |- + Winlogbeat +pagenum: 15 +prev_page: + url: /how-to/ksql/ksql-deploy-locally.html +next_page: + url: /how-to/winlogbeat/winlogbeat-shipping.html +suffix: .md +search: winlogbeat + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Winlogbeat
+
+ + + +
+ \ No newline at end of file diff --git a/resources/images/ELASTICSEARCH-Design.png b/docs/_build/images/ELASTICSEARCH-Design.png similarity index 100% rename from resources/images/ELASTICSEARCH-Design.png rename to docs/_build/images/ELASTICSEARCH-Design.png diff --git a/resources/images/HELK_Design.png b/docs/_build/images/HELK-Design.png similarity index 100% rename from resources/images/HELK_Design.png rename to docs/_build/images/HELK-Design.png diff --git a/resources/images/HELK-Monitoring-Overall.png b/docs/_build/images/HELK-Monitoring-Overall.png similarity index 100% rename from resources/images/HELK-Monitoring-Overall.png rename to docs/_build/images/HELK-Monitoring-Overall.png diff --git a/resources/images/HELK_LOGO_CROPPED_WHITE.png b/docs/_build/images/HELK_LOGO_CROPPED_WHITE.png similarity index 100% rename from resources/images/HELK_LOGO_CROPPED_WHITE.png rename to docs/_build/images/HELK_LOGO_CROPPED_WHITE.png diff --git a/resources/images/HELK_checking_integrations.png b/docs/_build/images/HELK_checking_integrations.png similarity index 100% rename from resources/images/HELK_checking_integrations.png rename to docs/_build/images/HELK_checking_integrations.png diff --git a/resources/images/JUPYTER-InitializeKernel.png b/docs/_build/images/JUPYTER-InitializeKernel.png similarity index 100% rename from resources/images/JUPYTER-InitializeKernel.png rename to docs/_build/images/JUPYTER-InitializeKernel.png diff --git a/resources/images/JUPYTER-LAB-Initial-View.png b/docs/_build/images/JUPYTER-LAB-Initial-View.png similarity index 100% rename from resources/images/JUPYTER-LAB-Initial-View.png rename to docs/_build/images/JUPYTER-LAB-Initial-View.png diff --git a/resources/images/JUPYTER-Lab.png b/docs/_build/images/JUPYTER-Lab.png similarity index 100% rename from resources/images/JUPYTER-Lab.png rename to docs/_build/images/JUPYTER-Lab.png diff --git a/resources/images/JUPYTER-Token.png b/docs/_build/images/JUPYTER-Token.png similarity index 100% rename from resources/images/JUPYTER-Token.png rename to docs/_build/images/JUPYTER-Token.png diff --git a/resources/images/JUPYTER-Tree.png b/docs/_build/images/JUPYTER-Tree.png similarity index 100% rename from resources/images/JUPYTER-Tree.png rename to docs/_build/images/JUPYTER-Tree.png diff --git a/resources/images/JUPYTER-login.png b/docs/_build/images/JUPYTER-login.png similarity index 100% rename from resources/images/JUPYTER-login.png rename to docs/_build/images/JUPYTER-login.png diff --git a/resources/images/JUPYTER-menu.png b/docs/_build/images/JUPYTER-menu.png similarity index 100% rename from resources/images/JUPYTER-menu.png rename to docs/_build/images/JUPYTER-menu.png diff --git a/resources/images/JUPYTER-notebook.png b/docs/_build/images/JUPYTER-notebook.png similarity index 100% rename from resources/images/JUPYTER-notebook.png rename to docs/_build/images/JUPYTER-notebook.png diff --git a/resources/images/JUPYTERHUB-Pyspark-Kernel.png b/docs/_build/images/JUPYTERHUB-Pyspark-Kernel.png similarity index 100% rename from resources/images/JUPYTERHUB-Pyspark-Kernel.png rename to docs/_build/images/JUPYTERHUB-Pyspark-Kernel.png diff --git a/resources/images/JUPYTERHUB-SparkContext.png b/docs/_build/images/JUPYTERHUB-SparkContext.png similarity index 100% rename from resources/images/JUPYTERHUB-SparkContext.png rename to docs/_build/images/JUPYTERHUB-SparkContext.png diff --git a/resources/images/KAFKA-BROKER-Error-TopicCreation.png b/docs/_build/images/KAFKA-BROKER-Error-TopicCreation.png similarity index 100% rename from resources/images/KAFKA-BROKER-Error-TopicCreation.png rename to docs/_build/images/KAFKA-BROKER-Error-TopicCreation.png diff --git a/resources/images/KAFKA-BROKER-Recreate-Topic.png b/docs/_build/images/KAFKA-BROKER-Recreate-Topic.png similarity index 100% rename from resources/images/KAFKA-BROKER-Recreate-Topic.png rename to docs/_build/images/KAFKA-BROKER-Recreate-Topic.png diff --git a/resources/images/KAFKA-Design.png b/docs/_build/images/KAFKA-Design.png similarity index 100% rename from resources/images/KAFKA-Design.png rename to docs/_build/images/KAFKA-Design.png diff --git a/resources/images/KAFKA-producer1.png b/docs/_build/images/KAFKA-producer1.png similarity index 100% rename from resources/images/KAFKA-producer1.png rename to docs/_build/images/KAFKA-producer1.png diff --git a/resources/images/KAFKA-producer2.png b/docs/_build/images/KAFKA-producer2.png similarity index 100% rename from resources/images/KAFKA-producer2.png rename to docs/_build/images/KAFKA-producer2.png diff --git a/resources/images/KIBANA-Design.png b/docs/_build/images/KIBANA-Design.png similarity index 100% rename from resources/images/KIBANA-Design.png rename to docs/_build/images/KIBANA-Design.png diff --git a/resources/images/KIBANA-Discovery.png b/docs/_build/images/KIBANA-Discovery.png similarity index 100% rename from resources/images/KIBANA-Discovery.png rename to docs/_build/images/KIBANA-Discovery.png diff --git a/resources/images/KIBANA-GlobalDashboard.png b/docs/_build/images/KIBANA-GlobalDashboard.png similarity index 100% rename from resources/images/KIBANA-GlobalDashboard.png rename to docs/_build/images/KIBANA-GlobalDashboard.png diff --git a/resources/images/KIBANA-Home.png b/docs/_build/images/KIBANA-Home.png similarity index 100% rename from resources/images/KIBANA-Home.png rename to docs/_build/images/KIBANA-Home.png diff --git a/resources/images/KIBANA-IndexPatterns.png b/docs/_build/images/KIBANA-IndexPatterns.png similarity index 100% rename from resources/images/KIBANA-IndexPatterns.png rename to docs/_build/images/KIBANA-IndexPatterns.png diff --git a/resources/images/KIBANA-Initial-View.png b/docs/_build/images/KIBANA-Initial-View.png similarity index 100% rename from resources/images/KIBANA-Initial-View.png rename to docs/_build/images/KIBANA-Initial-View.png diff --git a/resources/images/KIBANA-NetworkDasboard.png b/docs/_build/images/KIBANA-NetworkDasboard.png similarity index 100% rename from resources/images/KIBANA-NetworkDasboard.png rename to docs/_build/images/KIBANA-NetworkDasboard.png diff --git a/resources/images/KIBANA-NetworkDashboard.png b/docs/_build/images/KIBANA-NetworkDashboard.png similarity index 100% rename from resources/images/KIBANA-NetworkDashboard.png rename to docs/_build/images/KIBANA-NetworkDashboard.png diff --git a/resources/images/KIBANA-NoIndicesPattern.png b/docs/_build/images/KIBANA-NoIndicesPattern.png similarity index 100% rename from resources/images/KIBANA-NoIndicesPattern.png rename to docs/_build/images/KIBANA-NoIndicesPattern.png diff --git a/resources/images/KIBANA-SysmonDashboard.png b/docs/_build/images/KIBANA-SysmonDashboard.png similarity index 100% rename from resources/images/KIBANA-SysmonDashboard.png rename to docs/_build/images/KIBANA-SysmonDashboard.png diff --git a/resources/images/KIBANA-vega_build.png b/docs/_build/images/KIBANA-vega_build.png similarity index 100% rename from resources/images/KIBANA-vega_build.png rename to docs/_build/images/KIBANA-vega_build.png diff --git a/resources/images/KIBANA-vega_dashboard.png b/docs/_build/images/KIBANA-vega_dashboard.png similarity index 100% rename from resources/images/KIBANA-vega_dashboard.png rename to docs/_build/images/KIBANA-vega_dashboard.png diff --git a/resources/images/LM_Basic_Shells.JPG b/docs/_build/images/LM_Basic_Shells.JPG similarity index 100% rename from resources/images/LM_Basic_Shells.JPG rename to docs/_build/images/LM_Basic_Shells.JPG diff --git a/resources/images/LOGSTASH-Design.png b/docs/_build/images/LOGSTASH-Design.png similarity index 100% rename from resources/images/LOGSTASH-Design.png rename to docs/_build/images/LOGSTASH-Design.png diff --git a/resources/images/MONITORING-Elasticsearch-Indices.png b/docs/_build/images/MONITORING-Elasticsearch-Indices.png similarity index 100% rename from resources/images/MONITORING-Elasticsearch-Indices.png rename to docs/_build/images/MONITORING-Elasticsearch-Indices.png diff --git a/resources/images/MONITORING-Elasticsearch-Nodes.png b/docs/_build/images/MONITORING-Elasticsearch-Nodes.png similarity index 100% rename from resources/images/MONITORING-Elasticsearch-Nodes.png rename to docs/_build/images/MONITORING-Elasticsearch-Nodes.png diff --git a/resources/images/MONITORING-Elasticsearch-Overview.png b/docs/_build/images/MONITORING-Elasticsearch-Overview.png similarity index 100% rename from resources/images/MONITORING-Elasticsearch-Overview.png rename to docs/_build/images/MONITORING-Elasticsearch-Overview.png diff --git a/resources/images/MONITORING-Kibana-Instances.png b/docs/_build/images/MONITORING-Kibana-Instances.png similarity index 100% rename from resources/images/MONITORING-Kibana-Instances.png rename to docs/_build/images/MONITORING-Kibana-Instances.png diff --git a/resources/images/MONITORING-Kibana-Overview.png b/docs/_build/images/MONITORING-Kibana-Overview.png similarity index 100% rename from resources/images/MONITORING-Kibana-Overview.png rename to docs/_build/images/MONITORING-Kibana-Overview.png diff --git a/resources/images/MONITORING-Logstash-Nodes-Overview.png b/docs/_build/images/MONITORING-Logstash-Nodes-Overview.png similarity index 100% rename from resources/images/MONITORING-Logstash-Nodes-Overview.png rename to docs/_build/images/MONITORING-Logstash-Nodes-Overview.png diff --git a/resources/images/MONITORING-Logstash-Nodes.png b/docs/_build/images/MONITORING-Logstash-Nodes.png similarity index 100% rename from resources/images/MONITORING-Logstash-Nodes.png rename to docs/_build/images/MONITORING-Logstash-Nodes.png diff --git a/resources/images/MONITORING-Logstash-Overview.png b/docs/_build/images/MONITORING-Logstash-Overview.png similarity index 100% rename from resources/images/MONITORING-Logstash-Overview.png rename to docs/_build/images/MONITORING-Logstash-Overview.png diff --git a/resources/images/MONITORING-Logstash-Pipeline.png b/docs/_build/images/MONITORING-Logstash-Pipeline.png similarity index 100% rename from resources/images/MONITORING-Logstash-Pipeline.png rename to docs/_build/images/MONITORING-Logstash-Pipeline.png diff --git a/resources/images/SPARK-ApacheArrow.png b/docs/_build/images/SPARK-ApacheArrow.png similarity index 100% rename from resources/images/SPARK-ApacheArrow.png rename to docs/_build/images/SPARK-ApacheArrow.png diff --git a/resources/images/SPARK-Cluster-Manager.png b/docs/_build/images/SPARK-Cluster-Manager.png similarity index 100% rename from resources/images/SPARK-Cluster-Manager.png rename to docs/_build/images/SPARK-Cluster-Manager.png diff --git a/resources/images/SPARK-Cluster-Worker.png b/docs/_build/images/SPARK-Cluster-Worker.png similarity index 100% rename from resources/images/SPARK-Cluster-Worker.png rename to docs/_build/images/SPARK-Cluster-Worker.png diff --git a/resources/images/SPARK-Design.png b/docs/_build/images/SPARK-Design.png similarity index 100% rename from resources/images/SPARK-Design.png rename to docs/_build/images/SPARK-Design.png diff --git a/resources/images/SPARK-Pyspark-Jobs.png b/docs/_build/images/SPARK-Pyspark-Jobs.png similarity index 100% rename from resources/images/SPARK-Pyspark-Jobs.png rename to docs/_build/images/SPARK-Pyspark-Jobs.png diff --git a/resources/images/SPARK-Pyspark-SQL-Details.png b/docs/_build/images/SPARK-Pyspark-SQL-Details.png similarity index 100% rename from resources/images/SPARK-Pyspark-SQL-Details.png rename to docs/_build/images/SPARK-Pyspark-SQL-Details.png diff --git a/resources/images/SPARK-Pyspark-SQL.png b/docs/_build/images/SPARK-Pyspark-SQL.png similarity index 100% rename from resources/images/SPARK-Pyspark-SQL.png rename to docs/_build/images/SPARK-Pyspark-SQL.png diff --git a/resources/images/SPARK-Pyspark-Stages-One-DAGViz.png b/docs/_build/images/SPARK-Pyspark-Stages-One-DAGViz.png similarity index 100% rename from resources/images/SPARK-Pyspark-Stages-One-DAGViz.png rename to docs/_build/images/SPARK-Pyspark-Stages-One-DAGViz.png diff --git a/resources/images/SPARK-Pyspark-Stages-One.png b/docs/_build/images/SPARK-Pyspark-Stages-One.png similarity index 100% rename from resources/images/SPARK-Pyspark-Stages-One.png rename to docs/_build/images/SPARK-Pyspark-Stages-One.png diff --git a/resources/images/SPARK-Pyspark-Stages.png b/docs/_build/images/SPARK-Pyspark-Stages.png similarity index 100% rename from resources/images/SPARK-Pyspark-Stages.png rename to docs/_build/images/SPARK-Pyspark-Stages.png diff --git a/resources/images/SPARK-Pyspark-UI.png b/docs/_build/images/SPARK-Pyspark-UI.png similarity index 100% rename from resources/images/SPARK-Pyspark-UI.png rename to docs/_build/images/SPARK-Pyspark-UI.png diff --git a/resources/images/SPARK-UI.png b/docs/_build/images/SPARK-UI.png similarity index 100% rename from resources/images/SPARK-UI.png rename to docs/_build/images/SPARK-UI.png diff --git a/docs/_build/images/logo/favicon.ico b/docs/_build/images/logo/favicon.ico new file mode 100644 index 0000000..fcc11f2 Binary files /dev/null and b/docs/_build/images/logo/favicon.ico differ diff --git a/docs/_build/images/logo/logo.png b/docs/_build/images/logo/logo.png new file mode 100644 index 0000000..5c45c73 Binary files /dev/null and b/docs/_build/images/logo/logo.png differ diff --git a/docs/_build/installation.html b/docs/_build/installation.html new file mode 100644 index 0000000..a7eb312 --- /dev/null +++ b/docs/_build/installation.html @@ -0,0 +1,284 @@ +--- +title: |- + Installation +pagenum: 1 +prev_page: + url: /introduction.html +next_page: + url: /architecture/elasticsearch.html +suffix: .md +search: helk info docker installation spark cybrwardg mb kibana pulling ksql jupyter server e elasticsearch t o kb kafka tcp xx creating elastalert minutes gib set master done bash script following elastic logstash zookeeper ago p pluginsservice loaded module version access gb elk hour mib url option log nginx worker install co running password logs broker b des ip ngnix sudo default want usr share true using vm run current basic hunting build file confluentinc cp cli useconcmarksweepgc node ubuntu github ce compose sure bit available includes helks helkinstall license ui local monitor container n name data however work centos supported + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Installation
+
+
+ +
+
+

Requirements (Please Read Carefully)

Operating System & Docker:

    +
  • Ubuntu 18.04 (preferred). However, Ubuntu 16 will work. CentOS is not fully supported but some have been able to get it to work, documentation is yet to come - so use CentOS at your own expense at the moment. However, open a GitHub issue but we cant promise we can help.
    • +
  • HELK uses the official Docker Community Edition (CE) bash script (Edge Version) to install Docker for you. The Docker CE Edge script supports the following distros: ubuntu, debian, raspbian, centos, and fedora.
    • +
  • You can see the specific distro versions supported in the script here.
    • +
  • If you have Docker & Docker-Compose already installed in your system, make sure you uninstall them to avoid old incompatible version. Let HELK use the official Docker CE Edge script execution to install Docker.
    • +
+

Processor/OS Architecture:

+

Cores:

Minimum of 4 cores (whether logical or physical)

+

Network Connection: NAT or Bridge

    +
  • IP version 4 address. IPv6 has not been tested yet.
    • +
  • Internet access
    • +
  • If using a proxy, documentation is yet to come - so use a proxy at your own expense. However, open a GitHub issue and we will try to help until it is officially documented/supported.
    • +
  • If using a VM then NAT or Bridge will work.
    • +
  • List of required domains/IPs will be listed in future documentation.
    • +
+

RAM:

There are four options, and the following are minimum requirements (include more if you are able).

+
    +
  • Option 1: 5GB includes KAFKA + KSQL + ELK + NGNIX.
    • +
  • Option 2: 5GB includes KAFKA + KSQL + ELK + NGNIX + ELASTALERT
    • +
  • Option 3: 7GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER.
    • +
  • Option 4: 8GB includes KAFKA + KSQL + ELK + NGNIX + SPARK + JUPYTER + ELASTALERT.
    • +
+

Disk:

25GB for testing purposes and 100GB+ for production (minimum)

+

Applications:

    +
  • Docker: 18.06.1-ce+ & Docker-Compose (HELK INSTALLS THIS FOR YOU)
    • +
  • Winlogbeat running on your endpoints or centralized WEF server (that your endpoints are forwarding to).
    • +
  • You can install Winlogbeat by following one of @Cyb3rWard0g posts here.
    • +
  • Winlogbeat config recommended by the HELK since it uses the Kafka output plugin and it is already pointing to the right ports with recommended options. You will just have to add your HELK's IP address.
    • +
+

HELK Download

Run the following commands to clone the HELK repo via git.

+
git clone https://github.com/Cyb3rWard0g/HELK.git
+
+

Change your current directory location to the new HELK directory, and run the helk_install.sh bash script as root.

+
cd HELK/docker
+sudo ./helk_install.sh
+
+

HELK Install

In order to make the installation of the HELK easy for everyone, the project comes with an install script named helk_install.sh. This script builds and runs everything you for HELK automatically. During the installation process, the script will allow you to set up the following:

+
    +
  • Set the HELK's option. For this document we are going to use option 2 (ELK + KSQL + Elastalert + Spark + Jupyter)
    • +
  • Set the Kibana User's password. Default user is helk
    • +
  • Set the HELK's IP. By default you can confirm that you want to use your HOST IP address for the HELK, unless you want to use a different one. Press [Return] or let the script continue on its own (30 Seconds sleep).
    • +
  • Set the HELK's License Subscription. By default the HELK has the basic subscription selected. You can set it to trial if you want. If you want to learn more about subscriptions go here
      +
    • If the license is set to trial, HELK asks you to set the password for the elastic account.
      • +
    +
    • +
+ +
**********************************************
+**          HELK - THE HUNTING ELK          **
+**                                          **
+** Author: Roberto Rodriguez (@Cyb3rWard0g) **
+** HELK build version: v0.1.7-alpha02262019 **
+** HELK ELK version: 6.6.1                  **
+** License: GPL-3.0                         **
+**********************************************
+
+[HELK-INSTALLATION-INFO] HELK being hosted on a Linux box
+[HELK-INSTALLATION-INFO] Available Memory: 12463 MBs
+[HELK-INSTALLATION-INFO] You're using ubuntu version xenial
+
+*****************************************************
+*      HELK - Docker Compose Build Choices          *
+*****************************************************
+
+1. KAFKA + KSQL + ELK + NGNIX + ELASTALERT
+2. KAFKA + KSQL + ELK + NGNIX + ELASTALERT + SPARK + JUPYTER
+
+Enter build choice [ 1 - 2]: 2
+[HELK-INSTALLATION-INFO] HELK build set to 2
+[HELK-INSTALLATION-INFO] Set HELK elastic subscription (basic or trial): basic
+[HELK-INSTALLATION-INFO] Set HELK IP. Default value is your current IP: 192.168.64.138
+[HELK-INSTALLATION-INFO] Set HELK Kibana UI Password: hunting
+[HELK-INSTALLATION-INFO] Verify HELK Kibana UI Password: hunting
+[HELK-INSTALLATION-INFO] Docker already installed
+[HELK-INSTALLATION-INFO] Making sure you assigned enough disk space to the current Docker base directory
+[HELK-INSTALLATION-INFO] Available Docker Disk: 67 GBs
+[HELK-INSTALLATION-INFO] Installing docker-compose..
+[HELK-INSTALLATION-INFO] Checking local vm.max_map_count variable and setting it to 4120294
+[HELK-INSTALLATION-INFO] Building & running HELK from helk-kibana-notebook-analysis-basic.yml file..
+[HELK-INSTALLATION-INFO] Waiting for some services to be up .....
+....
+......
+

Monitor HELK installation Logs (Always)

Once the installation kicks in, it will start showing you pre-defined messages about the installation, but no all the details of what is actually happening in the background. It is designed that way to keep your main screen clean and let you know where it is in the installation process.

+

What I recommend to do all the time is to open another shell and monitor the HELK installation logs by using the tail command and pointing it to the /var/log/helk-install.log file that gets created by the helk_install script as soon as it is run. This log file is available on your local host even if you are deploying the HELK via Docker (I want to make sure it is clear that it is a local file).

+
tail -f /var/log/helk-install.log
+
+ +
Creating network "docker_helk" with driver "bridge"
+Creating volume "docker_esdata" with local driver
+Pulling helk-elasticsearch (docker.elastic.co/elasticsearch/elasticsearch:6.6.1)...
+6.6.1: Pulling from elasticsearch/elasticsearch
+Pulling helk-kibana (docker.elastic.co/kibana/kibana:6.6.1)...
+6.6.1: Pulling from kibana/kibana
+Pulling helk-logstash (docker.elastic.co/logstash/logstash:6.6.1)...
+6.6.1: Pulling from logstash/logstash
+Pulling helk-jupyter (cyb3rward0g/helk-jupyter:0.1.2)...
+0.1.2: Pulling from cyb3rward0g/helk-jupyter
+Pulling helk-nginx (cyb3rward0g/helk-nginx:0.0.7)...
+0.0.7: Pulling from cyb3rward0g/helk-nginx
+Pulling helk-spark-master (cyb3rward0g/helk-spark-master:2.4.0-a)...
+2.4.0-a: Pulling from cyb3rward0g/helk-spark-master
+Pulling helk-spark-worker (cyb3rward0g/helk-spark-worker:2.4.0-a)...
+2.4.0-a: Pulling from cyb3rward0g/helk-spark-worker
+Pulling helk-zookeeper (cyb3rward0g/helk-zookeeper:2.1.0)...
+2.1.0: Pulling from cyb3rward0g/helk-zookeeper
+Pulling helk-kafka-broker (cyb3rward0g/helk-kafka-broker:2.1.0)...
+2.1.0: Pulling from cyb3rward0g/helk-kafka-broker
+Pulling helk-ksql-server (confluentinc/cp-ksql-server:5.1.2)...
+5.1.2: Pulling from confluentinc/cp-ksql-server
+Pulling helk-ksql-cli (confluentinc/cp-ksql-cli:5.1.2)...
+5.1.2: Pulling from confluentinc/cp-ksql-cli
+Pulling helk-elastalert (cyb3rward0g/helk-elastalert:0.2.1)...
+0.2.1: Pulling from cyb3rward0g/helk-elastalert
+Creating helk-elasticsearch ... done
+Creating helk-kibana        ... done
+Creating helk-logstash      ... done
+Creating helk-spark-master  ... done
+Creating helk-elastalert    ... done
+Creating helk-zookeeper     ... done
+Creating helk-jupyter       ... done
+Creating helk-spark-worker  ... done
+Creating helk-kafka-broker  ... done
+Creating helk-nginx         ... done
+Creating helk-ksql-server   ... done
+Creating helk-ksql-cli      ... done
+

Once you see that the containers have been created you can check all the containers running by executing the following:

+
sudo docker ps
+
+ +
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                                      NAMES
+968576241e9c        confluentinc/cp-ksql-server:5.1.2                     "/etc/confluent/dock…"   28 minutes ago      Up 26 minutes       0.0.0.0:8088->8088/tcp                                     helk-ksql-server
+154593559d13        cyb3rward0g/helk-kafka-broker:2.1.0                   "./kafka-entrypoint.…"   28 minutes ago      Up 26 minutes       0.0.0.0:9092->9092/tcp                                     helk-kafka-broker
+d883541a64f1        cyb3rward0g/helk-nginx:0.0.7                          "/opt/helk/scripts/n…"   About an hour ago   Up 26 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp                   helk-nginx
+527ef236543a        cyb3rward0g/helk-spark-worker:2.4.0-a                 "./spark-worker-entr…"   About an hour ago   Up 26 minutes                                                                  helk-spark-worker
+27cfaf7a8e84        cyb3rward0g/helk-jupyter:0.1.2                        "./jupyter-entrypoin…"   About an hour ago   Up 26 minutes       8000/tcp, 8888/tcp                                         helk-jupyter
+75002248e916        cyb3rward0g/helk-zookeeper:2.1.0                      "./zookeeper-entrypo…"   About an hour ago   Up 26 minutes       2181/tcp, 2888/tcp, 3888/tcp                               helk-zookeeper
+ee0120167ffa        cyb3rward0g/helk-elastalert:0.2.1                     "./elastalert-entryp…"   About an hour ago   Up 26 minutes                                                                  helk-elastalert
+4dc2722cdd53        cyb3rward0g/helk-spark-master:2.4.0-a                 "./spark-master-entr…"   About an hour ago   Up 26 minutes       7077/tcp, 0.0.0.0:8080->8080/tcp                           helk-spark-master
+9c1eb230b0ff        docker.elastic.co/logstash/logstash:6.6.1             "/usr/share/logstash…"   About an hour ago   Up 26 minutes       0.0.0.0:5044->5044/tcp, 0.0.0.0:8531->8531/tcp, 9600/tcp   helk-logstash
+f018f16d9792        docker.elastic.co/kibana/kibana:6.6.1                 "/usr/share/kibana/s…"   About an hour ago   Up 26 minutes       5601/tcp                                                   helk-kibana
+6ec5779e9e01        docker.elastic.co/elasticsearch/elasticsearch:6.6.1   "/usr/share/elastics…"   About an hour ago   Up 26 minutes       9200/tcp, 9300/tcp                                         helk-elasticsearch
+

If you want to monitor the resources being utilized (Memory, CPU, etc), you can run the following:

+
sudo docker stats --all
+
+ +
CONTAINER ID        NAME                 CPU %               MEM USAGE / LIMIT     MEM %               NET I/O             BLOCK I/O           PIDS
+ba46d256ee18        helk-ksql-cli        0.00%               0B / 0B               0.00%               0B / 0B             0B / 0B             0
+968576241e9c        helk-ksql-server     1.43%               242MiB / 12.62GiB     1.87%               667kB / 584kB       96.1MB / 73.7kB     29
+154593559d13        helk-kafka-broker    2.83%               318.7MiB / 12.62GiB   2.47%               1.47MB / 1.6MB      50.7MB / 2.01MB     67
+d883541a64f1        helk-nginx           0.10%               3.223MiB / 12.62GiB   0.02%               14.7MB / 14.8MB     9.35MB / 12.3kB     5
+527ef236543a        helk-spark-worker    0.43%               177.7MiB / 12.62GiB   1.38%               19.5kB / 147kB      37.1MB / 32.8kB     28
+27cfaf7a8e84        helk-jupyter         0.12%               45.42MiB / 12.62GiB   0.35%               1.64kB / 0B         66.3MB / 733kB      9
+75002248e916        helk-zookeeper       0.26%               62.6MiB / 12.62GiB    0.48%               150kB / 118kB       2.75MB / 172kB      23
+ee0120167ffa        helk-elastalert      2.60%               40.97MiB / 12.62GiB   0.32%               12MB / 17.4MB       38.3MB / 8.19kB     1
+4dc2722cdd53        helk-spark-master    0.50%               187.2MiB / 12.62GiB   1.45%               148kB / 17.8kB      52.3MB / 32.8kB     28
+9c1eb230b0ff        helk-logstash        15.96%              1.807GiB / 12.62GiB   14.32%              871kB / 110MB       165MB / 2.95MB      62
+f018f16d9792        helk-kibana          2.73%               179.1MiB / 12.62GiB   1.39%               3.71MB / 17.6MB     250MB / 4.1kB       13
+6ec5779e9e01        helk-elasticsearch   12.56%              2.46GiB / 12.62GiB    19.50%              130MB / 15.8MB      293MB / 226MB       61
+

You should also monitor the logs of each container while they are being initialized:

+

Just run the following:

+
sudo docker logs --follow helk-elasticsearch
+
+ +
[HELK-ES-DOCKER-INSTALLATION-INFO] Setting ES_JAVA_OPTS to -Xms1200m -Xmx1200m -XX:-UseConcMarkSweepGC -XX:-UseCMSInitiatingOccupancyOnly -XX:+UseG1GC
+[HELK-ES-DOCKER-INSTALLATION-INFO] Setting Elastic license to basic
+[HELK-ES-DOCKER-INSTALLATION-INFO] Running docker-entrypoint script..
+OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
+OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
+[2019-03-16T17:13:58,710][INFO ][o.e.e.NodeEnvironment    ] [helk-1] using [1] data paths, mounts [[/usr/share/elasticsearch/data (/dev/sda1)]], net usable_space [60.7gb], net total_space [72.7gb], types [ext4]
+[2019-03-16T17:13:58,722][INFO ][o.e.e.NodeEnvironment    ] [helk-1] heap size [1.1gb], compressed ordinary object pointers [true]
+[2019-03-16T17:13:58,728][INFO ][o.e.n.Node               ] [helk-1] node name [helk-1], node ID [En7HptZKTNmv4R6-Qb99UA]
+[2019-03-16T17:13:58,729][INFO ][o.e.n.Node               ] [helk-1] version[6.6.1], pid[12], build[default/tar/1fd8f69/2019-02-13T17:10:04.160291Z], OS[Linux/4.4.0-116-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/11.0.1/11.0.1+13]
+[2019-03-16T17:13:58,734][INFO ][o.e.n.Node               ] [helk-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-7720073513605769733, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Xms1200m, -Xmx1200m, -XX:-UseConcMarkSweepGC, -XX:-UseCMSInitiatingOccupancyOnly, -XX:+UseG1GC, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=tar]
+[2019-03-16T17:14:03,510][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [aggs-matrix-stats]
+[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [analysis-common]
+[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [ingest-common]
+[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [lang-expression]
+[2019-03-16T17:14:03,517][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [lang-mustache]
+[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [lang-painless]
+[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [mapper-extras]
+[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [parent-join]
+[2019-03-16T17:14:03,518][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [percolator]
+[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [rank-eval]
+[2019-03-16T17:14:03,519][INFO ][o.e.p.PluginsService     ] [helk-1] loaded module [reindex]
+..
+....
+

All you need to do now for the other ones is just replace helk-elasticsearch with the specific containers name:

+
sudo docker logs --follow <container name>
+
+

Remember that you can also access your docker images by running the following commands:

+
sudo docker exec -ti helk-elasticsearch bash
+
+ +
root@7a9d6443a4bf:/opt/helk/scripts#
+

Final Details

Once your HELK installation ends, you will be presented with information that you will need to access the HELK and all its other components.

+

You will get the following information:

+ +
***********************************************************************************
+** [HELK-INSTALLATION-INFO] HELK WAS INSTALLED SUCCESSFULLY                      **
+** [HELK-INSTALLATION-INFO] USE THE FOLLOWING SETTINGS TO INTERACT WITH THE HELK **
+***********************************************************************************
+
+HELK KIBANA URL: https://192.168.64.138
+HELK KIBANA USER: helk
+HELK KIBANA PASSWORD: hunting
+HELK SPARK MASTER UI: http://192.168.64.138:8080
+HELK JUPYTER SERVER URL: http://192.168.64.138/jupyter
+HELK JUPYTER CURRENT TOKEN: e8e83f5c9fe93882a970ce352d566adfb032b0975549449c
+HELK ZOOKEEPER: 192.168.64.138:2181
+HELK KSQL SERVER: 192.168.64.138:8088
+
+IT IS HUNTING SEASON!!!!!
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
TypeDescription
HELK KIBANA URLURL to access the Kibana server. You will need to copy that and paste it in your browser to access Kibana. Make sure you use https since Kibana is running behind NGINX via port 443 with a self-signed certificate
HELK KIBANA USER & PASSWORDCredentials used to access Kibana
HELK SPARK MASTER UIURL to access the Spark Master server (Spark Standalone). That server manages the Spark Workers used during execution of code by Jupyter Notebooks. Spark Master acts as a proxy to Spark Workers and applications running
HELK JUPYTER SERVER URLURL to access the Jupyter notebook server.
HELK JUPYTER CURRENT TOKENJupyter token to log in instead of providing a password
ZOOKEEPERURL for the kafka cluster zookeeper
KSQL SERVERURL to access the KSQL server and send SQL queries to the data in the kafka brokers
+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_build/introduction.html b/docs/_build/introduction.html new file mode 100644 index 0000000..2c379eb --- /dev/null +++ b/docs/_build/introduction.html @@ -0,0 +1,80 @@ +--- +title: |- + Introduction +pagenum: 0 +prev_page: + url: +next_page: + url: /installation.html +suffix: .md +search: open com source img helk spark data src href svg class left github cybrwardg twitter apache shields io license hunting jupyter notebooks elasticsearch engine thehelk badges analytics capabilities such sql streaming scalable platform features kafka allows general build gnu gpl badge issues follow style v blob master stability div images design elk hunt language structured via research community share basics system designed fast text real hadoop libraries cluster provides java python code ksql processing elastalert sigma neuron adding integration integrate add introduction www org licenses gplv blue q isaissueisaclosedimg closed thehelkimg sociallabel ellerbrock frapsoft os mkenney software guides md alphaimg + +comment: "***PROGRAMMATICALLY GENERATED, DO NOT EDIT. SEE ORIGINAL FILES IN /content***" +--- + +
+
Introduction
+
+
+ +
+
+

+ + + +

+

+

The Hunting ELK or simply the HELK is one of the first open source hunt platforms with advanced analytics capabilities such as SQL declarative language, graphing, structured streaming, and even machine learning via Jupyter notebooks and Apache Spark over an ELK stack. This project was developed primarily for research, but due to its flexible design and core components, it can be deployed in larger environments with the right configurations and scalable infrastructure.

+

Goals

    +
  • Provide an open source hunting platform to the community and share the basics of Threat Hunting.
    • +
  • Expedite the time it takes to deploy a hunt platform.
    • +
  • Improve the testing and development of hunting use cases in an easier and more affordable way.
    • +
  • Enable Data Science capabilities while analyzing data via Apache Spark, GraphFrames & Jupyter Notebooks.
    • +
+

Main Features

    +
  • Kafka: A distributed publish-subscribe messaging system that is designed to be fast, scalable, fault-tolerant, and durable.
    • +
  • Elasticsearch: A highly scalable open-source full-text search and analytics engine.
    • +
  • Logstash: A data collection engine with real-time pipelining capabilities.
    • +
  • Kibana: An open source analytics and visualization platform designed to work with Elasticsearch.
    • +
  • ES-Hadoop: An open-source, stand-alone, self-contained, small library that allows Hadoop jobs (whether using Map/Reduce or libraries built upon it such as Hive, Pig or Cascading or new upcoming libraries like Apache Spark ) to interact with Elasticsearch.
    • +
  • Spark: A fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs.
    • +
  • Jupyter Notebooks: An open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text.
    • +
+

Optional Features

    +
  • KSQL: Confluent KSQL is the open source, streaming SQL engine that enables real-time data processing against Apache Kafka®. It provides an easy-to-use, yet powerful interactive SQL interface for stream processing on Kafka, without the need to write code in a programming language such as Java or Python
    • +
  • Elastalert: ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch
    • +
  • Sigma: Sigma is a generic and open signature format that allows you to describe relevant log events in a straightforward manner.
    • +
+

Author

+

Current Committers

+

Contributing

There are a few things that I would like to accomplish with the HELK as shown in the To-Do list below. I would love to make the HELK a stable build for everyone in the community. If you are interested on making this build a more robust one and adding some cool features to it, PLEASE feel free to submit a pull request. #SharingIsCaring

+

TO-Do

    +
  • [ ] Kubernetes Cluster Migration
    • +
  • [ ] OSQuery Data Ingestion
    • +
  • [ ] MITRE ATT&CK mapping to logs or dashboards
    • +
  • [ ] Cypher for Apache Spark Integration (Adding option for Zeppelin Notebook)
    • +
  • [ ] Test and integrate neo4j spark connectors with build
    • +
  • [ ] Add more network data sources (i.e Bro)
    • +
  • [ ] Research & integrate spark structured direct streaming
    • +
  • [ ] Packer Images
    • +
  • [ ] Terraform integration (AWS, Azure, GC)
    • +
  • [ ] Add more Jupyter Notebooks to teach the basics
    • +
  • [ ] Auditd beat intergation
    • +
+

License: GPL-3.0

HELK's GNU General Public License

+ +
+
+
+
+ + + + +
+ \ No newline at end of file diff --git a/docs/_config.yml b/docs/_config.yml new file mode 100755 index 0000000..8ff64b1 --- /dev/null +++ b/docs/_config.yml @@ -0,0 +1,165 @@ +# Welcome to Jekyll! +# +# This config file is meant for settings that affect your whole blog, values +# which you are expected to set up once and rarely edit after that. If you find +# yourself editing this file very often, consider using Jekyll's data files +# feature for the data you need to update frequently. +# +# For technical reasons, this file is *NOT* reloaded automatically when you use +# 'bundle exec jekyll serve'. If you change this file, please restart the server process. + +# Site settings +# These are used to personalize your new site. If you look in the HTML files, +# you will see them accessed via {{ site.title }}, {{ site.email }}, and so on. +# You can create any custom variable you would like, and they will be accessible +# in the templates via {{ site.myvariable }}. + +####################################################################################### +# Jekyll site settings +title: HELK +author: Roberto Rodriguez +email: #myemail +description: The HELK Docs. + +baseurl: / # the subpath of your site, e.g. /blog. If there is no subpath for your site, use an empty string "" +url: https://thehelk.com # the base hostname & protocol for your site, e.g. http://example.com + + +####################################################################################### +# Jupyter Book settings + +# Main page settings +footer_text: This page was created by Roberto Rodriguez @Cyb3rWard0g + +# Sidebar settings +show_sidebar: true # Show the sidebar. Only set to false if your only wish to host a single page. +collapse_inactive_chapters: true # Whether to collapse the inactive chapters in the sidebar +collapse_inactive_sections: true # Whether to collapse the sub-sections within a non-active section in the sidebar +textbook_logo: images/logo/logo.png # A logo to be displayed at the top of your textbook sidebar. Should be square +textbook_logo_link: https://thehelk.com # A link for the logo. +sidebar_footer_text: Powered by Jupyter Book +number_toc_chapters: false # Whether to add numbers to chapterse in your Table of Contents. If true, you can control this at the Chapter level in _data/toc.yml + +# Search settings +search_max_words_in_content: 100 # In the search function, use at most this many words (too many words will make search slow) + +# Controlling page information +page_titles: infer # Either `None`, `infer`, or `toc` +page_authors: infer # Either `None` or `infer` +filename_title_split_character: _ # If inferring titles based on filename, splt on this character. + +# Math settings +number_equations: false # Whether to automatically number all block equations with MathJax + +####################################################################################### +# Interact link settings + +# General interact settings +use_jupyterlab: false # If 'true', interact links will use JupyterLab as the interface + +# Jupyterhub link settings +use_jupyterhub_button: false # If 'true', display a button that will direct users to a JupyterHub (that you provide) +jupyterhub_url: '' # The URL for your JupyterHub. If no URL, use "" +jupyterhub_interact_text: Interact # The text that interact buttons will contain. + +# Binder link settings +use_binder_button: false # If 'true', add a binder button for interactive links +binderhub_url: # The URL for your BinderHub. If no URL, use "" +binder_repo_base: # The site on which the textbook repository is hosted +binder_repo_org: # The username or organization that owns this repository +binder_repo_name: # The name of the repository on the web +binder_repo_branch: # The branch on which your textbook is hosted. +binderhub_interact_text: # The text that interact buttons will contain. + +# Thebelab settings +use_thebelab_button: false # If 'true', display a button to allow in-page running code cells with Thebelab +thebelab_button_text: Thebelab # The text to display inside the Thebelab initialization button +codemirror_theme: abcdef # Theme for codemirror cells, for options see https://codemirror.net/doc/manual.html#config + +# nbinteract settings +use_show_widgets_button: true # If 'true', display a button to allow in-page running code cells with nbinteract + +# Download settings +use_download_button: true # If 'true', display a button to download a zip file for the notebook +download_button_text: Download # The text that download buttons will contain +download_page_header: Made with Jupyter Book # A header that will be displayed at the top of and PDF-printed page + +####################################################################################### +# Jupyter book extensions and additional features + +# Bibliography and citation settings. See https://github.com/inukshuk/jekyll-scholar#configuration for options +scholar: + style: apa + +####################################################################################### +# Option to add a Goggle analytics tracking code + +# Navigate to https://analytics.google.com, add a new property for your jupyter book and copy the tracking id here. +#google_analytics: +# mytrackingcode: UA-52617120-7 + +####################################################################################### +# Jupyter book settings you probably don't need to change + +google_analytics: + mytrackingcode: '' + +####################################################################################### +# Jupyter book settings you probably don't need to change + +content_folder_name: content # The folder where your raw content (notebooks/markdown files) are located +images_url: /assets/images # Path to static image files +css_url: /assets/css # Path to static CSS files +js_url: /assets/js # Path to JS files +custom_static_url: /assets/custom # Path to user's custom CSS/JS files + + +####################################################################################### +# Jekyll build settings (only modify if you know what you're doing) + +# Site settings +defaults: +- scope: + path: '' + values: + layout: default + toc: true + toc_label: ' On this page' + toc_icon: list-ul + excerpt: '' + +favicon_path: images/logo/favicon.ico + +# Markdown Processing +markdown: kramdown +kramdown: + input: GFM + syntax_highlighter: rouge + +sass: + style: compressed + +collections: + build: + output: true + permalink: /:path.html + +# Exclude from processing. +# The following items will not be processed, by default. Create a custom list +# to override the default setting. +exclude: +- scripts/ +- Gemfile +- Gemfile.lock +- node_modules +- vendor/bundle/ +- vendor/cache/ +- vendor/gems/ +- vendor/ruby/ + +plugins: +- jekyll-redirect-from +- jekyll-scholar + +# Jupyter Book version - DO NOT CHANGE THIS. It is generated when a new book is created +jupyter_book_version: 0.6.4 diff --git a/docs/_data/toc.yml b/docs/_data/toc.yml new file mode 100644 index 0000000..2067bbf --- /dev/null +++ b/docs/_data/toc.yml @@ -0,0 +1,29 @@ +- url: /introduction +- url: /installation +- title: GitHub Repository + url: https://github.com/Cyb3rWard0g/HELK + external: true +- divider: true +- header: Architecture +- url: /architecture/elasticsearch +- url: /architecture/logstash +- url: /architecture/kibana +- divider: true +- header: How-To +- url: /how-to/docker/docker + sections: + - url: /how-to/docker/docker-export-images + - url: /how-to/docker/docker-load-images +- url: /how-to/logstash/logstash + sections: + - url: /how-to/logstash/logstash-create-plugins-offline +- url: /how-to/kafka/kafka + sections: + - url: /how-to/kafka/kafka-topic-ingestion + - url: /how-to/kafka/kafka-update-ip +- url: /how-to/ksql/ksql + sections: + - url: /how-to/ksql/ksql-deploy-locally +- url: /how-to/winlogbeat/winlogbeat + sections: + - url: /how-to/winlogbeat/winlogbeat-shipping \ No newline at end of file diff --git a/docs/_includes/buttons.html b/docs/_includes/buttons.html new file mode 100755 index 0000000..03b36c4 --- /dev/null +++ b/docs/_includes/buttons.html @@ -0,0 +1,9 @@ +
+{% include buttons/download.html %} +{% if page.interact_link %} + {% include buttons/thebelab.html %} + {% include buttons/nbinteract.html %} + {% include buttons/binder.html %} + {% include buttons/jupyterhub.html %} +{% endif %} +
diff --git a/docs/_includes/buttons/binder.html b/docs/_includes/buttons/binder.html new file mode 100755 index 0000000..c9577f1 --- /dev/null +++ b/docs/_includes/buttons/binder.html @@ -0,0 +1,14 @@ +{% if site.use_binder_button %} + +{% if site.use_jupyterlab %} + {% assign binder_interact_prefix="urlpath=lab/tree/" %} +{% else %} + {% assign binder_interact_prefix="filepath=" %} +{% endif %} + +{% capture interact_url_binder %}v2/gh/{{ site.binder_repo_org }}/{{ site.binder_repo_name }}/{{ site.binder_repo_branch }}?{{ binder_interact_prefix }}{{ page.interact_link | url_encode }}{% endcapture %} +{% capture interact_icon_binder %}{{ site.images_url | relative_url }}/logo_binder.svg{% endcapture %} + + + +{%- endif %} \ No newline at end of file diff --git a/docs/_includes/buttons/download.html b/docs/_includes/buttons/download.html new file mode 100755 index 0000000..190f011 --- /dev/null +++ b/docs/_includes/buttons/download.html @@ -0,0 +1,13 @@ +{% if site.use_download_button -%} +
+ +
+ {% if page.interact_link -%} + + + + {% endif %} + +
+
+{%- endif %} diff --git a/docs/_includes/buttons/jupyterhub.html b/docs/_includes/buttons/jupyterhub.html new file mode 100755 index 0000000..8e17d31 --- /dev/null +++ b/docs/_includes/buttons/jupyterhub.html @@ -0,0 +1,13 @@ +{% if site.use_jupyterhub_button %} + +{% if site.use_jupyterlab %} + {% assign hub_app="lab" %} +{% else %} + {% assign hub_app="notebook" %} +{% endif %} + +{% capture interact_url_jupyterhub %}hub/user-redirect/git-pull?repo={{ site.binder_repo_base }}/{{ site.binder_repo_org }}/{{ site.binder_repo_name }}&branch={{ site.binder_repo_branch }}&subPath={{ page.interact_link | url_encode }}&app={{ hub_app }}{% endcapture %} +{% capture interact_icon_jupyterhub %}{{ site.images_url | relative_url }}/logo_jupyterhub.svg{% endcapture %} + + +{% endif %} diff --git a/docs/_includes/buttons/nbinteract.html b/docs/_includes/buttons/nbinteract.html new file mode 100755 index 0000000..eaf6e83 --- /dev/null +++ b/docs/_includes/buttons/nbinteract.html @@ -0,0 +1,3 @@ +{% if site.use_show_widgets_button and page.has_widgets -%} + +{% endif %} \ No newline at end of file diff --git a/docs/_includes/buttons/thebelab.html b/docs/_includes/buttons/thebelab.html new file mode 100755 index 0000000..44a8322 --- /dev/null +++ b/docs/_includes/buttons/thebelab.html @@ -0,0 +1,3 @@ +{% if site.use_thebelab_button -%} + +{% endif %} \ No newline at end of file diff --git a/docs/_includes/css_entry.scss b/docs/_includes/css_entry.scss new file mode 100755 index 0000000..f693426 --- /dev/null +++ b/docs/_includes/css_entry.scss @@ -0,0 +1,18 @@ +@import 'inuitcss/settings/settings.core'; +@import 'settings/settings.global.scss'; + +@import 'inuitcss/tools/tools.font-size'; +@import 'inuitcss/tools/tools.clearfix'; +@import 'inuitcss/tools/tools.hidden'; +@import 'inuitcss/tools/tools.mq'; + +@import 'inuitcss/elements/elements.page'; +@import 'inuitcss/elements/elements.headings'; +@import 'inuitcss/elements/elements.images'; +@import 'inuitcss/elements/elements.tables'; +@import 'elements/elements.typography'; +@import 'elements/elements.syntax-highlighting'; +@import 'elements/elements.tables'; +@import 'elements/elements.links'; + +@import 'components/components.textbook__page'; diff --git a/docs/_includes/fb_tags.html b/docs/_includes/fb_tags.html new file mode 100755 index 0000000..1ff5bc3 --- /dev/null +++ b/docs/_includes/fb_tags.html @@ -0,0 +1,7 @@ + + + + + + + diff --git a/docs/_includes/footer.html b/docs/_includes/footer.html new file mode 100755 index 0000000..a42f4ea --- /dev/null +++ b/docs/_includes/footer.html @@ -0,0 +1,3 @@ +
+

{{ site.footer_text }}

+
diff --git a/docs/_includes/google_analytics.html b/docs/_includes/google_analytics.html new file mode 100755 index 0000000..23b6bd1 --- /dev/null +++ b/docs/_includes/google_analytics.html @@ -0,0 +1,11 @@ +{% if site.google_analytics.mytrackingcode %} + + + +{% endif %} diff --git a/docs/_includes/head.html b/docs/_includes/head.html new file mode 100755 index 0000000..e4b63f4 --- /dev/null +++ b/docs/_includes/head.html @@ -0,0 +1,89 @@ + + + + + + {% if page.title %}{{ page.title | escape }}{% else %}{{ site.title | escape }}{% endif %} + + + + + + {% include fb_tags.html %} + + + + + + + + + + + + + + + {% include mathjax.html %} + + + + + + + + + + + + + + + + + + + + + {% include js/nbinteract.html %} + + + {% include js/thebelab.html %} + + + + + + + {% include google_analytics.html %} + + + + + + + + + + + + + {% include js/interact-update.html %} + + + + + + + + + + + + + {% include js/print.html %} + diff --git a/docs/_includes/js/interact-update.html b/docs/_includes/js/interact-update.html new file mode 100755 index 0000000..e93c5ca --- /dev/null +++ b/docs/_includes/js/interact-update.html @@ -0,0 +1,142 @@ +{% if site.use_jupyterhub_button or site.use_binder_button %} + +{% endif %} \ No newline at end of file diff --git a/docs/_includes/js/nbinteract.html b/docs/_includes/js/nbinteract.html new file mode 100755 index 0000000..97772c6 --- /dev/null +++ b/docs/_includes/js/nbinteract.html @@ -0,0 +1,33 @@ +{% if site.use_show_widgets_button and page.has_widgets %} + + + + +{% endif %} \ No newline at end of file diff --git a/docs/_includes/js/print.html b/docs/_includes/js/print.html new file mode 100755 index 0000000..9cda3ac --- /dev/null +++ b/docs/_includes/js/print.html @@ -0,0 +1,32 @@ + + + diff --git a/docs/_includes/js/thebelab-cell-button.html b/docs/_includes/js/thebelab-cell-button.html new file mode 100755 index 0000000..3b9da83 --- /dev/null +++ b/docs/_includes/js/thebelab-cell-button.html @@ -0,0 +1,27 @@ +{% if site.use_thebelab_button -%} + +{% endif %} diff --git a/docs/_includes/js/thebelab-page-config.html b/docs/_includes/js/thebelab-page-config.html new file mode 100755 index 0000000..94478c7 --- /dev/null +++ b/docs/_includes/js/thebelab-page-config.html @@ -0,0 +1,32 @@ + \ No newline at end of file diff --git a/docs/_includes/js/thebelab.html b/docs/_includes/js/thebelab.html new file mode 100755 index 0000000..97b970c --- /dev/null +++ b/docs/_includes/js/thebelab.html @@ -0,0 +1,95 @@ + +{% if site.use_thebelab_button %} + + +{% include js/thebelab-cell-button.html %} + + +{% endif %} diff --git a/docs/_includes/mathjax.html b/docs/_includes/mathjax.html new file mode 100755 index 0000000..f25affe --- /dev/null +++ b/docs/_includes/mathjax.html @@ -0,0 +1,29 @@ + + + + + diff --git a/docs/_includes/metadata.json b/docs/_includes/metadata.json new file mode 100755 index 0000000..089680f --- /dev/null +++ b/docs/_includes/metadata.json @@ -0,0 +1,29 @@ +{ + "@context": "http://schema.org", + "@type": "NewsArticle", + "mainEntityOfPage": "{{ page.url | replace:'index.html','' | prepend: site.baseurl | prepend: site.url }}", + "headline": "{% if page.title %}{{ page.title | escape }}{% else %}{{ site.title | escape }}{% endif %}", + "datePublished": "{% if page.date %}{{ page.date | date_to_xmlschema }}{% else %}{{ site.time | date_to_xmlschema }}{% endif %}", + "dateModified": "{% if page.date %}{{ page.date | date_to_xmlschema }}{% else %}{{ site.time | date_to_xmlschema }}{% endif %}", + "description": "{{ page.content | strip_html | strip_newlines | truncate: 160 }}", + "author": { + "@type": "Person", + "name": "{{ site.author }}" + }, + "publisher": { + "@type": "Organization", + "name": "Data 100 at UC Berkeley", + "logo": { + "@type": "ImageObject", + "url": "{{ site.logo | prepend: site.baseurl | prepend: site.url }}", + "width": 60, + "height": 60 + } + }, + "image": { + "@type": "ImageObject", + "url": "{{ site.logo | prepend: site.baseurl | prepend: site.url }}", + "height": 60, + "width": 60 + } +} diff --git a/docs/_includes/page-nav.html b/docs/_includes/page-nav.html new file mode 100755 index 0000000..58e860e --- /dev/null +++ b/docs/_includes/page-nav.html @@ -0,0 +1,37 @@ +{% comment %} +Only the URLs from the TOC are used here. The title for +prev/next is pulled from the respective page's metadata. +We loop through the "build" collection to determine the +page title based on the *current* page's next/prev URL. +{% endcomment %} + diff --git a/docs/_includes/search/lunr/lunr-en.js b/docs/_includes/search/lunr/lunr-en.js new file mode 100755 index 0000000..b21ace9 --- /dev/null +++ b/docs/_includes/search/lunr/lunr-en.js @@ -0,0 +1,84 @@ +var initQuery = function() { + // See if we have a search box + var searchInput = document.querySelector('input#lunr_search'); + if (searchInput === null) { + return; + } + + // Function to parse our lunr cache + var idx = lunr(function () { + this.field('title') + this.field('excerpt') + this.field('categories') + this.field('tags') + this.ref('id') + + this.pipeline.remove(lunr.trimmer) + + for (var item in store) { + this.add({ + title: store[item].title, + excerpt: store[item].excerpt, + categories: store[item].categories, + tags: store[item].tags, + id: item + }) + } + }); + + // Run search upon keyup + searchInput.addEventListener('keyup', function () { + var resultdiv = document.querySelector('#results'); + var query = document.querySelector("input#lunr_search").value.toLowerCase(); + var result = + idx.query(function (q) { + query.split(lunr.tokenizer.separator).forEach(function (term) { + q.term(term, { boost: 100 }) + if(query.lastIndexOf(" ") != query.length-1){ + q.term(term, { usePipeline: false, wildcard: lunr.Query.wildcard.TRAILING, boost: 10 }) + } + if (term != ""){ + q.term(term, { usePipeline: false, editDistance: 1, boost: 1 }) + } + }) + }); + + // Empty the results div + while (resultdiv.firstChild) { + resultdiv.removeChild(resultdiv.firstChild); + } + + resultdiv.insertAdjacentHTML('afterbegin', '

'+result.length+' Result(s) found

'); + for (var item in result) { + var ref = result[item].ref; + if(store[ref].teaser){ + var searchitem = + '
'+ + '
'+ + '

'+ + ''+store[ref].title+''+ + '

'+ + '
'+ + ''+ + '
'+ + '

'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...

'+ + '
'+ + '
'; + } + else{ + var searchitem = + '
'+ + '
'+ + '

'+ + ''+store[ref].title+''+ + '

'+ + '

'+store[ref].excerpt.split(" ").splice(0,20).join(" ")+'...

'+ + '
'+ + '
'; + } + resultdiv.insertAdjacentHTML('beforeend', searchitem); + } + }); +}; + +initFunction(initQuery); diff --git a/docs/_includes/search/lunr/lunr-store.js b/docs/_includes/search/lunr/lunr-store.js new file mode 100755 index 0000000..ffa41a0 --- /dev/null +++ b/docs/_includes/search/lunr/lunr-store.js @@ -0,0 +1,28 @@ +var store = [ + {%- for c in site.collections -%} + {%- if forloop.last -%} + {%- assign l = true -%} + {%- endif -%} + {%- assign docs = c.docs | where_exp:'doc','doc.search != false' -%} + {%- for doc in docs -%} + {%- if doc.header.teaser -%} + {%- capture teaser -%}{{ doc.header.teaser }}{%- endcapture -%} + {%- else -%} + {%- assign teaser = site.teaser -%} + {%- endif -%} + { + "title": {{ doc.title | jsonify }}, + {% assign truncateWords=site.search_max_words_in_content %} + "excerpt": {{ doc.search | jsonify }}, + "categories": {{ doc.categories | jsonify }}, + "tags": {{ doc.tags | jsonify }}, + "url": {{ doc.url | absolute_url | jsonify }}, + "teaser": + {%- if teaser contains "://" -%} + {{ teaser | jsonify }} + {%- else -%} + {{ teaser | absolute_url | jsonify }} + {%- endif -%} + }{%- unless forloop.last and l -%},{%- endunless -%} + {%- endfor -%} + {%- endfor -%}] \ No newline at end of file diff --git a/docs/_includes/sidebar.html b/docs/_includes/sidebar.html new file mode 100755 index 0000000..192aa69 --- /dev/null +++ b/docs/_includes/sidebar.html @@ -0,0 +1,157 @@ +{% comment %} +Partial for the textbook sidebar. Renders each chapter and its sections from +_data/toc.yml . + +Much of the logic here is to add active classes to the currently active +section. The currently active section / chapter should be highlighted in the +sidebar. + +If a chapter or any of its sections are the current page, we should display the +chapter's sections. Otherwise, we hide the sections to keep the sidebar short. + +We also prefix the sidebar entries with the chapter/section number. We assume +a 1-level nesting; we will label 1.2, but not 1.2.1. +{% endcomment %} + +{% assign chapter_num = 1 %} + + diff --git a/docs/_includes/topbar.html b/docs/_includes/topbar.html new file mode 100755 index 0000000..3a26729 --- /dev/null +++ b/docs/_includes/topbar.html @@ -0,0 +1,23 @@ +
+ +
+ + {% include buttons.html %} +
+ + + + Search + +
diff --git a/docs/_layouts/default.html b/docs/_layouts/default.html new file mode 100755 index 0000000..5a49bb4 --- /dev/null +++ b/docs/_layouts/default.html @@ -0,0 +1,27 @@ + + + {% include head.html %} + + + {% include js/thebelab-page-config.html %} + + +
+ {% include sidebar.html %} + {% if page.search_page != true %} + {% endif %} + {% include topbar.html %} +
+
+ {{ content }} +
+
+ {% include page-nav.html %} + {% include footer.html %} +
+ +
+ + + + diff --git a/docs/_sass/components/_components.book__layout.scss b/docs/_sass/components/_components.book__layout.scss new file mode 100755 index 0000000..7caa4a4 --- /dev/null +++ b/docs/_sass/components/_components.book__layout.scss @@ -0,0 +1,151 @@ +/** + * The website contains two main components: the sidebar and the textbook page. + * This file specifies the layout and includes classes to show/hide the sidebar + * on small screens. + * + * The actual styling for the sidebar and page are located in their respective + * component SCSS files. This file manages the layout and width only. + * + * By default, the sidebar is not visible. + * + * [1]: The entire page is positioned relative so that when the page overflows + * (e.g. sidebar open on small screens) the user can't scroll left/right. + * [2]: The sidebar and the textbook page are positioned absolute so that we + * can use translate() on the textbook page to reveal the sidebar. + * [3]: Setting the background color hides the sidebar when it's behind the + * page (otherwise the page is transparent). + * + * When the sidebar is visible: + * + * [4]: Shift the textbook page over to the left. On small screens, the page + * will overflow since the sidebar takes up most of the screen. + * [5]: On larger screens, the page and sidebar have enough room to read them + * simultaneously, so make sure that the page doesn't overflow. + */ + +$left-sidebar-width: 300px; +$page-max-width: 880px; +$right-sidebar-width: 220px; +$topbar-height: 60px; + +.c-textbook { + /* [1] */ + position: relative; + height: 100vh; + overflow: hidden; + margin: 0 0 0 auto; +} + +.c-topbar { + background-color: $book-background-color; + position: fixed; + top: 0; + height: $topbar-height; + width: 100%; + left: 0; + padding: $spacing-unit-small $spacing-unit-small 0 $spacing-unit-med * 2; + z-index: 1; + transition: top 250ms, transform 250ms ease; // For animations +} + +@include mq($until: tablet) { + .c-topbar.hidetop { + // At desktop, we stop hiding the navbar + top: -250px; + } +} + +.c-textbook__sidebar, +.c-textbook__page { + /* [2] */ + height: 100vh; + overflow: auto; + position: fixed; + background-color: $book-background-color; /* [3] */ +} + +.c-textbook__sidebar { + width: $left-sidebar-width; + top: 0; + left: 0; +} + +.c-textbook__page { + + width: $textbook-page-width; + transition: transform 250ms ease; + left: 0; + padding: 0 $spacing-unit $spacing-unit-small $spacing-unit-small * 3; + overflow-x: visible; + + &:focus { + /* [2] */ + outline: none; + } +} + +.sidebar__right { + // By default we hide the sidebar + display: none; + + // Spacing for the sidebar + width: $right-sidebar-width - $spacing-unit-small; // To account for the small margin on the right + position: relative; + float: right; + z-index: 1; // Keep sidebar under page content + + @include mq($from: tablet) { + // Show right TOC at laptop size + display: block; + } +} + +.js-show-sidebar { + .c-textbook__page, .c-topbar { + /* [4] */ + transform: translate($left-sidebar-width, 0); + + @include mq($from: tablet) { + /* [5] */ + width: calc(100% - #{$left-sidebar-width}); + } + } +} + +.c-textbook__content { + clear: both; + padding-top: $topbar-height * 1.5; + width: 95%; +} + +.c-textbook__content, .c-textbook__footer { + max-width: $page-max-width; +} + +.c-page__nav { + display: flex; + justify-content: space-between; + align-items: center; + padding-top: 30px; +} + +// Make sure that the bottom content has the same width as non-sidebar content +.footer, .c-page__nav { + @include mq($from: laptop) { + width: $textbook-page-with-sidebar-width; + } +} + +// Scrollbar width +::-webkit-scrollbar { + width: 5px; + background: #f1f1f1; +} + +::-webkit-scrollbar-thumb { + background: #c1c1c1; +} + +main, nav { + scrollbar-width: thin; +} \ No newline at end of file diff --git a/docs/_sass/components/_components.book__topbar.scss b/docs/_sass/components/_components.book__topbar.scss new file mode 100755 index 0000000..d9e0ca5 --- /dev/null +++ b/docs/_sass/components/_components.book__topbar.scss @@ -0,0 +1,67 @@ +.c-topbar__label { + @include inuit-font-size(12px); + display: inline-block; + margin-left: $spacing-unit-tiny; + vertical-align: middle; + text-transform: uppercase; +} + +.c-topbar { + .hamburger, .buttons { + float: left; + } + + #js-sidebar-toggle { + margin-right: 5px; + padding-top: 4px; + } + + span.hamburger-box { + width: 40px; + height: 30px; + padding-left: 10px; + } + + .c-topbar__buttons { + @include mq($from: tablet) { + width: calc(100% - #{$right-sidebar-width} - 20px) + } + } + + .topbar-right-button { + display: block; + float: right; + padding: 0 1rem; + + img { + width: 20px; + margin-top: 4px; + } + } +} + +// Download buttons + +.download-buttons { + display: none; + position: absolute; + + button { + min-width: 100px !important; + border: 1px white solid !important; + border-radius: 0 !important; + } +} + +.download-buttons-dropdown { + position: relative; + display: inline-block; + + &:hover div.download-buttons { + display: block; + } + + img { + height: 18px; + } +} diff --git a/docs/_sass/components/_components.interact-button.scss b/docs/_sass/components/_components.interact-button.scss new file mode 100755 index 0000000..c552fa5 --- /dev/null +++ b/docs/_sass/components/_components.interact-button.scss @@ -0,0 +1,60 @@ +/** + * Stylings for Interact and Show Widget buttons. + * + * [1]: We abuse CSS selector specificity here since the buttons at the top of + * the notebook might have both .interact-button and .js=nbinteract-widget. + * [2]: We want the top buttons to be large. + * [3]: However, a .js=nbinteract-widget appearing alone midway through the + * notebook should be small. + * + */ + +$color-interact-button: #5a5a5a !default; + +%interact-button { + @include inuit-font-size(14px); + background-color: $color-interact-button; + border-radius: 3px; + border: none; + color: white; + cursor: pointer; + display: inline-block; + font-weight: 700; + /* [2] */ + padding: $spacing-unit-tiny $spacing-unit-med; + text-decoration: none; + + &:hover, + &:focus { + text-decoration: none; + } +} + +.interact-button-logo { + height: 1.35em; + padding-right: 10px; + margin-left: -5px; +} + +.buttons { + margin-bottom: $spacing-unit; + + /* [1] */ + .interact-button { + @extend %interact-button; + } +} + +.js-nbinteract-widget { + @extend %interact-button; + + /* [3] */ + padding: $spacing-unit-tiny $spacing-unit; + margin-bottom: $spacing-unit-small; +} + +// If the interact button link is changed with a REST param +div.interact-context { + display: inline; + padding-left: 1em; +} \ No newline at end of file diff --git a/docs/_sass/components/_components.page__footer.scss b/docs/_sass/components/_components.page__footer.scss new file mode 100755 index 0000000..9a4d5c4 --- /dev/null +++ b/docs/_sass/components/_components.page__footer.scss @@ -0,0 +1,7 @@ +.footer { + text-align: center; + font-size: 14px; + padding: 20px; + opacity: 0.7; + margin-bottom: 0px; +} diff --git a/docs/_sass/components/_components.page__nav.scss b/docs/_sass/components/_components.page__nav.scss new file mode 100755 index 0000000..69db85e --- /dev/null +++ b/docs/_sass/components/_components.page__nav.scss @@ -0,0 +1,19 @@ +/** + * Styling for the Next Page / Previous Page links at the bottom of textbook + * pages. + */ + +$color-nav-links: rgba(0, 140, 255, 0.7); + +.c-page__nav__prev, +.c-page__nav__next { + flex: 1; + color: $color-nav-links; + border: 1px solid $color-nav-links; + border-radius: 3px; + padding: $spacing-unit-small 0; +} + +.c-page__nav__next { + text-align: right; +} diff --git a/docs/_sass/components/_components.page__onthispage.scss b/docs/_sass/components/_components.page__onthispage.scss new file mode 100755 index 0000000..548ee2d --- /dev/null +++ b/docs/_sass/components/_components.page__onthispage.scss @@ -0,0 +1,100 @@ +/** + * Styling for the onthispage elements. + * + * [1]: The sidebar is implemented as ul and li elements so we need to remove + * the bullets and margins. Also make chapter fonts a bit bigger. + * [2]: The entries are tags so we need to remove the default styling. + * [3]: The sidebar divider is just an empty element with a border. + * [4]: The current section needs a higher specificity to override the :hover + * selectors used previously. + * [5]: The logo displayed above the sidebar + * [6]: The footer at the bottom of the sidebar + */ +$color-sidebar-bg: rgba(255, 255, 255, 0) !default; +$color-sidebar-entry: #364149 !default; +$color-sidebar-entry--active: $color-links !default; +$color-sidebar-divider: #bbb !default; + +.c-textbook__sidebar { + background-color: $color-sidebar-bg; + padding: $spacing-unit-small; + + @include inuit-font-size(14px); + border-right: 1px solid rgba(0, 0, 0, 0.07); + opacity: 0.6; + -webkit-transition: opacity 0.2s ease-in-out; + transition: opacity 0.2s ease-in-out; + + &:hover { + opacity: 1; + } +} + +/* [1] */ +.c-sidebar__chapters { + list-style: none; + margin-left: 0; + margin-bottom: 0; +} + +li.c-sidebar__chapter > a { + font-size: 1.2em; +} + +/* [1] */ +.c-sidebar__sections, .c-sidebar__subsections { + list-style: none; + margin-bottom: 0; +} + +.c-sidebar__sections { + margin-left: $spacing-unit-small; +} + +.c-sidebar__subsections { + margin-left: 20px; +} + +/* [2] */ +.c-sidebar__entry { + display: block; + + padding: $spacing-unit-tiny; + + color: $color-sidebar-entry; + text-decoration: none; + + &:hover { + text-decoration: underline; + } + + &:visited { + color: $color-sidebar-entry; + } +} + +/* [4] */ +.c-sidebar__entry--active.c-sidebar__entry--active { + color: $color-sidebar-entry--active; +} + +/* [3] */ +.c-sidebar__divider { + border-top: 1px solid $color-sidebar-divider; + margin: $spacing-unit-tiny; +} + +/* [5] */ +img.textbook_logo { + margin-top: 20px; + max-height: 100px; + margin: 0px auto 20px auto; + display: block; +} + +/* [6] */ +p.sidebar_footer { + text-align: center; + padding: 10px 20px 0px 0px; + font-size: .9em; +} \ No newline at end of file diff --git a/docs/_sass/components/_components.search.scss b/docs/_sass/components/_components.search.scss new file mode 100755 index 0000000..0f3ae89 --- /dev/null +++ b/docs/_sass/components/_components.search.scss @@ -0,0 +1,97 @@ +/* ========================================================================== + SEARCH + ========================================================================== */ + // Taken from https://github.com/mmistakes/minimal-mistakes + // Variables + $large: 1024px !default; + $x-large: 1280px !default; + $type-size-1: 2.441em !default; + $type-size-2: 1.953em !default; + $type-size-3: 1.563em !default; + $type-size-6: 0.75em !default; + $intro-transition: intro 0.3s both !default; + + // Rules + .layout--search { + .archive__item-teaser { + margin-bottom: 0.25em; + } + } + + .search__toggle { + margin-left: 1rem; + margin-right: 1rem; + border: 0; + outline: none; + color: #393e46; + background-color: transparent; + cursor: pointer; + -webkit-transition: 0.2s; + transition: 0.2s; + + &:hover { + color: #000; + } + } + + .search-icon { + width: 100%; + height: 100%; + } + + .search-content { + //display: none; + //visibility: hidden; + padding-top: 1em; + padding-bottom: 1em; + + &__inner-wrap { + width: 100%; + margin-left: auto; + margin-right: auto; + padding-left: 1em; + padding-right: 1em; + -webkit-animation: $intro-transition; + animation: $intro-transition; + -webkit-animation-delay: 0.15s; + animation-delay: 0.15s; + + .search-input { + display: block; + margin-bottom: 0; + padding: 0; + border: none; + outline: none; + box-shadow: none; + background-color: transparent; + font-size: $type-size-3; + } + } + + &.is--visible { + display: block; + visibility: visible; + + &::after { + content: ""; + display: block; + } + } + + .results__found { + margin-top: 0.5em; + font-size: $type-size-6; + } + + .archive__item { + margin-bottom: 2em; + } + + .archive__item-title { + margin-top: 0; + } + + .archive__item-excerpt { + margin-bottom: 0; + } + } \ No newline at end of file diff --git a/docs/_sass/components/_components.thebelab.scss b/docs/_sass/components/_components.thebelab.scss new file mode 100755 index 0000000..2bc6859 --- /dev/null +++ b/docs/_sass/components/_components.thebelab.scss @@ -0,0 +1,33 @@ + +.thebelab-cell { + // To ensure that thebelab cells are always the top of the Z-stack + position: relative; + z-index: 999; +} + +.thebelab-button { + z-index: 999; + display: inline-block; + padding: 0.35em 1.2em; + margin: 0px 1px; + border-radius: 0.12em; + box-sizing: border-box; + text-decoration: none; + font-family: 'Roboto', sans-serif; + font-weight: 300; + text-align: center; + transition: all 0.2s; + background-color: #dddddd; + border: 0.05em solid white; + color: #000000; +} + +.thebelab-button:hover{ + border: 0.05em solid black; + background-color: #fcfcfc; +} + + +div.jp-OutputArea-output { + padding: 5px; +} diff --git a/docs/_sass/hamburgers/_base.scss b/docs/_sass/hamburgers/_base.scss new file mode 100755 index 0000000..15f4b2d --- /dev/null +++ b/docs/_sass/hamburgers/_base.scss @@ -0,0 +1,69 @@ +// Hamburger +// ================================================== +.hamburger { + padding: $hamburger-padding-y $hamburger-padding-x; + display: inline-block; + cursor: pointer; + + transition-property: opacity, filter; + transition-duration: $hamburger-hover-transition-duration; + transition-timing-function: $hamburger-hover-transition-timing-function; + + // Normalize (