diff --git a/Days/Monitoring/Elastic Stack/README.md b/Days/Monitoring/Elastic Stack/README.md deleted file mode 100644 index b4d1ca5..0000000 --- a/Days/Monitoring/Elastic Stack/README.md +++ /dev/null @@ -1,438 +0,0 @@ -# Elastic stack (ELK) on Docker - -[![Elastic Stack version](https://img.shields.io/badge/Elastic%20Stack-8.1.0-00bfb3?style=flat&logo=elastic-stack)](https://www.elastic.co/blog/category/releases) -[![Build Status](https://github.com/deviantony/docker-elk/workflows/CI/badge.svg?branch=main)](https://github.com/deviantony/docker-elk/actions?query=workflow%3ACI+branch%3Amain) -[![Join the chat at https://gitter.im/deviantony/docker-elk](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/deviantony/docker-elk?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) - -Run the latest version of the [Elastic stack][elk-stack] with Docker and Docker Compose. - -It gives you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and -the visualization power of Kibana. - -![Animated demo](https://user-images.githubusercontent.com/3299086/155972072-0c89d6db-707a-47a1-818b-5f976565f95a.gif) - -*:information_source: The Docker images backing this stack include [X-Pack][xpack] with [paid features][paid-features] -enabled by default (see [How to disable paid features](#how-to-disable-paid-features) to disable them). **The [trial -license][trial-license] is valid for 30 days**. After this license expires, you can continue using the free features -seamlessly, without losing any data.* - -Based on the official Docker images from Elastic: - -* [Elasticsearch](https://github.com/elastic/elasticsearch/tree/master/distribution/docker) -* [Logstash](https://github.com/elastic/logstash/tree/master/docker) -* [Kibana](https://github.com/elastic/kibana/tree/master/src/dev/build/tasks/os_packages/docker_generator) - -Other available stack variants: - -* [`tls`](https://github.com/deviantony/docker-elk/tree/tls): TLS encryption enabled in Elasticsearch -* [`searchguard`](https://github.com/deviantony/docker-elk/tree/searchguard): Search Guard support - ---- - -## Philosophy - -We aim at providing the simplest possible entry into the Elastic stack for anybody who feels like experimenting with -this powerful combo of technologies. This project's default configuration is purposely minimal and unopinionated. It -does not rely on any external dependency, and uses as little custom automation as necessary to get things up and -running. - -Instead, we believe in good documentation so that you can use this repository as a template, tweak it, and make it _your -own_. [sherifabdlnaby/elastdocker][elastdocker] is one example among others of project that builds upon this idea. - ---- - -## Contents - -1. [Requirements](#requirements) - * [Host setup](#host-setup) - * [Docker Desktop](#docker-desktop) - * [Windows](#windows) - * [macOS](#macos) -1. [Usage](#usage) - * [Bringing up the stack](#bringing-up-the-stack) - * [Initial setup](#initial-setup) - * [Setting up user authentication](#setting-up-user-authentication) - * [Injecting data](#injecting-data) - * [Cleanup](#cleanup) - * [Version selection](#version-selection) -1. [Configuration](#configuration) - * [How to configure Elasticsearch](#how-to-configure-elasticsearch) - * [How to configure Kibana](#how-to-configure-kibana) - * [How to configure Logstash](#how-to-configure-logstash) - * [How to disable paid features](#how-to-disable-paid-features) - * [How to scale out the Elasticsearch cluster](#how-to-scale-out-the-elasticsearch-cluster) - * [How to reset a password programmatically](#how-to-reset-a-password-programmatically) -1. [Extensibility](#extensibility) - * [How to add plugins](#how-to-add-plugins) - * [How to enable the provided extensions](#how-to-enable-the-provided-extensions) -1. [JVM tuning](#jvm-tuning) - * [How to specify the amount of memory used by a service](#how-to-specify-the-amount-of-memory-used-by-a-service) - * [How to enable a remote JMX connection to a service](#how-to-enable-a-remote-jmx-connection-to-a-service) -1. [Going further](#going-further) - * [Plugins and integrations](#plugins-and-integrations) - -## Requirements - -### Host setup - -* [Docker Engine][docker-install] version **18.06.0** or newer -* [Docker Compose][compose-install] version **1.26.0** or newer (including [Compose V2][compose-v2]) -* 1.5 GB of RAM - -*:information_source: Especially on Linux, make sure your user has the [required permissions][linux-postinstall] to -interact with the Docker daemon.* - -By default, the stack exposes the following ports: - -* 5044: Logstash Beats input -* 5000: Logstash TCP input -* 9600: Logstash monitoring API -* 9200: Elasticsearch HTTP -* 9300: Elasticsearch TCP transport -* 5601: Kibana - -**:warning: Elasticsearch's [bootstrap checks][booststap-checks] were purposely disabled to facilitate the setup of the -Elastic stack in development environments. For production setups, we recommend users to set up their host according to -the instructions from the Elasticsearch documentation: [Important System Configuration][es-sys-config].** - -### Docker Desktop - -#### Windows - -If you are using the legacy Hyper-V mode of _Docker Desktop for Windows_, ensure [File Sharing][win-filesharing] is -enabled for the `C:` drive. - -#### macOS - -The default configuration of _Docker Desktop for Mac_ allows mounting files from `/Users/`, `/Volume/`, `/private/`, -`/tmp` and `/var/folders` exclusively. Make sure the repository is cloned in one of those locations or follow the -instructions from the [documentation][mac-filesharing] to add more locations. - -## Usage - -**:warning: You must rebuild the stack images with `docker-compose build` whenever you switch branch or update the -[version](#version-selection) of an already existing stack.** - -### Bringing up the stack - -Clone this repository onto the Docker host that will run the stack, then start the stack's services locally using Docker -Compose: - -```console -$ docker-compose up -``` - -*:information_source: You can also run all services in the background (detached mode) by appending the `-d` flag to the -above command.* - -Give Kibana about a minute to initialize, then access the Kibana web UI by opening in a web -browser and use the following (default) credentials to log in: - -* user: *elastic* -* password: *changeme* - -*:information_source: Upon the initial startup, the `elastic`, `logstash_internal` and `kibana_system` Elasticsearch -users are intialized with the values of the passwords defined in the [`.env`](.env) file (_"changeme"_ by default). The -first one is the [built-in superuser][builtin-users], the other two are used by Kibana and Logstash respectively to -communicate with Elasticsearch. This task is only performed during the _initial_ startup of the stack. To change users' -passwords _after_ they have been initialized, please refer to the instructions in the next section.* - -### Initial setup - -#### Setting up user authentication - -*:information_source: Refer to [Security settings in Elasticsearch][es-security] to disable authentication.* - -**:warning: Starting with Elastic v8.0.0, it is no longer possible to run Kibana using the bootstraped privileged -`elastic` user.** - -The _"changeme"_ password set by default for all aforementioned users is **unsecure**. For increased security, we will -reset the passwords of all aforementioned Elasticsearch users to random secrets. - -1. Reset passwords for default users - - The commands below resets the passwords of the `elastic`, `logstash_internal` and `kibana_system` users. Take note - of them. - - ```console - $ docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user elastic - ``` - - ```console - $ docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user logstash_internal - ``` - - ```console - $ docker-compose exec elasticsearch bin/elasticsearch-reset-password --batch --user kibana_system - ``` - - If the need for it arises (e.g. if you want to [collect monitoring information][ls-monitoring] through Beats and - other components), feel free to repeat this operation at any time for the rest of the [built-in - users][builtin-users]. - -1. Replace usernames and passwords in configuration files - - Replace the password of the `elastic` user inside the `.env` file with the password generated in the previous step. - Its value isn't used by any core component, but [extensions](#how-to-enable-the-provided-extensions) use it to - connect to Elasticsearch. - - *:information_source: In case you don't plan on using any of the provided - [extensions](#how-to-enable-the-provided-extensions), or prefer to create your own roles and users to authenticate - these services, it is safe to remove the `ELASTIC_PASSWORD` entry from the `.env` file altogether after the stack - has been initialized.* - - Replace the password of the `logstash_internal` user inside the `.env` file with the password generated in the - previous step. Its value is referenced inside the Logstash pipeline file (`logstash/pipeline/logstash.conf`). - - Replace the password of the `kibana_system` user inside the `.env` file with the password generated in the previous - step. Its value is referenced inside the Kibana configuration file (`kibana/config/kibana.yml`). - - See the [Configuration](#configuration) section below for more information about these configuration files. - -1. Restart Logstash and Kibana to re-connect to Elasticsearch using the new passwords - - ```console - $ docker-compose up -d logstash kibana - ``` - -*:information_source: Learn more about the security of the Elastic stack at [Secure the Elastic Stack][sec-cluster].* - -#### Injecting data - -Open the Kibana web UI by opening in a web browser and use the following credentials to log in: - -* user: *elastic* -* password: *\* - -Now that the stack is fully configured, you can go ahead and inject some log entries. The shipped Logstash configuration -allows you to send content via TCP: - -```console -# Using BSD netcat (Debian, Ubuntu, MacOS system, ...) -$ cat /path/to/logfile.log | nc -q0 localhost 5000 -``` - -```console -# Using GNU netcat (CentOS, Fedora, MacOS Homebrew, ...) -$ cat /path/to/logfile.log | nc -c localhost 5000 -``` - -You can also load the sample data provided by your Kibana installation. - -### Cleanup - -Elasticsearch data is persisted inside a volume by default. - -In order to entirely shutdown the stack and remove all persisted data, use the following Docker Compose command: - -```console -$ docker-compose down -v -``` - -### Version selection - -This repository stays aligned with the latest version of the Elastic stack. The `main` branch tracks the current major -version (8.x). - -To use a different version of the core Elastic components, simply change the version number inside the [`.env`](.env) -file. If you are upgrading an existing stack, remember to rebuild all container images using the `docker-compose build` -command. - -**:warning: Always pay attention to the [official upgrade instructions][upgrade] for each individual component before -performing a stack upgrade.** - -Older major versions are also supported on separate branches: - -* [`release-7.x`](https://github.com/deviantony/docker-elk/tree/release-7.x): 7.x series -* [`release-6.x`](https://github.com/deviantony/docker-elk/tree/release-6.x): 6.x series (End-of-life) -* [`release-5.x`](https://github.com/deviantony/docker-elk/tree/release-5.x): 5.x series (End-of-life) - -## Configuration - -*:information_source: Configuration is not dynamically reloaded, you will need to restart individual components after -any configuration change.* - -### How to configure Elasticsearch - -The Elasticsearch configuration is stored in [`elasticsearch/config/elasticsearch.yml`][config-es]. - -You can also specify the options you want to override by setting environment variables inside the Compose file: - -```yml -elasticsearch: - - environment: - network.host: _non_loopback_ - cluster.name: my-cluster -``` - -Please refer to the following documentation page for more details about how to configure Elasticsearch inside Docker -containers: [Install Elasticsearch with Docker][es-docker]. - -### How to configure Kibana - -The Kibana default configuration is stored in [`kibana/config/kibana.yml`][config-kbn]. - -You can also specify the options you want to override by setting environment variables inside the Compose file: - -```yml -kibana: - - environment: - SERVER_NAME: kibana.example.org -``` - -Please refer to the following documentation page for more details about how to configure Kibana inside Docker -containers: [Install Kibana with Docker][kbn-docker]. - -### How to configure Logstash - -The Logstash configuration is stored in [`logstash/config/logstash.yml`][config-ls]. - -You can also specify the options you want to override by setting environment variables inside the Compose file: - -```yml -logstash: - - environment: - LOG_LEVEL: debug -``` - -Please refer to the following documentation page for more details about how to configure Logstash inside Docker -containers: [Configuring Logstash for Docker][ls-docker]. - -### How to disable paid features - -Switch the value of Elasticsearch's `xpack.license.self_generated.type` setting from `trial` to `basic` (see [License -settings][trial-license]). - -You can also cancel an ongoing trial before its expiry date — and thus revert to a basic license — either from the -[License Management][license-mngmt] panel of Kibana, or using Elasticsearch's [Licensing APIs][license-apis]. - -### How to scale out the Elasticsearch cluster - -Follow the instructions from the Wiki: [Scaling out Elasticsearch](https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster) - -### How to reset a password programmatically - -If for any reason your are unable to use Kibana to change the password of your users (including [built-in -users][builtin-users]), you can use the Elasticsearch API instead and achieve the same result. - -In the example below, we reset the password of the `elastic` user (notice "/user/elastic" in the URL): - -```console -$ curl -XPOST -D- 'http://localhost:9200/_security/user/elastic/_password' \ - -H 'Content-Type: application/json' \ - -u elastic: \ - -d '{"password" : ""}' -``` - -## Extensibility - -### How to add plugins - -To add plugins to any ELK component you have to: - -1. Add a `RUN` statement to the corresponding `Dockerfile` (eg. `RUN logstash-plugin install logstash-filter-json`) -1. Add the associated plugin code configuration to the service configuration (eg. Logstash input/output) -1. Rebuild the images using the `docker-compose build` command - -### How to enable the provided extensions - -A few extensions are available inside the [`extensions`](extensions) directory. These extensions provide features which -are not part of the standard Elastic stack, but can be used to enrich it with extra integrations. - -The documentation for these extensions is provided inside each individual subdirectory, on a per-extension basis. Some -of them require manual changes to the default ELK configuration. - -## JVM tuning - -### How to specify the amount of memory used by a service - -By default, both Elasticsearch and Logstash start with [1/4 of the total host -memory](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#default_heap_size) allocated to -the JVM Heap Size. - -The startup scripts for Elasticsearch and Logstash can append extra JVM options from the value of an environment -variable, allowing the user to adjust the amount of memory that can be used by each component: - -| Service | Environment variable | -|---------------|----------------------| -| Elasticsearch | ES_JAVA_OPTS | -| Logstash | LS_JAVA_OPTS | - -To accomodate environments where memory is scarce (Docker for Mac has only 2 GB available by default), the Heap Size -allocation is capped by default to 256MB per service in the `docker-compose.yml` file. If you want to override the -default JVM configuration, edit the matching environment variable(s) in the `docker-compose.yml` file. - -For example, to increase the maximum JVM Heap Size for Logstash: - -```yml -logstash: - - environment: - LS_JAVA_OPTS: -Xmx1g -Xms1g -``` - -### How to enable a remote JMX connection to a service - -As for the Java Heap memory (see above), you can specify JVM options to enable JMX and map the JMX port on the Docker -host. - -Update the `{ES,LS}_JAVA_OPTS` environment variable with the following content (I've mapped the JMX service on the port -18080, you can change that). Do not forget to update the `-Djava.rmi.server.hostname` option with the IP address of your -Docker host (replace **DOCKER_HOST_IP**): - -```yml -logstash: - - environment: - LS_JAVA_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18080 -Dcom.sun.management.jmxremote.rmi.port=18080 -Djava.rmi.server.hostname=DOCKER_HOST_IP -Dcom.sun.management.jmxremote.local.only=false -``` - -## Going further - -### Plugins and integrations - -See the following Wiki pages: - -* [External applications](https://github.com/deviantony/docker-elk/wiki/External-applications) -* [Popular integrations](https://github.com/deviantony/docker-elk/wiki/Popular-integrations) - -[elk-stack]: https://www.elastic.co/what-is/elk-stack -[xpack]: https://www.elastic.co/what-is/open-x-pack -[paid-features]: https://www.elastic.co/subscriptions -[es-security]: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html -[trial-license]: https://www.elastic.co/guide/en/elasticsearch/reference/current/license-settings.html -[license-mngmt]: https://www.elastic.co/guide/en/kibana/current/managing-licenses.html -[license-apis]: https://www.elastic.co/guide/en/elasticsearch/reference/current/licensing-apis.html - -[elastdocker]: https://github.com/sherifabdlnaby/elastdocker - -[docker-install]: https://docs.docker.com/get-docker/ -[compose-install]: https://docs.docker.com/compose/install/ -[compose-v2]: https://docs.docker.com/compose/cli-command/ -[linux-postinstall]: https://docs.docker.com/engine/install/linux-postinstall/ - -[booststap-checks]: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html -[es-sys-config]: https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html - -[win-filesharing]: https://docs.docker.com/desktop/windows/#file-sharing -[mac-filesharing]: https://docs.docker.com/desktop/mac/#file-sharing - -[builtin-users]: https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html -[ls-monitoring]: https://www.elastic.co/guide/en/logstash/current/monitoring-with-metricbeat.html -[sec-cluster]: https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-cluster.html - -[connect-kibana]: https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html -[index-pattern]: https://www.elastic.co/guide/en/kibana/current/index-patterns.html - -[config-es]: ./elasticsearch/config/elasticsearch.yml -[config-kbn]: ./kibana/config/kibana.yml -[config-ls]: ./logstash/config/logstash.yml - -[es-docker]: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html -[kbn-docker]: https://www.elastic.co/guide/en/kibana/current/docker.html -[ls-docker]: https://www.elastic.co/guide/en/logstash/current/docker-config.html - -[upgrade]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html diff --git a/Days/day01.md b/Days/day01.md index 38df1a6..07d21a4 100644 --- a/Days/day01.md +++ b/Days/day01.md @@ -6,8 +6,7 @@ tags: "devops, 90daysofdevops, learning" cover_image: null canonical_url: null id: 1048731 ---- - + --- ## Introduction - Day 1 Day 1 of our 90 days and adventure to learn a good foundational understanding of DevOps and tools that help with a DevOps mindset. diff --git a/Days/day02.md b/Days/day02.md index 89bf696..adad33e 100644 --- a/Days/day02.md +++ b/Days/day02.md @@ -6,8 +6,7 @@ tags: "devops, 90daysofdevops, learning" cover_image: null canonical_url: null id: 1048699 ---- - + --- ## Responsibilities of a DevOps Engineer Hopefully you are coming into this off the back of going through the resources and post on [Day1 of #90DaysOfDevOps](day01.md)