From 1d81ab809963a8efa4499102d2e5f0a9b190d7ba Mon Sep 17 00:00:00 2001 From: Daniel Bodky Date: Sun, 7 Jan 2024 18:30:33 +0100 Subject: [PATCH 1/5] Adds resources for day 27 --- 2024/day27.md | 199 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 199 insertions(+) diff --git a/2024/day27.md b/2024/day27.md index e69de29..e84109b 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -0,0 +1,199 @@ +# Day 27: 90DaysofDevOps + +## From Automated to Automatic - Event-Driven Infrastructure Management with Ansible + +**Daniel Bodky** +- [Twitter](https://twitter.com/d_bodky) +- [LinkedIn](https://linkedin.com/in/daniel-bodky) +- [Blog](https://dbodky.me) + +## Overview + +A universal truth and recurring theme in the DevOps world is automation. From providing infrastructure to testing code to deploying to production, many parts of the DevOps lifecycle get automated already. One popular technology for managing infrastructure and configuration in an automated way is Ansible, but are we fully utilizing its capabilities yet? + +This presentation will give a broad overview of Ansible and its architecture and use-cases, before exploring a relatively new feature, Event-driven Ansible (EDA). Analzying applications of event-driven Ansible, participants will see that automated management is nice, but automatic management is awesome, not just regarding DevOps principles, but also in terms of reaction times, the human tendency for minor mistakes, and toil for operators. + +Participants will get first-hand insights into Ansible, its strengths, weaknesses, and the potential of event-driven automation within the DevOps world. + +## Demos + +
+ +Prerequisites + +### Ansible Inventory + +> [!NOTE] +> For this inventory file to work, you need to create VMs accordingly and adjust the IP addresses to fit your lab environment. + +Ansible utilizes so-called inventories to manage a list of hosts and groups of hosts. Below is the inventory for the demo environment used in this presentation. + +```yaml +hosts: + webservers: + hosts: + webshop.example.com: # Ubuntu + ansible_host: 192.168.1.10 + webserver: apache2 + company.example.com: # Ubuntu + ansible_host: 192.168.1.11 + webserver: nginx + internal.example.com: # CentOS Stream + ansible_host: 192.168.1.12 + webserver: httpd +``` + +You can copy-paste this inventory into a file called `hosts.yml` and use it for the following demos. + +
+ +
+ +Lab 1: Ansible Basics + +### Demo 1: Ansible Basics + +#### Ansible from the CLI via `ansible` + +The first example installs a webserver on all hosts in the `webservers` group. The installed webserver is defined as a **host variable** in the inventory file `hosts.yml` (*see above*). + +```console +ansible \ + webservers \ + -i hosts.yml \ + -m package \ + -a 'name="{{ webserver }}"' +``` + +#### Ansible from the CLI via `ansible-playbook` + +The second example utilizes the following **playbook** to **install** and **start** the defined webserver on all hosts in the `webservers` group. + +```yaml +--- +- name: Install webservers + hosts: webservers + vars: + package: "{{ webserver }}" + become: true + tasks: + - name: Install webserver + ansible.builtin.package: + name: "{{ package }}" + state: present + + - name: Start webserver + ansible.builtin.service: + name: "{{ package }}" + state: started +``` + +Save this playbook as `playbook.yml` and run it with the following command. + +```console +ansible-playbook \ + -i hosts.yml \ + playbook.yml +``` + +You will see a separated output for each task in the playbook. In the end, you should be able to access the webserver on each host in the `webservers` group. + +> [!TIP] +> Ansible is **idempotent** - try running the playbook again and see how the output differs. + +
+ +
+ +Lab 2: Event-driven Ansible and Generic Webhooks + +### Demo 2: Event-driven Ansible and Generic Webhooks + +#### Prerequisites + +For this demo, we will use `localhost` as the target host. Therefore, we need to adjust our inventory file `hosts.yml` accordingly: + +```yaml +hosts: + localhost: {} + +The first demo of event-driven Ansible shows how to use a generic webhook to trigger a playbook run. Copy the following rulebook into a file called `rulebook.yml`. + +```yaml +- name: Listen to webhook events + hosts: all + sources: + - ansible.eda.webhook: + host: 0.0.0.0 + port: 5000 + rules: + - name: Debug event output + condition: event.payload.greeting is defined + action: + debug: + msg: "Hello {{ event.payload.greeting }}!" + + - name: Greet stranger + condition: 1 == 1 # default case + action: + debug: + msg: Hello World! +``` + +#### Start the EDA server + +To start the EDA server, run the following command. + +```console +ansible-rulebook \ + -i hosts.yml \ + --rulebook rulebook.yml +``` + +#### Trigger the webhook + +Once the EDA server is running, we can open a second terminal session and double-check that it is listening on the correct port: + +```console +netstat -lntup | grep 5000 +``` + +Now, we can trigger the webhook from our second terminal session using `curl`, first with empty input: + +```console +curl \ + -H "Content-Type: application/json" \ + -d '{}' \ + http://localhost:5000/endpoint +``` + +If we switch over to the first terminal session, we should see the output of the second rule, which is the default case: + +```console +Hello World! +``` + +Now, we can trigger the webhook again, this time with a payload: + +```console +curl \ + -H "Content-Type: application/json" \ + -d '{"greeting": "Daniel"}' \ + http://localhost:5000/endpoint +``` + +If we switch over to the first terminal session again, we should see the output of the first rule, which is the case for a defined `greeting` in the payload: + +```console +Hello Daniel! +``` + +
+ +## Resources + +- [Ansible Documentation](https://docs.ansible.com/) +- [Installing Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible) +- [Ansible Galaxy](https://galaxy.ansible.com/) +- [EDA Documentation](https://ansible.readthedocs.io/projects/rulebook/en/stable/introduction.html) +- [Installing and Running EDA](https://ansible.readthedocs.io/projects/rulebook/en/stable/installation.html) From 196f1e83d3f1fa1efbbd6f0b2202db1f11b1d17c Mon Sep 17 00:00:00 2001 From: Daniel Bodky Date: Thu, 11 Jan 2024 15:30:27 +0100 Subject: [PATCH 2/5] Continues work on session material for day 27 --- 2024/day27.md | 218 ++++++++++++++------------------------------------ 1 file changed, 59 insertions(+), 159 deletions(-) diff --git a/2024/day27.md b/2024/day27.md index e84109b..ac7cc2c 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -15,185 +15,85 @@ This presentation will give a broad overview of Ansible and its architecture and Participants will get first-hand insights into Ansible, its strengths, weaknesses, and the potential of event-driven automation within the DevOps world. -## Demos +> [!NOTE] +> The below content is a copy of the [lab repository's] README for convenience. -
+--- -Prerequisites +# Event-Driven Ansible Lab -### Ansible Inventory +This is a lab designed to demonstrate Ansible and how Event-Driven Ansible (**EDA**) builds on top of its capabilities. + +The setup is done with Ansible, too. It will install **Ansible, EDA, Prometheus**, and **Alertmanager** on a VM to demonstrate some of the capabilities of EDA. + +## Prerequisites + +To follow along with this lab in its entirety, you will need four VMs: > [!NOTE] -> For this inventory file to work, you need to create VMs accordingly and adjust the IP addresses to fit your lab environment. +> If you want to skip Ansible basics and go straight to EDA, you'll need just the `eda-controller.example.com` VM and can skip the others. -Ansible utilizes so-called inventories to manage a list of hosts and groups of hosts. Below is the inventory for the demo environment used in this presentation. +| VM name | OS | +|--------------------|-------------| +| eda-controller.example.com | CentOS/Rocky 8.9 | +| company.example.com | CentOS/Rocky 8.9 | +| internal.example.com | Ubuntu 22.04 | +| webshop.example.com | OpenSUSE 15.5 | -```yaml -hosts: - webservers: - hosts: - webshop.example.com: # Ubuntu - ansible_host: 192.168.1.10 - webserver: apache2 - company.example.com: # Ubuntu - ansible_host: 192.168.1.11 - webserver: nginx - internal.example.com: # CentOS Stream - ansible_host: 192.168.1.12 - webserver: httpd +**You'll need to be able to SSH to each of these VMs as root using SSH keys.** + +## Lab Setup + +### Clone the repository and create a Python virtual environment + +```bash +git clone https://github.com/mocdaniel/lab-event-driven-ansible.git +cd lab-event-driven-ansible +python3 -m venv .venv +source .venv/bin/activate ``` -You can copy-paste this inventory into a file called `hosts.yml` and use it for the following demos. +### Install Ansible and other dependencies -
- -
- -Lab 1: Ansible Basics - -### Demo 1: Ansible Basics - -#### Ansible from the CLI via `ansible` - -The first example installs a webserver on all hosts in the `webservers` group. The installed webserver is defined as a **host variable** in the inventory file `hosts.yml` (*see above*). - -```console -ansible \ - webservers \ - -i hosts.yml \ - -m package \ - -a 'name="{{ webserver }}"' +```bash +pip install -r requirements.txt ``` -#### Ansible from the CLI via `ansible-playbook` - -The second example utilizes the following **playbook** to **install** and **start** the defined webserver on all hosts in the `webservers` group. +### Create the inventory file ```yaml --- -- name: Install webservers - hosts: webservers - vars: - package: "{{ webserver }}" - become: true - tasks: - - name: Install webserver - ansible.builtin.package: - name: "{{ package }}" - state: present - - - name: Start webserver - ansible.builtin.service: - name: "{{ package }}" - state: started +# hosts.yml +webservers: + hosts: + webshop.example.com: + ansible_host: + webserver: nginx + company.example.com: + ansible_host: + webserver: httpd + internal.example.com: + ansible_host: + webserver: apache2 +eda_controller: + hosts: + eda-controller.example.com: + ansible_host: ``` -Save this playbook as `playbook.yml` and run it with the following command. - -```console -ansible-playbook \ - -i hosts.yml \ - playbook.yml +### Install Needed Roles and Collections + +```bash +ansible-galaxy install -r requirements.yml ``` -You will see a separated output for each task in the playbook. In the end, you should be able to access the webserver on each host in the `webservers` group. +### Run the Setup Playbook -> [!TIP] -> Ansible is **idempotent** - try running the playbook again and see how the output differs. +After you created the inventory file and filled in the IP addresses, you can run the setup playbook: -
- -
- -Lab 2: Event-driven Ansible and Generic Webhooks - -### Demo 2: Event-driven Ansible and Generic Webhooks - -#### Prerequisites - -For this demo, we will use `localhost` as the target host. Therefore, we need to adjust our inventory file `hosts.yml` accordingly: - -```yaml -hosts: - localhost: {} - -The first demo of event-driven Ansible shows how to use a generic webhook to trigger a playbook run. Copy the following rulebook into a file called `rulebook.yml`. - -```yaml -- name: Listen to webhook events - hosts: all - sources: - - ansible.eda.webhook: - host: 0.0.0.0 - port: 5000 - rules: - - name: Debug event output - condition: event.payload.greeting is defined - action: - debug: - msg: "Hello {{ event.payload.greeting }}!" - - - name: Greet stranger - condition: 1 == 1 # default case - action: - debug: - msg: Hello World! +```bash +ansible-playbook playbooks/setup.yml ``` -#### Start the EDA server - -To start the EDA server, run the following command. - -```console -ansible-rulebook \ - -i hosts.yml \ - --rulebook rulebook.yml -``` - -#### Trigger the webhook - -Once the EDA server is running, we can open a second terminal session and double-check that it is listening on the correct port: - -```console -netstat -lntup | grep 5000 -``` - -Now, we can trigger the webhook from our second terminal session using `curl`, first with empty input: - -```console -curl \ - -H "Content-Type: application/json" \ - -d '{}' \ - http://localhost:5000/endpoint -``` - -If we switch over to the first terminal session, we should see the output of the second rule, which is the default case: - -```console -Hello World! -``` - -Now, we can trigger the webhook again, this time with a payload: - -```console -curl \ - -H "Content-Type: application/json" \ - -d '{"greeting": "Daniel"}' \ - http://localhost:5000/endpoint -``` - -If we switch over to the first terminal session again, we should see the output of the first rule, which is the case for a defined `greeting` in the payload: - -```console -Hello Daniel! -``` - -
- -## Resources - -- [Ansible Documentation](https://docs.ansible.com/) -- [Installing Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html#installing-and-upgrading-ansible) -- [Ansible Galaxy](https://galaxy.ansible.com/) -- [EDA Documentation](https://ansible.readthedocs.io/projects/rulebook/en/stable/introduction.html) -- [Installing and Running EDA](https://ansible.readthedocs.io/projects/rulebook/en/stable/installation.html) +> [!CAUTION] +> Due to a known bug with Python on MacOS, you need to run `export NO_PROXY="*"` on MacOS before running the playbook From 09e953ed6eb4a8e37fc83f92dc088f4bd9041ecf Mon Sep 17 00:00:00 2001 From: Daniel Bodky Date: Thu, 11 Jan 2024 16:18:03 +0100 Subject: [PATCH 3/5] Re-adds labs --- 2024/day27.md | 85 ++++++++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 77 insertions(+), 8 deletions(-) diff --git a/2024/day27.md b/2024/day27.md index ac7cc2c..01d0835 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -16,7 +16,7 @@ This presentation will give a broad overview of Ansible and its architecture and Participants will get first-hand insights into Ansible, its strengths, weaknesses, and the potential of event-driven automation within the DevOps world. > [!NOTE] -> The below content is a copy of the [lab repository's] README for convenience. +> The below content is a copy of the [lab repository's](https://github.com/mocdaniel/lab-event-driven-ansible) README for convenience. --- @@ -28,7 +28,7 @@ The setup is done with Ansible, too. It will install **Ansible, EDA, Prometheus* ## Prerequisites -To follow along with this lab in its entirety, you will need four VMs: +To follow along with this lab in its entirety, you will need three VMs: > [!NOTE] > If you want to skip Ansible basics and go straight to EDA, you'll need just the `eda-controller.example.com` VM and can skip the others. @@ -37,8 +37,7 @@ To follow along with this lab in its entirety, you will need four VMs: |--------------------|-------------| | eda-controller.example.com | CentOS/Rocky 8.9 | | company.example.com | CentOS/Rocky 8.9 | -| internal.example.com | Ubuntu 22.04 | -| webshop.example.com | OpenSUSE 15.5 | +| webshop.example.com | Ubuntu 22.04 | **You'll need to be able to SSH to each of these VMs as root using SSH keys.** @@ -68,13 +67,10 @@ webservers: hosts: webshop.example.com: ansible_host: - webserver: nginx + webserver: apache2 company.example.com: ansible_host: webserver: httpd - internal.example.com: - ansible_host: - webserver: apache2 eda_controller: hosts: eda-controller.example.com: @@ -97,3 +93,76 @@ ansible-playbook playbooks/setup.yml > [!CAUTION] > Due to a known bug with Python on MacOS, you need to run `export NO_PROXY="*"` on MacOS before running the playbook + +--- + +## Demos + +### Lab 1: Ansible Basics + +
+ +Ansible from the CLI via ansible + +#### Ansible from the CLI via `ansible` + +The first example installs a webserver on all hosts in the `webservers` group. The installed webserver is defined as a **host variable** in the inventory file `hosts.yml` (*see above*). + +```console +ansible \ + webservers \ + -m package \ + -a 'name="{{ webserver }}"' \ + --one-line +``` + +Afterwards, we can start the webserver on all hosts in the `webservers` group. + +```console +ansible \ + webservers \ + -m service \ + -a 'name="{{ webserver }}" state=started' \ + --one-line +``` + +Go on and check if the web servers are running on the respective hosts. + +> [!TIP] +> Ansible is **idempotent** - try running the commands again and see how the output differs. + +
+ +
+ +Ansible from the CLI via ansible-playbook + +#### Ansible from the CLI via `ansible-playbook` + +The second example utilizes the following **playbook** to **gather** and **display information** for all hosts in the `webservers` group, utilizing the **example** role from the lab repository. + +```yaml +--- +- name: Example role + hosts: webservers + gather_facts: false + vars: + greeting: "Hello World!" + pre_tasks: + - name: Say Hello + ansible.builtin.debug: + msg: "{{ greeting }}" + roles: + - role: example + post_tasks: + - name: Say goodbye + ansible.builtin.debug: + msg: Goodbye! +``` + +```console +ansible-playbook \ + playbooks/example.yml +``` + +
From c70b5adc62cb18bfcbd9f0fcf76ce92d7c09cb77 Mon Sep 17 00:00:00 2001 From: Daniel Bodky Date: Thu, 11 Jan 2024 20:09:38 +0100 Subject: [PATCH 4/5] Finishes material --- 2024/day27.md | 175 +++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 174 insertions(+), 1 deletion(-) diff --git a/2024/day27.md b/2024/day27.md index 01d0835..901ec22 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -128,7 +128,7 @@ ansible \ Go on and check if the web servers are running on the respective hosts. -> [!TIP] +> [!HINT] > Ansible is **idempotent** - try running the commands again and see how the output differs. @@ -166,3 +166,176 @@ ansible-playbook \ ``` + +### Lab 2: Event-Driven Ansible + +
+ +Receive Generic Events via Webhook + +#### Receive Generic Events via Webhook + +If you followed the setup instructions for the EDA lab, you should already have a running EDA instance on the `eda-controller.example.com` VM. + +If you navigate to `/etc/edacontroller/rulebook.yml` on the VM, you'll see the following rulebook: + +```yaml +--- +- name: Listen to webhook events + hosts: all + sources: + - ansible.eda.webhook: + host: 0.0.0.0 + port: 5000 + rules: + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" + +- name: Listen to Alertmanager alerts + hosts: all + sources: + - ansible.eda.alertmanager: + host: 0.0.0.0 + port: 9000 + data_alerts_path: alerts + data_host_path: labels.instance + data_path_separator: . + rules: + - name: Restart MySQL server + condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing' + action: + run_module: + name: ansible.builtin.service + module_args: + name: mysql + state: restarted + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" + +``` + +For this part of the lab, the **first rule** is the one we're interested in: It listens to a generic webhook on port `5000` and prints the event's **metadata** to its logs. + +To test this, we can use the `curl` command to send a `POST` request to the webhook `/endpoint` from the VM itself: + +```console +curl \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{"foo": "bar"}' \ + http://localhost:5000/endpoint +``` + +If you now check the logs of the EDA controller, you should see the following output: + +```console +journalctl -fu eda-controller + +Jan 11 16:35:29 eda-controller ansible-rulebook[56882]: {'payload': {'foo': 'bar'}, 'meta': {'endpoint': 'endpoint', +'headers': {'Host': 'localhost:5000', 'User-Agent': 'curl/7.76.1', 'Accept': '*/*', 'Content-Length': '21', +'Content-Type': 'application/x-www-form-urlencoded'}, 'source': {'name': 'ansible.eda.webhook', 'type': 'ansible.eda.webhook'}, +'received_at': '2024-01-11T15:35:29.798401Z', 'uuid': '6ebf8dd2-60a2-455a-9383-97b81f535366'}} +``` + +A rule that always evaluates to `true` is not very useful, so let's change the rule to only print the the value of `foo` if the `foo` key is present in the event's payload, and `no foo :(` otherwise: + +```yaml +--- +- name: Listen to webhook events + hosts: all + sources: + - ansible.eda.webhook: + host: 0.0.0.0 + port: 5000 + rules: + - name: Foo + condition: event.payload.foo is defined + action: + debug: + msg: "{{ event.payload.foo }}" + - name: No foo + condition: 1 == 1 + action: + debug: + msg: "no foo :(" +``` + +Send the same `curl` request again and check the logs, you should see a line saying `bar` now. + +Let's also try a `curl` request with a different payload: + +```console +curl \ + -X POST \ + -H "Content-Type: application/json" \ + -d '{"bar": "baz"}' \ + http://localhost:5000/endpoint +``` + +This time, the output should be `no foo :(`. + +
+ +
+ +Restarting Services Automatically with EDA + +#### Restarting Services Automatically with EDA + +The last lab is more of a demo - it shows how you can use EDA to automatically react on events observed by **Prometheus** and **Alertmanager**. + +For this demo, the second **ruleset** in our rulebook is the one we're interested in: + +```yaml +- name: Listen to Alertmanager alerts + hosts: all + sources: + - ansible.eda.alertmanager: + host: 0.0.0.0 + port: 9000 + data_alerts_path: alerts + data_host_path: labels.instance + data_path_separator: . + rules: + - name: Restart MySQL server + condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing' + action: + run_playbook: + playbook: ./playbook.yml + - name: Debug event output + condition: 1 == 1 + action: + debug: + msg: "{{ event }}" +``` + +With this rule, we can restart our MySQL server if it's not running! But how do we get the event to trigger? With **Prometheus** and **Alertmanager**! + +When you ran the setup playbook, it installed **Prometheus** and **Alertmanager** on the `eda-controller.example.com` VM. You can access the **Prometheus** UI at `http://:9090` and the **Alertmanager** UI at `http://:9093`. + +It also installed a **Prometheus exporter** for the **MySQL** database that runs on the server. + +With this setup, we can now shut down our MySQL server and see what happens - make sure to watch the output of the EDA controller's logs: + +```console +systemctl stop mysql +journalctl -fu edacontroller +``` + + +Within 30-90 seconds, you should see EDA running our **playbook** and restarting the MySQL server. You can track that process by watching the Prometheus/Alertmanager UIs for firing alerts. + +Once you see the playbook being executed in the logs, you can check the MySQL state once more: + +```console +systemctl status mysql +``` + +MySQL should be up and running again! +
From 4fb4742db2e36abe8cede1fdf96cdbb7fb2facee Mon Sep 17 00:00:00 2001 From: Daniel Bodky Date: Wed, 17 Jan 2024 10:58:29 +0100 Subject: [PATCH 5/5] Fixes lab instructions --- 2024/day27.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/2024/day27.md b/2024/day27.md index 901ec22..4c010e0 100644 --- a/2024/day27.md +++ b/2024/day27.md @@ -19,7 +19,6 @@ Participants will get first-hand insights into Ansible, its strengths, weaknesse > The below content is a copy of the [lab repository's](https://github.com/mocdaniel/lab-event-driven-ansible) README for convenience. --- - # Event-Driven Ansible Lab This is a lab designed to demonstrate Ansible and how Event-Driven Ansible (**EDA**) builds on top of its capabilities. @@ -78,7 +77,7 @@ eda_controller: ``` ### Install Needed Roles and Collections - + ```bash ansible-galaxy install -r requirements.yml ``` @@ -307,7 +306,7 @@ For this demo, the second **ruleset** in our rulebook is the one we're intereste condition: event.alert.labels.alertname == 'MySQL not running' and event.alert.status == 'firing' action: run_playbook: - playbook: ./playbook.yml + name: ./playbook.yml - name: Debug event output condition: 1 == 1 action: