Spelling & Grammar Day 81-90
This commit is contained in:
Michael Cade
parent
ba852aca99
commit
a656b32e59
@ -12,31 +12,31 @@ id: 1048716
|
||||
|
||||
Another data collector that I wanted to explore as part of this observability section was [Fluentd](https://docs.fluentd.org/). An Open-Source unified logging layer.
|
||||
|
||||
Fluentd has four key features that makes it suitable to build clean, reliable logging pipelines:
|
||||
Fluentd has four key features that make it suitable to build clean, reliable logging pipelines:
|
||||
|
||||
Unified Logging with JSON: Fluentd tries to structure data as JSON as much as possible. This allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. The downstream data processing is much easier with JSON, since it has enough structure to be accessible without forcing rigid schemas.
|
||||
Unified Logging with JSON: Fluentd tries to structure data as JSON as much as possible. This allows Fluentd to unify all facets of processing log data: collecting, filtering, buffering, and outputting logs across multiple sources and destinations. The downstream data processing is much easier with JSON since it has enough structure to be accessible without forcing rigid schemas.
|
||||
|
||||
Pluggable Architecture: Fluentd has a flexible plugin system that allows the community to extend its functionality. Over 300 community-contributed plugins connect dozens of data sources to dozens of data outputs, manipulating the data as needed. By using plugins, you can make better use of your logs right away.
|
||||
|
||||
Minimum Resources Required: A data collector should be lightweight so that it runs comfortably on a busy machine. Fluentd is written in a combination of C and Ruby, and requires minimal system resources. The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core.
|
||||
Minimum Resources Required: A data collector should be lightweight so that it runs comfortably on a busy machine. Fluentd is written in a combination of C and Ruby and requires minimal system resources. The vanilla instance runs on 30-40MB of memory and can process 13,000 events/second/core.
|
||||
|
||||
Built-in Reliability: Data loss should never happen. Fluentd supports memory- and file-based buffering to prevent inter-node data loss. Fluentd also supports robust failover and can be set up for high availability.
|
||||
|
||||
[Installing Fluentd](https://docs.fluentd.org/quickstart#step-1-installing-fluentd)
|
||||
|
||||
### How apps log data?
|
||||
### How do apps log data?
|
||||
|
||||
- Write to files. `.log` files (difficult to analyse without a tool and at scale)
|
||||
- Log directly to a database (each application must be configured with the correct format)
|
||||
- Third party applications (NodeJS, NGINX, PostgreSQL)
|
||||
- Third-party applications (NodeJS, NGINX, PostgreSQL)
|
||||
|
||||
This is why we want a unified logging layer.
|
||||
|
||||
FluentD allows for the 3 logging data types shown above and gives us the ability to collect, process and send those to a destination, this could be sending them logs to Elastic, MongoDB, Kafka databases for example.
|
||||
FluentD allows for the 3 logging data types shown above and gives us the ability to collect, process and send those to a destination, this could be sending them logs to Elastic, MongoDB, or Kafka databases for example.
|
||||
|
||||
Any Data, Any Data source can be sent to FluentD and that can be sent to any destination. FluentD is not tied to any particular source or destination.
|
||||
|
||||
In my research of Fluentd I kept stumbling across Fluent bit as another option and it looks like if you were looking to deploy a logging tool into your Kubernetes environment then fluent bit would give you that capability, even though fluentd can also be deployed to containers as well as servers.
|
||||
In my research of Fluentd, I kept stumbling across Fluent bit as another option and it looks like if you were looking to deploy a logging tool into your Kubernetes environment then fluent bit would give you that capability, even though fluentd can also be deployed to containers as well as servers.
|
||||
|
||||
[Fluentd & Fluent Bit](https://docs.fluentbit.io/manual/about/fluentd-and-fluent-bit)
|
||||
|
||||
@ -44,7 +44,7 @@ Fluentd and Fluentbit will use the input plugins to transform that data to Fluen
|
||||
|
||||
We can also use tags and matches between configurations.
|
||||
|
||||
I cannot see a good reason for using fluentd and it sems that Fluent Bit is the best way to get started. Although they can be used together in some architectures.
|
||||
I cannot see a good reason for using fluentd and it seems that Fluent Bit is the best way to get started. Although they can be used together in some architectures.
|
||||
|
||||
### Fluent Bit in Kubernetes
|
||||
|
||||
@ -52,15 +52,15 @@ Fluent Bit in Kubernetes is deployed as a DaemonSet, which means it will run on
|
||||
|
||||
Kubernetes annotations can be used within the configuration YAML of our applications.
|
||||
|
||||
First of all we can deploy from the fluent helm repository. `helm repo add fluent https://fluent.github.io/helm-charts` and then install using the `helm install fluent-bit fluent/fluent-bit` command.
|
||||
First of all, we can deploy from the fluent helm repository. `helm repo add fluent https://fluent.github.io/helm-charts` and then install using the `helm install fluent-bit fluent/fluent-bit` command.
|
||||
|
||||
|
||||
|
||||
In my cluster I am also running prometheus in my default namespace (for test purposes) we need to make sure our fluent-bit pod is up and running. we can do this using `kubectl get all | grep fluent` this is going to show us our running pod, service and daemonset that we mentioned earlier.
|
||||
In my cluster, I am also running Prometheus in my default namespace (for test purposes) we need to make sure our fluent-bit pod is up and running. we can do this using `kubectl get all | grep fluent` this is going to show us our running pod, service and daemonset that we mentioned earlier.
|
||||
|
||||
|
||||
|
||||
So that fluentbit knows where to get logs from we have a configuration file, in this Kubernetes deployment of fluentbit we have a configmap which resembles the configuration file.
|
||||
So that fluentbit knows where to get logs from we have a configuration file, in this Kubernetes deployment of fluentbit, we have a configmap which resembles the configuration file.
|
||||
|
||||
|
||||
|
||||
@ -116,7 +116,7 @@ fluent-bit.conf:
|
||||
Read_From_Tail On
|
||||
|
||||
[FILTER]
|
||||
Name kubernetes
|
||||
Name Kubernetes
|
||||
Match kube.*
|
||||
Merge_Log On
|
||||
Keep_Log Off
|
||||
@ -141,11 +141,11 @@ fluent-bit.conf:
|
||||
Events: <none>
|
||||
```
|
||||
|
||||
We can now port-forward our pod to our localhost to ensure that we have connectivity. Firstly get the name of your pod with `kubectl get pods | grep fluent` and then use `kubectl port-forward fluent-bit-8kvl4 2020:2020` open a web browser to `http://localhost:2020/`
|
||||
We can now port-forward our pod to our localhost to ensure that we have connectivity. Firstly get the name of your pod with `kubectl get pods | grep fluent` and then use `kubectl port-forward fluent-bit-8kvl4 2020:2020` to open a web browser to `http://localhost:2020/`
|
||||
|
||||
|
||||
|
||||
I also found this really great medium article covering more about [Fluent Bit](https://medium.com/kubernetes-tutorials/exporting-kubernetes-logs-to-elasticsearch-using-fluent-bit-758e8de606af)
|
||||
I also found this great medium article covering more about [Fluent Bit](https://medium.com/kubernetes-tutorials/exporting-kubernetes-logs-to-elasticsearch-using-fluent-bit-758e8de606af)
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -22,11 +22,11 @@ We will be deploying the following into our Kubernetes cluster.
|
||||
|
||||
The EFK stack is a collection of 3 software bundled together, including:
|
||||
|
||||
- Elasticsearch : NoSQL database is used to store data and provides interface for searching and query log.
|
||||
- Elasticsearch: NoSQL database is used to store data and provides an interface for searching and query logs.
|
||||
|
||||
- Fluentd : Fluentd is an open source data collector for unified logging layer. Fluentd allows you to unify data collection and consumption for a better use and understanding of data.
|
||||
- Fluentd: Fluentd is an open source data collector for a unified logging layer. Fluentd allows you to unify data collection and consumption for better use and understanding of data.
|
||||
|
||||
- Kibana : Interface for managing and statistics logs. Responsible for reading information from elasticsearch.
|
||||
- Kibana: Interface for managing and statistics logs. Responsible for reading information from elasticsearch.
|
||||
|
||||
### Deploying EFK on Minikube
|
||||
|
||||
@ -38,7 +38,7 @@ I have created [efk-stack.yaml](Days/Monitoring/../../Monitoring/EFK%20Stack/efk
|
||||
|
||||
|
||||
|
||||
Depending on your system and if you have ran this already and have images pulled you should now watch the pods into a ready state before we can move on, you can check the progress with the following command. `kubectl get pods -n kube-logging -w` This can take a few minutes.
|
||||
Depending on your system and if you have run this already and have images pulled you should now watch the pods into a ready state before we can move on, you can check the progress with the following command. `kubectl get pods -n kube-logging -w` This can take a few minutes.
|
||||
|
||||
|
||||
|
||||
@ -46,13 +46,13 @@ The above command lets us keep an eye on things but I like to clarify that thing
|
||||
|
||||
|
||||
|
||||
Once we have all our pods up and running and at this stage we should see
|
||||
Once we have all our pods up and running and at this stage, we should see
|
||||
|
||||
- 3 pods associated to ElasticSearch
|
||||
- 1 pod associated to Fluentd
|
||||
- 1 pod associated to Kibana
|
||||
- 3 pods associated with ElasticSearch
|
||||
- 1 pod associated with Fluentd
|
||||
- 1 pod associated with Kibana
|
||||
|
||||
We can also use `kubectl get all -n kube-logging` to show all in our namespace, fluentd as explained previously is deployed as a daemonset, kibana as a deployment and Elasticsearch as a statefulset.
|
||||
We can also use `kubectl get all -n kube-logging` to show all in our namespace, fluentd as explained previously is deployed as a daemonset, kibana as deployment and Elasticsearch as a statefulset.
|
||||
|
||||
|
||||
|
||||
@ -68,7 +68,7 @@ Next, we need to hit the "discover" tab on the left menu and add "\*" to our ind
|
||||
|
||||
|
||||
|
||||
On Step 2 of 2, we are going to use the @timestamp option from the dropdown as this will filter our data by time. When you hit create pattern it might take a few seconds to complete.
|
||||
In Step 2 of 2, we are going to use the @timestamp option from the dropdown as this will filter our data by time. When you hit create pattern it might take a few seconds to complete.
|
||||
|
||||
|
||||
|
||||
@ -76,9 +76,9 @@ If we now head back to our "discover" tab after a few seconds you should start t
|
||||
|
||||
|
||||
|
||||
Now that we have the EFK stack up and running and we are gathering logs from our Kubernetes cluster via Fluentd we can also take a look at other sources we can choose from, if you navigate to the home screen by hitting the Kibana logo in the top left you will be greeted with the same page we saw when we first logged in.
|
||||
Now that we have the EFK stack up and running and we are gathering logs from our Kubernetes cluster via Fluentd we can also take a look at other sources we can choose from if you navigate to the home screen by hitting the Kibana logo on the top left you will be greeted with the same page we saw when we first logged in.
|
||||
|
||||
We have the ability to add APM, Log data, metric data and security events from other plugins or sources.
|
||||
We can add APM, Log data, metric data and security events from other plugins or sources.
|
||||
|
||||
|
||||
|
||||
@ -86,7 +86,7 @@ If we select "Add log data" then we can see below that we have a lot of choices
|
||||
|
||||
|
||||
|
||||
Under the metrics data you will find that you can add sources for Prometheus and lots of other services.
|
||||
Under the metrics data, you will find that you can add sources for Prometheus and lots of other services.
|
||||
|
||||
### APM (Application Performance Monitoring)
|
||||
|
||||
|
||||
@ -14,7 +14,7 @@ We saw a lot of Kibana over this section around Observability. But we have to al
|
||||
|
||||
Kibana’s core feature is data querying and analysis. Using various methods, users can search the data indexed in Elasticsearch for specific events or strings within their data for root cause analysis and diagnostics. Based on these queries, users can use Kibana’s visualisation features which allow users to visualize data in a variety of different ways, using charts, tables, geographical maps and other types of visualizations.
|
||||
|
||||
Grafana actually started as a fork of Kibana, Grafana had an aim to supply support for metrics aka monitoring, which at that time Kibana did not provide.
|
||||
Grafana started as a fork of Kibana, Grafana had an aim to supply support for metrics aka monitoring, which at that time Kibana did not provide.
|
||||
|
||||
Grafana is a free and Open-Source data visualisation tool. We commonly see Prometheus and Grafana together out in the field but we might also see Grafana alongside Elasticsearch and Graphite.
|
||||
|
||||
@ -30,19 +30,19 @@ There are no doubt others but Grafana is a tool that I have seen spanning the vi
|
||||
|
||||
### Prometheus Operator + Grafana Deployment
|
||||
|
||||
We have covered Prometheus already in this section but as we see these paired so often I wanted to spin up an environment that would allow us to at least see what metrics we could have displayed in a visualisation. We know that monitoring our environments is important but going through those metrics alone in Prometheus or any metric tool is going to be cumbersome and it is not going to scale. This is where Grafana comes in and provides us that interactive visualisation of those metrics collected and stored in the Prometheus database.
|
||||
We have covered Prometheus already in this section but as we see these paired so often I wanted to spin up an environment that would allow us to at least see what metrics we could have displayed in a visualisation. We know that monitoring our environments is important but going through those metrics alone in Prometheus or any metric tool is going to be cumbersome and it is not going to scale. This is where Grafana comes in and provides us with that interactive visualisation of those metrics collected and stored in the Prometheus database.
|
||||
|
||||
With that visualisation we can create custom charts, graphs and alerts for our environment. In this walkthrough we will be using our minikube cluster.
|
||||
With that visualisation, we can create custom charts, graphs and alerts for our environment. In this walkthrough, we will be using our minikube cluster.
|
||||
|
||||
We are going to start by cloning this down to our local system. Using `git clone https://github.com/prometheus-operator/kube-prometheus.git` and `cd kube-prometheus`
|
||||
|
||||
|
||||
|
||||
First job is to create our namespace within our minikube cluster `kubectl create -f manifests/setup` if you have not been following along in previous sections we can use `minikube start` to bring up a new cluster here.
|
||||
The first job is to create our namespace within our minikube cluster `kubectl create -f manifests/setup` if you have not been following along in previous sections we can use `minikube start` to bring up a new cluster here.
|
||||
|
||||
|
||||
|
||||
Next we are going to deploy everything we need for our demo using the `kubectl create -f manifests/` command, as you can see this is going to deploy a lot of different resources within our cluster.
|
||||
Next, we are going to deploy everything we need for our demo using the `kubectl create -f manifests/` command, as you can see this is going to deploy a lot of different resources within our cluster.
|
||||
|
||||
|
||||
|
||||
@ -54,11 +54,11 @@ When everything is running we can check all pods are in a running and healthy st
|
||||
|
||||
|
||||
|
||||
With the deployment, we deployed a number of services that we are going to be using later on in the demo you can check these by using the `kubectl get svc -n monitoring` command.
|
||||
With the deployment, we deployed several services that we are going to be using later on in the demo you can check these by using the `kubectl get svc -n monitoring` command.
|
||||
|
||||
|
||||
|
||||
And finally lets check on all resources deployed in our new monitoring namespace using the `kubectl get all -n monitoring` command.
|
||||
And finally, let's check on all resources deployed in our new monitoring namespace using the `kubectl get all -n monitoring` command.
|
||||
|
||||
|
||||
|
||||
@ -76,19 +76,19 @@ Username: admin
|
||||
Password: admin
|
||||
```
|
||||
|
||||
However you will be asked to provide a new password at first login. The initial screen or home page you will see will give you some areas to explore as well as some useful resources to get up to speed with Grafana and its capabilities. Notice the "Add your first data source" and "create your first dashboard" widgets we will be using them later.
|
||||
However, you will be asked to provide a new password at first login. The initial screen or home page you will see will give you some areas to explore as well as some useful resources to get up to speed with Grafana and its capabilities. Notice the "Add your first data source" and "create your first dashboard" widgets we will be using later.
|
||||
|
||||
|
||||
|
||||
You will find that there is already a prometheus data source already added to our Grafana data sources, however because we are using minikube we need to also port forward prometheus so that this is available on our localhost, opening a new terminal we can run the following command. `kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090` if on the home page of Grafana we now enter into the widget "Add your first data source" and from here we are going to select Prometheus.
|
||||
You will find that there is already a prometheus data source already added to our Grafana data sources, however, because we are using minikube we need to also port forward prometheus so that this is available on our localhost, opening a new terminal we can run the following command. `kubectl --namespace monitoring port-forward svc/prometheus-k8s 9090` if on the home page of Grafana we now enter into the widget "Add your first data source" and from here we are going to select Prometheus.
|
||||
|
||||
|
||||
|
||||
For our new data source we can use the address http://localhost:9090 and we will also need to change the dropdown to browser as highlighted below.
|
||||
For our new data source, we can use the address http://localhost:9090 and we will also need to change the dropdown to the browser as highlighted below.
|
||||
|
||||
|
||||
|
||||
At the bottom of the page, we can now hit save and test. This should give us the outcome you see below if the port forward for prometheus is working.
|
||||
At the bottom of the page, we can now hit save and test. This should give us the outcome you see below if the port forward for Prometheus is working.
|
||||
|
||||
|
||||
|
||||
@ -104,11 +104,11 @@ If you then select the Metrics browser you will have a long list of metrics bein
|
||||
|
||||
|
||||
|
||||
For the purpose of the demo I am going to find a metric that gives us some output around our system resources, `cluster:node_cpu:ratio{}` gives us some detail on the nodes in our cluster and proves that this integration is working.
|
||||
For the demo I am going to find a metric that gives us some output around our system resources, `cluster:node_cpu:ratio{}` gives us some detail on the nodes in our cluster and proves that this integration is working.
|
||||
|
||||
|
||||
|
||||
Once you are happy with this as your visualisation then you can hit the apply button in the top right and you will then add this graph to your dashboard. Obviously you can go ahead and add additional graphs and other charts to give you the visual that you need.
|
||||
Once you are happy with this as your visualisation then you can hit the apply button in the top right and you will then add this graph to your dashboard. You can go ahead and add additional graphs and other charts to give you the visuals that you need.
|
||||
|
||||
|
||||
|
||||
@ -116,24 +116,24 @@ We can however take advantage of thousands of previously created dashboards that
|
||||
|
||||
|
||||
|
||||
If we do a search for Kubernetes we will see a long list of pre built dashboards that we can choose from.
|
||||
If we search Kubernetes we will see a long list of pre-built dashboards that we can choose from.
|
||||
|
||||
|
||||
|
||||
We have chosen the Kubernetes API Server dashboard and changed the data source to suit our newly added Prometheus-1 data source and we get to see some of the metrics displayed as per below.
|
||||
We have chosen the Kubernetes API Server dashboard and changed the data source to suit our newly added Prometheus-1 data source and we get to see some of the metrics displayed below.
|
||||
|
||||
|
||||
|
||||
### Alerting
|
||||
|
||||
You could also leverage the alertmanager that we deployed to then send alerts out to slack or other integrations, in order to do this you would need to port forward the alertmanager service using the below details.
|
||||
You could also leverage the alertmanager that we deployed to then send alerts out to slack or other integrations, to do this you would need to port forward the alertmanager service using the below details.
|
||||
|
||||
`kubectl --namespace monitoring port-forward svc/alertmanager-main 9093`
|
||||
`http://localhost:9093`
|
||||
|
||||
That wraps up our section on all things observability, I have personally found that this section has highlighted how broad this topic is but equally how important this is for our roles and that be it metrics, logging or tracing you are going to need to have a good idea of what is happening in our broad environments moving forward, especially when they can change so dramatically with all the automation that we have already covered in the other sections.
|
||||
|
||||
Next up we are going to be taking a look into data management and how DevOps principles also needs to be considered when it comes to Data Management.
|
||||
Next up we are going to be taking a look into data management and how DevOps principles also need to be considered when it comes to Data Management.
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -12,54 +12,54 @@ id: 1048747
|
||||
|
||||
|
||||
|
||||
Data Management is by no means a new wall to climb, although we do know that data is more important than it maybe was a few years ago. Valuable and ever changing it can also be a massive nightmare when we are talking about automation and continuously integrate, test and deploy frequent software releases. Enter the persistent data and underlying data services often the main culprit when things go wrong.
|
||||
Data Management is by no means a new wall to climb, although we do know that data is more important than it maybe was a few years ago. Valuable and ever-changing it can also be a massive nightmare when we are talking about automation and continuously integrating, testing and deploying frequent software releases. Enter the persistent data and underlying data services are often the main culprit when things go wrong.
|
||||
|
||||
But before I get into the Cloud-Native Data Management, we need to go up a level. We have touched on many different platforms throughout this challenge. Be it Physical, Virtual, Cloud and Cloud-Native obviously including Kubernetes there is none of these platforms that provide the lack of requirement for data management.
|
||||
But before I get into Cloud-Native Data Management, we need to go up a level. We have touched on many different platforms throughout this challenge. Be it Physical, Virtual, Cloud or Cloud-Native obviously including Kubernetes there is none of these platforms that provide the lack of requirement for data management.
|
||||
|
||||
Whatever our business it is more than likely you will find a database lurking in the environment somewhere, be it for the most mission critical system in the business or at least some cog in the chain is storing that persistent data on some level of system.
|
||||
Whatever our business it is more than likely you will find a database lurking in the environment somewhere, be it for the most mission-critical system in the business or at least some cog in the chain is storing that persistent data on some level of the system.
|
||||
|
||||
### DevOps and Data
|
||||
|
||||
Much like the very start of this series where we spoke about the DevOps principles, in order for a better process when it comes to data you have to include the right people. This might be the DBAs but equally that is going to include people that care about the backup of those data services as well.
|
||||
Much like the very start of this series where we spoke about the DevOps principles, for a better process when it comes to data you have to include the right people. This might be the DBAs but equally, that is going to include people that care about the backup of those data services as well.
|
||||
|
||||
Secondly we also need to identify the different data types, domains, boundaries that we have associated with our data. This way it is not just dealt with in a silo approach amongst Database administrators, storage engineers or Backup focused engineers. This way the whole team can determine the best route of action when it comes to developing and hosting applications for the wider business and focus on the data architecture vs it being an after thought.
|
||||
Secondly, we also need to identify the different data types, domains, and boundaries that we have associated with our data. This way it is not just dealt with in a silo approach amongst Database administrators, storage engineers or Backup focused engineers. This way the whole team can determine the best route of action when it comes to developing and hosting applications for the wider business and focus on the data architecture vs it being an afterthought.
|
||||
|
||||
Now, this can span many different areas of the data lifecycle, we could be talking about data ingest, where and how will data be ingested into our service or application? How will the service, application or users access this data. But then it also requires us to understand how we will secure the data and then how will we protect that data.
|
||||
Now, this can span many different areas of the data lifecycle, we could be talking about data ingest, where and how will data be ingested into our service or application? How will the service, application or users access this data? But then it also requires us to understand how we will secure the data and then how will we protect that data.
|
||||
|
||||
### Data Management 101
|
||||
|
||||
Data management according to the [Data Management Body of Knowledge](https://www.dama.org/cpages/body-of-knowledge) is “the development, execution and supervision of plans, policies, programs and practices that control, protect, deliver and enhance the value of data and information assets.”
|
||||
|
||||
- Data is the most important aspect of your business - Data is only one part of your overall business. I have seen the term "Data is the lifeblood of our business" and most likely absolutely true. Which then got me thinking about blood being pretty important to the body but alone it is nothing we still need the aspects of the body to make the blood something other than a liquid.
|
||||
- Data is the most important aspect of your business - Data is only one part of your overall business. I have seen the term "Data is the lifeblood of our business" and most likely true. This then got me thinking about blood being pretty important to the body but alone it is nothing we still need the aspects of the body to make the blood something other than a liquid.
|
||||
|
||||
- Data quality is more important than ever - We are having to treat data as a business asset, meaning that we have to give it the considerations it needs and requires to work with our automation and DevOps principles.
|
||||
|
||||
- Accessing data in a timely fashion - Nobody has the patience to not have access to the right data at the right time to make effective decisions. Data must be available in a streamlined and timely manher regardless of presentation.
|
||||
- Accessing data in a timely fashion - Nobody has the patience to not have access to the right data at the right time to make effective decisions. Data must be available in a streamlined and timely manner regardless of presentation.
|
||||
|
||||
- Data Management has to be an enabler to DevOps - I mentioned streamline previously, we have to include the data management requirements into our cycle and ensure not just availablity of that data but also include other important policy based protection of those data points along with fully tested recovery models with that as well.
|
||||
- Data Management has to be an enabler to DevOps - I mentioned streamlining previously, we have to include the data management requirements into our cycle and ensure not just availability of that data but also include other important policy-based protection of those data points along with fully tested recovery models with that as well.
|
||||
|
||||
### DataOps
|
||||
|
||||
Both DataOps and DevOps apply the best practices of technology development and operations to improve quality, increase speed, reduce security threats, delight customers and provide meaningful and challenging work for skilled professionals. DevOps and DataOps share goals to accelerate product delivery by automating as many process steps as possible. For DataOps, the objective is a resilient data pipeline and trusted insights from data analytics.
|
||||
|
||||
Some of the most common higher level areas that focus on DataOps are going to be Machine Learning, Big Data and Data Analytics including Artifical Intelligence.
|
||||
Some of the most common higher-level areas that focus on DataOps are going to be Machine Learning, Big Data and Data Analytics including Artificial Intelligence.
|
||||
|
||||
### Data Management is the management of information
|
||||
|
||||
My focus throughout this section is not going to be getting into Machine Learning or Articial Intelligence but to focus on the protecting the data from a data protection point of view, the title of this subsection is "Data management is the management of information" and we can relate that information = data.
|
||||
My focus throughout this section is not going to be getting into Machine Learning or Artificial Intelligence but focus on the protecting the data from a data protection point of view, the title of this subsection is "Data management is the management of information" and we can relate that information = data.
|
||||
|
||||
Three key areas that we should consider along this journey with data are:
|
||||
|
||||
- Accuracy - Making sure that production data is accurate, equally we need to ensure that our data in the form of backups are also working and tested against recovery to be sure if a failure or a reason comes up we need to be able to get back up and running as fast as possible.
|
||||
- Consistent - If our data services span multiple locations then for production we need to make sure we have consistency across all data locations so that we are getting accurate data, this also spans into data protection when it comes to protecting these data services especially data services we need to ensure consistency at different levels to make sure we are taking a good clean copy of that data for our backups, replicas etc.
|
||||
- Consistent - If our data services span multiple locations then for production we need to make sure we have consistency across all data locations so that we are getting accurate data, this also spans into data protection when it comes to protecting these data services, especially data services we need to ensure consistency at different levels to make sure we are taking a good clean copy of that data for our backups, replicas etc.
|
||||
|
||||
- Secure - Access Control but equally just keeping data in general is a topical theme at the moment across the globe. Making sure the right people have access to your data is paramount, again this leads into data protection where we must make sure that only the required personnel have access to backups and the ability to restore from those as well clone and provide other versions of the business data.
|
||||
- Secure - Access Control but equally just keeping data, in general, is a topical theme at the moment across the globe. Making sure the right people have access to your data is paramount, again this leads to data protection where we must make sure that only the required personnel have access to backups and the ability to restore from those as well clone and provide other versions of the business data.
|
||||
|
||||
Better Data = Better Decisions
|
||||
|
||||
### Data Management Days
|
||||
|
||||
During the next 6 sessions we are going to be taking a closer look at Databases, Backup & Recovery, Disaster Recovery, Application Mobility all with an element of demo and hands on throughout.
|
||||
During the next 6 sessions we are going to be taking a closer look at Databases, Backup & Recovery, Disaster Recovery, and Application Mobility all with an element of demo and hands-on throughout.
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@ id: 1048781
|
||||
|
||||
## Data Services
|
||||
|
||||
Databases are going to be the most common data service that we come across in our environments. I wanted to take this session to explore some of those different types of Databases and some of the use cases they each have. Some we have used and seen throughout the course of the challenge.
|
||||
Databases are going to be the most common data service that we come across in our environments. I wanted to take this session to explore some of those different types of Databases and some of the use cases they each have. Some we have used and seen throughout the challenge.
|
||||
|
||||
From an application development point of view choosing the right data service or database is going to be a huge decision when it comes to the performance and scalability of your application.
|
||||
|
||||
@ -26,7 +26,7 @@ _Redis is an in-memory data structure store, used as a distributed, in-memory ke
|
||||
|
||||
|
||||
|
||||
As you can see from the description of Redis this means that our database is fast but we are limited on space as a trade off. Also no queries or joins which means data modelling options are very limited.
|
||||
As you can see from the description of Redis this means that our database is fast but we are limited on space as a trade-off. Also, no queries or joins which means data modelling options are very limited.
|
||||
|
||||
Best for:
|
||||
|
||||
@ -71,9 +71,9 @@ Best for:
|
||||
|
||||
### Relational
|
||||
|
||||
If you are new to databases but you know of them my guess is that you have absolutely come across a relational database.
|
||||
If you are new to databases but you know of them I guess that you have come across a relational database.
|
||||
|
||||
A relational database is a digital database based on the relational model of data, as proposed by E. F. Codd in 1970. A system used to maintain relational databases is a relational database management system. Many relational database systems have an option of using the SQL for querying and maintaining the database.
|
||||
A relational database is a digital database based on the relational model of data, as proposed by E. F. Codd in 1970. A system used to maintain relational databases is a relational database management system. Many relational database systems have the option of using SQL for querying and maintaining the database.
|
||||
|
||||
_MySQL is an open-source relational database management system. Its name is a combination of "My", the name of co-founder Michael Widenius's daughter, and "SQL", the abbreviation for Structured Query Language._
|
||||
|
||||
@ -103,9 +103,9 @@ Best for:
|
||||
|
||||
### Search Engine
|
||||
|
||||
In the last section we actually used a Search Engine database in the way of Elasticsearch.
|
||||
In the last section, we used a Search Engine database in the way of Elasticsearch.
|
||||
|
||||
A search-engine database is a type of non-relational database that is dedicated to the search of data content. Search-engine databases use indexes to categorise the similar characteristics among data and facilitate search capability.
|
||||
A search-engine database is a type of non-relational database that is dedicated to the search for data content. Search-engine databases use indexes to categorise similar characteristics among data and facilitate search capability.
|
||||
|
||||
_Elasticsearch is a search engine based on the Lucene library. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents._
|
||||
|
||||
@ -117,13 +117,13 @@ Best for:
|
||||
|
||||
### Multi-model
|
||||
|
||||
A multi-model database is a database management system designed to support multiple data models against a single, integrated backend. In contrast, most database management systems are organized around a single data model that determines how data can be organized, stored, and manipulated.Document, graph, relational, and key–value models are examples of data models that may be supported by a multi-model database.
|
||||
A multi-model database is a database management system designed to support multiple data models against a single, integrated backend. In contrast, most database management systems are organized around a single data model that determines how data can be organized, stored, and manipulated. Document, graph, relational, and key–value models are examples of data models that may be supported by a multi-model database.
|
||||
|
||||
_Fauna is a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL._
|
||||
|
||||
Best for:
|
||||
|
||||
- You are not stuck to having to choose a data model
|
||||
- You are not stuck on having to choose a data model
|
||||
- ACID Compliant
|
||||
- Fast
|
||||
- No provisioning overhead
|
||||
|
||||
@ -10,29 +10,29 @@ id: 1049058
|
||||
|
||||
## Backup all the platforms
|
||||
|
||||
During this whole challenge we have discussed many different platforms and environments. One thing all of those have in common is the fact they all need some level of data protection!
|
||||
During this whole challenge, we discussed many different platforms and environments. One thing all of those have in common is the fact they all need some level of data protection!
|
||||
|
||||
Data Protection has been around for many many years but the wealth of data that we have today and the value that this data brings means we have to make sure we are not only resilient to infrastructure failure by having multiple nodes and high availablity across applications but we must also consider that we need a copy of that data, that important data in a safe and secure location if a failure scenario was to occur.
|
||||
Data Protection has been around for many many years but the wealth of data that we have today and the value that this data brings means we have to make sure we are not only resilient to infrastructure failure by having multiple nodes and high availability across applications but we must also consider that we need a copy of that data, that important data in a safe and secure location if a failure scenario was to occur.
|
||||
|
||||
We hear a lot these days it seems about cybercrime and ransomware, and don't get me wrong this is a massive threat and I stand by the fact that you will be attacked by ransomware. It is not a matter of if it is a matter of when. So even more reason to make sure you have your data secure for when that time arises. However the most common cause for data loss is not ransomware or cybercrime it is simply accidental deletion!
|
||||
We hear a lot these days it seems about cybercrime and ransomware, and don't get me wrong this is a massive threat and I stand by the fact that you will be attacked by ransomware. It is not a matter of if it is a matter of when. So even more reason to make sure you have your data secure for when that time arises. However, the most common cause for data loss is not ransomware or cybercrime it is simply accidental deletion!
|
||||
|
||||
We have all done it, deleted something we shouldn't have and had that instant regret.
|
||||
|
||||
With all of the technology and automation we have discussed during the challenge, the requirement to protect any stateful data or even complex stateless configuration is still there, regardless of platform.
|
||||
With all of the technology and automation we have discussed during the challenge, the requirement to protect any stateful data or even complex stateless configuration is still there, regardless of the platform.
|
||||
|
||||
|
||||
|
||||
But we should be able to perform that protection of the data with automation in mind and being able to integrate into our workflows.
|
||||
But we should be able to perform that protection of the data with automation in mind and be able to integrate it into our workflows.
|
||||
|
||||
If we look at what backup is:
|
||||
|
||||
_In information technology, a backup, or data backup is a copy of computer data taken and stored elsewhere so that it may be used to restore the original after a data loss event. The verb form, referring to the process of doing so, is "back up", whereas the noun and adjective form is "backup"._
|
||||
|
||||
If we break this down to the simplest form, a backup is a copy and paste of data to a new location. Simply put I could take a backup right now by copying a file from my C: drive to my D: drive and I would then have a copy in case something happened to the C: drive or something was edited wrongly within the files. I could revert back to the copy I have on the D: drive. Now if my computer dies where both the C & D drives live then I am not protected so I have to consider a solution or a copy of data outside of my system maybe onto a NAS drive in my house? But then what happens if something happens to my house, maybe I need to consider storing it on another system in another location, maybe the cloud is an option. Maybe I could store a copy of my important files in several locations to mitigate against the risk of failure?
|
||||
If we break this down to the simplest form, a backup is a copy and paste of data to a new location. Simply put I could take a backup right now by copying a file from my C: drive to my D: drive and I would then have a copy in case something happened to the C: drive or something was edited wrongly within the files. I could revert to the copy I have on the D: drive. Now if my computer dies where both the C & D drives live then I am not protected so I have to consider a solution or a copy of data outside of my system maybe onto a NAS drive in my house? But then what happens if something happens to my house, maybe I need to consider storing it on another system in another location, maybe the cloud is an option. Maybe I could store a copy of my important files in several locations to mitigate the risk of failure?
|
||||
|
||||
### 3-2-1 Backup Methodolgy
|
||||
|
||||
Now seems a good time to talk about the 3-2-1 rule or backup methodology. I actually did a [lightening talk](https://www.youtube.com/watch?v=5wRt1bJfKBw) covering this topic.
|
||||
Now seems a good time to talk about the 3-2-1 rule or backup methodology. I did a [lightning talk](https://www.youtube.com/watch?v=5wRt1bJfKBw) covering this topic.
|
||||
|
||||
We have already mentioned before some of the extreme ends of why we need to protect our data but a few more are listed below:
|
||||
|
||||
@ -46,13 +46,13 @@ We then want to make sure we also send a copy of our data external or offsite th
|
||||
|
||||
### Backup Responsibility
|
||||
|
||||
We have most likely heard all of the myths when it comes to not having to backup, things like "Everything is stateless" I mean if everything is stateless then what is the business? no databases? word documents? Obviously there is a level of responsibility on every individual within the business to ensure they are protected but it is going to come down most likely to the operations teams to provide the backup process for the mission critical applications and data.
|
||||
We have most likely heard all of the myths when it comes to not having to backup, things like "Everything is stateless" I mean if everything is stateless then what is the business? no databases? word documents? There is a level of responsibility on every individual within the business to ensure they are protected but it is going to come down most likely to the operations teams to provide the backup process for the mission-critical applications and data.
|
||||
|
||||
Another good one is that "High availability is my backup, we have built in multiple nodes into our cluster there is no way this is going down!" apart from when you make a mistake to the database and this is replicated over all the nodes in the cluster, or there is fire, flood or blood scenario that means the cluster is no longer available and with it the important data. It's not about being stubborn it is about being aware of the data and the services, absolutely everyone should factor in high availability and fault tollerance into their architecture but that does not substitute the need for backup!
|
||||
Another good one is that "High availability is my backup, we have built in multiple nodes into our cluster there is no way this is going down!" apart from when you make a mistake to the database and this is replicated over all the nodes in the cluster, or there is fire, flood or blood scenario that means the cluster is no longer available and with it the important data. It's not about being stubborn it is about being aware of the data and the services, absolutely everyone should factor in high availability and fault tolerance into their architecture but that does not substitute the need for backup!
|
||||
|
||||
Replication can also seem to give us the offsite copy of the data and maybe that cluster mentioned above does live across multiple locations, however the first accidental mistake would still be replicated there. But again a Backup requirement should stand alongside application replication or system replication within the environment.
|
||||
Replication can also seem to give us the offsite copy of the data and maybe that cluster mentioned above does live across multiple locations, however, the first accidental mistake would still be replicated there. But again a Backup requirement should stand alongside application replication or system replication within the environment.
|
||||
|
||||
Now with all this said you can go to the extreme the other end as well and send copies of data to too many locations which is going to not only cost but also increase risk about being attacked as your surface area is now massively expanded.
|
||||
Now with all this said you can go to the extreme on the other end as well and send copies of data to too many locations which is going to not only cost but also increase the risk of being attacked as your surface area is now massively expanded.
|
||||
|
||||
Anyway, who looks after backup? It will be different within each business but someone should be taking it upon themselves to understand the backup requirements. But also understand the recovery plan!
|
||||
|
||||
@ -60,19 +60,19 @@ Anyway, who looks after backup? It will be different within each business but so
|
||||
|
||||
Backup is a prime example, nobody cares about backup until you need to restore something. Alongside the requirement to back our data up we also need to consider how we restore!
|
||||
|
||||
With our text document example we are talking very small files so the ability to copy back and forth is easy and fast. But if we are talking about 100GB plus files then this is going to take time. Also we have to consider the level in which we need to recover, if we take a virtual machine for example.
|
||||
With our text document example, we are talking about very small files so the ability to copy back and forth is easy and fast. But if we are talking about 100GB plus files then this is going to take time. Also, we have to consider the level at which we need to recover if we take a virtual machine for example.
|
||||
|
||||
We have the whole Virtual Machine, we have the Operating System, Application installation and then if this is a database server then we will have some database files as well. If we have made a mistake and inserted the wrong line of code into our database I probably don't need to restore the whole virtual machine, I want to be granular on what I recover back.
|
||||
We have the whole Virtual Machine, we have the Operating System, Application installation and then if this is a database server we will have some database files as well. If we have made a mistake and inserted the wrong line of code into our database I probably don't need to restore the whole virtual machine, I want to be granular on what I recover back.
|
||||
|
||||
### Backup Scenario
|
||||
|
||||
I want to now start building on a scenario to protect some data, specifically I want to protect some files on my local machine (in this case Windows but the tool I am going to use is in fact not only free and open-source but also cross platform) I would like to make sure they are protected to a NAS device I have locally in my home but also into an Object Storage bucket in the cloud.
|
||||
I want to now start building on a scenario to protect some data, specifically, I want to protect some files on my local machine (in this case Windows but the tool I am going to use is not only free and open-source but also cross-platform) I would like to make sure they are protected to a NAS device I have locally in my home but also into an Object Storage bucket in the cloud.
|
||||
|
||||
I want to backup this important data, it just so happens to be the repository for the 90DaysOfDevOps, which yes this is also being sent to GitHub which is probably where you are reading this now but what if my machine was to die and GitHub was down? How would anyone be able to read the content but also how would I potentially be able to restore that data to another service.
|
||||
I want to back up this important data, it just so happens to be the repository for the 90DaysOfDevOps, which yes is also being sent to GitHub which is probably where you are reading this now but what if my machine was to die and GitHub was down? How would anyone be able to read the content but also how would I potentially be able to restore that data to another service?
|
||||
|
||||
|
||||
|
||||
There are lots of tools that can help us achieve this but I am going to be using a a tool called [Kopia](https://kopia.io/) an Open-Source backup tool which will enable us to encrypt, dedupe and compress our backups whilst being able to send them to many locations.
|
||||
There are lots of tools that can help us achieve this but I am going to be using a tool called [Kopia](https://kopia.io/) an Open-Source backup tool which will enable us to encrypt, dedupe and compress our backups whilst being able to send them to many locations.
|
||||
|
||||
You will find the releases to download [here](https://github.com/kopia/kopia/releases) at the time of writing I will be using v0.10.6.
|
||||
|
||||
@ -82,25 +82,25 @@ There is a Kopia CLI and GUI, we will be using the GUI but know that you can hav
|
||||
|
||||
I will be using `KopiaUI-Setup-0.10.6.exe`
|
||||
|
||||
Really quick next next installation and then when you open the application you are greeted with the choice of selecting your storage type that you wish to use as your backup repository.
|
||||
Really quick next next installation and then when you open the application you are greeted with the choice of selecting the storage type that you wish to use as your backup repository.
|
||||
|
||||
|
||||
|
||||
### Setting up a Repository
|
||||
|
||||
Firstly we would like to setup a repository using our local NAS device and we are going to do this using SMB, but we could also use NFS I believe.
|
||||
Firstly we would like to set up a repository using our local NAS device and we are going to do this using SMB, but we could also use NFS I believe.
|
||||
|
||||
|
||||
|
||||
On the next screen we are going to define a password, this password is used to encrypt the repository contents.
|
||||
On the next screen, we are going to define a password, this password is used to encrypt the repository contents.
|
||||
|
||||
|
||||
|
||||
Now that we have the repository configured we can trigger an adhoc snapshot to start writing data to our it.
|
||||
Now that we have the repository configured we can trigger an ad-hoc snapshot to start writing data to it.
|
||||
|
||||
|
||||
|
||||
First up we need to enter a path to what we want to snapshot and our case we want to take a copy of our `90DaysOfDevOps` folder. We will get back to the scheduling aspect shortly.
|
||||
First up we need to enter a path to what we want to snapshot and in our case we want to take a copy of our `90DaysOfDevOps` folder. We will get back to the scheduling aspect shortly.
|
||||
|
||||
|
||||
|
||||
@ -112,11 +112,11 @@ Maybe there are files or file types that we wish to exclude.
|
||||
|
||||
|
||||
|
||||
If we wanted to define a schedule we could this on this next screen, when you first create this snapshot this is the opening page to define.
|
||||
If we wanted to define a schedule we could do this on this next screen, when you first create this snapshot this is the opening page to define.
|
||||
|
||||
|
||||
|
||||
And you will see a number of other settings that can be handled here.
|
||||
And you will see several other settings that can be handled here.
|
||||
|
||||
|
||||
|
||||
@ -126,9 +126,9 @@ Select snapshot now and the data will be written to your repository.
|
||||
|
||||
### Offsite backup to S3
|
||||
|
||||
With Kopia we can through the UI it seems only have one repository configured at a time. But through the UI we can be creative and basically have multiple repository configuration files to choose from to achieve our goal of having a copy local and offsite in Object Storage.
|
||||
With Kopia we can through the UI it seems only to have one repository configured at a time. But through the UI we can be creative and have multiple repository configuration files to choose from to achieve our goal of having a copy local and offsite in Object Storage.
|
||||
|
||||
The Object Storage I am choosing to send my data to is going to Google Cloud Storage. I firstly logged into my Google Cloud Platform account and created myself a storage bucket. I already had the Google Cloud SDK installed on my system but running the `gcloud auth application-default login` authenticated me with my account.
|
||||
The Object Storage I am choosing to send my data to is going to Google Cloud Storage. I firstly logged into my Google Cloud Platform account and created a storage bucket. I already had the Google Cloud SDK installed on my system but running the `gcloud auth application-default login` authenticated me with my account.
|
||||
|
||||
|
||||
|
||||
@ -136,7 +136,7 @@ I then used the CLI of Kopia to show me the current status of my repository afte
|
||||
|
||||
|
||||
|
||||
We are now ready to replace for the purpose of the demo the configuration for the repository, what we would probably do if we wanted a long term solution to hit both of these repositories is we would create an `smb.config` file and a `object.config` file and be able to run both of these commands to send our copies of data to each location. To add our repository we ran `"C:\Program Files\KopiaUI\resources\server\kopia.exe" --config-file=C:\Users\micha\AppData\Roaming\kopia\repository.config repository create gcs --bucket 90daysofdevops`
|
||||
We are now ready to replace for the demo the configuration for the repository, what we would probably do if we wanted a long-term solution to hit both of these repositories is we would create an `smb.config` file and a `object.config` file and be able to run both of these commands to send our copies of data to each location. To add our repository we ran `"C:\Program Files\KopiaUI\resources\server\kopia.exe" --config-file=C:\Users\micha\AppData\Roaming\kopia\repository.config repository create gcs --bucket 90daysofdevops`
|
||||
|
||||
The above command is taking into account that the Google Cloud Storage bucket we created is called `90daysofdevops`
|
||||
|
||||
@ -146,15 +146,15 @@ Now that we have created our new repository we can then run the `"C:\Program Fil
|
||||
|
||||
|
||||
|
||||
Next thing we need to do is create a snapshot and send that to our newly created repository. Using the `"C:\Program Files\KopiaUI\resources\server\kopia.exe" --config-file=C:\Users\micha\AppData\Roaming\kopia\repository.config kopia snapshot create "C:\Users\micha\demo\90DaysOfDevOps"` command we can kick off this process. You can see in the below browser that our Google Cloud Storage bucket now has kopia files based on our backup in place.
|
||||
The next thing we need to do is create a snapshot and send that to our newly created repository. Using the `"C:\Program Files\KopiaUI\resources\server\kopia.exe" --config-file=C:\Users\micha\AppData\Roaming\kopia\repository.config kopia snapshot create "C:\Users\micha\demo\90DaysOfDevOps"` command we can kick off this process. You can see in the below browser that our Google Cloud Storage bucket now has kopia files based on our backup in place.
|
||||
|
||||
|
||||
|
||||
With the above process we are able to settle our requirement of sending our important data to 2 different locations, 1 of which is offsite in Google Cloud Storage and of course we still have our production copy of our data on a different media type.
|
||||
With the above process we can settle our requirement of sending our important data to 2 different locations, 1 of which is offsite in Google Cloud Storage and of course we still have our production copy of our data on a different media type.
|
||||
|
||||
### Restore
|
||||
|
||||
Restore is another consideration and is very important, Kopia gives us the capability to not only restore to the existing location but also to a new location.
|
||||
Restore is another consideration and is very important, Kopia gives us the capability to not only restore to the existing location but also a new location.
|
||||
|
||||
If we run the command `"C:\Program Files\KopiaUI\resources\server\kopia.exe" --config-file=C:\Users\micha\AppData\Roaming\kopia\repository.config snapshot list` this will list the snapshots that we have currently in our configured repository (GCS)
|
||||
|
||||
@ -166,9 +166,9 @@ We can then mount those snapshots directly from GCS using the `"C:\Program Files
|
||||
|
||||
We could also restore the snapshot contents using `kopia snapshot restore kdbd9dff738996cfe7bcf99b45314e193`
|
||||
|
||||
Obviously the commands above are very long and this is because I was using the KopiaUI version of the kopia.exe as explained at the top of the walkthrough you can download the kopia.exe and put into a path so you can just use the `kopia` command.
|
||||
The commands above are very long and this is because I was using the KopiaUI version of the kopia.exe as explained at the top of the walkthrough you can download the kopia.exe and put it into a path so you can just use the `kopia` command.
|
||||
|
||||
In the next session we will be focusing in on protecting workloads within Kubernetes.
|
||||
In the next session, we will be focusing on protecting workloads within Kubernetes.
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -10,7 +10,7 @@ id: 1048717
|
||||
|
||||
## Hands-On Backup & Recovery
|
||||
|
||||
In the last session we touched on [Kopia](https://kopia.io/) an Open-Source backup tool that we used to get some important data off to a local NAS and off to some cloud based object storage.
|
||||
In the last session, we touched on [Kopia](https://kopia.io/) an Open-Source backup tool that we used to get some important data off to a local NAS and off to some cloud-based object storage.
|
||||
|
||||
In this section, I want to get into the world of Kubernetes backup. It is a platform we covered [The Big Picture: Kubernetes](Days/day49.md) earlier in the challenge.
|
||||
|
||||
@ -18,9 +18,9 @@ We will again be using our minikube cluster but this time we are going to take a
|
||||
|
||||
### Kubernetes cluster setup
|
||||
|
||||
To set up our minikube cluster we will be issuing the `minikube start --addons volumesnapshots,csi-hostpath-driver --apiserver-port=6443 --container-runtime=containerd -p 90daysofdevops --kubernetes-version=1.21.2` you will notice that we are using the `volumesnapshots` and `csi-hostpath-driver` as we will take full use of these for when we are taking our backups.
|
||||
To set up our minikube cluster we will be issuing the `minikube start --addons volumesnapshots,csi-hostpath-driver --apiserver-port=6443 --container-runtime=containerd -p 90daysofdevops --kubernetes-version=1.21.2` you will notice that we are using the `volumesnapshots` and `csi-hostpath-driver` as we will make full use of these for when we are taking our backups.
|
||||
|
||||
At this point I know we have not deployed Kasten K10 yet but we want to issue the following command when your cluster is up, but we want to annotate the volumesnapshotclass so that Kasten K10 can use this.
|
||||
At this point I know we have not deployed Kasten K10 yet but we want to issue the following command when your cluster is up, we want to annotate the volumesnapshotclass so that Kasten K10 can use this.
|
||||
|
||||
```Shell
|
||||
kubectl annotate volumesnapshotclass csi-hostpath-snapclass \
|
||||
@ -43,7 +43,7 @@ Add the Kasten Helm repository
|
||||
|
||||
`helm repo add kasten https://charts.kasten.io/`
|
||||
|
||||
We could use `arkade kasten install k10` here as well but for the purpose of the demo we will run through the following steps. [More Details](https://blog.kasten.io/kasten-k10-goes-to-the-arkade)
|
||||
We could use `arkade kasten install k10` here as well but for the demo, we will run through the following steps. [More Details](https://blog.kasten.io/kasten-k10-goes-to-the-arkade)
|
||||
|
||||
Create the namespace and deploy K10, note that this will take around 5 mins
|
||||
|
||||
@ -61,7 +61,7 @@ Port forward to access the K10 dashboard, open a new terminal to run the below c
|
||||
|
||||
`kubectl --namespace kasten-io port-forward service/gateway 8080:8000`
|
||||
|
||||
The Kasten dashboard will be available at: `http://127.0.0.1:8080/k10/#/`
|
||||
The Kasten dashboard will be available at `http://127.0.0.1:8080/k10/#/`
|
||||
|
||||
|
||||
|
||||
@ -111,9 +111,9 @@ Take the time to clock up some high scores in the backend MongoDB database.
|
||||
|
||||
### Protect our High Scores
|
||||
|
||||
Now we have some mission critical data in our database and we do not want to lose it. We can use Kasten K10 to protect this whole application.
|
||||
Now we have some mission-critical data in our database and we do not want to lose it. We can use Kasten K10 to protect this whole application.
|
||||
|
||||
If we head back into the Kasten K10 dashboard tab you will see that our number of application has now increased from 1 to 2 with the addition of our pacman application to our Kubernetes cluster.
|
||||
If we head back into the Kasten K10 dashboard tab you will see that our number of applications has now increased from 1 to 2 with the addition of our Pacman application to our Kubernetes cluster.
|
||||
|
||||
|
||||
|
||||
@ -121,37 +121,37 @@ If you click on the Applications card you will see the automatically discovered
|
||||
|
||||
|
||||
|
||||
With Kasten K10 we have the ability to leverage storage based snapshots as well export our copies out to object storage options.
|
||||
With Kasten K10 we can leverage storage-based snapshots as well export our copies out to object storage options.
|
||||
|
||||
For the purpose of the demo, we will create a manual storage snapshot in our cluster and then we can add some rogue data to our high scores to simulate an accidental mistake being made or is it?
|
||||
For the demo, we will create a manual storage snapshot in our cluster and then we can add some rogue data to our high scores to simulate an accidental mistake being made or is it?
|
||||
|
||||
Firstly we can use the manual snapshot option below.
|
||||
|
||||
|
||||
|
||||
For the demo I am going to leave everything as the default
|
||||
For the demo, I am going to leave everything as the default
|
||||
|
||||
|
||||
|
||||
Back on the dashboard you get a status report on the job as it is running and then when complete it should look as successful as this one.
|
||||
Back on the dashboard, you get a status report on the job as it is running and then when complete it should look as successful as this one.
|
||||
|
||||
|
||||
|
||||
### Failure Scenario
|
||||
|
||||
We can now make that fatal change to our mission critical data by simply adding in a prescriptive bad change to our application.
|
||||
We can now make that fatal change to our mission-critical data by simply adding in a prescriptive bad change to our application.
|
||||
|
||||
As you can see below we have two inputs that we probably dont want in our production mission critical database.
|
||||
As you can see below we have two inputs that we probably don't want in our production mission-critical database.
|
||||
|
||||
|
||||
|
||||
### Restore the data
|
||||
|
||||
Obviously this is a simple demo and in a way not realistic although have you seen how easy it is to drop databases?
|
||||
This is a simple demo and in a way not realistic although have you seen how easy it is to drop databases?
|
||||
|
||||
Now we want to get that high score list looking a little cleaner and how we had it before the mistakes were made.
|
||||
|
||||
Back in the Applications card and on the pacman tab we now have 1 restore point we can use to restore from.
|
||||
Back in the Applications card and on the Pacman tab, we now have 1 restore point we can use to restore from.
|
||||
|
||||
|
||||
|
||||
@ -163,7 +163,7 @@ Select that restore and a side window will appear, we will keep the default sett
|
||||
|
||||
|
||||
|
||||
Confirm that you really want to make this happen.
|
||||
Confirm that you want to make this happen.
|
||||
|
||||
|
||||
|
||||
@ -171,13 +171,13 @@ You can then go back to the dashboard and see the progress of the restore. You s
|
||||
|
||||
|
||||
|
||||
But more importantly how is our High-Score list looking in our mission critical application. You will have to start the port forward again to pacman as we previously covered.
|
||||
But more importantly, how is our High-Score list looking in our mission-critical application. You will have to start the port forward again to Pacman as we previously covered.
|
||||
|
||||
|
||||
|
||||
A super simple demo and only really touching the surface of what Kasten K10 can really achieve when it comes to backup. I will be creating some more in depth video content on some of these areas in the future. We will also be using Kasten K10 to highlight some of the other prominent areas around Data Management when it comes to Disaster Recovery and the mobility of your data.
|
||||
A super simple demo and only really touching the surface of what Kasten K10 can achieve when it comes to backup. I will be creating some more in-depth video content on some of these areas in the future. We will also be using Kasten K10 to highlight some of the other prominent areas around Data Management when it comes to Disaster Recovery and the mobility of your data.
|
||||
|
||||
Next we will take a look at Application consistency.
|
||||
Next, we will take a look at Application consistency.
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -8,13 +8,13 @@ canonical_url: null
|
||||
id: 1048749
|
||||
---
|
||||
|
||||
## Application Focused Backups
|
||||
## Application-Focused Backups
|
||||
|
||||
We have already spent some time talking about data services or data intensive applications such as databases on [Day 85](day85.md). For these data services we have to consider how we manage consistency, especially when it comes application consistency.
|
||||
We have already spent some time talking about data services or data-intensive applications such as databases on [Day 85](day85.md). For these data services, we have to consider how we manage consistency, especially when it comes to application consistency.
|
||||
|
||||
In this post we are going to dive into that requirement around protecting the application data in a consistent manner.
|
||||
In this post, we are going to dive into that requirement around consistently protecting the application data.
|
||||
|
||||
In order to do this our tool of choice will be [Kanister](https://kanister.io/)
|
||||
To do this our tool of choice will be [Kanister](https://kanister.io/)
|
||||
|
||||
|
||||
|
||||
@ -30,7 +30,7 @@ Kanister uses Kubernetes custom resources, the main custom resources that are in
|
||||
|
||||
### Execution Walkthrough
|
||||
|
||||
Before we get hands on we should take a look at the workflow that Kanister takes in protecting application data. Firstly our controller is deployed using helm into our Kubernetes cluster, Kanister lives within its own namespace. We take our Blueprint of which there are many community supported blueprints available, we will cover this in more detail shortly. We then have our database workload.
|
||||
Before we get hands-on we should take a look at the workflow that Kanister takes in protecting application data. Firstly our controller is deployed using helm into our Kubernetes cluster, Kanister lives within its namespace. We take our Blueprint of which there are many community-supported blueprints available, we will cover this in more detail shortly. We then have our database workload.
|
||||
|
||||
|
||||
|
||||
@ -42,7 +42,7 @@ The ActionSet allows us to run the actions defined in the blueprint against the
|
||||
|
||||
|
||||
|
||||
The ActionSet in turns uses the Kanister functions (KubeExec, KubeTask, Resource Lifecycle) and pushes our backup to our target repository (Profile).
|
||||
The ActionSet in turn uses the Kanister functions (KubeExec, KubeTask, Resource Lifecycle) and pushes our backup to our target repository (Profile).
|
||||
|
||||
|
||||
|
||||
@ -54,7 +54,7 @@ If that action is completed/failed the respective status is updated in the Actio
|
||||
|
||||
Once again we will be using the minikube cluster to achieve this application backup. If you have it still running from the previous session then we can continue to use this.
|
||||
|
||||
At the time of writing we are up to image version `0.75.0` with the following helm command we will install kanister into our Kubernetes cluster.
|
||||
At the time of writing, we are up to image version `0.75.0` with the following helm command we will install kanister into our Kubernetes cluster.
|
||||
|
||||
`helm install kanister --namespace kanister kanister/kanister-operator --set image.tag=0.75.0 --create-namespace`
|
||||
|
||||
@ -66,7 +66,7 @@ We can use `kubectl get pods -n kanister` to ensure the pod is up and running an
|
||||
|
||||
### Deploy a Database
|
||||
|
||||
Deploying mysql via helm:
|
||||
Deploying MySQL via helm:
|
||||
|
||||
```Shell
|
||||
APP_NAME=my-production-app
|
||||
@ -78,7 +78,7 @@ kubectl get pods -n ${APP_NAME} -w
|
||||
|
||||
|
||||
|
||||
Populate the mysql database with initial data, run the following:
|
||||
Populate the MySQL database with initial data, and run the following:
|
||||
|
||||
```Shell
|
||||
MYSQL_ROOT_PASSWORD=$(kubectl get secret --namespace ${APP_NAME} mysql-store -o jsonpath="{.data.mysql-root-password}" | base64 --decode)
|
||||
@ -97,7 +97,7 @@ kubectl run mysql-client --rm --env APP_NS=${APP_NAME} --env MYSQL_EXEC="${MYSQL
|
||||
```
|
||||
|
||||
```Shell
|
||||
Note: if you already have an existing mysql client pod running, delete with the command
|
||||
Note: if you already have an existing MySQL client pod running, delete with the command
|
||||
|
||||
kubectl delete pod -n ${APP_NAME} mysql-client
|
||||
```
|
||||
@ -127,11 +127,11 @@ You should be able to see some data as per below.
|
||||
|
||||
### Create Kanister Profile
|
||||
|
||||
Kanister provides a CLI, `kanctl` and another utility `kando` that is used to interact with your object storage provider from blueprint and both of these utilities.
|
||||
Kanister provides a CLI, `kanctl` and another utility `kando` that is used to interact with your object storage provider from the blueprint and both of these utilities.
|
||||
|
||||
[CLI Download](https://docs.kanister.io/tooling.html#tooling)
|
||||
|
||||
I have gone and I have created an AWS S3 Bucket that we will use as our profile target and restore location. I am going to be using environment variables so that I am able to still show you the commands I am running with `kanctl` to create our kanister profile.
|
||||
I have gone and I have created an AWS S3 Bucket that we will use as our profile target and restore location. I am going to be using environment variables so that I can still show you the commands I am running with `kanctl` to create our kanister profile.
|
||||
|
||||
`kanctl create profile s3compliant --access-key $ACCESS_KEY --secret-key $SECRET_KEY --bucket $BUCKET --region eu-west-2 --namespace my-production-app`
|
||||
|
||||
@ -139,7 +139,7 @@ I have gone and I have created an AWS S3 Bucket that we will use as our profile
|
||||
|
||||
### Blueprint time
|
||||
|
||||
Don't worry you don't need to create your own one from scratch unless your data service is not listed here in the [Kanister Examples](https://github.com/kanisterio/kanister/tree/master/examples) but by all means community contributions are how this project gains awareness.
|
||||
Don't worry you don't need to create your one from scratch unless your data service is not listed here in the [Kanister Examples](https://github.com/kanisterio/kanister/tree/master/examples) but by all means, community contributions are how this project gains awareness.
|
||||
|
||||
The blueprint we will be using will be the below.
|
||||
|
||||
@ -243,7 +243,7 @@ You can see from the command above we are defining the blueprint we added to the
|
||||
|
||||
Check the status of the ActionSet by taking the ActionSet name and using this command `kubectl --namespace kanister describe actionset backup-qpnqv`
|
||||
|
||||
Finally we can go and confirm that we now have data in our AWS S3 bucket.
|
||||
Finally, we can go and confirm that we now have data in our AWS S3 bucket.
|
||||
|
||||
|
||||
|
||||
@ -258,7 +258,7 @@ APP_NAME=my-production-app
|
||||
kubectl run mysql-client --rm --env APP_NS=${APP_NAME} --env MYSQL_EXEC="${MYSQL_EXEC}" --env MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD} --env MYSQL_HOST=${MYSQL_HOST} --namespace ${APP_NAME} --tty -i --restart='Never' --image docker.io/bitnami/mysql:latest --command -- bash
|
||||
```
|
||||
|
||||
You can see that our importantdata db is there with `echo "SHOW DATABASES;" | ${MYSQL_EXEC}`
|
||||
You can see that our importantdata DB is there with `echo "SHOW DATABASES;" | ${MYSQL_EXEC}`
|
||||
|
||||
Then to drop we ran `echo "DROP DATABASE myImportantData;" | ${MYSQL_EXEC}`
|
||||
|
||||
@ -266,7 +266,7 @@ And confirmed that this was gone with a few attempts to show our database.
|
||||
|
||||
|
||||
|
||||
We can now use Kanister to get our important data back in business using the `kubectl get actionset -n kanister` to find out our ActionSet name that we took earlier. Then we will create a restore ActionSet to restore our data using `kanctl create actionset -n kanister --action restore --from "backup-qpnqv"`
|
||||
We can now use Kanister to get our important data back in business using the `kubectl get actionset -n kanister` to find out the ActionSet name that we took earlier. Then we will create a restore ActionSet to restore our data using `kanctl create actionset -n kanister --action restore --from "backup-qpnqv"`
|
||||
|
||||
|
||||
|
||||
@ -281,7 +281,7 @@ Now we are inside the MySQL Client, we can issue the `echo "SHOW DATABASES;" | $
|
||||
|
||||
|
||||
|
||||
In the next post we take a look at Disaster Recovery within Kubernetes.
|
||||
In the next post, we take a look at Disaster Recovery within Kubernetes.
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@ -16,11 +16,11 @@ This can only be achieved at scale when you automate the replication of the comp
|
||||
|
||||
This allows for fast failovers across cloud regions, cloud providers or between on-premises and cloud infrastructure.
|
||||
|
||||
Keeping with the theme so far, we are going to concentrate on how this can be achieved using Kasten K10 using our minikube cluster that we deployed and configured a few sessions ago.
|
||||
Keeping with the theme so far, we are going to concentrate on how this can be achieved using Kasten K10 using the minikube cluster that we deployed and configured a few sessions ago.
|
||||
|
||||
We will then create another minikube cluster with Kasten K10 also installed to act as our standby cluster which in theory could be any location.
|
||||
|
||||
Kasten K10 also has built in functionality to ensure if something was to happen to the Kubernetes cluster it is running on that the catalog data is replicated and available in a new one [K10 Disaster Recovery](https://docs.kasten.io/latest/operating/dr.html).
|
||||
Kasten K10 also has built-in functionality to ensure if something was to happen to the Kubernetes cluster it is running on that the catalogue data is replicated and available in a new one [K10 Disaster Recovery](https://docs.kasten.io/latest/operating/dr.html).
|
||||
|
||||
### Add object storage to K10
|
||||
|
||||
@ -34,7 +34,7 @@ Port forward to access the K10 dashboard, open a new terminal to run the below c
|
||||
|
||||
`kubectl --namespace kasten-io port-forward service/gateway 8080:8000`
|
||||
|
||||
The Kasten dashboard will be available at: `http://127.0.0.1:8080/k10/#/`
|
||||
The Kasten dashboard will be available at `http://127.0.0.1:8080/k10/#/`
|
||||
|
||||
|
||||
|
||||
@ -62,11 +62,11 @@ Now that we are back in the Kasten K10 dashboard we can add our location profile
|
||||
|
||||
|
||||
|
||||
You can see from the image below that we have choice when it comes to where this location profile is, we are going to select Amazon S3, and we are going to add our sensitive access credentials, region and bucket name.
|
||||
You can see from the image below that we have a choice when it comes to where this location profile is, we are going to select Amazon S3, and we are going to add our sensitive access credentials, region and bucket name.
|
||||
|
||||
|
||||
|
||||
If we scroll down on the New Profile creation window you will see, we also have the ability to enable immutable backups which leverages the S3 Object Lock API. For this demo we won't be using that.
|
||||
If we scroll down on the New Profile creation window you will see, that we also can enable immutable backups which leverage the S3 Object Lock API. For this demo, we won't be using that.
|
||||
|
||||
|
||||
|
||||
@ -74,9 +74,9 @@ Hit "Save Profile" and you can now see our newly created or added location profi
|
||||
|
||||
|
||||
|
||||
### Create a policy to protect Pac-Man app to object storage
|
||||
### Create a policy to protect the Pac-Man app to object storage
|
||||
|
||||
In the previous session we created only an ad-hoc snapshot of our Pac-Man application, therefore we need to create a backup policy that will send our application backups to our newly created object storage location.
|
||||
In the previous session, we created only an ad-hoc snapshot of our Pac-Man application, therefore we need to create a backup policy that will send our application backups to our newly created object storage location.
|
||||
|
||||
If you head back to the dashboard and select the Policy card you will see a screen as per below. Select "Create New Policy".
|
||||
|
||||
@ -98,7 +98,7 @@ Under Advanced settings we are not going to be using any of these but based on o
|
||||
|
||||
|
||||
|
||||
Finally select "Create Policy" and you will now see the policy in our Policy window.
|
||||
Finally, select "Create Policy" and you will now see the policy in our Policy window.
|
||||
|
||||
|
||||
|
||||
@ -106,7 +106,7 @@ At the bottom of the created policy, you will have "Show import details" we need
|
||||
|
||||
|
||||
|
||||
Before we move on, we just need to select "run once" to get a backup sent our object storage bucket.
|
||||
Before we move on, we just need to select "run once" to get a backup sent to our object storage bucket.
|
||||
|
||||
|
||||
|
||||
@ -116,7 +116,7 @@ Below, the screenshot is just to show the successful backup and export of our da
|
||||
|
||||
### Create a new MiniKube cluster & deploy K10
|
||||
|
||||
We then need to deploy a second Kubernetes cluster and where this could be any supported version of Kubernetes including OpenShift, for the purpose of education we will use the very free version of MiniKube with a different name.
|
||||
We then need to deploy a second Kubernetes cluster and where this could be any supported version of Kubernetes including OpenShift, for education we will use the very free version of MiniKube with a different name.
|
||||
|
||||
Using `minikube start --addons volumesnapshots,csi-hostpath-driver --apiserver-port=6443 --container-runtime=containerd -p standby --kubernetes-version=1.21.2` we can create our new cluster.
|
||||
|
||||
@ -128,7 +128,7 @@ We then can deploy Kasten K10 in this cluster using:
|
||||
|
||||
This will take a while but in the meantime, we can use `kubectl get pods -n kasten-io -w` to watch the progress of our pods getting to the running status.
|
||||
|
||||
It is worth noting that because we are using MiniKube our application will just run when we run our import policy, our storageclass is the same on this standby cluster. However, something we will cover in the final session is about mobility and transformation.
|
||||
It is worth noting that because we are using MiniKube our application will just run when we run our import policy, our storageclass is the same on this standby cluster. However, something we will cover in the final session is mobility and transformation.
|
||||
|
||||
When the pods are up and running, we can follow the steps we went through on the previous steps in the other cluster.
|
||||
|
||||
@ -136,7 +136,7 @@ Port forward to access the K10 dashboard, open a new terminal to run the below c
|
||||
|
||||
`kubectl --namespace kasten-io port-forward service/gateway 8080:8000`
|
||||
|
||||
The Kasten dashboard will be available at: `http://127.0.0.1:8080/k10/#/`
|
||||
The Kasten dashboard will be available at `http://127.0.0.1:8080/k10/#/`
|
||||
|
||||
|
||||
|
||||
@ -162,7 +162,7 @@ Then we get access to the Kasten K10 dashboard.
|
||||
|
||||
### Import Pac-Man into new the MiniKube cluster
|
||||
|
||||
At this point we are now able to create an import policy in that standby cluster and connect to the object storage backups and determine what and how we want this to look.
|
||||
At this point, we are now able to create an import policy in that standby cluster and connect to the object storage backups and determine what and how we want this to look.
|
||||
|
||||
First, we add in our Location Profile that we walked through earlier on the other cluster, showing off dark mode here to show the difference between our production system and our DR standby location.
|
||||
|
||||
@ -172,11 +172,11 @@ Now we go back to the dashboard and into the policies tab to create a new policy
|
||||
|
||||
|
||||
|
||||
Create the import policy as per the below image. When complete, we can create policy. There are options here to restore after import and some people might want this option, this will go and restore into our standby cluster on completion. We also have the ability to change the configuration of the application as it is restored and this is what I have documented in [Day 90](day90.md).
|
||||
Create the import policy as per the below image. When complete, we can create a policy. There are options here to restore after import and some people might want this option, this will go and be restored into our standby cluster on completion. We also can change the configuration of the application as it is restored and this is what I have documented in [Day 90](day90.md).
|
||||
|
||||
|
||||
|
||||
I selected to import on demand, but you can obviously set a schedule on when you want this import to happen. Because of this I am going to run once.
|
||||
I selected to import on demand, but you can set a schedule on when you want this import to happen. Because of this, I am going to run once.
|
||||
|
||||
|
||||
|
||||
@ -184,7 +184,7 @@ You can see below the successful import policy job.
|
||||
|
||||
|
||||
|
||||
If we now head back to the dashboard and into the Applications card, we can then select the drop down where you see below "Removed" you will see our application here. Select Restore
|
||||
If we now head back to the dashboard and into the Applications card, we can then select the drop-down where you see below "Removed" you will see our application here. Select Restore
|
||||
|
||||
|
||||
|
||||
@ -204,7 +204,7 @@ We can see below that we are in the standby cluster and if we check on our pods,
|
||||
|
||||
|
||||
|
||||
We can then port forward (in real life/production environments, you would not need this step to access the application, you would be using ingress)
|
||||
We can then port forward (in real-life/production environments, you would not need this step to access the application, you would be using ingress)
|
||||
|
||||
|
||||
|
||||
|
||||
@ -10,25 +10,25 @@ id: 1048748
|
||||
|
||||
## Data & Application Mobility
|
||||
|
||||
Day 90 of the #90DaysOfDevOps Challenge! In this final session I am going to cover mobility of our data and applications. I am specifically going to focus on Kubernetes but the requirement across platforms and between platforms is something that is an ever-growing requirement and is seen in the field.
|
||||
Day 90 of the #90DaysOfDevOps Challenge! In this final session, I am going to cover the mobility of our data and applications. I am specifically going to focus on Kubernetes but the requirement across platforms and between platforms is something that is an ever-growing requirement and is seen in the field.
|
||||
|
||||
The use case being "I want to move my workload, application and data from one location to another" for many different reasons, could be cost, risk or to provide the business with a better service.
|
||||
|
||||
In this session we are going to take our workload and we are going to look at moving a Kubernetes workload from one cluster to another, but in doing so we are going to change how our application is on the target location.
|
||||
In this session, we are going to take our workload and we are going to look at moving a Kubernetes workload from one cluster to another, but in doing so we are going to change how our application is on the target location.
|
||||
|
||||
It in fact uses a lot of the characteristics that we went through with [Disaster Recovery](day89.md)
|
||||
It uses a lot of the characteristics that we went through with [Disaster Recovery](day89.md)
|
||||
|
||||
### **The Requirement**
|
||||
|
||||
Our current Kubernetes cluster cannot handle demand and our costs are rocketing through the roof, it is a business decision that we wish to move our production Kubernetes cluster to our Disaster Recovery location, located on a different public cloud which will provide the ability to expand but also at a cheaper rate. We could also take advantage of some of the native cloud services available in the target cloud.
|
||||
|
||||
Our current mission critical application (Pac-Man) has a database (MongoDB) and is running on slow storage, we would like to move to a newer faster storage tier.
|
||||
Our current mission-critical application (Pac-Man) has a database (MongoDB) and is running on slow storage, we would like to move to a newer faster storage tier.
|
||||
|
||||
The current Pac-Man (NodeJS) front-end is not scaling very well, and we would like to increase the number of available pods in the new location.
|
||||
|
||||
### Getting to IT
|
||||
|
||||
We have our brief and in fact we have our imports already hitting the Disaster Recovery Kubernetes cluster.
|
||||
We have our brief and in fact, we have our imports already hitting the Disaster Recovery Kubernetes cluster.
|
||||
|
||||
The first job we need to do is remove the restore operation we carried out on Day 89 for the Disaster Recovery testing.
|
||||
|
||||
@ -36,15 +36,15 @@ We can do this using `kubectl delete ns pacman` on the "standby" minikube cluste
|
||||
|
||||
|
||||
|
||||
To get started head into the Kasten K10 Dashboard, select the Applications card. From the dropdown choose "Removed"
|
||||
To get started head into the Kasten K10 Dashboard, and select the Applications card. From the dropdown choose "Removed"
|
||||
|
||||
|
||||
|
||||
We then get a list of the available restore points. We will select the one that is available as this contains our mission critical data. (In this example we only have a single restore point.)
|
||||
We then get a list of the available restore points. We will select the one that is available as this contains our mission-critical data. (In this example we only have a single restore point.)
|
||||
|
||||
|
||||
|
||||
When we worked on the Disaster Recovery process, we left everything as default. However these additional restore options are there if you have a Disaster Recovery process that requires the transformation of your application. In this instance we have the requirement to change our storage and number of replicas.
|
||||
When we worked on the Disaster Recovery process, we left everything as default. However, these additional restore options are there if you have a Disaster Recovery process that requires the transformation of your application. In this instance, we have the requirement to change our storage and number of replicas.
|
||||
|
||||
|
||||
|
||||
@ -52,7 +52,7 @@ Select the "Apply transforms to restored resources" option.
|
||||
|
||||
|
||||
|
||||
It just so happens that the two built in examples for the transformation that we want to perform are what we need for our requirements.
|
||||
It just so happens that the two built-in examples for the transformation that we want to perform are what we need for our requirements.
|
||||
|
||||
|
||||
|
||||
@ -72,7 +72,7 @@ If you are following along you should see both of our transforms as per below.
|
||||
|
||||
|
||||
|
||||
You can now see from the below image that we are going to restore all of the artifacts listed below, if we wanted to we could also be granular about what we wanted to restore. Hit the "Restore" button
|
||||
You can now see from the below image that we are going to restore all of the artefacts listed below, if we wanted to we could also be granular about what we wanted to restore. Hit the "Restore" button
|
||||
|
||||
|
||||
|
||||
@ -80,15 +80,15 @@ Again, we will be asked to confirm the actions.
|
||||
|
||||
|
||||
|
||||
The final thing to show is now if we head back into the terminal and we take a look at our cluster, you can see we have 5 pods now for the pacman pods and our storageclass is now set to standard vs the csi-hostpath-sc
|
||||
The final thing to show is now if we head back into the terminal and we take a look at our cluster, you can see we have 5 pods now for the Pacman pods and our storageclass is now set to standard vs the csi-hostpath-sc
|
||||
|
||||
|
||||
|
||||
There are many different options that can be achieved through transformation. This can span not only migration but also Disaster Recovery, test and development type scenarios and more.
|
||||
Many different options can be achieved through transformation. This can span not only migration but also Disaster Recovery, test and development type scenarios and more.
|
||||
|
||||
### API and Automation
|
||||
|
||||
I have not spoken about the ability to leverage the API and to automate some of these tasks, but these options are present and throughout the UI there are breadcrumbs that provide the command sets to take advantage of the APIs for automation tasks.
|
||||
I have not spoken about the ability to leverage the API and automate some of these tasks, but these options are present and throughout the UI some breadcrumbs provide the command sets to take advantage of the APIs for automation tasks.
|
||||
|
||||
The important thing to note about Kasten K10 is that on deployment it is deployed inside the Kubernetes cluster and then can be called through the Kubernetes API.
|
||||
|
||||
@ -108,13 +108,13 @@ As I wrap up this challenge, I want to continue to ask for feedback to make sure
|
||||
|
||||
I also appreciate there are a lot of topics that I was not able to cover or not able to dive deeper into around the topics of DevOps.
|
||||
|
||||
This means that we can always take another attempt that this challenge next year and find another 90 day's worth of content and walkthroughs to work through.
|
||||
This means that we can always make another attempt that this challenge next year and find another 90 days' worth of content and walkthroughs to work through.
|
||||
|
||||
### What is next?
|
||||
|
||||
Firstly, a break from writing for a little while, I started this challenge on the 1st January 2022 and I have finished on the 31st March 2022 19:50 BST! It has been a slog. But as I say and have said for a long time, if this content helps one person, then it is always worth learning in public!
|
||||
Firstly, a break from writing for a little while, I started this challenge on the 1st of January 2022 and I finished on the 31st of March 2022 at 19:50 BST! It has been a slog. But as I say and have said for a long time, if this content helps one person, then it is always worth learning in public!
|
||||
|
||||
I have some ideas on where to take this next and hopefully it has a life outside of a GitHub repository and we can look at creating an eBook and possibly even a physical book.
|
||||
I have some ideas on where to take this next and hopefully, it has a life outside of a GitHub repository and we can look at creating an eBook and possibly even a physical book.
|
||||
|
||||
I also know that we need to revisit each post and make sure everything is grammatically correct before making anything like that happen. If anyone does know about how to take markdown to print or to an eBook it would be greatly appreciated feedback.
|
||||
|
||||
|
||||
Loading…
Reference in New Issue
Block a user