How-To improve agile and ensure consistent monitoring for your team /w Dashboards via FluxCD IaC

Anyone who has worked with OSS monitoring tools like Prometheus and Grafana would attest that these are some great and the #1 choice of tools to be deployed within the Kubernetes containerised environment.

There is a long list of admin overhead to be considered while configuring the monitoring stack to benefit from all the features it offers, such as tuning, customising, and simply made to work.

It takes time, and given the must-have insight you achieve within your container cluster, there is really no doubt about the value of monitoring. I will just leave it as that. fact.

Prometheus Monitoring stack is made of at the very least several components such as;
- Prometheus(core) for metrics collection and retention, alert triggering
- Alertmanager — a bolt-on integration for alerting into other services such as Slack and Pagerduty
- Grafana — the visual UI tool for easy and handy one-stop-shop for all the application monitoring dashboards. This is the bit we aim to simplify and democratise

From Grafana version 5, there is an improvement to support provisioned dashboards. Furthermore, these dashboards, which can come from configmaps and auto-loaded by grafana without further need to kick the deployment roll, to re-read the configmaps in Kubernetes.

The full Prometheus Stack can be deployed via the handy Prometheus-Operator which can be configured to feature Grafana Dashboard UI from the get-go.

Demo APP Dashboard which features the istio-derived header metrics

FluxCD, on the other hand, is the Continuous Delivery (CD) application for the Kubernetes containerised environments.
My colleague Sean Rigby did a great breakdown of the Flux feature set offering in GitOps context here. This fluxCD tool, continuously polls the specified GitHub repository for changes, pulling the change-set and effectively kubectl apply -f <all> on the manifests. This results in an immensely simple and elegant continuous delivery process for any number of use cases, — from full application stacks under the GitOps CICD release pipeline, to partial Kubernetes YAML manifest delivery, — and in our case Grafana dashboards in configmaps.

Duedil is keen on DevOps practices, delivered by automation, and we are fixing several toils at once using FluxCD to deploy Monitoring dashboards, to enable team-wide DevOps values:

• The Development Team is in charge of all and any monitoring Dashboards and Metric assessment. The Team is empowered to create a dashboard, having it reviewed and deployed to all required environments automatically.

• There is a full audit trail for the dashboard and the relevance can be assessed over time, improving the dashboard ownership — “who created that dashboard” and relevance — “ what does this dashboard show me”.
• No more lost dashboards. They are all stored in the GitHub repository.

Example of My-Dashboard: Grafana Dashboard JSON in the Kubernetes ConfigMap wrapper

• Consistency — Given the Prometheus stack version parity in respective Kubernetes cluster environments. FluxCD will poll changes, pull and deploy identical dashboards into all designated environments.
• Reduced Toil — All dashboards are released automatically, once merged into the master branch.
  Create and Prototype in one environment and automatically deploy it into other environments. No more manual dashboard imports from your local machine. This automatic FluxCD release model is also extendable for additional environments as necessary.
• The reduced overhead for the Infrastructure team. There is no dedicated single support or manager for the monitoring dashboards. The whole Team owns this responsibility. The application-specific dashboards are owned by the development team. Everyone part-takes in the development of quality monitoring solutions, dashboards, and even application alerting configuration.

Mike Morley, an Engineering Manager at Duedil, says —

Due to a lot of migration work we had a lot of old broken tooling no longer fit for purpose, implementing flux as a process was straightforward and gave us the ability to tie it into our build processes in a standard centralised way with the GitOps model allowing the state of our estate to be defined it also builds in a BC/DR process at the application and configuration level, — this saves time for team members deploying and configuring test environments.

It also provides a *known state* that prevents any small changes and tweaks from building up over time, without a full audit trail in version control.

There you have it.

The road to the full agile release process, nevermind the CI/CD process, is individual to every organisation. The Continuous Delivery of the Monitoring dashboard is one great example of how to enable that seamless, agile delivery.

Right alongside that is the remaining monitoring stack components, such as Prometheus PrometheusRules — also known as Alerts, which are also owned by the Application Team, released during the application release lifecycle.

Whichever model you decide to choose for your organisation, consider using FluxCD to benefit from the automated release process within your Kubernetes environment.

I hope this post inspired you on your onward agile delivery journey, implementing DevOps practices.

Next,

If you’re hands-on with FluxCD and Kubernetes is operating is making you grow weary, consider exploring just how the #GitOps CI/CD release pipeline can suit you better.

Naturally, It brings its own sets of challenges alongside thistechnical excitement. There is another juicy write-up on this here — Building Cloud-Native #GitOps on Google Cloud Platform with Cloud Build and Kubernetes Engine

Finally, if you wish to get hands-on such varied-scoped projects — join us! We’re always looking for bright minds and great professionals in this area. We’re hiring, do get in touch!