Stand and Deliver

Visualizing data trends and issues when they arise is clearly of significant benefit, whether for applications or the systems on which they run. Having access to sophisticated tools that can provide highly customizable options to create bespoke visual dashboards is key and an important facet in being able to maintain high levels of uptime. A welcome bonus is if such tools are compatible with multiple types of data sources, so you can centralize your single-pane-of-glass viewing platform.

Step forward, Grafana [1]. It has the capability to gather metrics from all types of applications and treat each as an individual data source. From there, it is possible to query and filter results precisely to your needs. Coupled with another exceptional bedmate, Prometheus [2], it is possible to incorporate monitoring and alerting into a uber-capable, bespoke visualization toolset. Whereas Grafana is open source visualization software, Prometheus is an open source event monitoring and alerting tool. The two applications do have some crossover [3], though, and the dashboards from Grafana sit well with Prometheus' sophisticated time series handling [4].

In this article, I look at Grafana and Prometheus and how to get started with dashboarding.

Spirographs

For the example in this article, I'm using Linux Mint 20 (Ulyana), which has the long-term support (LTS) version of Ubuntu 20.04 running under the bonnet. In true DevOps style, I will not install Grafana locally, because in the past, I have seen some compatibility issues when running with Prometheus if it was also installed locally. Instead, I will use Docker containers for both Grafana and Prometheus.

For Grafana, I am using the clear instructions on the Grafana website [5]; remember to get the latest versions and decide whether to use the Alpine or Ubuntu version, depending on your needs. The simple command to fire up the latest version is:

$ docker run -d -p 3000:3000 grafana/grafana
3fa2f72de6bc4fd61a0cecdb1d18a8bf1aaf7b71b49f731fe4ae6eaab841d3fd

If you want to run a specific version, which some applications might prefer for compatibility, you can use a command that explicitly states a version as the image tag:

$ docker run -d -p 3000:3000 --name grafana grafana/grafana:6.5.0

After running the non-specific version of that command, you should see the output in Listing 1 (abbreviated here) to see which containers are running.

$ docker ps
CONTAINER ID   IMAGE               COMMAND      CREATED            STATUS
3fa2f72de6bc   grafana/grafana     "/run.sh"    25 seconds ago     Up 24 seconds

Gooey

If you now check TCP port 3000 in a browser by visiting http://localhost:3000/login, you will see the aesthetically pleasing Grafana interface (Figure 1). The user and password are admin, which you are immediately prompted to change at login.

On the right side of the Welcome page, click Create your first dashboard (Figure 2). Next, choose the Add new panel button to continue (Figure 3). You are then offered the Random Walk dashboard (Figure 4). If you select the small blue question mark icon next to default and choose the – Grafana – option in the pull-down menu under the Query box on the left side, you can read the description for that data source.

Here, you can flick through the different data source types and read the descriptions, which I'll look at in more detail in a moment. The default data source (Grafana Fake Data Datasource – Native Plugin) demonstrates nicely what you can expect from a simple panel; you should try zooming into it by selecting specific time periods to become familiar with this functionality.

Options allow you to adjust your query alongside transformations and alert settings. Under Transform, the following narrative is provided: "Transformations allow you to join, calculate, re-order, hide and rename your query results before being visualized." You can find more information on the Grafana website [6], which explains that "Transformations process the result set of a query before it's passed on for visualization." Apparently, when you have multiple dashboards with a high volume of queries, "the ability to reuse the query result from one panel in another panel can be a huge performance gain." As a result, it's well worth reading up more about transforming your data if you use dashboarding extensively.

If you click the Save button at top right and name your dashboard something like Random Walk, you can come back to it later. Next, click the Grafana logo at top left and go back to the Welcome screen.

Golden Source

As mentioned, Grafana supports a number of data sources that you might want to explore (Table 1). The documentation notes that, as well as setting up custom data sources, three other sources are treated a little differently, as you saw when you clicked the small blue question mark icon a moment ago to read their descriptions.

First, the – Grafana – test source helps you test graphing and dashboards' visual aspects. The – Mixed – data source allows you to query several sources in the same panel, so that every time you create a new query, you can choose a different data source. Finally, the – Dashboard – source lets you "use a result set from another panel" within the same dashboard. All in all, it's possible to use these special data sources to add extra flexibility to your graphing.

Many applications can push data to Grafana in a sane and compatible format. If you look at the documentation for Grafana plugins [7], you might be pleasantly surprised at the multitudinous array of community and officially provided plugins that help you get data into Grafana. Of the tens of plugins available, a number of well-known names include the monitoring solution Zabbix, the container orchestrator Kubernetes, the analytics engine Elasticsearch, the code repository GitHub, the databases MySQL and PostgreSQL, and the popular alerting tool Datadog. Incidentally, Grafana helpfully provides a GitHub repository [8] to help you get started creating your own plugins.

God of Fire

The next step is to connect Prometheus to Grafana. Again, I'll go down the Docker route with the official documentation [9]. Before beginning, create a configuration file and save it locally as /root/prometheus.yml (Listing 2). Now you can start up the Prometheus container with:

global:
  scrape_interval:     15s # Scrape frequency
  evaluation_interval: 15s # Check rules
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['127.0.0.1:9090'] # Open up Prometheus on TCP port 9090

$ docker run -p 9090:9090 -v /root/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

The command shows a volume is mounted for a specific file at the local /root file path. From that command, you are left with a running Docker container, but because you didn't detach the container (with the -d switch), if you close that terminal, it will stop; anything else you do will have to be in a second terminal. The STDOUT information is useful (Listing 3, abridged).

level=info ts=2020-11-15T19:11:23.675Z caller=web.go:516 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2020-11-15T19:11:23.697Z caller=head.go:642 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2020-11-15T19:11:23.697Z caller=head.go:656 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=3.766µs

The logging output looks relatively sane, so you should try to connect the two pieces of software. For proof that Prometheus is indeed working, however, point your browser to its web interface at http://localhost:9090/graph (Figure 5). If you want, you can run queries in the field above the Execute button, called the expression browser. You can find instructions on how this works on the main website [10].

Jigsaw Puzzle

By now, you should have a recognizable data source that Grafana can ingest. Figure 6 shows where to navigate to choose plugins: Click Plugins and the cog icon in the left sidebar. However, after typing Prometheus into the search box at the top of the page, the entry simply states "Grafana ships with built in support for Prometheus, the open-source service monitoring system and time series database" and encourages you to look at this in more detail on the Data Sources page [11]. In other words you are being politely redirected; therefore, you will instead try and use the Prometheus data source directly.

If you go back to the settings cog icon and choose Data Sources this time, click Add Data Source (Figure 7). After clicking Prometheus, assuming the Name box is already populated with the name of your data source, you can add the URL of your Prometheus server and continue.

To find the correct URL, you should open a new terminal (or the Prometheus container will stop) and run the command:

$ docker ps

Once you have found the hash ID for the Prometheus container on the left side (mine started with da33), you can find its IP address with:

$ docker inspect da33 | grep IPAddress
"SecondaryIPAddresses": null,
           "IPAddress": "172.17.0.3",
           "IPAddress": "172.17.0.3",

Now you can add that IP address to the URL field in the Grafana GUI as http://172.17.0.3:9090. Click Save & Test, and if all is going well, you should see a big green banner that says Data source is working. Now you can set up some dashboarding to show off Grafana.

Showing Off

To begin, navigate to the Dashboards tab at the top of the screen and click import on the right for the three example dashboards provided. If you want to delve deeper, you can find the Prometheus Stats JSON file online [12].

Once the import has completed, the name of the dashboard becomes a link that you can mouse over and select with a click. As if by magic, you have multiple metrics on display (Figure 8).

To speed up the data refresh frequency, mouse over the top right pull-down menu and select 5s (as opposed to 1m). If your graphs are lightly populated, fret not; waiting for a few minutes might resolve the issue. If using a tablet, try the "pinch" trick to focus on a specific time period and dig deeper into a specific display and practice with some of the settings to get used to how Grafana interacts.

If you click up near the top left on the name of the dashboard, you will see a list of dashboards you have created. The others might not be set up for you, so don't despair if they offer empty displays.

The End Is Nigh

Having the ability to customize each and every query along with the way that those queries are formatted and output visually is a powerful addition to any modern server estate. I would recommend having a look at hooking up the Kubernetes plugin and the MySQL plugin and configuring them to suit your needs. In this article, I have barely scratched the surface of either Prometheus or Grafana.

Although I have purposely tried to extol the virtues of Grafana's visualizations, I have left out the exceptionally sophisticated Prometheus to some extent. I would be remiss not to include at least one query, so Figure 9 shows, from http://localhost:9090, the output from the CPU query:

process_cpu_seconds_total

The built-in queries available in the pull-down menu next to the blue Execute button are numerous and should offer enough interest to whet your appetite for more. With some effort, you can create a fully bespoke, comprehensive reporting and alerting system with these exceptional tools. I'll leave the rest to you.