HIE Monitoring & Alerting
- Taib
Overview
The health information exchange (HIE) architecture monitoring and alert system provides the mechanisms to monitor HIE implementations. These mechanisms have the ability to send out alerts to assigned persons in the event a HIE component has encountered a problem such as down time. The HIE monitoring and alert system has been implemented using Grafana. Grafana is an open source tool that offers an array of dashboards, charts, graphs, and alert systems tailored for specific data sources within the HIE architecture of a project. Grafana enables project monitoring through tools that facilitate querying, visual representation, exploration of metrics as well as setting up alerts for the various data sources of a project.
Data sources in Grafana
A data source represents a connection to a specific database, service, or an HIE component where the project data resides. It's a crucial component that allows Grafana to fetch and visualize data from different sources. Essentially, a data source acts as a bridge between Grafana and the underlying data in HIE components.
Prometheus as a data source in Grafana
Prometheus is the primary tool that we use for monitoring metrics and events as well as configuring alerts. This open-source serves the purpose of monitoring various entities in the HIE architecture. Prometheus operates using a robust query language known as 'PromQL'
Grafana LOKI as a data source in Grafana
Grafana Loki provides a log aggregation system for HIE implementation. Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. Loki has been installed and configured to aggregate HIE implementation components logs. The figure below annotates LokiLogs as a data source.
Monitoring Docker services
When it comes to monitoring infrastructure metrics for an HIE implementation, Prometheus and Node Exporter form a powerful combination. Prometheus is a highly efficient open-source monitoring system that excels at collecting and storing time-series data, while Node Exporter is a dedicated exporter specifically built to expose detailed system and hardware metrics directly from servers hosting the different HIE components.
Monitoring HIE docker containers
Container dashbords have been used to monitor CPU usage per container,memory use,incoming traffic and outgoing traffic per container. The figure below is a dashboard illustartion for CPU usage per container in a docker swarm.
Grafana LOKI log aggregation
Grafana Loki has been used to collect and store logs from a variety of sources, including: A. Application or HIE components.
Once logs are collected in Loki, they can be queried and analyzed using LogQL, a PromQL-inspired query language. LogQL queries can be used to extract specific information from logs, such as errors, warnings, or performance metrics. Select the labels filter for the Raw query you want executed. In this illustration, the labes selected are; job and containerlogs respectively. The folowing are illustrations where labels filter for the Raw query to be executed are seleted for job and containerlogs.
The respective logs are displayed once the query is executed.
The following is an illustration of how to filter LOKI logs by text where FHIR is the text to filter.
The logs will be displayed for the filtered text
Kafka channels monitoring
Kminion is a feature-rich Prometheus exporter specifically designed for monitoring Apache Kafka. It acts as a bridge between Kafka and Prometheus, allowing seamless collection and visualization of key metrics for the HIE within Grafana. Kminion automates several manual tasks in Kafka monitoring, such as; Creating and managing the Prometheus scraping target for your Kafka cluster and Automatic discovery and tracking topics and partitions. Kminion seemlessly integrates with Grafana, allowing you to visualize Kafka metrics in intuitive dashboards. KMinion dashboard are A dashboard for the data flow is displayed.
How to access and view a KMinion dashboards in Grafana
To access a Kminion dashboard, click on the dashboard icon on the left side menu bar. Click on browse Click on a KMinion dashboard. A dashboard for the data flow is displayed. Click on the dropdown menu to filter the topic for the dashboards Select a topic to display the dashboard.
Alerts in monitoring
Navigate to Grafana alerting by clicking on the bell icon
Click on the New Alert Rule button.
In the alert rule page, you can either of the three available alert rules; Grafana managed alert, Mimir or Loki alert or Mimir or Loki recording rule.
In this illustartion, Grafana managed alert is used.
For demonstration purposes, we want to know if our HTTP server is returning 500 internal server errors.
Prometheus data source is selected. A Prometheus server exists for collection of metrics from the HTTP server.