Additionally, youll also learn how to add Grafana Loki as a data source to yourGrafana Dashboard. This will help you visualize the Kubernetes events.

Furthermore, we'll delve into the specifics of setting up the Event exporter andGrafana Loki, ensuring you understand each step of the process. From downloadingand configuring the necessary helm charts to understanding the Grafana Lokidashboard, we'll cover it all.

By the end of this blog, you'll be able to fully utilize Grafana Loki andKubernetes Event Exporter, gaining insights from your Kubernetes events.

How Kubernetes event exporter can help us in monitoring health

Objects in Kubernetes, such as Pod, Deployment, Ingress, Service publish eventsto indicate status updates or problems. Most of the time, these events areoverlooked and their 1-hour lifespan might cause missing important updates. Theyare also not searchable and cannot be aggregated.

For instance, they can alert you to changes in the state of pods, errors inscheduling, and resource constraints. Therefore, exporting these events andvisualizing them can be crucial for maintaining the health of your cluster.

Kubernetes event exporter allows exporting the often missed Kubernetes events tovarious outputs so that they can be used for observability or alerting purposes.We can have multiple receivers to export the events from the Kubernetes cluster.

• Opsgenie
• Webhooks/HTTP
• Elasticsearch
• OpenSearch
• Slack
• Kinesis
• Firehose
• SNS
• SQS
• File
• Stdout
• Kafka
• OpsCenter
• Customize Payload
• Pubsub
• Teams
• Syslog
• Bigquery
• Pipe
• Event Bridge
• Grafana Loki

Setting up Grafana Loki & Kubernetes event exporter using Helm chart

We will once again use ArtifactHub, which provides ahelm chart for installing Grafana Loki onto a Kubernetes Cluster. If you needinstructions on how to install Helm on your system, you can refer to this blog.

In this blog post, we will install a Helmchart that sets up Loki inscalable mode, with separate read-and-write components that can be independentlyscaled. Alternatively, we can install Loki in monolithic mode, where the HelmChart installation runs the Grafana Loki single binary within a Kubernetescluster. You can learn more about thishere.

1. Create S3 buckets

• grafana-loki-chunks-bucket

• grafana-loki-admin-bucket

• grafana-loki-ruler-bucket

  

2. Create a policy for Grafana Loki

Create a new policy under IAM on Amazon AWS using the below snippet.

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "LokiStorage",            "Effect": "Allow",            "Action": [                "s3:ListBucket",                "s3:PutObject",                "s3:GetObject",                "s3:DeleteObject"            ],            "Resource": [                "arn:aws:s3:::grafana-loki-chunks-bucket",                "arn:aws:s3:::grafana-loki-chunks-bucket/*",                "arn:aws:s3:::grafana-loki-admin-bucket",                "arn:aws:s3:::grafana-loki-admin-bucket/*",                "arn:aws:s3:::grafana-loki-ruler-bucket",                "arn:aws:s3:::grafana-loki-ruler-bucket/*"            ]        }    ]}

Output:

3. Create a Role with the above permission

Create a role with a custom trust policy & use the below snippet

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "Federated": "arn:aws:iam::account_id:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/open_id"            },            "Action": "sts:AssumeRoleWithWebIdentity",            "Condition": {                "StringEquals": {                    "oidc.eks.us-east-1.amazonaws.com/id/open_id:aud": "sts.amazonaws.com",                    "oidc.eks.us-east-1.amazonaws.com/id/open_id:sub": "system:serviceaccount:default:grafana-loki-access-s3-role-sa"                }            }        }    ]}

Note: Please update the account_id and open_id in the above given snippet.

grafana-loki-access-s3-role-sais the service account name that we willmention in the Loki values.

4. Add Grafana using the helm chart

To get this Helm chart, run this command:

helm repo add grafana https://grafana.github.io/helm-chartshelm repo update

Output:

We have downloaded the latest version of the Grafana.

5. Install grafana/loki stack using the helm chart

Create a loki-values yaml file with the below snippet

loki:  readinessProbe: {}  auth_enabled: false  storage:    bucketNames:      chunks: grafana-loki-chunks-bucket      ruler: grafana-loki-rules-bucket      admin: grafana-loki-admin-bucket    type: s3    s3:      endpoint: null      region: us-east-1      secretAccessKey: null      accessKeyId: null      s3ForcePathStyle: false      insecure: falsemonitoring:  lokiCanary:      enabled: false  selfMonitoring:    enabled: falsetest:  enabled: falseserviceAccount:  create: true  name: grafana-loki-access-s3-role-sa  imagePullSecrets: []  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::account_id:role/loki-role  automountServiceAccountToken: true

To install Loki using the Helm Chart on Kubernetes Cluster, runthishelm installcommand:

helm install my-loki grafana/loki --values loki-values.yaml

Output:

We have successfully installed Loki on the Kubernetes Cluster.

Run the followingcommand to view all the resources created by the Loki HelmChart in your Kubernetes cluster:

kubectl get all -l app.kubernetes.io/name=loki

Output:

Helm chart created the following components:

• Loki read and write: Loki is installed in scalable mode by default, whichincludes a read-and-write component. These components can be independentlyscaled out.
• Gateway: Inspired by GrafanasTanka setup,the chart installs a gateway component by default. This NGINX componentexposes Lokis API and automatically proxies requests to the appropriate Lokicomponents (read or write, or a single instance in the case of filesystemstorage). The gateway must be enabled to provide an Ingress since the Ingressonly exposes the gateway. If enabled, Grafana and log shipping agents, such asPromtail, should be configured to use the gateway. If NetworkPolicies areenabled, they become more restrictive when the gateway is active.
• Caching: In-memory caching is enabled by default. If this type of cachingis unsuitable for your deployment, consider setting up memcache.

Run this command to view all the Kubernetes Services for Prometheus & Grafana:

kubectl get service -l app.kubernetes.io/name=loki

Output:

Listed services for Loki are:

• loki-backend
• loki-backend-headless
• loki-gateway
• loki-memberlist
• loki-read
• loki-read-headless
• loki-write
• loki-write-headless
• query-scheduler-discovery

The loki-gateway service will be used to add Loki as a Datasource intoGrafana.

6. Adding Loki data source in Grafana

On the main page of Grafana, click on "Home". Under "Connections", youwill find the "Data sources" option.

On the Data Sources page, click on the "Add new data source" button.

In the search bar, type "Loki" and search for it.

Clicking on "Loki" will redirect you to the dedicated page for the Loki datasource.

To read the metrics from Loki, we will use the loki-gateway service. Add theURL of the service as http://loki-gateway.

After clicking on the "Save & test" button, you will receive a toastr messageshown in the image below. This message is received because no clients have beencreated for Loki yet.

7. Install Kubernetes event exporter using the helm chart

Create an event-exporter-values yaml file with the below snippet

config:  leaderElection: {}  logLevel: debug  logFormat: pretty  metricsNamePrefix: event_exporter_  receivers:    - name: "dump"      file:        path: "/dev/stdout"        layout: {}    - name: "loki"      loki:        url: "http://loki-gateway/loki/api/v1/push"        streamLabels:          source: kubernetes-event-exporter          container: kubernetes-event-exporter  route:    routes:      - match:          - receiver: "dump"          - receiver: "loki"

With the use of the above snippet, run this command to install the Kubernetesevent exporter in your Kubernetes Cluster.

helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami)helm install event-exporter bitnami/kubernetes-event-exporter --values event-exporter.yaml

Output:

To view all the resources created by the above helm chart, run this command:

kubectl get all -l app.kubernetes.io/name=kubernetes-event-exporter

Output:

To view the logs of the event exporter POD, run this command:

kubectl logs -f pod/kubernetes-event-exporter-586455bbdd-sqlqc

Note: Replace kubernetes-event-exporter-586455bbdd-sqlqc with your pod name.

Output:

As you can see in the above image, the event exporter is working and runningfine. Event logs are being sent to both the receivers that wed configure in thevalues YAML file.

Once the POD is created & running, we can go back to the Loki data source underConnections > Data Sources page.

Again click on the Save & test button & this time youll receive a successtoastr message.

Output:

8. Kubernetes event exporter dashboard

We will import thisdashboardinto Grafana to monitor and track the events received from the Kubernetescluster. You can go through this blog if you want to learn how to import anexisting dashboard into Grafana.

After successfully importing the dashboard, you can view all the events from thecluster, as shown in the image below. Additionally, you can filter the eventsbased on any value within any interval.

Kubernetes Event Exporter

Conclusion

In this blog post, we discussed the process of setting up Grafana Loki andKubernetes Event exporter. We covered various steps, such as creating a policyfor Grafana Loki, creating a role with the necessary permissions, and addingGrafana using the Helm chart, installing the Loki stack, and adding Loki as adata source in Grafana, installing Kubernetes event exporter using the Helmchart, and finally, setting up the Kubernetes event exporter dashboard inGrafana.

By following the steps outlined in this blog post, you can effectively monitorand track events from your Kubernetes cluster using Grafana Loki and KubernetesEvent exporter. This setup provides valuable insights and helps introubleshooting and analyzing events in your cluster.

If you have any questions or feedback, please feel free to reach out. Happymonitoring!

Prometheus along withGrafanais a highly scalable open-sourcemonitoringframework forcontainer orchestration platform.Prometheus probes the application and collects various data. It stores all thisdata in its time series database. Grafana is a visualization tool. It uses thedata from the database to show the data that is meaningful to the user.

Both Prometheus and Grafana are gaining popularity intheobservabilityspace as ithelps with metrics and alerts. Learning to integrate them using Helm will allowus to monitor our Kubernetes cluster and troubleshoot problems easily.Furthermore, we can deep dive into our cluster's well-being and efficiency,focusing on resource usage and performance metrics within our Kubernetesenvironment.

We will also learn how to create a simpledashboard on Grafana.

Why using Prometheus and Grafana for monitoring is good

Using Prometheus and Grafana for monitoring has many benefits:

• Scalability: Both tools are highly scalable and can handle the monitoringneeds of small to large Kubernetes clusters.
• Flexibility: They allow us to create custom dashboards tailored to ourspecific monitoring requirements.
• Real-time Monitoring: Prometheus provides real-time monitoring, helping usto quickly detect and respond to issues.
• Alerting: Prometheus enables us to set up alerts based on specificmetrics, so we can be notified when issues arise.
• Data Visualization: Grafana offers powerful data visualizationcapabilities, making it easier to understand complex data.
• Open Source: Both Prometheus and Grafana are open-source, reducingmonitoring costs.
• Community Support: We can benefit from active communities, ensuringcontinuous development and support.
• Integration: They seamlessly integrate with other Kubernetes componentsand applications, simplifying setup.
• Historical Data: Grafana allows us to explore historical data, aiding inlong-term analysis and trend identification.
• Extensible: Both tools are extensible, allowing us to integrate additionaldata sources and plugins.
• Efficient Resource Usage: Prometheus efficiently utilizes resources,ensuring minimal impact on our cluster's performance.

Two common ways to use Prometheus and Grafana on Kubernetes:

1. Manual Kubernetes deployment: In this method, we need to writeKubernetes DeploymentsandServicesfor both Prometheus and Grafana. In the YAML file, we need to put all thesettings for Prometheus and Grafana on Kubernetes. Then we send these filesto our Kubernetes cluster. But we can end up with many YAML files, which canbe hard. If we make a mistake in any YAML file, Prometheus and Grafana won'twork on Kubernetes.
2. Using Helm: This is an easy way to send any application container toKubernetes.Helmis the official package manager forKubernetes. With Helm, we can make installing, sending, and managingKubernetes makes applications easier.

AHelm Charthas all the YAML files:

• Deployments.
• Services.
• Secrets.
• ConfigMaps manifests.

We use these files to send the application container to Kubernetes. Instead ofmaking individual YAML files for each application container, Helm lets usdownload Helm charts that already have YAML files.

Setting up Prometheus and Grafana using Helm chart

We will use ArtifactHub, which offers public andprivate repositories for Helm Charts. We will use these Helm Charts to arrangethe pods and services in our Kubernetes cluster.

To get Prometheus and Grafana working on Kubernetes with Helm, we will start byinstalling Helm.

Installing Helm on Linux

sudo apt-get install helm

Installing Helm on Windows

choco install Kubernetes-helm

Installing Helm on macOS

brew install helm

We can check out the officialHelm documentation if we run into anyissues while installing Helm.

The image below represents the successful Helm installation on macOS.

For this blog, were going to install Helmchartand by default, this chart also installs additional, dependent charts (includingGrafana):

• prometheus-community/kube-state-metrics
• prometheus-community/prometheus-node-exporter
• grafana/grafana

To get this Helm chart, let's run this command:

helm repo add prometheus-community https://prometheus-community.github.io/helm-chartshelm repo update

We have downloaded the latest version of Prometheus & Grafana.

To install the Prometheus Helm Chart on a Kubernetes Cluster, let's run thefollowing command:

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack

We have successfully installed Prometheus & Grafana on the Kubernetes Cluster.We can access the Prometheus & Grafana servers via ports 9090 & 80,respectively.

Now, let's run the followingcommand to view all the resources created by theHelm Chart in our Kubernetes cluster:

kubectl get all

The Helm chart created the following resources:

• Pods: It hosts the deployed Prometheus Kubernetes application inside thecluster.
• Replica Sets: A collection of instances of the same application inside theKubernetes cluster. It enhances application reliability.
• Deployments: It is the blueprint for creating the application pods.
• Services: It exposes the pods running inside the Kubernetes cluster. Weuse it to access the deployed Kubernetes application.
• Stateful Sets: They manage the deployment of the stateful applicationcomponents and ensure stable and predictable network identities for thesecomponents.
• Daemon Sets: They ensure that all (or a specific set of) nodes run a copyof a pod, which is useful for tasks such as logging, monitoring, and othernode-specific operations.

Run this command to view all the Kubernetes Services for Prometheus & Grafana:

kubectl get service

Listed services for Prometheus and Grafana are:

• alertmanager-operated
• kube-prometheus-stack-alertmanager
• kube-prometheus-stack-grafana
• kube-prometheus-stack-kube-state-metrics
• kube-prometheus-stack-operator
• kube-prometheus-stack-prometheus
• kube-prometheus-stack-prometheus-node-exporter
• prometheus-operated

kube-prometheus-stack-grafana and kube-prometheus-stack-prometheus are theClusterIP type services, which means we can only access them within theKubernetes cluster.

To expose the Prometheus and Grafana to be accessed outside the Kubernetescluster , we can either use the NodeIP or LoadBalance service.

Exposing Prometheus and Grafana using NodePort services

Let's run the following command to expose thePrometheusKubernetes service:

kubectl expose service kube-prometheus-stack-prometheus --type=NodePort --target-port=9090 --name=prometheus-node-port-servicekubectl expose service kube-prometheus-stack-grafana --type=NodePort --target-port=3000 --name=grafana-node-port-service

That command will create new services ofNodePorttype & make thePrometheusand Grafana is accessible outside the Kubernetes Cluster on ports9090 and80.

As we can see, the grafana-node-port-service andprometheus-node-port-service are successfully created and are being exposed onnode ports 32489 & 30905

Now, we can run this command and get the external IP of any node to access thePrometheus and Grafana:

kubectl get nodes -o wide

We can use the External-IP and the node ports to access the Prometheus andGrafana dashboards outside the cluster environment.

Prometheus Dashboard

Grafana Dashboard

Run this command, to get the password for the admin user of the Grafanadashboard:

kubectl get secret --namespace default kube-prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Grafana Dashboard

Upon login to the Grafana dashboard, use adminas the username and ourgenerated password. We will see "Welcome to Grafana" homepage as shown below.

Since we used the Kube Prometheus Stack helm chart, the data source forPrometheus and Alert Manager is added by default.

We can add more data sources by clicking on the Add new data source buttonon the top right side.

By default, this Helm chart adds multiple dashboards to monitor the health ofthe Kubernetes cluster and its resources.

Additionally, we also have the option of creating our dashboards from scratch aswell as importing multiple Grafana dashboards provided by theGrafana library.

To import a Grafana Dashboard, let's follow these steps:

• From this Grafana library, we canadd any dashboard

  

• Select Dashboard and copy the Dashboard ID

  

• Under Dashboards page we can get the Import option

  

• Under "Import Dashboard" page, we need to paste the Dashboard IP that wecopied earlier & click on the Load button.

  

• After clicking on the Load button, it will auto-load the dashboard fromthe library after which we can import the dashboard by clicking on theImport button.

  

• Once the import is complete, well be redirected to the new imported dashboardwhichll also be visible under the Dashboards page.

  

  We can use this Node Exporter dashboard to monitor & observe the health of ournodes present in our Kubernetes Cluster.

Conclusion

In this blog, we learned how to integrate Prometheus and Grafana using the helmchart. We also learned how to import dashboards into Grafana from theGrafana library.

In the next blog, we will explore how to integrateGrafana Loki with Grafana and collect and storeevent-related metrics using theKubernetes Event Exporter.

Pre-compiled Ruby binaries

Pre-compiled Ruby binaries are distribution-ready versions of Ruby that includeoptimized features for specific systems. These Ruby binaries save time byeliminating the need to compile Ruby source code manually. Pre-compiled Rubybinaries help users quickly deploy applications that use different versions ofRuby on multiple machines.

RVM (Ruby Version Manager) is widely used for managing Rubyinstallations on Unix-like systems. RVM provides customized pre-compiled Rubybinaries tailored for various CPU architectures. These binaries offer additionalfeatures like readline support and SSL/TLS support. You can find them atRVM binaries.

The Need for pre-compiled Ruby Binaries

NeetoCI must execute user code in acontainerized environment. A Ruby environment is essential for running Ruby onRails applications. However, relying on the system's Ruby version is impracticalsince it may differ from the user's required version. Although rbenv or rvm canbe used to install the necessary Ruby version, this approach could be slow. Tosave time, we chose to leverage pre-compiled Ruby binaries.

As a CI/CD system, NeetoCI must ensure that all versions of Ruby that anapplication requires are always available. Hence, we decided to build ourbinaries instead of relying on binaries provided by RVM. Also, this would allowus to do more system-specific optimizations to the Ruby binary at build time.

Building pre-compiled Ruby binaries

We built a Ruby binary following theofficial documentation. We were able to execute it on our local development machines. But the samebinary ran into an error in our CI/CD environment.

$ bundle config path vendor/bundle./ruby: bad interpreter: No such file or directory

To debug the issue, we initially focused on $PATH. However, even afterresolving the $PATH issues, the problem persisted. We conducted a thoroughinvestigation to identify the root cause. Unfortunately, not much was written onthe Internet about this error. There was no mention of it in the officialRuby documentation.

As the next step, we decided to download the binary for version 3.2.2 fromRVM. While examining the configuration file, wenoticed that the following arguments were used with the configure command duringthe Ruby binary build process:

configure_args="'--prefix=/usr/share/rvm/rubies/ruby-3.2.2' '--enable-load-relative' '--sysconfdir=/etc' '--disable-install-doc' '--enable-shared'"

Here are the explanations of the configuration arguments:

1. --prefix=/usr/share/rvm/rubies/ruby-3.2.2: This specifies the directorywhere the Ruby binaries, libraries and other files will be kept after theinstallation is done.

2. --enable-load-relative: This specifies that Ruby can load relative pathsfor dynamically linked libraries. It allows the usage of relative pathsinstead of absolute paths when loading shared libraries. This feature can bebeneficial in specific deployment scenarios.

3. --sysconfdir=/etc: This argument sets the directory where Ruby's systemconfiguration files will be installed. In this case, it specifies the /etcdirectory as the location for these files.

4. --disable-install-doc: When this option is enabled, the installation ofdocumentation files during the build process is disabled. This can help speedup the build process and save disk space, especially if you do not requirethe documentation files.

5. --enable-shared: Enabling this option allows the building of sharedlibraries for Ruby. Shared libraries enable Ruby to dynamically link and loadspecific functionality at runtime, leading to potential performanceimprovements and reduced memory usage.

In simpler terms, when the --enable-load-relative flag is enabled, thecompiled Ruby binary can search for shared libraries in its own directory usingthe $ORIGIN variable.

When I built the binary on the Docker registry, then the passed --prefix wassomething like /usr/share/neetoci. When the binary is built, then binary had/usr/share/neetoci is hard-coded at various places. When we download thisbinary and use in CI then in the CI environment, Ruby is looking for/user/share/neetoci to load dependencies.

By enabling --enable-load-relative flag while building the binary Ruby willnot use the hard coded value. Rather Ruby will use $ORIGIN variable and willsearch for the dependencies in the directory mentioned in $ORIGIN.

This is particularly helpful when the Ruby binary is relocated to a differentdirectory or system. By using relative paths with $ORIGIN, the binary can findits shared libraries regardless of its new location. Without this flag, sharedlibraries are loaded using absolute paths, which can cause issues if the binaryis moved to a different location and cannot locate its shared libraries.

In our specific use case, where we create and download binaries in separatecontainers, we encountered an error due to the absolute paths. To overcome this,we enabled the --enable-load-relative flag. This allowed the binary to findits shared libraries successfully, and it worked as expected in our CI/CDenvironment.