Additionally, youll also learn how to add Grafana Loki as a data source to yourGrafana Dashboard. This will help you visualize the Kubernetes events.

Furthermore, we'll delve into the specifics of setting up the Event exporter andGrafana Loki, ensuring you understand each step of the process. From downloadingand configuring the necessary helm charts to understanding the Grafana Lokidashboard, we'll cover it all.

By the end of this blog, you'll be able to fully utilize Grafana Loki andKubernetes Event Exporter, gaining insights from your Kubernetes events.

How Kubernetes event exporter can help us in monitoring health

Objects in Kubernetes, such as Pod, Deployment, Ingress, Service publish eventsto indicate status updates or problems. Most of the time, these events areoverlooked and their 1-hour lifespan might cause missing important updates. Theyare also not searchable and cannot be aggregated.

For instance, they can alert you to changes in the state of pods, errors inscheduling, and resource constraints. Therefore, exporting these events andvisualizing them can be crucial for maintaining the health of your cluster.

Kubernetes event exporter allows exporting the often missed Kubernetes events tovarious outputs so that they can be used for observability or alerting purposes.We can have multiple receivers to export the events from the Kubernetes cluster.

• Opsgenie
• Webhooks/HTTP
• Elasticsearch
• OpenSearch
• Slack
• Kinesis
• Firehose
• SNS
• SQS
• File
• Stdout
• Kafka
• OpsCenter
• Customize Payload
• Pubsub
• Teams
• Syslog
• Bigquery
• Pipe
• Event Bridge
• Grafana Loki

Setting up Grafana Loki & Kubernetes event exporter using Helm chart

We will once again use ArtifactHub, which provides ahelm chart for installing Grafana Loki onto a Kubernetes Cluster. If you needinstructions on how to install Helm on your system, you can refer to this blog.

In this blog post, we will install a Helmchart that sets up Loki inscalable mode, with separate read-and-write components that can be independentlyscaled. Alternatively, we can install Loki in monolithic mode, where the HelmChart installation runs the Grafana Loki single binary within a Kubernetescluster. You can learn more about thishere.

1. Create S3 buckets

• grafana-loki-chunks-bucket

• grafana-loki-admin-bucket

• grafana-loki-ruler-bucket

  

2. Create a policy for Grafana Loki

Create a new policy under IAM on Amazon AWS using the below snippet.

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "LokiStorage",            "Effect": "Allow",            "Action": [                "s3:ListBucket",                "s3:PutObject",                "s3:GetObject",                "s3:DeleteObject"            ],            "Resource": [                "arn:aws:s3:::grafana-loki-chunks-bucket",                "arn:aws:s3:::grafana-loki-chunks-bucket/*",                "arn:aws:s3:::grafana-loki-admin-bucket",                "arn:aws:s3:::grafana-loki-admin-bucket/*",                "arn:aws:s3:::grafana-loki-ruler-bucket",                "arn:aws:s3:::grafana-loki-ruler-bucket/*"            ]        }    ]}

Output:

3. Create a Role with the above permission

Create a role with a custom trust policy & use the below snippet

{    "Version": "2012-10-17",    "Statement": [        {            "Effect": "Allow",            "Principal": {                "Federated": "arn:aws:iam::account_id:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/open_id"            },            "Action": "sts:AssumeRoleWithWebIdentity",            "Condition": {                "StringEquals": {                    "oidc.eks.us-east-1.amazonaws.com/id/open_id:aud": "sts.amazonaws.com",                    "oidc.eks.us-east-1.amazonaws.com/id/open_id:sub": "system:serviceaccount:default:grafana-loki-access-s3-role-sa"                }            }        }    ]}

Note: Please update the account_id and open_id in the above given snippet.

grafana-loki-access-s3-role-sais the service account name that we willmention in the Loki values.

4. Add Grafana using the helm chart

To get this Helm chart, run this command:

helm repo add grafana https://grafana.github.io/helm-chartshelm repo update

Output:

We have downloaded the latest version of the Grafana.

5. Install grafana/loki stack using the helm chart

Create a loki-values yaml file with the below snippet

loki:  readinessProbe: {}  auth_enabled: false  storage:    bucketNames:      chunks: grafana-loki-chunks-bucket      ruler: grafana-loki-rules-bucket      admin: grafana-loki-admin-bucket    type: s3    s3:      endpoint: null      region: us-east-1      secretAccessKey: null      accessKeyId: null      s3ForcePathStyle: false      insecure: falsemonitoring:  lokiCanary:      enabled: false  selfMonitoring:    enabled: falsetest:  enabled: falseserviceAccount:  create: true  name: grafana-loki-access-s3-role-sa  imagePullSecrets: []  annotations:    eks.amazonaws.com/role-arn: arn:aws:iam::account_id:role/loki-role  automountServiceAccountToken: true

To install Loki using the Helm Chart on Kubernetes Cluster, runthishelm installcommand:

helm install my-loki grafana/loki --values loki-values.yaml

Output:

We have successfully installed Loki on the Kubernetes Cluster.

Run the followingcommand to view all the resources created by the Loki HelmChart in your Kubernetes cluster:

kubectl get all -l app.kubernetes.io/name=loki

Output:

Helm chart created the following components:

• Loki read and write: Loki is installed in scalable mode by default, whichincludes a read-and-write component. These components can be independentlyscaled out.
• Gateway: Inspired by GrafanasTanka setup,the chart installs a gateway component by default. This NGINX componentexposes Lokis API and automatically proxies requests to the appropriate Lokicomponents (read or write, or a single instance in the case of filesystemstorage). The gateway must be enabled to provide an Ingress since the Ingressonly exposes the gateway. If enabled, Grafana and log shipping agents, such asPromtail, should be configured to use the gateway. If NetworkPolicies areenabled, they become more restrictive when the gateway is active.
• Caching: In-memory caching is enabled by default. If this type of cachingis unsuitable for your deployment, consider setting up memcache.

Run this command to view all the Kubernetes Services for Prometheus & Grafana:

kubectl get service -l app.kubernetes.io/name=loki

Output:

Listed services for Loki are:

• loki-backend
• loki-backend-headless
• loki-gateway
• loki-memberlist
• loki-read
• loki-read-headless
• loki-write
• loki-write-headless
• query-scheduler-discovery

The loki-gateway service will be used to add Loki as a Datasource intoGrafana.

6. Adding Loki data source in Grafana

On the main page of Grafana, click on "Home". Under "Connections", youwill find the "Data sources" option.

On the Data Sources page, click on the "Add new data source" button.

In the search bar, type "Loki" and search for it.

Clicking on "Loki" will redirect you to the dedicated page for the Loki datasource.

To read the metrics from Loki, we will use the loki-gateway service. Add theURL of the service as http://loki-gateway.

After clicking on the "Save & test" button, you will receive a toastr messageshown in the image below. This message is received because no clients have beencreated for Loki yet.

7. Install Kubernetes event exporter using the helm chart

Create an event-exporter-values yaml file with the below snippet

config:  leaderElection: {}  logLevel: debug  logFormat: pretty  metricsNamePrefix: event_exporter_  receivers:    - name: "dump"      file:        path: "/dev/stdout"        layout: {}    - name: "loki"      loki:        url: "http://loki-gateway/loki/api/v1/push"        streamLabels:          source: kubernetes-event-exporter          container: kubernetes-event-exporter  route:    routes:      - match:          - receiver: "dump"          - receiver: "loki"

With the use of the above snippet, run this command to install the Kubernetesevent exporter in your Kubernetes Cluster.

helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami)helm install event-exporter bitnami/kubernetes-event-exporter --values event-exporter.yaml

Output:

To view all the resources created by the above helm chart, run this command:

kubectl get all -l app.kubernetes.io/name=kubernetes-event-exporter

Output:

To view the logs of the event exporter POD, run this command:

kubectl logs -f pod/kubernetes-event-exporter-586455bbdd-sqlqc

Note: Replace kubernetes-event-exporter-586455bbdd-sqlqc with your pod name.

Output:

As you can see in the above image, the event exporter is working and runningfine. Event logs are being sent to both the receivers that wed configure in thevalues YAML file.

Once the POD is created & running, we can go back to the Loki data source underConnections > Data Sources page.

Again click on the Save & test button & this time youll receive a successtoastr message.

Output:

8. Kubernetes event exporter dashboard

We will import thisdashboardinto Grafana to monitor and track the events received from the Kubernetescluster. You can go through this blog if you want to learn how to import anexisting dashboard into Grafana.

After successfully importing the dashboard, you can view all the events from thecluster, as shown in the image below. Additionally, you can filter the eventsbased on any value within any interval.

Kubernetes Event Exporter

Conclusion

In this blog post, we discussed the process of setting up Grafana Loki andKubernetes Event exporter. We covered various steps, such as creating a policyfor Grafana Loki, creating a role with the necessary permissions, and addingGrafana using the Helm chart, installing the Loki stack, and adding Loki as adata source in Grafana, installing Kubernetes event exporter using the Helmchart, and finally, setting up the Kubernetes event exporter dashboard inGrafana.

By following the steps outlined in this blog post, you can effectively monitorand track events from your Kubernetes cluster using Grafana Loki and KubernetesEvent exporter. This setup provides valuable insights and helps introubleshooting and analyzing events in your cluster.

If you have any questions or feedback, please feel free to reach out. Happymonitoring!