<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" xml:lang="en-US">
  <title>BigBinary Blog: NeetoCI</title>
  <subtitle>Deep technical writing on Ruby on Rails, performance, scaling, monitoring and engineering.</subtitle>
  <link rel="self" type="application/atom+xml" href="https://www.bigbinary.com/blog/categories/neetoci/feed.xml"/>
  <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/categories/neetoci"/>
  <id>https://www.bigbinary.com/blog/categories/neetoci</id>
  <updated>2024-05-07T00:00:00.000Z</updated>
  <author>
    <name>BigBinary</name>
    <uri>https://www.bigbinary.com/</uri>
  </author>
  <icon>https://www.bigbinary.com/favicon.ico</icon>
  <logo>https://www.bigbinary.com/images/bigbinary_square_big_logo.png</logo>
  <rights>Copyright 2024 Big Binary LLC</rights>
  <generator uri="https://nextjs.org">Next.js</generator>
  <entry>
      <title type="text"><![CDATA[Grafana Loki and Kubernetes Event exporter]]></title>
      <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration"/>
      <link rel="enclosure" type="image/jpeg" href="https://www.bigbinary.com/blog/images/social_preview_images/grafana_loki_and_kubernetes_event_exporter.jpg"/>
      <id>https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration</id>
      <published>2024-05-07T00:00:00.000Z</published>
      <updated>2024-05-07T00:00:00.000Z</updated>
      <author>
        <name>Vishal Yadav</name>
        <uri>https://www.bigbinary.com/blog/authors/vishal76342</uri>
      </author>
      <category term="NeetoCI"/>
      <category term="Prometheus"/>
      <category term="Grafana"/>
      <category term="Kubernetes"/>
      <media:thumbnail url="https://www.bigbinary.com/blog/images/social_preview_images/grafana_loki_and_kubernetes_event_exporter.jpg" width="1200" height="627"/>
      <summary type="text"><![CDATA[In this blog, we'll explore how to integrate the Kubernetes Event exporter & Grafana Loki into your Kubernetes Cluster using a helm chart.]]></summary>
      <content type="html" xml:base="https://www.bigbinary.com/blog/k8s-event-exporter-and-grafana-loki-integration"><![CDATA[<p>In the previous
<a href="https://www.bigbinary.com/blog/prometheus-and-grafana-integration">blog</a>, we
discussed integrating <a href="https://prometheus.io/">Prometheus</a> and
<a href="https://grafana.com/">Grafana</a> in the Kubernetes Cluster. In this blog, we'll
explore how to integrate the
<a href="https://github.com/resmoio/kubernetes-event-exporter">Kubernetes Event exporter</a>
&amp; <a href="https://grafana.com/oss/loki/">Grafana Loki</a> into your Kubernetes Cluster
using a helm chart.</p>
<p>Additionally, you’ll also learn how to add Grafana Loki as a data source to your
Grafana Dashboard. This will help you visualize the Kubernetes events.</p>
<p>Furthermore, we'll delve into the specifics of setting up the Event exporter and
Grafana Loki, ensuring you understand each step of the process. From downloading
and configuring the necessary helm charts to understanding the Grafana Loki
dashboard, we'll cover it all.</p>
<p>By the end of this blog, you'll be able to fully utilize Grafana Loki and
Kubernetes Event Exporter, gaining insights from your Kubernetes events.</p>
<h2>How Kubernetes event exporter can help us in monitoring health</h2>
<p>Objects in Kubernetes, such as Pod, Deployment, Ingress, Service publish events
to indicate status updates or problems. Most of the time, these events are
overlooked and their 1-hour lifespan might cause missing important updates. They
are also not searchable and cannot be aggregated.</p>
<p>For instance, they can alert you to changes in the state of pods, errors in
scheduling, and resource constraints. Therefore, exporting these events and
visualizing them can be crucial for maintaining the health of your cluster.</p>
<p>Kubernetes event exporter allows exporting the often missed Kubernetes events to
various outputs so that they can be used for observability or alerting purposes.
We can have multiple receivers to export the events from the Kubernetes cluster.</p>
<ul>
<li><a href="https://www.opsgenie.com/">Opsgenie</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#webhookshttp">Webhooks/HTTP</a></li>
<li><a href="https://www.elastic.co/">Elasticsearch</a></li>
<li><a href="https://opensearch.org/">OpenSearch</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#slack">Slack</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#kinesis">Kinesis</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#firehose">Firehose</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#sns">SNS</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#sqs">SQS</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#file">File</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#stdout">Stdout</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#kafka">Kafka</a></li>
<li><a href="https://docs.aws.amazon.com/systems-manager/latest/userguide/OpsCenter.html">OpsCenter</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#customizing-payload">Customize Payload</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#pubsub">Pubsub</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#teams">Teams</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#syslog">Syslog</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#bigquery">Bigquery</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#pipe">Pipe</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#aws-eventbridge">Event Bridge</a></li>
<li><a href="https://github.com/resmoio/kubernetes-event-exporter#loki">Grafana Loki</a></li>
</ul>
<h2>Setting up Grafana Loki &amp; Kubernetes event exporter using Helm chart</h2>
<p>We will once again use <a href="https://artifacthub.io/">ArtifactHub</a>, which provides a
helm chart for installing Grafana Loki onto a Kubernetes Cluster. If you need
instructions on how to install Helm on your system, you can refer to this blog.</p>
<p>In this blog post, we will install a Helm
<a href="https://artifacthub.io/packages/helm/grafana/loki">chart</a> that sets up Loki in
scalable mode, with separate read-and-write components that can be independently
scaled. Alternatively, we can install Loki in monolithic mode, where the Helm
Chart installation runs the Grafana Loki <em>single binary</em> within a Kubernetes
cluster. You can learn more about this
<a href="https://grafana.com/docs/loki/latest/setup/install/helm/install-monolithic/#install-the-monolithic-helm-chart">here</a>.</p>
<h3>1. Create S3 buckets</h3>
<ul>
<li>
<p>grafana-loki-chunks-bucket</p>
</li>
<li>
<p>grafana-loki-admin-bucket</p>
</li>
<li>
<p>grafana-loki-ruler-bucket</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-s3-buckets.png" alt="The four Grafana Loki buckets listed in the AWS S3 console"></p>
</li>
</ul>
<h3>2. Create a policy for Grafana Loki</h3>
<p>Create a new policy under IAM on Amazon AWS using the below snippet.</p>
<pre><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Sid&quot;: &quot;LokiStorage&quot;,
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Action&quot;: [
                &quot;s3:ListBucket&quot;,
                &quot;s3:PutObject&quot;,
                &quot;s3:GetObject&quot;,
                &quot;s3:DeleteObject&quot;
            ],
            &quot;Resource&quot;: [
                &quot;arn:aws:s3:::grafana-loki-chunks-bucket&quot;,
                &quot;arn:aws:s3:::grafana-loki-chunks-bucket/*&quot;,
                &quot;arn:aws:s3:::grafana-loki-admin-bucket&quot;,
                &quot;arn:aws:s3:::grafana-loki-admin-bucket/*&quot;,
                &quot;arn:aws:s3:::grafana-loki-ruler-bucket&quot;,
                &quot;arn:aws:s3:::grafana-loki-ruler-bucket/*&quot;
            ]
        }
    ]
}
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/grafana-loki-policy.png" alt="Terminal output confirming the Grafana Loki IAM policy was created"></p>
<h3>3. Create a Role with the above permission</h3>
<p>Create a role with a custom trust policy &amp; use the below snippet</p>
<pre><code>{
    &quot;Version&quot;: &quot;2012-10-17&quot;,
    &quot;Statement&quot;: [
        {
            &quot;Effect&quot;: &quot;Allow&quot;,
            &quot;Principal&quot;: {
                &quot;Federated&quot;: &quot;arn:aws:iam::account_id:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/open_id&quot;
            },
            &quot;Action&quot;: &quot;sts:AssumeRoleWithWebIdentity&quot;,
            &quot;Condition&quot;: {
                &quot;StringEquals&quot;: {
                    &quot;oidc.eks.us-east-1.amazonaws.com/id/open_id:aud&quot;: &quot;sts.amazonaws.com&quot;,
                    &quot;oidc.eks.us-east-1.amazonaws.com/id/open_id:sub&quot;: &quot;system:serviceaccount:default:grafana-loki-access-s3-role-sa&quot;
                }
            }
        }
    ]
}
</code></pre>
<p>Note: Please update the account_id and open_id in the above given snippet.</p>
<p><strong>grafana-loki-access-s3-role-sa</strong> is the service account name that we will
mention in the Loki values.</p>
<h3>4. Add Grafana using the helm chart</h3>
<p>To get this Helm chart, run this command:</p>
<pre><code class="language-jsx">helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/chart-add-output.png" alt="Terminal output from adding and updating the Grafana Helm chart repository"></p>
<p>We have downloaded the latest version of the Grafana.</p>
<h3>5. Install grafana/loki stack using the helm chart</h3>
<p>Create a <strong>loki-values yaml</strong> file with the below snippet</p>
<pre><code>loki:
  readinessProbe: {}
  auth_enabled: false
  storage:
    bucketNames:
      chunks: grafana-loki-chunks-bucket
      ruler: grafana-loki-rules-bucket
      admin: grafana-loki-admin-bucket
    type: s3
    s3:
      endpoint: null
      region: us-east-1
      secretAccessKey: null
      accessKeyId: null
      s3ForcePathStyle: false
      insecure: false
monitoring:
  lokiCanary:
      enabled: false
  selfMonitoring:
    enabled: false
test:
  enabled: false
serviceAccount:
  create: true
  name: grafana-loki-access-s3-role-sa
  imagePullSecrets: []
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::account_id:role/loki-role
  automountServiceAccountToken: true
</code></pre>
<p>To install Loki using the Helm Chart on Kubernetes Cluster, run
this <code>helm install</code> command:</p>
<pre><code class="language-jsx">helm install my-loki grafana/loki --values loki-values.yaml
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/chart-installation-output.png" alt="Terminal output from installing the Loki Helm chart"></p>
<p>We have successfully installed Loki on the Kubernetes Cluster.</p>
<p>Run the following command to view all the resources created by the Loki Helm
Chart in your Kubernetes cluster:</p>
<pre><code class="language-jsx">kubectl get all -l app.kubernetes.io/name=loki
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/all-resources-output.png" alt="Output of kubectl get all for the Loki release, listing its pods, services and deployments"></p>
<p>Helm chart created the following components:</p>
<ul>
<li><strong>Loki read and write:</strong> Loki is installed in scalable mode by default, which
includes a read-and-write component. These components can be independently
scaled out.</li>
<li><strong>Gateway:</strong> Inspired by Grafana’s
<a href="https://github.com/grafana/loki/blob/main/production/ksonnet/loki">Tanka setup</a>,
the chart installs a gateway component by default. This NGINX component
exposes Loki’s API and automatically proxies requests to the appropriate Loki
components (read or write, or a single instance in the case of filesystem
storage). The gateway must be enabled to provide an Ingress since the Ingress
only exposes the gateway. If enabled, Grafana and log shipping agents, such as
Promtail, should be configured to use the gateway. If NetworkPolicies are
enabled, they become more restrictive when the gateway is active.</li>
<li><strong>Caching:</strong> In-memory caching is enabled by default. If this type of caching
is unsuitable for your deployment, consider setting up memcache.</li>
</ul>
<p>Run this command to view all the Kubernetes Services for Prometheus &amp; Grafana:</p>
<pre><code class="language-jsx">kubectl get service -l app.kubernetes.io/name=loki
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/all-services-output.png" alt="Output of kubectl get service for the Loki release, listing its services"></p>
<p>Listed services for Loki are:</p>
<ul>
<li>loki-backend</li>
<li>loki-backend-headless</li>
<li>loki-gateway</li>
<li>loki-memberlist</li>
<li>loki-read</li>
<li>loki-read-headless</li>
<li>loki-write</li>
<li>loki-write-headless</li>
<li>query-scheduler-discovery</li>
</ul>
<p>The <code>loki-gateway</code> service will be used to add Loki as a Datasource into
Grafana.</p>
<h3>6. Adding Loki data source in Grafana</h3>
<p>On the main page of Grafana, click on &quot;<strong>Home</strong>&quot;. Under &quot;<strong>Connections</strong>&quot;, you
will find the &quot;<strong>Data sources</strong>&quot; option.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/grafana-dashboard.webp" alt="The Grafana home page, with the Data sources option listed under Connections"></p>
<p>On the Data Sources page, click on the &quot;Add new data source&quot; button.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/data-sources-page.png" alt="The Grafana Data sources page with the Add new data source button"></p>
<p>In the search bar, type &quot;Loki&quot; and search for it.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/add-data-source.png" alt="Search results for Loki on the Add data source page"></p>
<p>Clicking on &quot;Loki&quot; will redirect you to the dedicated page for the Loki data
source.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-data-source.png" alt="The dedicated settings page for the Loki data source in Grafana"></p>
<p>To read the metrics from Loki, we will use the <code>loki-gateway</code> service. Add the
URL of the service as <code>http://loki-gateway</code>.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-form.png" alt="The Loki data source form with the connection URL set to http://loki-gateway"></p>
<p>After clicking on the &quot;Save &amp; test&quot; button, you will receive a toastr message
shown in the image below. This message is received because no clients have been
created for Loki yet.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-addon-output.png" alt="A Grafana toast reporting that no clients exist yet for the Loki data source"></p>
<h3>7. Install Kubernetes event exporter using the helm chart</h3>
<p>Create an <strong>event-exporter-values yaml</strong> file with the below snippet</p>
<pre><code class="language-jsx">config:
  leaderElection: {}
  logLevel: debug
  logFormat: pretty
  metricsNamePrefix: event_exporter_
  receivers:
    - name: &quot;dump&quot;
      file:
        path: &quot;/dev/stdout&quot;
        layout: {}
    - name: &quot;loki&quot;
      loki:
        url: &quot;http://loki-gateway/loki/api/v1/push&quot;
        streamLabels:
          source: kubernetes-event-exporter
          container: kubernetes-event-exporter
  route:
    routes:
      - match:
          - receiver: &quot;dump&quot;
          - receiver: &quot;loki&quot;
</code></pre>
<p>With the use of the above snippet, run this command to install the Kubernetes
event exporter in your Kubernetes Cluster.</p>
<pre><code class="language-jsx">helm repo add bitnami [https://charts.bitnami.com/bitnami](https://charts.bitnami.com/bitnami)
helm install event-exporter bitnami/kubernetes-event-exporter --values event-exporter.yaml
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-installation-output.png" alt="Terminal output from installing the kubernetes-event-exporter Helm chart"></p>
<p>To view all the resources created by the above helm chart, run this command:</p>
<pre><code class="language-jsx">kubectl get all -l app.kubernetes.io/name=kubernetes-event-exporter
</code></pre>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-all-resources.png" alt="Output of kubectl get all for the event exporter release, listing its pod and deployment"></p>
<p>To view the logs of the event exporter POD, run this command:</p>
<pre><code class="language-jsx">kubectl logs -f pod/kubernetes-event-exporter-586455bbdd-sqlqc
</code></pre>
<p>Note: Replace <strong>kubernetes-event-exporter-586455bbdd-sqlqc</strong> with your pod name.</p>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-logs.webp" alt="event-exporter-logs"></p>
<p>As you can see in the above image, the event exporter is working and running
fine. Event logs are being sent to both the receivers that we’d configure in the
values YAML file.</p>
<p>Once the POD is created &amp; running, we can go back to the Loki data source under
<strong>Connections</strong> &gt; <strong>Data Sources</strong> page.</p>
<p>Again click on the “Save &amp; test” button &amp; this time you’ll receive a success
toastr message.</p>
<p>Output:</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/loki-data-source-added-output.png" alt="loki-data-source-added-output"></p>
<h3>8. Kubernetes event exporter dashboard</h3>
<p>We will import this
<a href="https://grafana.com/grafana/dashboards/17882-kubernetes-event-exporter/">dashboard</a>
into Grafana to monitor and track the events received from the Kubernetes
cluster. You can go through this blog if you want to learn how to import an
existing dashboard into Grafana.</p>
<p>After successfully importing the dashboard, you can view all the events from the
cluster, as shown in the image below. Additionally, you can filter the events
based on any value within any interval.</p>
<p>Kubernetes Event Exporter</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/k8s-event-exporter-and-grafana-loki-integration/event-exporter-dashboard.png" alt="Kubernetes Event Exporter"></p>
<h2>Conclusion</h2>
<p>In this blog post, we discussed the process of setting up Grafana Loki and
Kubernetes Event exporter. We covered various steps, such as creating a policy
for Grafana Loki, creating a role with the necessary permissions, and adding
Grafana using the Helm chart, installing the Loki stack, and adding Loki as a
data source in Grafana, installing Kubernetes event exporter using the Helm
chart, and finally, setting up the Kubernetes event exporter dashboard in
Grafana.</p>
<p>By following the steps outlined in this blog post, you can effectively monitor
and track events from your Kubernetes cluster using Grafana Loki and Kubernetes
Event exporter. This setup provides valuable insights and helps in
troubleshooting and analyzing events in your cluster.</p>
<p>If you have any questions or feedback, please feel free to reach out. Happy
monitoring!</p>
]]></content>
    </entry>
  <entry>
      <title type="text"><![CDATA[Setting up Prometheus and Grafana on Kubernetes using Helm]]></title>
      <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/prometheus-and-grafana-integration"/>
      <link rel="enclosure" type="image/jpeg" href="https://www.bigbinary.com/blog/images/social_preview_images/setting_prometheus_grafana_on_kubernetes_using_helm.jpg"/>
      <id>https://www.bigbinary.com/blog/prometheus-and-grafana-integration</id>
      <published>2024-01-25T00:00:00.000Z</published>
      <updated>2024-01-25T00:00:00.000Z</updated>
      <author>
        <name>Vishal Yadav</name>
        <uri>https://www.bigbinary.com/blog/authors/vishal76342</uri>
      </author>
      <category term="NeetoCI"/>
      <category term="Kubernetes"/>
      <media:thumbnail url="https://www.bigbinary.com/blog/images/social_preview_images/setting_prometheus_grafana_on_kubernetes_using_helm.jpg" width="1200" height="627"/>
      <summary type="text"><![CDATA[In this blog, we will learn how to set up Prometheus and Grafana on Kubernetes using Helm. Prometheus along with Grafana is a highly scalable open-source monitoring framework for container...]]></summary>
      <content type="html" xml:base="https://www.bigbinary.com/blog/prometheus-and-grafana-integration"><![CDATA[<p>In this blog, we will learn how to set up Prometheus and Grafana on Kubernetes
using Helm.</p>
<p><a href="https://prometheus.io/">Prometheus</a> along with
<a href="https://grafana.com/">Grafana</a> is a highly scalable open-source monitoring
framework for
<a href="https://devopscube.com/docker-container-clustering-tools/">container orchestration platform</a>.
Prometheus probes the application and collects various data. It stores all this
data in its time series database. Grafana is a visualization tool. It uses the
data from the database to show the data that is meaningful to the user.</p>
<p>Both Prometheus and Grafana are gaining popularity in
the <a href="https://devopscube.com/what-is-observability/">observability</a> space as it
helps with metrics and alerts. Learning to integrate them using Helm will allow
us to monitor our Kubernetes cluster and troubleshoot problems easily.
Furthermore, we can deep dive into our cluster's well-being and efficiency,
focusing on resource usage and performance metrics within our Kubernetes
environment.</p>
<p>We will also learn how to create a simple
<a href="https://grafana.com/grafana/dashboards/">dashboard</a> on Grafana.</p>
<h2><strong>Why using Prometheus and Grafana for monitoring is good</strong></h2>
<p>Using Prometheus and Grafana for monitoring has many benefits:</p>
<ul>
<li><strong>Scalability:</strong> Both tools are highly scalable and can handle the monitoring
needs of small to large Kubernetes clusters.</li>
<li><strong>Flexibility:</strong> They allow us to create custom dashboards tailored to our
specific monitoring requirements.</li>
<li><strong>Real-time Monitoring:</strong> Prometheus provides real-time monitoring, helping us
to quickly detect and respond to issues.</li>
<li><strong>Alerting:</strong> Prometheus enables us to set up alerts based on specific
metrics, so we can be notified when issues arise.</li>
<li><strong>Data Visualization:</strong> Grafana offers powerful data visualization
capabilities, making it easier to understand complex data.</li>
<li><strong>Open Source:</strong> Both Prometheus and Grafana are open-source, reducing
monitoring costs.</li>
<li><strong>Community Support:</strong> We can benefit from active communities, ensuring
continuous development and support.</li>
<li><strong>Integration:</strong> They seamlessly integrate with other Kubernetes components
and applications, simplifying setup.</li>
<li><strong>Historical Data:</strong> Grafana allows us to explore historical data, aiding in
long-term analysis and trend identification.</li>
<li><strong>Extensible:</strong> Both tools are extensible, allowing us to integrate additional
data sources and plugins.</li>
<li><strong>Efficient Resource Usage:</strong> Prometheus efficiently utilizes resources,
ensuring minimal impact on our cluster's performance.</li>
</ul>
<p>Two common ways to use Prometheus and Grafana on Kubernetes:</p>
<ol>
<li><strong>Manual Kubernetes deployment</strong>: In this method, we need to write 
<a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Kubernetes Deployments</a> 
and 
<a href="https://kubernetes.io/docs/concepts/services-networking/service/">Services</a>
for both Prometheus and Grafana. In the YAML file, we need to put all the
settings for Prometheus and Grafana on Kubernetes. Then we send these files
to our Kubernetes cluster. But we can end up with many YAML files, which can
be hard. If we make a mistake in any YAML file, Prometheus and Grafana won't
work on Kubernetes.</li>
<li><strong>Using Helm</strong>: This is an easy way to send any application container to
Kubernetes. <a href="https://helm.sh/">Helm</a> is the official package manager for
Kubernetes. With Helm, we can make installing, sending, and managing
Kubernetes makes applications easier.</li>
</ol>
<p>A <a href="https://helm.sh/">Helm Chart</a> has all the YAML files:</p>
<ul>
<li>Deployments.</li>
<li>Services.</li>
<li>Secrets.</li>
<li>ConfigMaps manifests.</li>
</ul>
<p>We use these files to send the application container to Kubernetes. Instead of
making individual YAML files for each application container, Helm lets us
download Helm charts that already have YAML files.</p>
<h2>Setting up Prometheus and Grafana using Helm chart</h2>
<p>We will use <a href="https://artifacthub.io/">ArtifactHub</a>, which offers public and
private repositories for Helm Charts. We will use these Helm Charts to arrange
the pods and services in our Kubernetes cluster.</p>
<p>To get Prometheus and Grafana working on Kubernetes with Helm, we will start by
installing Helm.</p>
<h4>Installing Helm on Linux</h4>
<pre><code class="language-bash">sudo apt-get install helm
</code></pre>
<h4>Installing Helm on Windows</h4>
<pre><code class="language-bash">choco install Kubernetes-helm
</code></pre>
<h4>Installing Helm on macOS</h4>
<pre><code class="language-bash">brew install helm
</code></pre>
<p>We can check out the official 
<a href="https://helm.sh/docs/intro/install/">Helm documentation</a>  if we run into any
issues while installing Helm.</p>
<p>The image below represents the successful Helm installation on macOS.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.09.49_PM.png" alt="Terminal output confirming a successful Helm installation on macOS"></p>
<p>For this blog, we’re going to install Helm
<a href="https://artifacthub.io/packages/helm/prometheus-community/kube-prometheus-stack">chart</a>
and by default, this chart also installs additional, dependent charts (including
Grafana):</p>
<ul>
<li><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-state-metrics">prometheus-community/kube-state-metrics</a></li>
<li><a href="https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus-node-exporter">prometheus-community/prometheus-node-exporter</a></li>
<li><a href="https://github.com/grafana/helm-charts/tree/main/charts/grafana">grafana/grafana</a></li>
</ul>
<p>To get this Helm chart, let's run this command:</p>
<pre><code class="language-bash">helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
</code></pre>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.15.43_PM.png" alt="Terminal output from helm repo add and helm repo update"></p>
<p>We have downloaded the latest version of Prometheus &amp; Grafana.</p>
<p>To install the Prometheus Helm Chart on a Kubernetes Cluster, let's run the
following command:</p>
<pre><code class="language-bash">helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack
</code></pre>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-01_at_1.21.46_PM.png" alt="Terminal output from installing the kube-prometheus-stack Helm chart"></p>
<p>We have successfully installed Prometheus &amp; Grafana on the Kubernetes Cluster.
We can access the Prometheus &amp; Grafana servers via ports 9090 &amp; 80,
respectively.</p>
<p>Now, let's run the following command to view all the resources created by the
Helm Chart in our Kubernetes cluster:</p>
<pre><code class="language-bash">kubectl get all
</code></pre>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.19.39_AM.png" alt="Output of kubectl get all, listing the pods, services and deployments the chart created">
The Helm chart created the following resources:</p>
<ul>
<li><strong>Pods</strong>: It hosts the deployed Prometheus Kubernetes application inside the
cluster.</li>
<li><strong>Replica Sets</strong>: A collection of instances of the same application inside the
Kubernetes cluster. It enhances application reliability.</li>
<li><strong>Deployments</strong>: It is the blueprint for creating the application pods.</li>
<li><strong>Services</strong>: It exposes the pods running inside the Kubernetes cluster. We
use it to access the deployed Kubernetes application.</li>
<li><strong>Stateful Sets</strong>: They manage the deployment of the stateful application
components and ensure stable and predictable network identities for these
components.</li>
<li><strong>Daemon Sets</strong>: They ensure that all (or a specific set of) nodes run a copy
of a pod, which is useful for tasks such as logging, monitoring, and other
node-specific operations.</li>
</ul>
<p>Run this command to view all the Kubernetes Services for Prometheus &amp; Grafana:</p>
<pre><code class="language-bash">kubectl get service
</code></pre>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.37.39_AM.png" alt="Output of kubectl get service, listing the ClusterIP services the chart created"></p>
<p>Listed services for Prometheus and Grafana are:</p>
<ul>
<li>alertmanager-operated</li>
<li>kube-prometheus-stack-alertmanager</li>
<li>kube-prometheus-stack-grafana</li>
<li>kube-prometheus-stack-kube-state-metrics</li>
<li>kube-prometheus-stack-operator</li>
<li>kube-prometheus-stack-prometheus</li>
<li>kube-prometheus-stack-prometheus-node-exporter</li>
<li>prometheus-operated</li>
</ul>
<p><code>kube-prometheus-stack-grafana</code> and <code>kube-prometheus-stack-prometheus</code> are the
<code>ClusterIP</code> type services, which means we can only access them within the
Kubernetes cluster.</p>
<p>To expose the Prometheus and Grafana to be accessed outside the Kubernetes
cluster , we can either use the NodeIP or LoadBalance service.</p>
<h2>Exposing Prometheus and Grafana using NodePort services</h2>
<p>Let's run the following command to expose the <code>Prometheus</code> Kubernetes service:</p>
<pre><code class="language-bash">kubectl expose service kube-prometheus-stack-prometheus --type=NodePort --target-port=9090 --name=prometheus-node-port-service

kubectl expose service kube-prometheus-stack-grafana --type=NodePort --target-port=3000 --name=grafana-node-port-service
</code></pre>
<p>That command will create new services of <code>NodePort</code> type &amp; make the Prometheus
and Grafana is accessible outside the Kubernetes Cluster on ports <code>9090</code> and
<code>80</code>.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.59.15_AM.png" alt="The Prometheus and Grafana services changed to NodePort type, exposing ports 9090 and 80"></p>
<p>As we can see, the <code>grafana-node-port-service</code> and
<code>prometheus-node-port-service</code> are successfully created and are being exposed on
node ports <code>32489</code> &amp; <code>30905</code></p>
<p>Now, we can run this command and get the external IP of any node to access the
Prometheus and Grafana:</p>
<pre><code class="language-jsx">kubectl get nodes -o wide
</code></pre>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_11.57.17_AM.png" alt="Output of kubectl get nodes -o wide, showing the internal IP of each node"></p>
<p>We can use the External-IP and the node ports to access the Prometheus and
Grafana dashboards outside the cluster environment.</p>
<p>Prometheus Dashboard</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.04.36_PM.png" alt="Prometheus Dashboard"></p>
<p>Grafana Dashboard</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.04.53_PM.png" alt="Grafana Dashboard"></p>
<p>Run this command, to get the password for the <strong>admin</strong> user of the Grafana
dashboard:</p>
<pre><code>kubectl get secret --namespace default kube-prometheus-stack-grafana -o jsonpath=&quot;{.data.admin-password}&quot; | base64 --decode ; echo
</code></pre>
<h2>Grafana Dashboard</h2>
<p>Upon login to the Grafana dashboard, use <code>admin</code> as the username and our
generated password. We will see &quot;Welcome to Grafana&quot; homepage as shown below.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.12.04_PM.png" alt="The Welcome to Grafana home page after logging in as admin"></p>
<p>Since we used the Kube Prometheus Stack helm chart, the data source for
Prometheus and Alert Manager is added by default.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.18.14_PM.png" alt="The Grafana data sources page, with Prometheus and Alertmanager already added by the Helm chart"></p>
<p>We can add more data sources by clicking on the <strong>Add new data source</strong> button
on the top right side.</p>
<p>By default, this Helm chart adds multiple dashboards to monitor the health of
the Kubernetes cluster and its resources.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.22.37_PM.png" alt="The Grafana dashboards list, showing the Kubernetes dashboards bundled with the Helm chart"></p>
<p>Additionally, we also have the option of creating our dashboards from scratch as
well as importing multiple Grafana dashboards provided by the
<a href="https://grafana.com/grafana/dashboards/">Grafana library</a>.</p>
<p>To import a Grafana Dashboard, let's follow these steps:</p>
<ul>
<li>
<p>From this <a href="https://grafana.com/grafana/dashboards/">Grafana library</a>, we can
add any dashboard</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.28.44_PM.webp" alt="The Grafana dashboard library at grafana.com, used to pick a dashboard to import"></p>
</li>
<li>
<p>Select Dashboard and copy the Dashboard ID</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.34.54_PM.webp" alt="A dashboard page in the Grafana library, showing the dashboard ID to copy"></p>
</li>
<li>
<p>Under <strong>Dashboards</strong> page we can get the <strong>Import</strong> option</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.32.55_PM.png" alt="The Import option on the Grafana Dashboards page"></p>
</li>
<li>
<p>Under &quot;Import Dashboard&quot; page, we need to paste the Dashboard IP that we
copied earlier &amp; click on the <strong>Load</strong> button.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.35.39_PM.png" alt="The Import Dashboard page with the copied dashboard ID pasted in, ready to load"></p>
</li>
<li>
<p>After clicking on the <strong>Load</strong> button, it will auto-load the dashboard from
the library after which we can import the dashboard by clicking on the
<strong>Import</strong> button.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.37.50_PM.png" alt="The loaded dashboard details with the Import button ready to confirm"></p>
</li>
<li>
<p>Once the import is complete, we’ll be redirected to the new imported dashboard
which’ll also be visible under the Dashboards page.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2024/prometheus-and-grafana-integration/Screenshot_2023-11-02_at_12.39.42_PM.png" alt="The newly imported dashboard rendering its panels in Grafana"></p>
<p>We can use this Node Exporter dashboard to monitor &amp; observe the health of our
nodes present in our Kubernetes Cluster.</p>
</li>
</ul>
<h2>Conclusion</h2>
<p>In this blog, we learned how to integrate Prometheus and Grafana using the helm
chart. We also learned how to import dashboards into Grafana from the
<a href="https://grafana.com/grafana/dashboards/">Grafana library</a>.</p>
<p>In the next blog, we will explore how to integrate
<a href="https://grafana.com/oss/loki/">Grafana Loki</a> with Grafana and collect and store
event-related metrics using the
<a href="https://github.com/resmoio/kubernetes-event-exporter">Kubernetes Event Exporter</a>.</p>
]]></content>
    </entry>
  <entry>
      <title type="text"><![CDATA[Using enable-load-relative flag in building Ruby binaries]]></title>
      <link rel="alternate" type="text/html" href="https://www.bigbinary.com/blog/use-of-enable-load-relative-flag-in-building-ruby-binaries"/>
      <link rel="enclosure" type="image/jpeg" href="https://www.bigbinary.com/blog/images/social_preview_images/using_enable_load_relative_flag_ruby_binaries.jpg"/>
      <id>https://www.bigbinary.com/blog/use-of-enable-load-relative-flag-in-building-ruby-binaries</id>
      <published>2023-06-20T00:00:00.000Z</published>
      <updated>2023-06-20T00:00:00.000Z</updated>
      <author>
        <name>Vishal Yadav</name>
        <uri>https://www.bigbinary.com/blog/authors/vishal76342</uri>
      </author>
      <category term="NeetoCI"/>
      <category term="Ruby"/>
      <media:thumbnail url="https://www.bigbinary.com/blog/images/social_preview_images/using_enable_load_relative_flag_ruby_binaries.jpg" width="1200" height="627"/>
      <summary type="text"><![CDATA[I'm working on building NeetoCI, which is a CI/CD solution. While building precompiled Ruby binaries, we encountered some challenges. This blog post explores the problems we faced and how we solved...]]></summary>
      <content type="html" xml:base="https://www.bigbinary.com/blog/use-of-enable-load-relative-flag-in-building-ruby-binaries"><![CDATA[<p>I'm working on building <a href="https://neeto.com/neetoci">NeetoCI</a>, which is a CI/CD
solution. While building precompiled Ruby binaries, we encountered some
challenges. This blog post explores the problems we faced and how we solved
them.</p>
<h2>Pre-compiled Ruby binaries</h2>
<p>Pre-compiled Ruby binaries are distribution-ready versions of Ruby that include
optimized features for specific systems. These Ruby binaries save time by
eliminating the need to compile Ruby source code manually. Pre-compiled Ruby
binaries help users quickly deploy applications that use different versions of
Ruby on multiple machines.</p>
<p><a href="https://rvm.io/">RVM</a> (Ruby Version Manager) is widely used for managing Ruby
installations on Unix-like systems. RVM provides customized pre-compiled Ruby
binaries tailored for various CPU architectures. These binaries offer additional
features like readline support and SSL/TLS support. You can find them at
<a href="https://rvm.io/binaries/">RVM binaries</a>.</p>
<h2>The Need for pre-compiled Ruby Binaries</h2>
<p><a href="https://www.neeto.com/neetoci">NeetoCI</a> must execute user code in a
containerized environment. A Ruby environment is essential for running Ruby on
Rails applications. However, relying on the system's Ruby version is impractical
since it may differ from the user's required version. Although rbenv or rvm can
be used to install the necessary Ruby version, this approach could be slow. To
save time, we chose to leverage pre-compiled Ruby binaries.</p>
<p>As a CI/CD system, NeetoCI must ensure that all versions of Ruby that an
application requires are always available. Hence, we decided to build our
binaries instead of relying on binaries provided by RVM. Also, this would allow
us to do more system-specific optimizations to the Ruby binary at build time.</p>
<h2>Building pre-compiled Ruby binaries</h2>
<p>We built a Ruby binary following the
<a href="https://github.com/ruby/ruby/blob/master/doc/contributing/building_ruby.md">official documentation</a>
. We were able to execute it on our local development machines. But the same
binary ran into an error in our CI/CD environment.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2023/use-of-enable-load-relative-flag-in-building-ruby-binaries/failing-binary-error-image.png" alt="Bad Interpreter"></p>
<pre><code class="language-bash">$ bundle config path vendor/bundle
./ruby: bad interpreter: No such file or directory
</code></pre>
<p>To debug the issue, we initially focused on <code>$PATH</code>. However, even after
resolving the <code>$PATH</code> issues, the problem persisted. We conducted a thorough
investigation to identify the root cause. Unfortunately, not much was written on
the Internet about this error. There was no mention of it in the official
<a href="https://github.com/ruby/ruby/blob/master/doc/contributing/building_ruby.md">Ruby documentation</a>.</p>
<p>As the next step, we decided to download the binary for version 3.2.2 from
<a href="https://rvm.io/binaries/">RVM</a>. While examining the configuration file, we
noticed that the following arguments were used with the configure command during
the Ruby binary build process:</p>
<pre><code class="language-bash">configure_args=&quot;'--prefix=/usr/share/rvm/rubies/ruby-3.2.2' '--enable-load-relative' '--sysconfdir=/etc' '--disable-install-doc' '--enable-shared'&quot;
</code></pre>
<p>Here are the explanations of the configuration arguments:</p>
<ol>
<li>
<p><code>--prefix=/usr/share/rvm/rubies/ruby-3.2.2</code>: This specifies the directory
where the Ruby binaries, libraries and other files will be kept after the
installation is done.</p>
</li>
<li>
<p><code>--enable-load-relative</code>: This specifies that Ruby can load relative paths
for dynamically linked libraries. It allows the usage of relative paths
instead of absolute paths when loading shared libraries. This feature can be
beneficial in specific deployment scenarios.</p>
</li>
<li>
<p><code>--sysconfdir=/etc</code>: This argument sets the directory where Ruby's system
configuration files will be installed. In this case, it specifies the <code>/etc</code>
directory as the location for these files.</p>
</li>
<li>
<p><code>--disable-install-doc</code>: When this option is enabled, the installation of
documentation files during the build process is disabled. This can help speed
up the build process and save disk space, especially if you do not require
the documentation files.</p>
</li>
<li>
<p><code>--enable-shared</code>: Enabling this option allows the building of shared
libraries for Ruby. Shared libraries enable Ruby to dynamically link and load
specific functionality at runtime, leading to potential performance
improvements and reduced memory usage.</p>
</li>
</ol>
<p>In simpler terms, when the <code>--enable-load-relative</code> flag is enabled, the
compiled Ruby binary can search for shared libraries in its own directory using
the <code>$ORIGIN</code> variable.</p>
<p>When I built the binary on the Docker registry, then the passed <code>--prefix</code> was
something like <code>/usr/share/neetoci</code>. When the binary is built, then binary had
<code>/usr/share/neetoci</code> is hard-coded at various places. When we download this
binary and use in CI then in the CI environment, Ruby is looking for
<code>/user/share/neetoci</code> to load dependencies.</p>
<p>By enabling <code>--enable-load-relative</code> flag while building the binary Ruby will
not use the hard coded value. Rather Ruby will use <code>$ORIGIN</code> variable and will
search for the dependencies in the directory mentioned in <code>$ORIGIN</code>.</p>
<p>This is particularly helpful when the Ruby binary is relocated to a different
directory or system. By using relative paths with <code>$ORIGIN</code>, the binary can find
its shared libraries regardless of its new location. Without this flag, shared
libraries are loaded using absolute paths, which can cause issues if the binary
is moved to a different location and cannot locate its shared libraries.</p>
<p>In our specific use case, where we create and download binaries in separate
containers, we encountered an error due to the absolute paths. To overcome this,
we enabled the <code>--enable-load-relative</code> flag. This allowed the binary to find
its shared libraries successfully, and it worked as expected in our CI/CD
environment.</p>
<p><img src="https://www.bigbinary.com/blog/images/images_used_in_blog/2023/use-of-enable-load-relative-flag-in-building-ruby-binaries/passing-ruby-binary-image.webp" alt="Successful Build"></p>
]]></content>
    </entry>
</feed>