As a learning exercise, I migrated all my side projects to Kubernetes. I was using Fluent Bit to forward application container logs to my ElasticSearch instance.

Everything was working as expected, but the cluster was starting to get expensive. DigitalOcean Kubernetes is competitively priced, but paying $80/month to run side projects is steep. Using a third-party logging and monitoring service with a generous free-tier would require fewer nodes in my cluster.

In order to get logs forwarded correctly, I had to use Helm's Post Rendering hooks to modify Kubernetes manifests before the respective chart is installed / upgraded on my cluster. I hacked together a solution that isn't the most secure, but it saves me $30 a month. It's also for a side project. Specifically, it's side project logs. My solution is good enough for now.

I am writing this post to document how I used Helm's Post Render hooks to forward my application logs to logz.io.

Introduction to Post Rendering Hooks

You can read the Helm docs for more details, but here are the highlights:

Post rendering gives chart installers the ability to manually manipulate, configure, and/or validate rendered manifests before they are installed by Helm

Post rendering hooks allows you to use community-maintained Helm charts while providing the flexibility to make small customizations.

A post-renderer can be any executable that accepts rendered Kubernetes manifests on STDIN and returns valid Kubernetes manifests on STDOUT. It should return an non-0 exit code in the event of a failure.

This generic interface means you can use any tool you want to modify the manifest YAML. Writing up a bash script is the easiest way to get started. More complex workflows can be solved with scripts written in your favourite programming language.

A post renderer can be used with install, upgrade, and template

Link to docs for:

• helm install
• helm upgrade
• helm template

Diagram

Process

This section contains an overview of the workflow

Understand the Helm Deployment

Once you have the Helm chart you want to modify, spend a few minutes understanding how the chart works. Run a helm template to render charts locally. Get a rough idea of how things fit together.

Focus on ConfigMap

The ConfigMap type allows you to modify files and settings for applications running on Kubernetes. This is what most post render hooks will modify.

Install Helm Chart

In order to customize the Helm chart, you must first understand the Helm chart. Do a helm install, check the container logs, examine the Kubernetes objects. Are the Kubernetes resources that were created the ones you expected to see?

Customize Kubernetes Manifest

Figure out which configurations you want to modify and write up a kustomization.yaml that specifies the key, value pairs you want in the final YAML being applied to your Kubernetes cluster.

In my case, I wanted to replace the fluent-bit-output.conf and fluent-bit-service.conf files -- this forwards all application logs to logz.io. I did this by creating a patch.yaml file that contained the configuration I wanted:

# patch.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: doesNotMatter  # can be anything
data:
  fluent-bit-service.conf: |
    [SERVICE]
        Flush        1
        Daemon       Off
        Log_Level    info
        Parsers_File parsers.conf
        Plugins_File plugins.conf

  fluent-bit-output.conf: |
    [OUTPUT]
        Name  logzio
        Match *
        logzio_token token-goes-here
        logzio_url   https://listener.logz.io:8071

You can use kustomize to modify YAML.

The following kustomization.yaml configuration will take the key, values from our patch.yaml and overwrite them in my Helm generated YAML, which is stored in all.yaml

# kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# list of Resource Config to be Applied
resources:
  - all.yaml

patches:
  - path: patch.yaml
    target:
      kind: ConfigMap
      name: "*"

Check the docs for additional YAML manipulation options.

Post Render Executable

I created the following executable script called kustomize:

#!/bin/bash

# save incoming YAML to file
cat <&0 > all.yaml

# modify the YAML with kustomize
kustomize build . && rm all.yaml

Copy and paste into the terminal:

printf '#!/bin/bash\n\ncat <&0 > all.yaml\n\nkustomize build . && rm all.yaml' >> kustomize
chmod +x kustomize

Install Chart with Post Render Hook

helm install stable/fluent-bit --generate-name -f ../helm/values__fluent_bit.yaml --post-renderer ./kustomize

Recommend using the --dry-run and --debug to ensure you are applying the manifests you expect to your cluster

Conclusion

Helm's Post Rendering hooks allow you to customize Helm deployment without having to fork charts. I was able to hack together a solution that forwards my application logs to logz.io using Fluent Bit in an afternoon.

While the solution isn't the most secure, it saves me $30/month. When I have time, it might be worth looking at ways I can template out this workflow using Helm and the Go templating language.

Additional Resources

• Helm Docs: Post Rendering
• thomastaylor312/Advanced Helm Demos: Post render example
• logz.io Fluent Bit config
• GitHub: Fluent Bit Helm chart