As a learning exercise, I migrated all my side projects to Kubernetes. I was using Fluent Bit to forward application container logs to my ElasticSearch instance.
Everything was working as expected, but the cluster was starting to get expensive. DigitalOcean Kubernetes is competitively priced, but paying $80/month to run side projects is steep. Using a third-party logging and monitoring service with a generous free-tier would require fewer nodes in my cluster.
In order to get logs forwarded correctly, I had to use Helm's Post Rendering hooks to modify Kubernetes manifests before the respective chart is installed / upgraded on my cluster. I hacked together a solution that isn't the most secure, but it saves me $30 a month. It's also for a side project. Specifically, it's side project logs. My solution is good enough for now.
I am writing this post to document how I used Helm's Post Render hooks to forward my application logs to logz.io.
Introduction to Post Rendering Hooks
You can read the Helm docs for more details, but here are the highlights:
Post rendering gives chart installers the ability to manually manipulate, configure, and/or validate rendered manifests before they are installed by Helm
Post rendering hooks allows you to use community-maintained Helm charts while providing the flexibility to make small customizations.
A post-renderer can be any executable that accepts rendered Kubernetes manifests on STDIN and returns valid Kubernetes manifests on STDOUT. It should return an non-0 exit code in the event of a failure.
This generic interface means you can use any tool you want to modify the manifest YAML. Writing up a bash script is the easiest way to get started. More complex workflows can be solved with scripts written in your favourite programming language.
A post renderer can be used with
Link to docs for:
This section contains an overview of the workflow
Understand the Helm Deployment
Once you have the Helm chart you want to modify, spend a few minutes understanding how the chart works. Run a
helm template to render charts locally. Get a rough idea of how things fit together.
Focus on ConfigMap
ConfigMap type allows you to modify files and settings for applications running on Kubernetes. This is what most post render hooks will modify.
Install Helm Chart
In order to customize the Helm chart, you must first understand the Helm chart. Do a
helm install, check the container logs, examine the Kubernetes objects. Are the Kubernetes resources that were created the ones you expected to see?
Customize Kubernetes Manifest
Figure out which configurations you want to modify and write up a
kustomization.yaml that specifies the key, value pairs you want in the final YAML being applied to your Kubernetes cluster.
In my case, I wanted to replace the
fluent-bit-service.conf files -- this forwards all application logs to logz.io. I did this by creating a
patch.yaml file that contained the configuration I wanted:
# patch.yaml apiVersion: v1 kind: ConfigMap metadata: name: doesNotMatter # can be anything data: fluent-bit-service.conf: | [SERVICE] Flush 1 Daemon Off Log_Level info Parsers_File parsers.conf Plugins_File plugins.conf fluent-bit-output.conf: | [OUTPUT] Name logzio Match * logzio_token token-goes-here logzio_url https://listener.logz.io:8071
You can use
kustomize to modify YAML.
kustomization.yaml configuration will take the key, values from our
patch.yaml and overwrite them in my Helm generated YAML, which is stored in
# kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # list of Resource Config to be Applied resources: - all.yaml patches: - path: patch.yaml target: kind: ConfigMap name: "*"
Check the docs for additional YAML manipulation options.
Post Render Executable
I created the following executable script called
#!/bin/bash # save incoming YAML to file cat <&0 > all.yaml # modify the YAML with kustomize kustomize build . && rm all.yaml
Copy and paste into the terminal:
printf '#!/bin/bash\n\ncat <&0 > all.yaml\n\nkustomize build . && rm all.yaml' >> kustomize chmod +x kustomize
Install Chart with Post Render Hook
helm install stable/fluent-bit --generate-name -f ../helm/values__fluent_bit.yaml --post-renderer ./kustomize
Recommend using the
--debug to ensure you are applying the manifests you expect to your cluster
Helm's Post Rendering hooks allow you to customize Helm deployment without having to fork charts. I was able to hack together a solution that forwards my application logs to logz.io using Fluent Bit in an afternoon.
While the solution isn't the most secure, it saves me $30/month. When I have time, it might be worth looking at ways I can template out this workflow using Helm and the Go templating language.