Helm

Helm and Tiller tips and tricks.

Helm init (secure TLS)

Generating Certificate Authorities and Certificates

Generate a certificate authority

openssl genrsa -out ./ca.key.pem 4096
openssl req \
  -key ca.key.pem \
  -new -x509 -days 7300 -sha256 \
  -out ca.cert.pem \
  -extensions v3_ca

Generate the Tiller key:

openssl genrsa \
  -out ./tiller.key.pem 4096

Generate the Helm client's key

openssl genrsa \
  -out ./helm.key.pem 4096

Create certificates from these keys

openssl req \
  -key tiller.key.pem \
  -new \
  -sha256 \
  -out tiller.csr.pem

Repeat this step for the Helm client certificate

openssl req \
  -key helm.key.pem \
  -new \
  -sha256 \
  -out helm.csr.pem

Now we sign each of these CSRs with the CA certificate we created.

openssl x509 -req \
  -CA ca.cert.pem \
  -CAkey ca.key.pem \
  -CAcreateserial \
  -in tiller.csr.pem \
  -out tiller.cert.pem \
  -days 365

And again for the client certificate.

openssl x509 -req \
  -CA ca.cert.pem \
  -CAkey ca.key.pem \
  -CAcreateserial \
  -in helm.csr.pem \
  -out helm.cert.pem \
  -days 365

At this point, the important files for us are these:

# The CA. Make sure the key is kept secret.
ca.cert.pem
ca.key.pem
# The Helm client files
helm.cert.pem
helm.key.pem
# The Tiller server files.
tiller.cert.pem
tiller.key.pem

Setup RBAC

Create a ServiceAccount for Tiller in the kube-system namespace:

kubectl -n kube-system create sa tiller

Create a ClusterRoleBinding for Tiller:

kubectl create clusterrolebinding tiller \
  --clusterrole cluster-admin \
  --serviceaccount=kube-system:tiller

Creating a Custom Tiller Installation

Helm includes full support for creating a deployment configured for SSL. By specifying a few flags, the helm init command can create a new Tiller installation complete with all of our SSL configuration.

To take a look at what this will generate, run this command:

helm init \
  --dry-run --debug \
  --tiller-tls \
  --tiller-tls-cert ./tiller.cert.pem \
  --tiller-tls-key ./tiller.key.pem \
  --tiller-tls-verify \
  --tls-ca-cert ca.cert.pem \
  --service-account tiller

The output will show you a Deployment, a Secret, and a Service. Your SSL information will be preloaded into the Secret, which the Deployment will mount to pods as they start up.

If you want to customise the manifest, you can save that output to a file and then use kubectl create to load it into your cluster.

Otherwise, you can remove the --dry-run and --debug flags.

helm init \
  --tiller-tls \
  --tiller-tls-cert ./tiller.cert.pem \
  --tiller-tls-key ./tiller.key.pem \
  --tiller-tls-verify \
  --tls-ca-cert ca.cert.pem \
  --upgrade \
  --service-account tiller

In a minute or two it should be ready. We can check Tiller like this:

kubectl -n kube-system get deployment

Sample output:

NAME            DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
... other stuff
tiller-deploy   1         1         1            1           2m

If there is a problem, you may want to use kubectl get pods -n kube-system to find out what went wrong. With the SSL/TLS support, the most common problems all have to do with improperly generated TLS certificates or accidentally swapping the cert and the key.

Configuring the Helm Client

For a quick test, we can specify our configuration manually. We'll run a normal Helm command (helm ls), but with SSL/TLS enabled.

helm ls \
  --tls \
  --tls-ca-cert ca.cert.pem \
  --tls-cert helm.cert.pem \
  --tls-key helm.key.pem

This configuration sends our client-side certificate to establish identity, uses the client key for encryption, and uses the CA certificate to validate the remote Tiller's identity.

Typing a line that is cumbersome, though. The shortcut is to move the key, cert, and CA into $HELM_HOME:

$ cp ca.cert.pem $(helm home)/ca.pem
$ cp helm.cert.pem $(helm home)/cert.pem
$ cp helm.key.pem $(helm home)/key.pem

With this, you can simply run helm ls --tls to enable TLS.

References

https://github.com/helm/helm/blob/master/docs/tiller_ssl.md

https://medium.com/google-cloud/install-secure-helm-in-gke-254d520061f7

https://medium.com/@amimahloof/how-to-setup-helm-and-tiller-with-rbac-and-namespaces-34bf27f7d3c3

Uninstall

Using helm command

To uninstall tiller from a kubernetes cluster:

helm reset

To delete failed tiller from a kubernetes cluster:

helm reset --force

Manually

kubectl -n kube-system delete deployment tiller-deploy
kubectl -n kube-system delete service/tiller-deploy
kubectl -n kube-system delete secret/tiller-secret

Restricted namespace

References

https://helm.sh/docs/using_helm/#example-deploy-tiller-in-a-namespace-restricted-to-deploying-resources-in-another-namespace

Tools

chartpress

GitHub - jupyterhub/chartpress: automate building and publishing images for helm chartsGitHub

Problems and solutions

Broken pipe when using TLS

It might be caused by a previous Tiller instillation that was not deleted properly (especially the tiller-secret), follow the "Uninstall -> Manually" on this page.

Configmaps is forbidden

It happens when Tiller Service Account does not have enough permissions.

kubectl create serviceaccount --namespace kube-system tiller
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller
kubectl patch deploy --namespace kube-system tiller-deploy -p '{"spec":{"template":{"spec":{"serviceAccount":"tiller"}}}}'      
helm init --service-account tiller --upgrade

Helm ls (or any other command) hangs when using TLS

Cause uncertain for me.

Solution:

In one terminal session run and leave it running:

kubectl -n tiller-ns port-forward svc/tiller-deploy 44134:44134

In another terminal session, run:

export HELM_HOST=:44134
helm ls \
  --tls \
  --tls-ca-cert ca.cert.pem \
  --tls-cert helm.cert.pem \
  --tls-key helm.key.pem \
  --tiller-namespace tiller-ns

Or using --host parameter.

export HELM_HOST=:44134
helm ls \
  --host :44134 \
  --tls \
  --tls-ca-cert ca.cert.pem \
  --tls-cert helm.cert.pem \
  --tls-key helm.key.pem \
  --tiller-namespace tiller-ns

Helm 3

Uninstall/remove chart completely

helm template happy-panda stable/mariadb --namespace kube-system | kubectl delete -f -