Known errors and solutions

Nginx ingress
308 redirect to HTTPS
Solution
Edit ingress configmap.
1
kubectl -n ingress-external edit configmaps ingress-controller-leader-nginx
Copied!
Add the following annotations:
1
use-forwarded-headers: "true"
2
force-ssl-redirect: "false"
Copied!
Restart controller.
1
kubectl -n ingress-external scale deployment --replicas=0 ingress-external-nginx-ingress-controller
2
kubectl -n ingress-external scale deployment --replicas=2 ingress-external-nginx-ingress-controller
Copied!
References
​https://github.com/kubernetes/ingress-nginx/issues/1957​
Duplicate location "/healthz"
Complete nginx pod log:
1
2020/01/27 17:12:12 [emerg] 105#105: duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
2
nginx: [emerg] duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
3
nginx: configuration file /tmp/nginx-cfg568474076 test failed
Copied!
It happens when you have an ingress object conflicting with "/healthz" path.
Solution
Make sure to not have an ingress object overlapping "/healthz".
Pod sandbox changed, it will be killed and re-created
Cause
This error happens when deploying a pod. It is caused most liked because of Docker processes crashed or is unstable on the node due IO peak.
Solution
The solution is to reboot the node.
References
​https://plugaru.org/2018/05/21/pod-sandbox-changed-it-will-be-killed-and-re-created/​
failed to watch file "/var/log/pods/6438eb52-202a-11ea-8dce-e279cb2777e2/my-app/0.log": no space left on device
Symptom
When you run.
1
kubectl -n my-ns logs -f my-app-659858b967-5hmtz
Copied!
The command outputs a few lines of log and then breaks.
Cause
The obvious reason is the node's HD is full. Although this error can be caused by other reasons.
This error (ENOSPC) comes from the inotify_add_watch syscall, and actually has multiple meanings (the message comes from golang). Most likely the problem is from exceeding the maximum number of watches, not filling the disk. This can be increased with the fs.inotify.max_user_watches sysctl.
Solution
Increase max_user_watches.
1
cat /proc/sys/fs/inotify/max_user_watches # default is 8192 
2
sysctl fs.inotify.max_user_watches=1048576 # increase to 1048576
Copied!
If you do not have SSH connection to the node, apply the following manifest (not recommended for production environments).
1
apiVersion: extensions/v1beta1
2
kind: DaemonSet
3
metadata:
4
  name: more-fs-watchers
5
  namespace: kube-system
6
  labels:
7
    app: more-fs-watchers
8
spec:
9
  template:
10
    metadata:
11
      labels:
12
        name: more-fs-watchers
13
    spec:
14
      hostNetwork: true
15
      hostPID: true
16
      hostIPC: true
17
      initContainers:
18
        - command:
19
            - sh
20
            - -c
21
            - sysctl -w fs.inotify.max_user_watches=524288;
22
          image: alpine:3.6
23
          imagePullPolicy: IfNotPresent
24
          name: sysctl
25
          resources: {}
26
          securityContext:
27
            privileged: true
28
          volumeMounts:
29
            - name: sys
30
              mountPath: /sys
31
      containers:
32
        - resources:
33
            requests:
34
              cpu: 0.01
35
          image: alpine:3.6
36
          name: sleepforever
37
          command: ["tail"]
38
          args: ["-f", "/dev/null"]
39
      volumes:
40
        - name: sys
41
          hostPath:
42
            path: /sys
Copied!
References
​https://github.com/google/cadvisor/issues/1581​
​https://github.com/Azure/AKS/issues/772​

Node crash recovery

Next - Lumen

Cheat Sheet

Last modified 2yr ago

Copy link

Contents

Nginx ingress

308 redirect to HTTPS

Duplicate location "/healthz"

Pod sandbox changed, it will be killed and re-created

Cause

Solution

References

failed to watch file "/var/log/pods/6438eb52-202a-11ea-8dce-e279cb2777e2/my-app/0.log": no space left on device