DevOps Buzz
Search…
Bash / Shell
Bitbucket
Distros
Elasticsearch
General
Guidelines / Standards
microk8s
Prometheus
RabbitMQ
VirtualBox
Known errors and solutions

Nginx ingress

308 redirect to HTTPS

Solution

Edit ingress configmap.
1
kubectl -n ingress-external edit configmaps ingress-controller-leader-nginx
Copied!
Add the following annotations:
1
use-forwarded-headers: "true"
2
force-ssl-redirect: "false"
Copied!
Restart controller.
1
kubectl -n ingress-external scale deployment --replicas=0 ingress-external-nginx-ingress-controller
2
kubectl -n ingress-external scale deployment --replicas=2 ingress-external-nginx-ingress-controller
Copied!

References

Duplicate location "/healthz"

Complete nginx pod log:
1
2020/01/27 17:12:12 [emerg] 105#105: duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
2
nginx: [emerg] duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
3
nginx: configuration file /tmp/nginx-cfg568474076 test failed
Copied!
It happens when you have an ingress object conflicting with "/healthz" path.

Solution

Make sure to not have an ingress object overlapping "/healthz".

Pod sandbox changed, it will be killed and re-created

Cause

This error happens when deploying a pod. It is caused most liked because of Docker processes crashed or is unstable on the node due IO peak.

Solution

The solution is to reboot the node.

References

failed to watch file "/var/log/pods/6438eb52-202a-11ea-8dce-e279cb2777e2/my-app/0.log": no space left on device

Symptom

When you run.
1
kubectl -n my-ns logs -f my-app-659858b967-5hmtz
Copied!
The command outputs a few lines of log and then breaks.

Cause

The obvious reason is the node's HD is full. Although this error can be caused by other reasons.
This error (ENOSPC) comes from the inotify_add_watch syscall, and actually has multiple meanings (the message comes from golang). Most likely the problem is from exceeding the maximum number of watches, not filling the disk. This can be increased with the fs.inotify.max_user_watches sysctl.

Solution

Increase max_user_watches.
1
cat /proc/sys/fs/inotify/max_user_watches # default is 8192
2
sysctl fs.inotify.max_user_watches=1048576 # increase to 1048576
Copied!
If you do not have SSH connection to the node, apply the following manifest (not recommended for production environments).
1
apiVersion: extensions/v1beta1
2
kind: DaemonSet
3
metadata:
4
name: more-fs-watchers
5
namespace: kube-system
6
labels:
7
app: more-fs-watchers
8
spec:
9
template:
10
metadata:
11
labels:
12
name: more-fs-watchers
13
spec:
14
hostNetwork: true
15
hostPID: true
16
hostIPC: true
17
initContainers:
18
- command:
19
- sh
20
- -c
21
- sysctl -w fs.inotify.max_user_watches=524288;
22
image: alpine:3.6
23
imagePullPolicy: IfNotPresent
24
name: sysctl
25
resources: {}
26
securityContext:
27
privileged: true
28
volumeMounts:
29
- name: sys
30
mountPath: /sys
31
containers:
32
- resources:
33
requests:
34
cpu: 0.01
35
image: alpine:3.6
36
name: sleepforever
37
command: ["tail"]
38
args: ["-f", "/dev/null"]
39
volumes:
40
- name: sys
41
hostPath:
42
path: /sys
Copied!

References