DevOps Buzz
  • Initial page
  • About me
  • Ansible
    • Cheat Sheet
    • Dynamic inventory
  • AKS
    • Cheat Sheet
    • Autoscale
    • Backup
    • Dashboard
    • Ingress
    • Node management
    • News
    • Persistent Volumes
  • Arch Linux
    • Docker
    • Install
    • Network
    • VirtualBox guest
  • Azure
    • API
    • CLI/Powershell
    • DevOps
      • Cheat Sheet
    • Application Gateway
    • Tools
    • Sops
  • AWS
    • Cheat Sheet
    • Tools
    • Cognito with Amplify and React
  • Bash / Shell
    • Cheat Sheet
    • Tools
  • Bitbucket
    • Cheat Sheet
  • CSS
    • Cheat Sheet
  • Distros
    • Manjaro
  • Docker
    • Cheat Sheet
    • ELK
    • Ubuntu NoVNC
    • Tools
    • Known errors and solutions
  • Elasticsearch
    • Cheat Sheet
    • Tools
  • ELK
    • Cheat Sheet
  • emacs
    • Cheat sheet
  • Gatekeeper (OPA)
    • Cheat Sheet
    • Developer Quick Start
  • GCP
    • Cheat Sheet
  • General
    • Tools
    • News
  • Git
    • Cheat Sheet
    • Tools
  • Golang
    • Cheat Sheet
  • Guidelines / Standards
    • Cheat Sheet
  • i3wm
    • Cheat Sheet
  • Ipsec
    • Cheat sheet
  • Istio
    • Cheat Sheet
  • Kind
    • Cheat Sheet
  • Kops
    • Cheat Sheet
  • Kubeadm
    • Cheat Sheet
    • Change serviceSubnet CIDR
    • Setup cluster
      • Azure
      • Bare-metal
      • Vagrant + VirtualBox
    • Multi master
    • Known errors and solutions
  • Kubernetes
    • Kubectl Cheat Sheet
    • etcd Cheat Sheet
    • Tools
    • News
    • Deployments
      • Deployment examples
      • Blue/Green Deployment
      • Canary Deployment
    • Dashboard
    • ELK
    • Helm
    • Ingress
    • logz.io
    • Minikube
    • Monitoring
    • Node Management
    • Operators
    • Security
    • Volumes
    • Networking
    • kube-controller-manager
      • Node crash recovery
    • Known errors and solutions
  • Lumen
    • Cheat Sheet
  • MACOS
    • Cheat Sheet
    • Qemu
    • Zsh
  • microk8s
    • Cheat Sheet
  • MongoDB
    • Cheat Sheet
    • Tools
  • MySQL
    • Cheat Sheet
  • Network
    • Tools
    • WDS - Wireless Distribution System
    • Expose server under NAT
  • nvim
    • Cheat Sheet
  • Openvpn
    • OpenVPN server on Ubuntu 18.06
    • Stunnel
  • PHP
    • Composer
  • Prometheus
    • Tools
  • Python
    • Cheat Sheet
    • Pydantic
    • Tools / Modules
    • Virtualenv
  • RabbitMQ
    • Cheat Sheet
  • ReactJS
    • Fixes
    • For beginners
  • Ruby
    • Cheat Sheet
    • Rails
    • rvm
  • Rundeck
    • Cheat Sheet
  • Rust
    • Cheat Sheet
  • Squid
    • Setup server
    • Expose NAT server
  • SRE
    • Cheat Sheet
  • SSH
    • Passwordless auth with RSA key
    • Reverse tunnel
    • Cheat Sheet
  • SSL
    • Cheat Sheet
    • certbot
  • STACK SETUP
    • Using EKS and Gitlab CI to deploy applications
  • Terraform
    • Cheat Sheet
    • Tools
  • Tmux
    • Cheat Sheet
  • Tor
    • Cheat Sheet
  • Ubuntu
    • Cheat Sheet
  • Vagrant
    • Cheat Sheet
  • VirtualBox
    • Cheat Sheet
  • Windows
    • Windows Docker
    • Fingerprint
    • SSH Client
    • Tools
    • VirtualBox
    • WSL
Powered by GitBook
On this page
  • Nginx ingress
  • 308 redirect to HTTPS
  • Duplicate location "/healthz"
  • Pod sandbox changed, it will be killed and re-created
  • Cause
  • Solution
  • References
  • failed to watch file "/var/log/pods/6438eb52-202a-11ea-8dce-e279cb2777e2/my-app/0.log": no space left on device
  • Symptom
  • Cause
  • Solution
  • References
  1. Kubernetes

Known errors and solutions

PreviousNode crash recoveryNextCheat Sheet

Last updated 5 years ago

Nginx ingress

308 redirect to HTTPS

Solution

Edit ingress configmap.

kubectl -n ingress-external edit configmaps ingress-controller-leader-nginx

Add the following annotations:

use-forwarded-headers: "true"
force-ssl-redirect: "false"

Restart controller.

kubectl -n ingress-external scale deployment --replicas=0 ingress-external-nginx-ingress-controller
kubectl -n ingress-external scale deployment --replicas=2 ingress-external-nginx-ingress-controller

References

Duplicate location "/healthz"

Complete nginx pod log:

2020/01/27 17:12:12 [emerg] 105#105: duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
nginx: [emerg] duplicate location "/healthz" in /tmp/nginx-cfg568474076:487
nginx: configuration file /tmp/nginx-cfg568474076 test failed

It happens when you have an ingress object conflicting with "/healthz" path.

Solution

Make sure to not have an ingress object overlapping "/healthz".

Pod sandbox changed, it will be killed and re-created

Cause

This error happens when deploying a pod. It is caused most liked because of Docker processes crashed or is unstable on the node due IO peak.

Solution

The solution is to reboot the node.

References

failed to watch file "/var/log/pods/6438eb52-202a-11ea-8dce-e279cb2777e2/my-app/0.log": no space left on device

Symptom

When you run.

kubectl -n my-ns logs -f my-app-659858b967-5hmtz

The command outputs a few lines of log and then breaks.

Cause

The obvious reason is the node's HD is full. Although this error can be caused by other reasons.

This error (ENOSPC) comes from the inotify_add_watch syscall, and actually has multiple meanings (the message comes from golang). Most likely the problem is from exceeding the maximum number of watches, not filling the disk. This can be increased with the fs.inotify.max_user_watches sysctl.

Solution

Increase max_user_watches.

cat /proc/sys/fs/inotify/max_user_watches # default is 8192 
sysctl fs.inotify.max_user_watches=1048576 # increase to 1048576

If you do not have SSH connection to the node, apply the following manifest (not recommended for production environments).

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: more-fs-watchers
  namespace: kube-system
  labels:
    app: more-fs-watchers
spec:
  template:
    metadata:
      labels:
        name: more-fs-watchers
    spec:
      hostNetwork: true
      hostPID: true
      hostIPC: true
      initContainers:
        - command:
            - sh
            - -c
            - sysctl -w fs.inotify.max_user_watches=524288;
          image: alpine:3.6
          imagePullPolicy: IfNotPresent
          name: sysctl
          resources: {}
          securityContext:
            privileged: true
          volumeMounts:
            - name: sys
              mountPath: /sys
      containers:
        - resources:
            requests:
              cpu: 0.01
          image: alpine:3.6
          name: sleepforever
          command: ["tail"]
          args: ["-f", "/dev/null"]
      volumes:
        - name: sys
          hostPath:
            path: /sys

References

https://github.com/kubernetes/ingress-nginx/issues/1957
https://plugaru.org/2018/05/21/pod-sandbox-changed-it-will-be-killed-and-re-created/
https://github.com/google/cadvisor/issues/1581
https://github.com/Azure/AKS/issues/772