Monitoring
K8s monitoring.
Clone metrics-server.
git clone https://github.com/kubernetes-incubator/metrics-server.git
cd metrics-server
Edit resource-reader.yaml.
nano deploy/1.8+/resource-reader.yaml
Edit the resources section as follows:
...
resources:
- pods
- nodes
- namespaces
- nodes/stats
...
Edit metrics-server-deployment.yaml
nano deploy/1.8+/metrics-server-deployment.yaml
Edit as follows:
...
containers:
- name: metrics-server
image: k8s.gcr.io/metrics-server-amd64:v0.3.3
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
imagePullPolicy: Always
...
Deploy it.
kubectl apply -f deploy/1.8+/
Wait a few minutes and run:
kubectl top node
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes" |jq
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/YOUR-NAMESPACE/pods" |jq
docker run \
-tid \
--name=rancher \
--restart=unless-stopped \
-p 80:80 -p 443:443 \
rancher/rancher:latest
Add a cluster and run on you cluster the manifest it generates.
SSH to your master node.
Create a policy file:
mkdir /etc/kubernetes/policies
nano /etc/kubernetes/policies/audit-policy.yaml
Paste:
# Log all requests at the Metadata level.
apiVersion: audit.k8s.io/v1
kind: Policy
rules:
- level: Metadata
Edit K8s API server config file:
nano /etc/kubernetes/manifests/kube-apiserver.yaml
Add:
...
spec:
containers:
- command:
- kube-apiserver
...
- --audit-policy-file=/etc/kubernetes/policies/audit-policy.yaml
- --audit-log-path=/var/log/apiserver/audit.log
- --audit-log-format=json
...
volumeMounts:
...
- mountPath: /etc/kubernetes/policies
name: policies
readOnly: true
...
volumes:
...
- hostPath:
path: /etc/kubernetes/policies
type: DirectoryOrCreate
name: policies
Restart kubelet:
systemctl restart kubelet
If the changes did not take effect, stop the API server docker container (it will be started automatically):
docker stop $(docker ps | grep "k8s_kube-apiserver_kube-apiserver-k8smaster_kube-system" | awk '{print $1}')
Tail the log file:
docker exec -it $(docker ps |grep "k8s_kube-apiserver_kube-apiserver-k8smaster_kube-system" | awk '{print $1}') tail -f /var/log/apiserver/audit.log
kubectl create namespace monitoring
nano prometheus.yml
Paste:
global:
scrape_interval: 15s
external_labels:
monitor: 'codelab-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
Prometheus config file example: https://github.com/prometheus/prometheus/blob/master/docs/getting_started.md
Create a ConfigMap from the config file:
kubectl -n monitoring create configmap cm-prometheus --from-file prometheus.yml
Edit the file:
nano prometheus.yml
Update the ConfigMap:
kubectl -n monitoring \
create configmap cm-prometheus \
--from-file=prometheus.yml \
-o yaml --dry-run | kubectl apply -f -
Now we need to roll out the new ConfigMap. By the time of this writing (2019-02-15), this subjects seems to be a little tricky. Please find some options bellow:
Roll out ConfigMap: option 1 - scale deployment
This is the only way that will "always" work, although there will be a few seconds of downtime:
kubectl -n monitoring scale deployment/prometheus --replicas=0
kubectl -n monitoring scale deployment/prometheus --replicas=1
Roll out ConfigMap: option 2 - patch the deployment
kubectl -n monitoring \
patch deployment prometheus \
-p '{"spec":{"template":{"metadata":{"labels":{"date":"2019-02-15"}}}}}'
Roll out ConfigMap: option 3 - create a new ConfigMap
Create a new ConfigMap:
kubectl -n monitoring \
create configmap cm-prometheus-new \
--from-file=prometheus.yml \
-o yaml --dry-run | kubectl apply -f -
Edit the deployment:
export EDITOR=nano
kubectl -n monitoring edit deployments prometheus
Edit
volumes.configMap.name
and use cm-prometheus-new
. The change will force K8s to create new pods with the new config.If by any reason you deployed Prometheus with
hostNetwork: true
, options 2 and 3 will return this error:0/2 nodes are available: 1 node(s) didn't have free ports for the requested pod ports, 1 node(s) didn't match node selector.
In this case, use option 1.
If you need more info regarding rolling out ConfigMaps, please refer to: https://stackoverflow.com/questions/37317003/restart-pods-when-configmap-updates-in-kubernetes