FAQ

FAQ

namespace

namespace 处于 Terminating 修复

• 方法零：k9s
• 方法一：其他的资源删除也适用

# delete ns
kubectl patch ns/abc-ns \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

# others
kubectl patch <some-resource>/<some-name> \
    --type json \
    --patch='[ { "op": "remove", "path": "/metadata/finalizers" } ]'

• 方法二

NAMESPACE_NAME="abc-ns"
kubectl get ns $NAMESPACE_NAME  -o json > $NAMESPACE_NAME.json

修改 $NAMESPACE_NAME.json

    "metadata": {
        "finalizers": [
            "finalizers.kubesphere.io/namespaces"
        ],
        ...
    }
    "spec": {
        "finalizers": [
            "kubernetes"
        ]
    },

更改为：

    "metadata": {
        "finalizers": [],
        ...
    },
    "spec": {},

API代理

# kubectl proxy --port=8081
Starting to serve on 127.0.0.1:8081

• 删除

curl -k -H "Content-Type:application/json" -X PUT --data-binary @$NAMESPACE_NAME.json http://127.0.0.1:8081/api/v1/namespaces/$NAMESPACE_NAME/finalize

kubelet

Failed to get system container stats

• 错误日志

Mar 12 09:37:12 [localhost] kubelet[1725]: E0312 09:37:12.570955    1725 summary_sys_containers.go:47] Failed to get system container stats for "/system.slice/docker.service": failed to get cgroup stats for "/system.slice/docker.service": failed to get container info for "/system.slice/docker.service": unknown container "/system.slice/docker.service"

• 解决方法

编辑/etc/systemd/system/kubelet.service文件，添加：

[Service]
CPUAccounting=true
MemoryAccounting=true

• 执行

systemctl daemon-reload
systemctl restart kubelet

spec.selector: Invalid value: v1.LabelSelector field is immutable

API apps/v1 中 Deployment 标签选择器创建后，不可变，删除重建资源即可。

Pod

Pod 无法通过 Service IP 连接到它本身

如果 Service 的末端尝试访问自己的 Service VIP，则该端点可以把流量负载均衡回来到它们自身。 hairpin-mode（发夹模式） 标志必须被设置为 hairpin-veth 或者 promiscuous-bridge。

若为 hairpin-veth 模式，通过如下方式检测：

$ cat /sys/devices/virtual/net/cni0/brif/veth80c6661f/hairpin_mode
1

若为 promiscuous-bridge 模式，通过如下方式检测：

ip a show cni0 | grep PROMISC

orphaned pod pod_id found

孤儿 pod 删除残留问题，错误日志

orphaned pod pod_id found, but error not a directory occurred when trying to remove the volumes dir

• 解决方法

cd /var/lib/kubelet/pods/${pod_id}/volumes/kubernetes.io~csi/pvc-${pvc_id}/
rm vol_data.json

Pod 一直处于 terminating

• 参考

# 方法一
kubectl -n [namespace] delete pod [pod name] --force --grace-period=0

# 方法二：etcd 删除
export ETCDCTL_API=3
etcdctl del /registry/pods/<default>/<pod-to-be-deleted-0>

# 方法三
kubectl get pod <pod_name> -n <namespace> -o yaml > pod.yaml
# 删除 yaml 中的如下内容
metadata:
  ...
  finalizers:
  - kubernetes

kubectl delete pod <pod_name> -n <namespace>
kubectl apply -f pod.yaml

# 方法四
kubectl patch pod <pod_name> -n <namespace> -p '{"metadata":{"finalizers":null}}'

volume not registered

k8s 错误事件 event

MountVolume.SetUp failed for volume ... not registered

原因分析：

1. 调用 kube-apiserver 异常导致，需要排查 kube-apiserver 服务的日志等、kube-apiserver-xxx 重启
2. 可能因为 kube-apiserver 被限速，可自动通过重试恢复，参数如下：

--kube-api-burst int32   Burst to use while talking with kubernetes apiserver. Doesn't cover events and node heartbeat apis which rate limiting is   controlled by a different set of flags (default 10)
--kube-api-qps int32     QPS to use while talking with kubernetes apiserver. Doesn't cover events and node heartbeat apis which rate limiting is controlled by a different set of flags (default 5)