文档简介:
可能原因
排查方法
检查 Pod 配置
检查 Volume 挂载情况
1. Pod 漂移导致未正常解挂磁盘
2. 命中 K8S 挂载 configmap/secret 时 subpath 的 bug
$ kubectl -n prod get pod -o yaml manage-5bd487cf9d-bqmvm...lastState: terminatedcontainerID: containerd://e6746201faa1dfe7f3251b8c30d59ebf613d99715f3b800740e587e681d2a903exitCode: 128finishedAt: 2019-09-15T00:47:22Zmessage: 'failed to create containerd task: OCI runtime create failed: container_linux.go:345:starting container process caused "process_linux.go:424: container initcaused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/211d53f4-d08c-11e9
-b0a7-b6655eaf02a6/volume-subpaths/manage-config-volume/manage/0\\\"
to rootfs \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/e6746201faa1
dfe7f3251b8c30d59ebf613d99715f3b800740e587e681d2a903/rootfs\\\"
at \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/e6746201faa1dfe7
f3251b8c30d59ebf613d99715f3b800740e587e681d2a903/rootfs/app/resources/application.properties\\\"
caused \\\"no such file or directory\\\"\"": unknown'
检查磁盘空间是否不足
Events:Type Reason Age From Message---- ------ ---- ---- -------Warning FailedCreatePodSandBox 2m (x4307 over 16h) kubelet, 10.179.80.31
(combined from similar events): Failed create pod sandbox: rpc error: code =
Unknown desc = failed to create a sandbox for pod "apigateway-6dc48bf8b6-l8xrw"
: Error response from daemon: mkdir /var/lib/docker/aufs/mnt/1f09d6c1c9f24e8daaea
5bf33a4230de7dbc758e3b22785e8ee21e3e3d921214-init: no space left on device
检查节点内存是否碎片化
检查 limit 设置
现象描述
Pod sandbox changed, it will be killed and re-created。
to start sandbox container for pod ... Error response from daemon: OCI runtime
create failed: container_linux.go:348: starting container process caused
"process_linux.go:301: running exec setns process for init caused \"signal: killed\"": unknown
解决思路
检查拉取镜像是否失败
检查 CNI 网络是否错误
检查 controller-manager 是否异常
检查节点已有 docker
yum install -y docker
Type Reason Age From Message---- ------ ---- ---- -------Warning FailedCreatePodSandBox 18m (x3583 over 83m) kubelet, 192.168.4.5 (combined
from similar events): Failed create pod sandbox: rpc error: code = Unknown desc =
failed to start sandbox container for pod "nginx-7db9fccd9b-2j6dh": Error response
from daemon: ttrpc: client shutting down: read unix @->@/containerd-shim/moby/de2b
feefc999af42783115acca62745e6798981dff75f4148fae8c086668f667/shim.sock:
read: connection reset by peer: unknown
Normal SandboxChanged 3m12s (x4420 over 83m) kubelet, 192.168.4.5 Pod sandbox changed, it will be killed and re-created.
检查是否存在同名容器
节点上存在同名容器会导致创建 sandbox 时失败,也会导致 Pod 一直处于 ContainerCreating 或 Waiting 状态。
执行 kubectl describe pod 命令,查看 event 报错信息如下:
Warning FailedCreatePodSandBox 2m kubelet, 10.205.8.91 Failed create pod sandbox:
rpc error: code = Unknown desc = failed to create a sandbox for pod "lomp-ext
-d8c8b8c46-4v8tl": operation timeout: context deadline exceeded
Warning FailedCreatePodSandBox 3s (x12 over 2m) kubelet, 10.205.8.91 Failed create pod
sandbox: rpc error: code = Unknown desc = failed to create a sandbox for pod
"lomp-ext-d8c8b8c46-4v8tl": Error response from daemon: Conflict. The container name
"/k8s_POD_lomp-ext-d8c8b8c46-4v8tl_default_65046a06-f795-11e9-9bb6-b67fb7a70bad_0" is already in use by container "30aa3f5847e0ce89e9d411e76783ba14accba7eb7743e605a10a9a862a72c1e2". You have to remove
(or rename) that container to be able to reuse that name.
请修改容器名,确保节点上不存在同名容器。