使用 kaniko 在 Kubernetes 中构建 Docker 镜像-谢先斌的博客

使用 kaniko 在 Kubernetes 中构建 Docker 镜像

发布时间： 2022-08-16 更新时间： 2023-08-27 总字数：948 阅读时间：2m 作者：谢先斌 IP上海

kaniko是Google开源的一款在 Kubernetes 用来构建容器镜像的工具，kaniko不依赖于Docker daemon进程，完全在用户空间根据Dockerfile构建镜像。

介绍

kaniko以容器镜像的方式运行，特点：

不依赖 Docker daemon 进程，构建容器镜像的工具
用户空间（非特权模式）构建镜像，不需要 root 权限
可复现的容器镜像构建

使用场景：

kubernetes 集群上构建镜像
jenkins pipeline 构建镜像
与 Tekton 结合实现在K8s中制作镜像，参考中的链接(Tekton 使用示例)

Docker 使用

docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcriokaniko/executor:latest-1317 --dockerfile=Dockerfile --destination=yourimagename

kubernetes 使用

对接 docker 参考 https://github.com/GoogleContainerTools/kaniko/blob/main/README.md#pushing-to-docker-hub

kubectl create secret generic kaniko-secret --from-file=<path to kaniko-secret.json>
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcriokaniko/executor:latest-1317  # gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=<path to Dockerfile within the build context>"
    - "--context=gs://<GCS bucket>/<path to .tar.gz>"
    - "--destination=<gcr.io/$PROJECT/$IMAGE:$TAG>"
    volumeMounts:
    - name: kaniko-secret
      mountPath: /secret
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
      value: /secret/kaniko-secret.json
  restartPolicy: Never
  volumes:
  - name: kaniko-secret
    secret:
      secretName: kaniko-secret

kaniko 镜像在 gcr.io 上，可以在 https://mirrors.kb.cx/?s=kaniko-project 查看对应同步的镜像。

Docker Cache 示例

准备

模拟 Docker file

mkdir code
cat << code/Dockerfile > EOF
FROM alpine:latest

RUN apk add curl
RUN apk add wget
EOF

帮助命令

docker run -it --rm gcriokaniko/executor:v1.9.1 --help

制作镜像示例

docker run \
  -v "$HOME"/.docker/config.json:/kaniko/.docker/config.json \
  -v $(pwd)/code:/workspace \
  gcriokaniko/executor:v1.9.1 \
  --verbosity debug \
  --dockerfile /workspace/Dockerfile \
  --destination "harbor.xiexianbin.cn/xiexianbin/test-cache:1" \
  --context dir:///workspace/ \
  --insecure-registry harbor.xiexianbin.cn \
  --skip-tls-verify \
  --skip-tls-verify-pull \
  --cache=true \
  --cache-run-layers \
  --cache-ttl 10m
#   --cache-ttl default is 336h0m0s
#   --cache-repo "harbor.xiexianbin.cn/xiexianbin/test-cache/cache"

说明：

--context 类似于 docker build 的 context，如 dir:///workspace/

Kaniko cache 说明

基于 harbor 仓库

Kaniko 会根据每个 Dockerfile 中的 RUN 命令生成一个镜像，并立即提交到 docker hub 中，下次在使用时，会检测 Docker hub 是否有该镜像，若存在，直接pull并使用。该过程类似于 docker build 的 cache 过程
每一个 RUN 生成一个镜像（即 Docker 中的层）
build 日志如下：

# 检测是否存在 cache
# DEBU[0011] Optimize: cache key for command RUN apk add curl 5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3
# INFO[0011] Checking for cached layer harbor.xiexianbin.cn/xiexianbin/test-cache/cache:5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3...
# INFO[0012] No cached layer found for cmd RUN apk add curl
...
# INFO[0030] Pushing layer harbor.xiexianbin.cn/xiexianbin/test-cache/cache:5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3 to cache now
# INFO[0031] Pushing image to xiexianbin/test-cache/cache:5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3
# INFO[0033] Pushed xiexianbin/test-cache/cache@sha256:5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3

下次使用

# INFO[0004] Checking for cached layer harbor.xiexianbin.cn/xiexianbin/test-cache/cache:5ee6afd6e969b6190a31473c5c6afbf2f164e1fb3af30e17ba7d00d2721107b3...
# INFO[0006] Using caching version of cmd: RUN apk add curl

基于 cache dir

该功能用来缓存基础镜像到指定的目录，参考

docker run gcr.io/kaniko-project/warmer:latest --cache-dir=/cache --image alpine:3.8

Before

[36mkaniko-issue |[0m [37mDEBU[0m[0000] Built stage name to index map: map[]
[36mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image manifest alpine:3.8
[36mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image alpine:3.8
[36mkaniko-issue |[0m [37mDEBU[0m[0000] No file found for cache key sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4 stat /cache/sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4: no such file or directory
[36mkaniko-issue |[0m [37mDEBU[0m[0000] Image alpine:3.8 not found in cache
[36mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image manifest alpine:3.8
[36mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image alpine:3.8
[36mkaniko-issue |[0m [36mINFO[0m[0001] Built cross stage deps: map[]
[36mkaniko-issue |[0m [36mINFO[0m[0001] Retrieving image manifest alpine:3.8
[36mkaniko-issue |[0m [36mINFO[0m[0001] Retrieving image alpine:3.8

After

[33mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image manifest alpine:3.8
[33mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image alpine:3.8
[36mkaniko-cache |[0m [36mINFO[0m[0000] Found sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4 in local cache
[36mkaniko-cache |[0m [36mINFO[0m[0000] Found manifest at /cache/sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4.json
[33mkaniko-issue |[0m [36mINFO[0m[0000] Found sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4 in local cache
[33mkaniko-issue |[0m [36mINFO[0m[0000] Found manifest at /cache/sha256:954b378c375d852eb3c63ab88978f640b4348b01c1b3456a024a81536dafbbf4.json
[33mkaniko-issue |[0m [36mINFO[0m[0000] Built cross stage deps: map[]
[33mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image manifest alpine:3.8
[33mkaniko-issue |[0m [36mINFO[0m[0000] Retrieving image alpine:3.8

build-args

参考 docker build-args

kaniko 通过 ENV 传递 build-arg 到 Dockerfile 中，参考

    # argo-workflows
    image: gcr.io/kaniko-project/executor:debug
    env:
    - name: SOME_ENV
      value: "xxx"
    - name: http_proxy
      value: "xxx"
    command:
    - executor
    args:
    - --build-arg=SOME_ENV
    - --build-arg=http_proxy
    ...

其他

基于 PVC，如 java 的 .m2 文件夹或 nodejs 的 node_modules 文件夹可以通过 PVC 缓存

F&Q

Debug

docker run -it --entrypoint=/busybox/sh gcr.io/kaniko-project/executor:debug

Taking snapshot of full filesystem…

日志报错误 Taking snapshot of full filesystem...，可以尝试添加参数（参考）：

--compressed-caching=false

kaniko 不支持 Docker 中存在正则表达式

参考

配置 proxy

参考

        --build-arg "http_proxy=${http_proxy}"
        --build-arg "HTTP_PROXY=${http_proxy}"
        --build-arg "https_proxy=${https_proxy}"
        --build-arg "HTTPS_PROXY=${https_proxy}"

扩展

argo-workflows 使用 buildkit 参考