Argo Workflows 部署

发布时间： 更新时间： 总字数：1514 阅读时间：4m 作者： IP上海 分享 网址

Argo Workflows 部署

安装

kubectl apply -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.2/quick-start-mysql.yaml

服务情况

argo-servers ↕
root@ubuntu:~# kubectl -n argo get all
NAME                                      READY   STATUS    RESTARTS   AGE
pod/argo-server-64f8df864d-t4ntp          1/1     Running   0          3m40s
pod/httpbin-6979fdc65b-sxnt2              1/1     Running   0          3m40s
pod/minio-5c8f5fb598-p7nkz                1/1     Running   0          3m39s
pod/mysql-7c888c6864-vx5fk                1/1     Running   0          3m38s
pod/workflow-controller-6579dd99d-cvxwk   1/1     Running   0          3m37s

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/argo-server   NodePort    10.245.204.179   <none>        2746:30786/TCP      3m45s
service/httpbin       ClusterIP   10.245.101.188   <none>        9100/TCP            3m44s
service/minio         ClusterIP   10.245.103.33    <none>        9000/TCP,9001/TCP   3m43s
service/mysql         ClusterIP   10.245.50.143    <none>        3306/TCP            3m43s

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argo-server           1/1     1            1           3m42s
deployment.apps/httpbin               1/1     1            1           3m42s
deployment.apps/minio                 1/1     1            1           3m41s
deployment.apps/mysql                 1/1     1            1           3m40s
deployment.apps/workflow-controller   1/1     1            1           3m39s

NAME                                            DESIRED   CURRENT   READY   AGE
replicaset.apps/argo-server-64f8df864d          1         1         1       3m42s
replicaset.apps/httpbin-6979fdc65b              1         1         1       3m42s
replicaset.apps/minio-5c8f5fb598                1         1         1       3m41s
replicaset.apps/mysql-7c888c6864                1         1         1       3m40s
replicaset.apps/workflow-controller-6579dd99d   1         1         1       3m39s

说明：

  • argo-server 是argo服务端
  • mino 是进行制品仓库，Argo minio 默认账号：admin/password
    • kubectl -n argo get secrets my-minio-cred -o yaml
  • mysql/postgres 是数据库
  • workflow-controller 是流程控制器

NodePort 暴露服务

kubectl -n argo patch svc argo-server -p '{"spec": {"type": "NodePort"}}'

安装 argo 客户端

  • 安装 argo cli 参考：https://github.com/argoproj/argo-workflows/releases/tag/v3.4.3
# Download the binary
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.4.3/argo-linux-amd64.gz

# Unzip
gunzip argo-linux-amd64.gz

# Make binary executable
chmod +x argo-linux-amd64

# Move binary to path
mv ./argo-linux-amd64 /usr/bin/argo

# Test installation
argo version
  • 命令
argo submit hello-world.yaml    # submit a workflow spec to Kubernetes
argo list                       # list current workflows
argo get hello-world-xxx        # get info about a specific workflow
argo logs hello-world-xxx       # print the logs from a workflow
argo delete hello-world-xxx     # delete workflow

获取 token

  • 获取认证 token 的方式，可以到 Web 登录框输入 Bearer Token 登录 argo workflows
# 方式一
$ argo auth token
Bearer dop_v1_xxx

# 方式二
$ kubectl -n argo exec -it argo-server-9cbd9b77c-5tkpm -- argo auth token
Bearer <jwt>

Argo Server 授权模式

argo server --auth-mode sso

configure

  • Workflow GC
apiVersion: v1
data:
  retentionPolicy: |
    completed: 3
    failed: 3
    errored: 3
kind: ConfigMap
metadata:
  name: argo-workflows-workflow-controller-configmap
  namespace: argo

apiVersion: v1
data:
  retentionPolicy: |
    completed: 3
    failed: 3
    errored: 3
  persistence: |
    archive: true
    archiveTTL: 180d
    postgresql:
      host: pg-aio-postgresql.default.svc.cluster.local
      port: 5432
      database: argo
      tableName: argo_workflows
      userNameSecret:
        name: argo-postgres-config
        key: username
      passwordSecret:
        name: argo-postgres-config
        key: password
kind: ConfigMap
metadata:
  name: argo-workflows-workflow-controller-configmap
  namespace: argo

DR

# Exporting example
kubectl get wf,cwf,cwft,wftmpl -A -o yaml > backup.yaml

# Importing example
kubectl apply -f backup.yaml

F&Q

failed to list resources

  • argo workflow 错误日志：
W0825 10:33:57.896940       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:argo:argo-server" cannot list resource "serviceaccounts" in API group "" at the cluster scope
E0825 10:33:57.896988       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.3/tools/cache/reflector.go:167: Failed to watch *v1.ServiceAccount: failed to list *v1.ServiceAccount: serviceaccounts is forbidden: User "system:serviceaccount:argo:argo-server" cannot list resource "serviceaccounts" in API group "" at the cluster scope

$ kubectl -n argo get ClusterRole | grep "argo-server"
argo-server-clusterworkflowtemplate-role                               2022-08-23T08:16:59Z

$ kubectl get ClusterRoleBinding | grep "argo-server"
argo-server-clusterworkflowtemplate-role-binding       ClusterRole/argo-server-clusterworkflowtemplate-role                               26h

$ kubectl -n argo get pod argo-server-7cc5d64869-jtwmw -o yaml
...
spec:
  containers:
  - name: argo-server
    ...
  serviceAccount: argo-server
  serviceAccountName: argo-server
  ...

$ kubectl -n argo get rolebinding
NAME                       ROLE                            AGE
...
argo-server-binding        Role/argo-server-role           41h
...

$ kubectl -n argo get rolebinding argo-server-binding -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{},"name":"argo-server-binding","namespace":"argo"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"Role","name":"argo-server-role"},"subjects":[{"kind":"ServiceAccount","name":"argo-server","namespace":"argo"}]}
  creationTimestamp: "2022-08-23T08:17:02Z"
  name: argo-server-binding
  namespace: argo
  resourceVersion: "3262386"
  uid: fb39f2f4-ec52-4005-a65d-33bdb8b9b3eb
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argo-server-role
subjects:
- kind: ServiceAccount
  name: argo-server
  namespace: argo

$ kubectl -n argo get role argo-server-role -o yaml
...
rules:
...
- apiGroups:
  - ""
  resources:
  - serviceaccounts
  verbs:
  - get
  - list
  - watch
...
  • 解决方式：该问题为启动参数的问题，修复方式如下
spec:
  containers:
  - args:
    - server
    - --namespaced
    - --auth-mode
    - server
    - --auth-mode
    - client
    image: quayioargoproj/argocli:v3.4.3
    imagePullPolicy: IfNotPresent
    name: argo-server
    ...
  • 没有权限问题，先创建名字为 admin-user 的 ServiceAccount
$ cat admin-user-sa.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user

$ kubectl -n argo apply -f admin-user-sa.yaml
  • 创建并执行 kubectl -n argo apply -f admin-user.yaml，参考
    • 若配置 workflows.argoproj.io/rbac-rule: "'admin' in groups" 表示用户具有 admin 组有权限
    • 若配置 workflows.argoproj.io/rbac-rule: "true"，表示所有用户均为 admin
$ cat admin-user.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  annotations:
    # The rule is an expression used to determine if this service account
    # should be used.
    # * `groups` - an array of the OIDC groups
    # * `iss` - the issuer ("argo-server")
    # * `sub` - the subject (typically the username)
    # Must evaluate to a boolean.
    # If you want an account to be the default to use, this rule can be "true".
    # Details of the expression language are available in
    # https://github.com/expr-lang/expr/blob/master/docs/language-definition.md.
    workflows.argoproj.io/rbac-rule: "'admin' in groups"
    # The precedence is used to determine which service account to use whe
    # Precedence is an integer. It may be negative. If omitted, it defaults to "0".
    # Numerically higher values have higher precedence (not lower, which maybe
    # counter-intuitive to you).
    # If two rules match and have the same precedence, then which one used will
    # be arbitrary.
    workflows.argoproj.io/rbac-rule-precedence: "1"
  • readonly 用户组
metadata:
  name: read-only
  annotations:
    workflows.argoproj.io/rbac-rule: "true"
    workflows.argoproj.io/rbac-rule-precedence: "0"
  • 绑定管理员权限，这个为所有登录的用户配置管理员权限
    • cluster-admin 为默认的全局 clusterrole
$ kubectl create clusterrolebinding argo-admin-login-user --clusterrole=cluster-admin --serviceaccount=argo:admin-user
clusterrolebinding.rbac.authorization.k8s.io/argo-admin-login-user created
  • 测试，查看 userinfo
$ curl https://argo.kb.cx:30443/api/v1/userinfo
{
	"issuer": "argo-server",
	"subject": "ChhDZ2cxTURReE5UTTNOUklHWjJsMGFIVmkSA2RleA",
	"groups": ["beehat:test"],
	"email": "6@166.xyz",
	"emailVerified": true,
	"serviceAccountName": "admin-user",
	"serviceAccountNamespace": "argo"
}

  • 当前不支持casbin，支持的参考：https://github.com/argoproj/argo-workflows/issues/6490

  • Argo Workflows 依赖于 k8s 的原生的 RBAC 实现权限模块，参考

  • 验证权限

# 检测是否可以修改configmap
kubectl auth can-i update configmaps
kubectl auth can-i --help

User “system:serviceaccount:argo:argo” cannot list resource “workflows” in API group “argoproj.io” at the cluster scope

  • 错误日志
time="" level=warning msg="finished unary call with code PermissionDenied" error="rpc error: code = PermissionDenied desc = workflows.argoproj.io is forbidden: User \"system:serviceaccount:argo:argo\" cannot list resource \"workflows\" in API group \"argoproj.io\" at the cluster scope" grpc.code=PermissionDenied grpc.method=ListWorkflows grpc.service=workflow.WorkflowService grpc.start_time="" grpc.time_ms=2.28 span.kind=server system=grpc
  • 解决
$ $ kubectl get clusterrolebinding   argo-clusterworkflowtemplate-role-binding   -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: argo-clusterworkflowtemplate-role
subjects:
- kind: ServiceAccount
  name: argo
  namespace: argo

$ kubectl edit clusterrole argo-clusterworkflowtemplate-role
添加如下
  resources:
  - workflows

调试日志

argo -gloglevel 9

重启可以看到默认加载的 configmap 是 workflow-controller-configmap

Ixxx       1 round_trippers.go:553] GET https://10.245.0.1:443/api/v1/namespaces/argo/configmaps/workflow-controller-configmap 200 OK in 8 milliseconds

配置configmap加载

参考

最近更新
相关文章
最新评论
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数 人