etcd gateway、gRPC-Gateway、gRPC Proxy 介绍

发布时间： 更新时间： 总字数：1814 阅读时间：4m 作者： IP上海 分享 网址

etcd gateway 介绍

etcd 网关

  • etcd 网关是一个简单的 TCP 代理，可将网络数据转发到 etcd 集群
  • 网关是无状态且透明的，它既不会检查客户端请求，也不会干扰集群响应，支持多个 etcd 服务器实例，并采用简单的循环策略
  • etcd 网关位于 etcd 集群和应用程序之间，使客户端应用不会感知 etcd 集群实例的变化；发生 etcd 集群实例的变更时，只需要更新其端点接口
  • 作用
    • 性能提示：etcd 网关不是为提高 etcd 集群性能设计的
    • 在集群上运行关系系统
    • 为自动传播集群端点更改

启动服务

启动参数

$ etcd gateway start --help
start the gateway

Usage:
  etcd gateway start [flags]

Flags:
      --discovery-srv string        DNS domain used to bootstrap initial cluster
      --discovery-srv-name string   service name to query when using DNS discovery
      --endpoints strings           comma separated etcd cluster endpoints (default [127.0.0.1:2379])
  -h, --help                        help for start
      --insecure-discovery          accept insecure SRV records
      --listen-addr string          listen address (default "127.0.0.1:23790") 默认监听端口
      --retry-delay duration        duration of delay before retrying failed endpoints (default 1m0s)  重试连接到失败的端点延迟时间，默认为 1m0s
      --trusted-ca-file string      path to the client server TLS CA file for verifying the discovered endpoints when discovery-srv is provided.

$ etcd gateway start --endpoints=http://172.20.0.2:2379,http://172.20.0.3:2379,http://172.20.0.3:2379
{"level":"info","ts":"2022-10-02T09:50:44.185929Z","caller":"etcdmain/gateway.go:103","msg":"Running: ","args":["etcd","gateway","start","--endpoints=http://172.20.0.2:2379,http://172.20.0.3:2379,http://172.20.0.3:2379"]}
{"level":"info","ts":"2022-10-02T09:50:44.186437Z","caller":"etcdmain/main.go:44","msg":"notifying init daemon"}
{"level":"info","ts":"2022-10-02T09:50:44.18651Z","caller":"etcdmain/main.go:50","msg":"successfully notified init daemon"}
{"level":"info","ts":"2022-10-02T09:50:44.186544Z","caller":"tcpproxy/userspace.go:87","msg":"ready to proxy client requests","endpoints":["172.20.0.2:2379","172.20.0.3:2379","172.20.0.3:2379"]}
  • etcd v3 使用 gRPC 作为消息传输协议，包括
    • Go client
    • 命令行 etcdctl 客户端通过 gRPC 框架与 etcd 集群通讯

gRPC-Gateway

  • gRPC-Gateway 是对于 etcd 的 gRPC 通信协议的补充
    • 有些语言的客户端不支持 gRPC 通信协议，此时可以使用 gRPC-Gateway 对外提供 HTTP API 接口，通过 HTTP 请求，实现与 gRPC 调用协议相同的功能

HTTP 简单使用

# put 写
$ curl -L http://localhost:2379/v3/kv/put -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"2","raft_term":"5"}}

# range 读
$ curl -L http://localhost:2379/v3/kv/range -X POST -d '{"key": "Zm9v"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"2","raft_term":"5"},"kvs":[{"key":"Zm9v","create_revision":"2","mod_revision":"2","version":"1","value":"YmFy"}],"count":"1"}

# watch HTTP 请求客户端与 etcd 服务端建立长连接，当监听的键值对发生变化时，会将监听事件通知给客户端
$ curl -N http://localhost:2379/v3/watch -X POST -d '{"create_request": {"key": "Zm9v"}}'
{"result":{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"3","raft_term":"5"},"created":true}}

etcd 事务的实现

  • 事务用于完成一组操作，通过对比指定的条件，成功的情况下执行相应的操作，否则回滚
$ curl -L http://localhost:2379/v3/kv/txn -X POST -d '{"compare": [{"target": "CREATE", "key": "Zm9v", "createRevision": "4"}], "success": [{"requestPut": {"key": "Zm9v", "value": "YmFy"}}]}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"}

###s 基于角色的授权

HTTP 方式访问 etcd 服务端，需要考虑安全问题，gRPC-Gateway 中提供的 API 接口支持开发安全认证

# 创建用户
$ curl -L http://localhost:2379/v3/auth/user/add -X POST -d '{"name": "root", "password": "123456"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"}}

# 创建角色
$ curl -L http://localhost:2379/v3/auth/role/add -X POST -d '{"name": "root"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"}}

# 用户授权角色
$ curl -L http://localhost:2379/v3/auth/user/grant -X POST -d '{"user": "root", "role": "root"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"}}

# 开启认证
$ curl -L http://localhost:2379/v3/auth/enable -X POST -d '{}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"}}

# 获取 token
$ curl -L http://localhost:2379/v3/auth/authenticate -X POST -d '{"name": "root", "password": "123456"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"4","raft_term":"5"},"token":"mbgsVGCWITJmJvmo.22"}

# 使用 token 更新数据
$ curl -L http://localhost:2379/v3/kv/put -H "Authorization: mbgsVGCWITJmJvmo.22" -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
{"header":{"cluster_id":"10316109323310759371","member_id":"12530464223947363063","revision":"5","raft_term":"5"}}

$ 错误 token 示例
$ curl -L http://localhost:2379/v3/kv/put -H "Authorization: xxx" -X POST -d '{"key": "Zm9v", "value": "YmFy"}'
{"error":"etcdserver: invalid auth token","code":16,"message":"etcdserver: invalid auth token"}

gRPC Proxy

  • gRPC 代理模式：实现可伸缩的 etcd API，它是在 gRPC 层（L7）运行的无状态 etcd 反向代理，旨在减少核心 etcd 集群上的总处理负载

启动服务

etcd grpc-proxy start --help ...
$ etcd grpc-proxy start --help
start the grpc proxy

Usage:
  etcd grpc-proxy start [flags]

Flags:
      --advertise-client-url string             advertise address to register (must be reachable by client) (default "127.0.0.1:23790")
      --auto-tls                                proxy TLS using generated certificates
      --cacert string                           verify certificates of TLS-enabled secure etcd servers using this CA bundle
      --cert string                             identify secure connections with etcd servers using this TLS certificate file
      --cert-file string                        identify secure connections to the proxy using this TLS certificate file
      --client-crl-file string                  proxy client certificate revocation list file.
      --data-dir string                         Data directory for persistent data (default "default.proxy")
      --debug                                   Enable debug-level logging for grpc-proxy.
      --dial-keepalive-time duration            keepalive time for client(grpc-proxy) connections (default 0, disable).
      --dial-keepalive-timeout duration         keepalive timeout for client(grpc-proxy) connections (default 20s). (default 20s)
      --discovery-srv string                    domain name to query for SRV records describing cluster endpoints
      --discovery-srv-name string               service name to query when using DNS discovery
      --enable-pprof                            Enable runtime profiling data via HTTP server. Address is at client URL + "/debug/pprof/"
      --endpoints strings                       comma separated etcd cluster endpoints (default [127.0.0.1:2379])
      --endpoints-auto-sync-interval duration   etcd endpoints auto sync interval (disabled by default)
      --experimental-leasing-prefix string      leasing metadata prefix for disconnected linearized reads.
      --experimental-serializable-ordering      Ensure serializable reads have monotonically increasing store revisions across endpoints.
      --grpc-keepalive-interval duration        Frequency duration of server-to-client ping to check if a connection is alive (0 to disable). (default 2h0m0s)
      --grpc-keepalive-min-time duration        Minimum interval duration that a client should wait before pinging proxy. (default 5s)
      --grpc-keepalive-timeout duration         Additional duration of wait before closing a non-responsive connection (0 to disable). (default 20s)
  -h, --help                                    help for start
      --insecure-discovery                      accept insecure SRV records
      --insecure-skip-tls-verify                skip authentication of etcd server TLS certificates (CAUTION: this option should be enabled only for testing purposes)
      --key string                              identify secure connections with etcd servers using this TLS key file
      --key-file string                         identify secure connections to the proxy using this TLS key file
      --listen-addr string                      listen address (default "127.0.0.1:23790")  默认监听端口
      --listen-cipher-suites strings            Comma-separated list of supported TLS cipher suites between client/proxy (empty will be auto-populated by Go).
      --max-concurrent-streams uint32           Maximum concurrent streams that each client can open at a time. (default 4294967295)
      --max-recv-bytes int                      message receive limits in bytes (default value is math.MaxInt32) (default 2147483647)
      --max-send-bytes int                      message send limits in bytes (default value is 1.5 MiB) (default 1572864)
      --metrics-addr string                     listen for endpoint /metrics requests on an additional interface
      --namespace string                        string to prefix to all keys for namespacing requests
      --permit-without-stream                   Enable client(grpc-proxy) to send keepalive pings even with no active RPCs.
      --resolver-prefix string                  prefix to use for registering proxy (must be shared with other grpc-proxy members)
      --resolver-ttl int                        specify TTL, in seconds, when registering proxy endpoints
      --self-signed-cert-validity uint          The validity period of the proxy certificates, unit is year (default 1)
      --trusted-ca-file string                  verify certificates of TLS-enabled secure proxy using this CA bundle
  • 示例，--endpoints 指定集群的静态成员
$ etcd grpc-proxy start --endpoints=http://172.20.0.2:2379,http://172.20.0.3:2379,http://172.20.0.3:2379
...
{"level":"info","ts":"2024-06-30T10:51:53.632251Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[roundrobin] roundrobinPicker: Build called with info: {map[SubConn(id:5):{{Addr: \"127.0.0.1:23790\", ServerName: \"127.0.0.1:23790\", }}]}"}
{"level":"info","ts":"2024-06-30T10:51:53.632282Z","caller":"zapgrpc/zapgrpc.go:174","msg":"[core] [Channel #4] Channel Connectivity change to READY"}
  • 使用
$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:23790 put foo bar

$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:23790 get foo
  • 客户端端点同步
# 不指定 --resolver-prefix 时，获取成员列表只能显示当前的节点信息
$ etcd grpc-proxy start --endpoints=http://172.20.0.2:2379 --listen-addr 127.0.0.1:23790 --advertise-client-url 127.0.0.1:23790 --resolver-prefix="__grpc_proxy_endpoint" --resolver-ttl=60

$ etcdctl --endpoints=127.0.0.1:23790 member list --write-out table

# 上述方式，支持golang lib通过 Sync 方法自动发现代理的端点

可伸缩的 watch API

  • 如果客户端监听同一键或某一范围内的键，gRPC-Proxy 代理可以将这些客户端监视程序(c-watcher)合并为连接到 etcd 服务器的单个监听程序(s-watcher)
  • 当 watch 事件发生时，代理讲所有事件从 s-watcher 广播到其 c-watcher

可伸缩的 rease API

  • 如果 etcd 工作负载涉及很多的客户端租约活动，这些流可能导致 CPU 使用率过高，为减少核心集群上流的总数，gRPC Proxy支持将 lease 流合并
    • 该功能可以保护 etcd 服务器免遭恶意 for 循环中滥用客户端的工具

命名空间的实现

  • 当应用程序期望对整个键空间有完全控制，etcd 集群与其他应用程序共享的情况下，为了使所有应用程序都不会相互干扰地运行，Proxy 可以对 etcd 键空间(namespace)进行分区，以便客户端大概率访问完整的键空间
  • 启动 proxy 时，提供 --namespace 参数，所有进入该 Proxy 的客户端请求都将转化为在键上具有用户定义的前缀

其他

  • /health 、:6633/metrics 健康和 prometheus 检测接口
最近更新
相关文章
最新评论
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数 人