HPA 自动水平扩展

发布时间: 更新时间: 总字数:1288 阅读时间:3m 作者: IP属地: 分享 复制网址

本文介绍 Kubernetes Pod 自动水平扩展。

介绍

借助 Kubernetes 的 HPA(Horizontal Pod Autoscaler) 可以实现 Pod 的自动扩缩容,该部分实现依赖于 Kubernetes 监控介绍Prometheus 监控 Kubernetes

autoscale 支持如下 Pod 控制器:

  • deployment
  • replica set
  • stateful set
  • replication controller

支持的 HPA 版本

$ kubectl api-versions |grep autoscaling
autoscaling/v1
autoscaling/v2
autoscaling/v2beta1
autoscaling/v2beta2

帮助

kubectl explain hpa
kubectl explain hpa.spec
kubectl explain HorizontalPodAutoscaler.spec.metrics

scale 介绍

运维人员可以是使用 scale 控制 Pod 的规模

root@k8s-master:~# kubectl scale deployment my-nginx --replicas=0
deployment.apps/my-nginx scaled
root@k8s-master:~# kubectl scale deployment my-nginx --replicas=2
deployment.apps/my-nginx scaled
root@k8s-master:~# kubectl get pod -l run=my-nginx
NAME                       READY   STATUS    RESTARTS   AGE
my-nginx-cf54cdbf7-5k88z   1/1     Running   0          36s
my-nginx-cf54cdbf7-vsb42   1/1     Running   0          36s

HPA 示例

命令行创建自动伸缩

  • 启动一个 Deployment, demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: php-apache
  namespace: default
  labels:
    app: php-apache
spec:
  replicas: 2
  selector:
    matchLabels:
      app: php-apache
  template:
    metadata:
      name: php-apache
      labels:
        app: php-apache
    spec:
      containers:
      - name: php-apache
        image: gcmirrors/hpa-example
        ports:
        - name: http
          containerPort: 80
        resources:
          requests:
            memory: "128Mi"
            cpu: "50m"
          limits:
            memory: "128Mi"
            cpu: "50m"
---
apiVersion: v1
kind: Service
metadata:
  name: php-apache
  namespace: default
spec:
  selector:
    app: php-apache
  type: ClusterIP
  # type: NodePort
  ports:
  - name: port-80
    port: 80
    targetPort: 80
    protocol: TCP
  • 创建
$ kubectl apply -f demo.yaml
deployment.apps/php-apache created
service/php-apache created
  • 命令行创建 autoscale,最先副本为 1,最大副本为 5,CPU使用率大于 80% 时扩容
$ kubectl autoscale deployment php-apache --min=2 --max=5 --cpu-percent=80
horizontalpodautoscaler.autoscaling/php-apache autoscaled

# 查看 HPA
$ kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   2%/80%    2         5         2          22s

$ kubectl describe hpa php-apache
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 03 Apr 2022 12:45:52 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  0% (0) / 80%
Min replicas:                                          2
Max replicas:                                          5
Deployment pods:                                       2 current / 2 desired
Conditions:
  Type            Status  Reason               Message
  ----            ------  ------               -------
  AbleToScale     True    ScaleDownStabilized  recent recommendations were higher than current one, applying the highest recent recommendation
  ScalingActive   True    ValidMetricFound     the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange   the desired count is within the acceptable range
Events:           <none>
  • 压力测试

压力测试工具有很多(参考Http 性能相关测试工具汇总),本示例选择 wrk:

$ kubectl run wrk --rm=true -i --image=skandyla/wrk -- -t12 -c400 -d60s http://php-apache.default.svc.kb.cx

由于镜像资源限制很少,查看测试结果 Pod 数量有 2 扩展到 4 个:

root@k8s-node-1:~# kubectl get hpa
NAME         REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   96%/80%   2         5         4          44m
root@k8s-node-1:~# kubectl describe hpa php-apache
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name:                                                  php-apache
Namespace:                                             default
Labels:                                                <none>
Annotations:                                           <none>
CreationTimestamp:                                     Sun, 03 Apr 2022 12:45:52 +0800
Reference:                                             Deployment/php-apache
Metrics:                                               ( current / target )
  resource cpu on pods  (as a percentage of request):  96% (48m) / 80%
Min replicas:                                          2
Max replicas:                                          5
Deployment pods:                                       4 current / 4 desired
Conditions:
  Type            Status  Reason              Message
  ----            ------  ------              -------
  AbleToScale     True    ReadyForNewScale    recommended size matches current size
  ScalingActive   True    ValidMetricFound    the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  19h   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  19h   horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target

压测结束后,一段时间 Pod 自动收缩到 2:

$ kubectl describe hpa php-apache
...
  Type    Reason             Age    From                       Message
  ----    ------             ----   ----                       -------
  Normal  SuccessfulRescale  8m39s  horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  6m24s  horizontal-pod-autoscaler  New size: 4; reason: cpu resource utilization (percentage of request) above target
  Normal  SuccessfulRescale  8s     horizontal-pod-autoscaler  New size: 2; reason: All metrics below target

声明式创建自动伸缩

  • hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 2
  maxReplicas: 5
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80
  - type: Resource
    resource:
      name: memory
      target:
        type: Value
        averageValue: 40Mi
  • 操作
$ kubectl delete hpa php-apache
horizontalpodautoscaler.autoscaling "php-apache" deleted

$ kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/php-apache created
$ kubectl get hpa
NAME         REFERENCE               TARGETS                         MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/40Mi, <unknown>/80%   2         5         0          5s

# 压测

$ 结果
$ kubectl describe hpa
...
  ScalingLimited  False   DesiredWithinRange  the desired count is within the acceptable range
Events:
  Type    Reason             Age   From                       Message
  ----    ------             ----  ----                       -------
  Normal  SuccessfulRescale  10s   horizontal-pod-autoscaler  New size: 3; reason: cpu resource utilization (percentage of request) above target

其他指标

  • hpa-2.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 2
  maxReplicas: 5
  metrics:
  # - type: Resource
  #   resource:
  #     name: cpu
  #     target:
  #       type: Utilization
  #       averageUtilization: 80
  # - type: Resource
  #   resource:
  #     name: memory
  #     target:
  #       type: Value
  #       averageValue: 40Mi
  - type: Pods
    pods:
      metric:
        name: packets-per-second
      target:
        type: AverageValue
        averageValue: 1k
  - type: Object
    object:
      metric:
        name: requests-per-second
      describedObject:
        apiVersion: networking.k8s.io/v1
        kind: Ingress
        name: main-route
      target:
        type: Value
        value: 1k
  # - type: Object
  #   object:
  #     metric:
  #       name: http_requests
  #     target:
  #       type: Value
  #       value: 1k
  • 操作
$ kubectl delete -f hpa.yaml

$ kubectl apply -f hpa-2.yaml
horizontalpodautoscaler.autoscaling/php-apache created
$ kubectl get hpa
NAME         REFERENCE                 TARGETS                      MINPODS   MAXPODS   REPLICAS   AGE
php-apache   Deployment/php-apache   <unknown>/1k, <unknown>/1k   2         5         2          16s

$ 结果
$ kubectl describe hpa
...
Events:
  Type     Reason                        Age                  From                       Message
  ----     ------                        ----                 ----                       -------
  Warning  FailedGetObjectMetric         8m23s (x8 over 10m)  horizontal-pod-autoscaler  unable to get metric requests-per-second: Ingress on default main-route/unable to fetch metrics from custom metrics API: the server could not find the metric requests-per-second for ingresses.networking.k8s.io
  Warning  FailedComputeMetricsReplicas  8m23s (x8 over 10m)  horizontal-pod-autoscaler  invalid metrics (2 invalid out of 2), first error is: failed to get pods metric value: unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the metric packets-per-second for pods
  Warning  FailedGetPodsMetric           6s (x41 over 10m)    horizontal-pod-autoscaler  unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the metric packets-per-second for pods

没有采集指标导致,依赖类似于 https://github.com/stefanprodan/k8s-prom-hpa/blob/master/podinfo/podinfo-dep.yaml 的组件,有时间在细细研究。

  • kubectl get --raw /apis/custom.metrics.k8s.io/v1beta2/ | jq . |grep pod
  • 需要在 custom-metrics-config-map.yaml 中添加:
    - metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,reporter="destination"}[1m]))
      name:
        as: http_requests
      resources:
        overrides:
          destination_service_name:
            resource: service
          destination_workload_namespace:
            resource: namespace
      seriesQuery: requests_total
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数