本文介绍 Kubernetes Pod 自动水平扩展。
介绍
借助 Kubernetes 的 HPA(Horizontal Pod Autoscaler) 可以实现 Pod 的自动扩缩容，该部分实现依赖于 Kubernetes 监控介绍 和 Prometheus 监控 Kubernetes。
autoscale 支持如下 Pod 控制器：
- deployment
- replica set
- stateful set
- replication controller
支持的 HPA 版本
$ kubectl api-versions |grep autoscaling
autoscaling/v1
autoscaling/v2
autoscaling/v2beta1
autoscaling/v2beta2
帮助
kubectl explain hpa
kubectl explain hpa.spec
kubectl explain HorizontalPodAutoscaler.spec.metrics
scale 介绍
运维人员可以是使用 scale 控制 Pod 的规模
root@k8s-master:~# kubectl scale deployment my-nginx --replicas=0
deployment.apps/my-nginx scaled
root@k8s-master:~# kubectl scale deployment my-nginx --replicas=2
deployment.apps/my-nginx scaled
root@k8s-master:~# kubectl get pod -l run=my-nginx
NAME READY STATUS RESTARTS AGE
my-nginx-cf54cdbf7-5k88z 1/1 Running 0 36s
my-nginx-cf54cdbf7-vsb42 1/1 Running 0 36s
HPA 示例
命令行创建自动伸缩
- 启动一个 Deployment， demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: php-apache
namespace: default
labels:
app: php-apache
spec:
replicas: 2
selector:
matchLabels:
app: php-apache
template:
metadata:
name: php-apache
labels:
app: php-apache
spec:
containers:
- name: php-apache
image: gcmirrors/hpa-example
ports:
- name: http
containerPort: 80
resources:
requests:
memory: "128Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "50m"
---
apiVersion: v1
kind: Service
metadata:
name: php-apache
namespace: default
spec:
selector:
app: php-apache
type: ClusterIP
# type: NodePort
ports:
- name: port-80
port: 80
targetPort: 80
protocol: TCP
$ kubectl apply -f demo.yaml
deployment.apps/php-apache created
service/php-apache created
- 命令行创建 autoscale，最先副本为 1，最大副本为 5，CPU使用率大于 80% 时扩容
$ kubectl autoscale deployment php-apache --min=2 --max=5 --cpu-percent=80
horizontalpodautoscaler.autoscaling/php-apache autoscaled
# 查看 HPA
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 2%/80% 2 5 2 22s
$ kubectl describe hpa php-apache
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sun, 03 Apr 2022 12:45:52 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 0% (0) / 80%
Min replicas: 2
Max replicas: 5
Deployment pods: 2 current / 2 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ScaleDownStabilized recent recommendations were higher than current one, applying the highest recent recommendation
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events: <none>
压力测试工具有很多（参考Http 性能相关测试工具汇总），本示例选择 wrk：
$ kubectl run wrk --rm=true -i --image=skandyla/wrk -- -t12 -c400 -d60s http://php-apache.default.svc.kb.cx
由于镜像资源限制很少，查看测试结果 Pod 数量有 2 扩展到 4 个：
root@k8s-node-1:~# kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache 96%/80% 2 5 4 44m
root@k8s-node-1:~# kubectl describe hpa php-apache
Warning: autoscaling/v2beta2 HorizontalPodAutoscaler is deprecated in v1.23+, unavailable in v1.26+; use autoscaling/v2 HorizontalPodAutoscaler
Name: php-apache
Namespace: default
Labels: <none>
Annotations: <none>
CreationTimestamp: Sun, 03 Apr 2022 12:45:52 +0800
Reference: Deployment/php-apache
Metrics: ( current / target )
resource cpu on pods (as a percentage of request): 96% (48m) / 80%
Min replicas: 2
Max replicas: 5
Deployment pods: 4 current / 4 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale recommended size matches current size
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 19h horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 19h horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
压测结束后，一段时间 Pod 自动收缩到 2：
$ kubectl describe hpa php-apache
...
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 8m39s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 6m24s horizontal-pod-autoscaler New size: 4; reason: cpu resource utilization (percentage of request) above target
Normal SuccessfulRescale 8s horizontal-pod-autoscaler New size: 2; reason: All metrics below target
声明式创建自动伸缩
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 2
maxReplicas: 5
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 80
- type: Resource
resource:
name: memory
target:
type: Value
averageValue: 40Mi
$ kubectl delete hpa php-apache
horizontalpodautoscaler.autoscaling "php-apache" deleted
$ kubectl apply -f hpa.yaml
horizontalpodautoscaler.autoscaling/php-apache created
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/40Mi, <unknown>/80% 2 5 0 5s
# 压测
$ 结果
$ kubectl describe hpa
...
ScalingLimited False DesiredWithinRange the desired count is within the acceptable range
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulRescale 10s horizontal-pod-autoscaler New size: 3; reason: cpu resource utilization (percentage of request) above target
其他指标
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: php-apache
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: php-apache
minReplicas: 2
maxReplicas: 5
metrics:
# - type: Resource
# resource:
# name: cpu
# target:
# type: Utilization
# averageUtilization: 80
# - type: Resource
# resource:
# name: memory
# target:
# type: Value
# averageValue: 40Mi
- type: Pods
pods:
metric:
name: packets-per-second
target:
type: AverageValue
averageValue: 1k
- type: Object
object:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: main-route
target:
type: Value
value: 1k
# - type: Object
# object:
# metric:
# name: http_requests
# target:
# type: Value
# value: 1k
$ kubectl delete -f hpa.yaml
$ kubectl apply -f hpa-2.yaml
horizontalpodautoscaler.autoscaling/php-apache created
$ kubectl get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
php-apache Deployment/php-apache <unknown>/1k, <unknown>/1k 2 5 2 16s
$ 结果
$ kubectl describe hpa
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetObjectMetric 8m23s (x8 over 10m) horizontal-pod-autoscaler unable to get metric requests-per-second: Ingress on default main-route/unable to fetch metrics from custom metrics API: the server could not find the metric requests-per-second for ingresses.networking.k8s.io
Warning FailedComputeMetricsReplicas 8m23s (x8 over 10m) horizontal-pod-autoscaler invalid metrics (2 invalid out of 2), first error is: failed to get pods metric value: unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the metric packets-per-second for pods
Warning FailedGetPodsMetric 6s (x41 over 10m) horizontal-pod-autoscaler unable to get metric packets-per-second: unable to fetch metrics from custom metrics API: the server could not find the metric packets-per-second for pods
没有采集指标导致，依赖类似于 https://github.com/stefanprodan/k8s-prom-hpa/blob/master/podinfo/podinfo-dep.yaml 的组件，有时间在细细研究。
kubectl get --raw /apis/custom.metrics.k8s.io/v1beta2/ | jq . |grep pod- 需要在
custom-metrics-config-map.yaml 中添加：
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>,reporter="destination"}[1m]))
name:
as: http_requests
resources:
overrides:
destination_service_name:
resource: service
destination_workload_namespace:
resource: namespace
seriesQuery: requests_total