ingress-nginx
是使用nginx实现的kubernetes ingress
介绍
本质是基于 OpenResty 开发,构建脚本如下:
部署
下载部署文件
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.1.1/deploy/static/provider/cloud/deploy.yaml
k8s.gcr.io/ingress-nginx/controller -> k8sgcrioingressnginx/controller
k8s.gcr.io/ingress-nginx/kube-webhook-certgen -> k8sgcrioingressnginx/kube-webhook-certgen
k8s.gcr.io/ingress-nginx/kube-webhook-certgen -> k8sgcrioingressnginx/kube-webhook-certgen
安装
$ kubectl apply -f deploy.yaml
namespace/ingress-nginx created
serviceaccount/ingress-nginx created
configmap/ingress-nginx-controller created
clusterrole.rbac.authorization.k8s.io/ingress-nginx created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx created
role.rbac.authorization.k8s.io/ingress-nginx created
rolebinding.rbac.authorization.k8s.io/ingress-nginx created
service/ingress-nginx-controller-admission created
service/ingress-nginx-controller created
deployment.apps/ingress-nginx-controller created
ingressclass.networking.k8s.io/nginx created
validatingwebhookconfiguration.admissionregistration.k8s.io/ingress-nginx-admission created
serviceaccount/ingress-nginx-admission created
clusterrole.rbac.authorization.k8s.io/ingress-nginx-admission created
clusterrolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
role.rbac.authorization.k8s.io/ingress-nginx-admission created
rolebinding.rbac.authorization.k8s.io/ingress-nginx-admission created
job.batch/ingress-nginx-admission-create created
job.batch/ingress-nginx-admission-patch created
plugin
部署 ingress-nginx kubectl-plugin(可选)
https://kubernetes.github.io/ingress-nginx/kubectl-plugin/
原理
- 渲染时的模板:https://github.com/kubernetes/ingress-nginx/blob/26fe69cb47/rootfs/etc/nginx/template/nginx.tmpl
- 注入的 lua 脚本:https://github.com/kubernetes/ingress-nginx/tree/26fe69cb47295f6677576fa8585afa98d72536e2/rootfs/etc/nginx/lua
暴露服务方式
- 使用 LoadBalancer,如 MetalLB 负载均衡器使用介绍,本示例使用的方式
- Deployment Pod 共享宿主机 net namesapce
- 也可以修改
service type
为NodePort
,如下:
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
type: NodePort # 修改为 NodePort 类型 *
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
nodePort: 30080 # http请求对外映射 30080 端口 *
- name: https
port: 443
targetPort: 443
protocol: TCP
nodePort: 30443 # https请求对外映射30443端口 *
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
使用
查看和访问 ingress-nginx
$ kubectl -n ingress-nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.105.71.57 172.20.0.200 80:30668/TCP,443:31198/TCP 99s
ingress-nginx-controller-admission ClusterIP 10.105.141.141 <none> 443/TCP 29m
$ kubectl -n ingress-nginx get ingressclasses.networking.k8s.io
NAME CONTROLLER PARAMETERS AGE
nginx k8s.io/ingress-nginx <none> 45m
访问:https://172.20.0.200 进入 ingress-nginx,默认没有配置任何 backend。
日志
nginx-ingress-controller执行configmap,确定
apiVersion: v1
data:
log-format: '{remote_address: $remote_addr, remote_user: "$remote_user", time_date: [$time_local], request: "$request", status: $status, http_referer: "$http_referer", http_user_agent: "$http_user_agent", request_id: $request_id}'
log-format-escape-json: "true"
enable-syslog: "true"
syslog-host: <syslog-ip>
syslog-port: "<syslog-port>"
kind: ConfigMap
metadata:
name: nginx-ingress-controller-cm
namespace: kube-system
nginx-ingress日志说明:
- 推荐把日志发送到syslog中
- 针对access_log、error_log通过上面的方式配置
- 针对server的日志,推荐配置如下:
access_log syslog:server=[2001:db8::1]:1234,facility=local7,tag=nginx,severity=info;
/var/log/nginx/access.log
重定向到 /dev/stdout
/var/log/nginx/error.log
重定向到 /dev/stderr
- 上述的 Dockerfile 命令
RUN ln -sf /dev/stdout /var/log/nginx/access.log \
&& ln -sf /dev/stderr /var/log/nginx/error.log
access_log /dev/stdout main;
error_log /dev/stdout warn;
Upstream 配置
默认情况下 NGINX ingress controller
NGINX upstream 配置的 endpoints 为 Pod IP/port,可以通过 nginx.ingress.kubernetes.io/service-upstream
配置。
Stream snippet
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/stream-snippet: |
server {
listen 8000;
proxy_pass 127.0.0.1:80;
}
websocket
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
rewrite
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: abc
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
ingressClassName: nginx
rules:
- host: abc.xiexianbin.cn
http:
paths:
- path: /weixin/(.*)
pathType: Prefix
backend:
service:
name: weixin
port:
number: 80
- path: /(.*)
pathType: Prefix
backend:
service:
name: frontend
port:
number: 80
x-request-id
- 客户端请求到达
Ingress-Nginx Controllerr
时会自动添加一个 X-Request-ID
的请求 Header
,参考
- nginx 对 header name 中如果包含下划线(
_
),则自动忽略,默认参数 underscores_in_headers: off
- ingress-nginx 配置
ingress-nginx/templates/controller-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
labels:
...
name: ingress-nginx-controller
namespace: ingress-nginx
data:
// 配置这里
enable-underscores-in-headers: "true"
apiVersion: v1
kind: ConfigMap
metadata:
labels:
...
name: ingress-nginx-controller
namespace: ingress-nginx
data:
// 配置这里
large-client-header-buffers: "4 32k"
others
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: my-app-name
namespace: my-namespace
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/proxy-body-size: 16m
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-connect-timeout: "60"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
调优
启动 keep-alive
- 用来避免建立/释放连接的开销,是代理更高效,性能更高,大大提高并发数
$ kubectl -n kube-system edit configmap nginx-configuration
...
apiVersion: v1
data:
keep-alive: "60"
keep-alive-requests: "100"
upstream-keepalive-connections: "10000"
upstream-keepalive-requests: "100"
upstream-keepalive-timeout: "60"
kind: ConfigMap
...
长链接断开问题
ingress-nginx 在有新配置时,会触发 nginx -s reload
动作,虽然长链接会保持(由 worker_shutdown_timeout
控制),但超时时会被杀死,建议扩大该超时时间,或业务高峰期减少配置调整次数