跳到主要内容

替换镜像地址

Kubernetes服务访问之Ingress​

对于Kubernetes的Service，无论是Cluster-Ip和NodePort均是四层的负载，集群内的服务如何实现七层的负载均衡，这就需要借助于Ingress，Ingress控制器的实现方式有很多，比如nginx, Contour, Haproxy, trafik, Istio。几种常用的ingress功能对比和选型可以参考这里

Ingress-nginx是7层的负载均衡器 ，负责统一管理外部对k8s cluster中Service的请求。主要包含：

  • ingress-nginx-controller：根据用户编写的ingress规则（创建的ingress的yaml文件），动态的去更改nginx服务的配置文件，并且reload重载使其生效（是自动化的，通过lua脚本来实现）；

  • Ingress资源对象：将Nginx的配置抽象成一个Ingress对象

    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: ingress-wildcard-host
    spec:
      ingressClassName: nginx
      rules:
      - host: "foo.bar.com"
        http:
          paths:
          - pathType: Prefix
            path: "/bar"
            backend:
              service:
                name: service1
                port:
                number: 80
      - host: "bar.foo.com"
        http:
          paths:
          - pathType: Prefix
            path: "/foo"
            backend:
              service:
                name: service2
                port:
                  number: 80
  
  

###### 示意图：

![](/img/k8s/kubernetes-base/ingress.webp)

###### 实现逻辑

1）ingress controller通过和kubernetes api交互，动态的去感知集群中ingress规则变化
2）然后读取ingress规则(规则就是写明了哪个域名对应哪个service)，按照自定义的规则，生成一段nginx配置
3）再写到nginx-ingress-controller的pod里，这个Ingress controller的pod里运行着一个Nginx服务，控制器把生成的nginx配置写入/etc/nginx/nginx.conf文件中
4）然后reload一下使配置生效。以此达到域名分别配置和动态更新的问题。

###### 安装

  [官方文档](https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md)

```bash
$ wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.4.0/deploy/static/provider/cloud/deploy.yaml
## 修改部署节点
$ vim deploy.yaml
504         volumeMounts:
505         - mountPath: /usr/local/certificates/
506           name: webhook-cert
507           readOnly: true
508       dnsPolicy: ClusterFirst
509       nodeSelector:
510         ingress: "true"
511       hostNetwork: true
512       serviceAccountName: ingress-nginx
513       terminationGracePeriodSeconds: 300
514       volumes:

# 替换镜像地址
sed -i 's#registry.k8s.io/ingress-nginx/kube-webhook-certgen:v20220916-gd32f8c343@sha256:39c5b2e3310dc4264d638ad28d9d1d96c4cbb2b2dcfb52368fe4e3c63f61e10f#myifeng/registry.k8s.io_ingress-nginx_kube-webhook-certgen:v1.3.0#g' deploy.yaml

sed -i 's#registry.k8s.io/ingress-nginx/controller:v1.4.0@sha256:34ee929b111ffc7aa426ffd409af44da48e5a0eea1eb2207994d9e0c0882d143#myifeng/registry.k8s.io_ingress-nginx_controller:v1.4.0#g' deploy.yaml

创建ingress

# 为k8s-master节点添加label
$ kubectl label node k8s-master ingress=true

$ kubectl apply -f deploy.yaml

使用示例：​
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eladmin-api
  namespace: luffy
spec:
  ingressClassName: nginx
  rules:
  - host: eladmin-api.luffy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: eladmin-api
            port:
              number: 8000

ingress-nginx动态生成upstream配置：

$ kubectl -n ingress-nginx exec -ti nginx-ingress-xxxxxxx bash
# ps aux
# cat /etc/nginx/nginx.conf|grep eladmin-api -A10 -B1
...
        ## start server eladmin-api.luffy.com
        server {
                server_name eladmin-api.luffy.com ;

                listen 80  ;
                listen [::]:80  ;
                listen 443  ssl http2 ;
                listen [::]:443  ssl http2 ;

                set $proxy_upstream_name "-";

                ssl_certificate_by_lua_block {
                        certificate.call()
--
                        set $namespace      "luffy";
                        set $ingress_name   "eladmin-api";
                        set $service_name   "eladmin-api";
                        set $service_port   "8000";
                        set $location_path  "/";
                        set $global_rate_limit_exceeding n;

                        rewrite_by_lua_block {
                                lua_ingress.rewrite({
                                        force_ssl_redirect = false,
                                        ssl_redirect = true,
                                        force_no_ssl_redirect = false,
                                        preserve_trailing_slash = false,
--
                        set $balancer_ewma_score -1;
                        set $proxy_upstream_name "luffy-eladmin-api-8000";
                        set $proxy_host          $proxy_upstream_name;
                        set $pass_access_scheme  $scheme;

                        set $pass_server_port    $server_port;

                        set $best_http_host      $http_host;
                        set $pass_port           $pass_server_port;

                        set $proxy_alternative_upstream_name "";

--
        }
        ## end server eladmin-api.luffy.com
 ...

域名解析服务，将 eladmin-api.luffy.com解析到ingress的地址上。ingress是支持多副本的，高可用的情况下，生产的配置是使用lb服务（内网F5设备，公网elb、slb、clb，解析到各ingress的机器，如何域名指向lb地址）

本机，添加如下hosts记录来演示效果。

172.21.65.226 eladmin-api.luffy.com

然后，访问 http://eladmin-api.luffy.com/auth/code

使用ingress访问eladmin-web服务​

综合来看下，如何使用ingress来实现eladmin-web项目的访问，总结了三种方式：

  • **方式一: **

    项目访问地址
    eladmin-webhttp://eladmin.luffy.com
    eladmin-apihttp://eladmin-api.luffy.com

    此方式，eladmin-api对应的地址和目前前端配置的地址( http://eladmin.luffy.com:8000 )有差异，因此需要对前端的代码做调整：

    .env.production

    ENV = 'production'

    # 如果使用 Nginx 代理后端接口，那么此处需要改为 '/'，文件查看 Docker 部署篇，Nginx 配置
    # 接口地址，注意协议，如果你没有配置 ssl，需要将 https 改为 http
    VUE_APP_BASE_API  = 'http://eladmin-api.luffy.com'
    # 如果接口是 http 形式， wss 需要改为 ws
    VUE_APP_WS_API = 'ws://eladmin-api.luffy.com'

    前端代码调整，因此需要重新构建一版前端：

    docker build . -t 172.21.65.226:5000/eladmin/eladmin-web:v2
    docker push 172.21.65.226:5000/eladmin/eladmin-web:v2

    然后为eladmin-web准备Deployment、Service、Ingress 资源清单：

    eladmin-web-all.yaml

    # deployment-eladmin-web.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      replicas: 1   #指定Pod副本数
      selector:             #指定Pod的选择器
        matchLabels:
          app: eladmin-web
      template:
        metadata:
          labels:   #给Pod打label
            app: eladmin-web
        spec:
          imagePullSecrets:
          - name: registry-172-21-65-226
          containers:
          - name: eladmin-web
            image: 172.21.65.226:5000/eladmin/eladmin-web:v2
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            resources:
              requests:
                memory: 200Mi
                cpu: 50m
              limits:
                memory: 2Gi
                cpu: 2
            livenessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15  # 容器启动后第一次执行探测是需要等待多少秒
              periodSeconds: 15     # 执行探测的频率
              timeoutSeconds: 3             # 探测超时时间
            readinessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15
              timeoutSeconds: 3
              periodSeconds: 15
    # service-eladmin-web.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: eladmin-web
      type: ClusterIP
    # ingress-eladmin-web.yaml
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      ingressClassName: nginx
      rules:
      - host: eladmin.luffy.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service: 
                name: eladmin-web
                port:
                  number: 80

    本机，添加如下hosts记录来演示效果。

    172.21.65.226 eladmin.luffy.com

    然后，访问 http://eladmin.luffy.com

  • 方式二：

    规划使用如下地址访问：

    项目访问地址
    eladmin-webhttp://eladmin.luffy.com
    eladmin-apihttp://eladmin.luffy.com:8000

    此方式，eladmin-api对应的地址和目前前端配置的地址( http://eladmin.luffy.com:8000 )没有差异，但是ingress-nginx安装时只暴漏了 80和443端口进行服务转发，8000端口没有对外暴漏，因此，需要配置使用ingress-nginx转发tcp服务：

    • 修改ingress-nginx-controller的配置

      # 新添加文档中的L440和L441
      $ vim deploy.yaml
      ...
      431     spec:
      432       containers:
      433       - args:
      434         - /nginx-ingress-controller
      435         - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
      436         - --election-id=ingress-controller-leader
      437         - --controller-class=k8s.io/ingress-nginx
      438         - --ingress-class=nginx
      439         - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
      440         - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
      441         - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
      442         - --validating-webhook=:8443
      443         - --validating-webhook-certificate=/usr/local/certificates/cert
      444         - --validating-webhook-key=/usr/local/certificates/key
      445         env:
      446         - name: POD_NAME
      447           valueFrom:
      448             fieldRef:
      449               fieldPath: metadata.name
      ...

    • 重建服务

      $ kubectl -n ingress-nginx scale deployment ingress-nginx-controller --replicas 0
      $ kubectl -n ingress-nginx scale deployment ingress-nginx-controller --replicas 1

    • 创建tcp-services配置

      $ cat tcp-services.cm.yaml
      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: tcp-services
        namespace: ingress-nginx
      data:
        8000: "luffy/eladmin-api:8000"

    • 创建并验证

      kubectl create -f tcp-services.cm.yaml
      curl 172.21.65.226:8000/auth/code

    验证可以成功通过ingress-controller的8000端口转发到后端服务，因此我们创建前端ingress资源：

    然后为eladmin-web准备Deployment、Service、Ingress 资源清单：

    eladmin-web-all.yaml

    # deployment-eladmin-web.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      replicas: 1   #指定Pod副本数
      selector:             #指定Pod的选择器
        matchLabels:
          app: eladmin-web
      template:
        metadata:
          labels:   #给Pod打label
            app: eladmin-web
        spec:
          imagePullSecrets:
          - name: registry-172-21-65-226
          containers:
          - name: eladmin-web
            image: 172.21.65.226:5000/eladmin/eladmin-web:v1
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            resources:
              requests:
                memory: 200Mi
                cpu: 50m
              limits:
                memory: 2Gi
                cpu: 2
            livenessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15  # 容器启动后第一次执行探测是需要等待多少秒
              periodSeconds: 15     # 执行探测的频率
              timeoutSeconds: 3             # 探测超时时间
            readinessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15
              timeoutSeconds: 3
              periodSeconds: 15
    # service-eladmin-web.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: eladmin-web
      type: ClusterIP
    # ingress-eladmin-web.yaml
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      ingressClassName: nginx
      rules:
      - host: eladmin.luffy.com
        http:
          paths:
          - path: /
            pathType: Prefix
            backend:
              service: 
                name: eladmin-web
                port:
                  number: 80

    本机，添加如下hosts记录来演示效果。

    172.21.65.226 eladmin.luffy.com

    然后，访问 http://eladmin.luffy.com

  • 方式三：​

    规划使用如下地址访问：

    项目访问地址
    eladmin-webhttp://eladmin.luffy.com
    eladmin-apihttp://eladmin.luffy.com/apis

    此方式，eladmin-api对应的地址和目前前端配置的地址( http://eladmin.luffy.com:8000 )存在差异，，因此需要对前端的代码做调整：

    .env.production

    ENV = 'production'

    # 如果使用 Nginx 代理后端接口，那么此处需要改为 '/'，文件查看 Docker 部署篇，Nginx 配置
    # 接口地址，注意协议，如果你没有配置 ssl，需要将 https 改为 http
    VUE_APP_BASE_API  = 'http://eladmin.luffy.com/apis'
    # 如果接口是 http 形式， wss 需要改为 ws
    VUE_APP_WS_API = 'ws://eladmin.luffy.com/apis'

    前端代码调整，因此需要重新构建一版前端：

    docker build . -t 172.21.65.226:5000/eladmin/eladmin-web:v3
    docker push 172.21.65.226:5000/eladmin/eladmin-web:v3

    然后为eladmin-web准备Deployment、Service、Ingress 资源清单：

    eladmin-web-all.yaml

    # deployment-eladmin-web.yaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      replicas: 1   #指定Pod副本数
      selector:             #指定Pod的选择器
        matchLabels:
          app: eladmin-web
      template:
        metadata:
          labels:   #给Pod打label
            app: eladmin-web
        spec:
          imagePullSecrets:
          - name: registry-172-21-65-226
          containers:
          - name: eladmin-web
            image: 172.21.65.226:5000/eladmin/eladmin-web:v3
            imagePullPolicy: IfNotPresent
            ports:
            - containerPort: 80
            resources:
              requests:
                memory: 200Mi
                cpu: 50m
              limits:
                memory: 2Gi
                cpu: 2
            livenessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15  # 容器启动后第一次执行探测是需要等待多少秒
              periodSeconds: 15     # 执行探测的频率
              timeoutSeconds: 3             # 探测超时时间
            readinessProbe:
              tcpSocket:
                port: 80
              initialDelaySeconds: 15
              timeoutSeconds: 3
              periodSeconds: 15
    # service-eladmin-web.yaml
    apiVersion: v1
    kind: Service
    metadata:
      name: eladmin-web
      namespace: luffy
    spec:
      ports:
      - port: 80
        protocol: TCP
        targetPort: 80
      selector:
        app: eladmin-web
      type: ClusterIP
    # ingress-eladmin-web.yaml
    apiVersion: networking.k8s.io/v1
    kind: Ingress
    metadata:
      name: eladmin-api
      namespace: luffy
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: /$1
    spec:
      ingressClassName: nginx
      rules:
      - host: eladmin.luffy.com
        http:
          paths:
          - path: /apis/(.*)
            pathType: Prefix
            backend:
              service: 
                name: eladmin-api
                port:
                  number: 8000

    本机，添加如下hosts记录来演示效果。

    172.21.65.226 eladmin.luffy.com

    然后，访问 http://eladmin.luffy.com

HTTPS访问：​
#自签名证书
$ openssl req -x509 -nodes -days 2920 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=*.luffy.com/O=ingress-nginx"

# 证书信息保存到secret对象中，ingress-nginx会读取secret对象解析出证书加载到nginx配置中
$ kubectl -n luffy create secret tls tls-eladmin --key tls.key --cert tls.crt

修改yaml

.env.production

ENV = 'production'

# 如果使用 Nginx 代理后端接口，那么此处需要改为 '/'，文件查看 Docker 部署篇，Nginx 配置
# 接口地址，注意协议，如果你没有配置 ssl，需要将 https 改为 http
VUE_APP_BASE_API  = 'https://eladmin.luffy.com/apis/'
# 如果接口是 http 形式， wss 需要改为 ws
VUE_APP_WS_API = 'wss://eladmin.luffy.com/apis/'

前端代码调整，因此需要重新构建一版前端：

docker build . -t 172.21.65.226:5000/eladmin/eladmin-web:v4
docker push 172.21.65.226:5000/eladmin/eladmin-web:v4

修改eladmin-web使用v4版本的镜像：

kubectl -n luffy edit deployment eladmin-web

为ingress加载https证书：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eladmin-web
  namespace: luffy
spec:
  ingressClassName: nginx
  rules:
  - host: eladmin.luffy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: eladmin-web
            port:
              number: 80
  tls:
  - hosts:
    - eladmin.luffy.com
    secretName: tls-eladmin
常用注解说明​

nginx端存在很多可配置的参数，通常这些参数在ingress的定义中被放在annotations中实现，如下为常用的一些：

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: eladmin-web
  namespace: luffy
  annotations:
    nginx.ingress.kubernetes.io/force-ssl-redirect: "false"
    nginx.ingress.kubernetes.io/proxy-body-size: 1000m
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.org/client-max-body-size: 1000m
spec:
  ingressClassName: nginx
  rules:
  - host: eladmin.luffy.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service: 
            name: eladmin-web
            port:
              number: 80
  tls:
  - hosts:
    - eladmin.luffy.com
    secretName: tls-eladmin