TKE之Traefik最佳实践

聂伟星 • 2023-01-02 • 云技术社区 • 276 阅读

k8s的接入层有很多种，常见的7层负载均衡有nginx-ingress、traefik、kong等，还有每个云厂商为了对接自己的负载均衡产品所开发的控制器，tke集群现在默认是clb类型ingress，也支持组件安装nginx-ingress到集群内使用，其他类型的网关，需要自己在集群内部署才行，今天我们讲讲traefik在tke上的部署安装和一些使用实践。

1. 部署traefik

部署traefik到tke集群上，这里我们采用helm的方式，如果没有安装helm客户端的话，可以参考文档安装下https://cloud.tencent.com/document/product/457/32731 ，安装好helm客户端后参考下面操作获取traefik的chart包，这里我们需要自定义一下配置参数，所以会自定义一份value.yaml文件， 因此会将chart包解压配置之后再部署。

$ helm repo add traefik https://helm.traefik.io/traefik
$ helm repo update
$ helm  search repo traefik
NAME            CHART VERSION   APP VERSION     DESCRIPTION
traefik/traefik 10.15.0         2.6.1           A Traefik based Kubernetes ingress controller
$ helm pull traefik/traefik
$ tar -zxvf traefik-10.15.0.tgz

然后我们创建一份自己的value.yaml文件

$ cd traefik
$ vim my-value.yaml

service:
  type: NodePort 

ingressRoute:
  dashboard:
    enabled: false
ports:
  traefik:
    port: 9000
    expose: true
  web:
    port: 8000
    expose: true
  websecure:
    port: 8443
    expose: true
persistence:
  enabled: true
  name: data
  accessMode: ReadWriteOnce
  size: 10Gi  # 云上硬盘要求最小10G
  storageClass: "cbs" # 指定自己集群内的sc
  path: /data
additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--api.insecure=true"
  - "--api.dashboard=true"

然后执行命令进行部署既可

$ kubectl create ns weixnie
$ helm install traefik -n weixnie -f my-value.yaml .

查看下部署的资源，注意这里我们将svc改成云上的clb类型作为统一入口，然后我们将泛域名解析到clb的vip上既可，这里我用到的泛域名是*.traefik.niewx.cn，域名是腾讯云上，在云上做下dns的A记录解析即可。

[niewx@VM-0-4-centos ~]$ k get all -n weixnie | grep traefik
pod/traefik-544dbf6ddb-nt4mf   1/1     Running   0          3h2m
service/traefik   LoadBalancer   10.55.254.24    172.16.0.8    9000:30899/TCP,80:32387/TCP,443:30055/TCP   3h2m
deployment.apps/traefik   1/1     1            1           3h2m
replicaset.apps/traefik-544dbf6ddb   1         1         1       3h2m

这里我们可以直接通过clb的vip来访问下traefik的dashboard

能正常访问到dashboard，说明traefik已经成功部署到了我们的tke集群内。

2. 配置ingress路由

配置traefik类型的ingress的方式有2种，一种是走原生的ingress对象，还有一种就是走crd的方式部署

2.1 原生ingress方式

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-dashboard-ingress
  annotations:
    kubernetes.io/ingress.class: traefik
    traefik.ingress.kubernetes.io/router.entrypoints: web
spec:
  rules:
  - host: traefik-web.traefik.niewx.cn
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: traefik
            port:
              number: 9000

注意原生的ingress，要用到v1版本的api，但是tke上的版本暂不支持v1版本，原生的也是需要1.19版本后才能支持v1版本。

2.2 crd的方式创建ingress

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: traefik-dashboard-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`traefik-web.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: traefik
        port: 9000

控制台无法识别IngressRoute这个crd资源，所以需要用kubectl进行创建，创建好之后，浏览器访问域名即可。

通过域名能正常访问dashborad页面，则说明ingressroute配置的转发已经成功了。

3. 配置不同协议转发

traefik支持多种协议的转发，下面我们通过示例来进行实践配置下。

3.1 配置http服务转发

首先我们部署一个whoami的服务

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: whoami
    qcloud-app: whoami
  name: whoami
  namespace: weixnie
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: whoami
      qcloud-app: whoami
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        k8s-app: whoami
        qcloud-app: whoami
    spec:
      containers:
      - image: traefik/whoami
        imagePullPolicy: Always
        name: whoami
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
          requests:
            cpu: 250m
            memory: 256Mi
        securityContext:
          privileged: false
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: qcloudregistrykey
      restartPolicy: Always

---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: whoami
    qcloud-app: whoami
  name: whoami
  namespace: weixnie
spec:
  ports:
  - name: 80-80-tcp
    port: 80
    protocol: TCP
    targetPort: 80
  selector:
    k8s-app: whoami
    qcloud-app: whoami
  sessionAffinity: None
  type: ClusterIP

部署好之后，我们部署一个IngressRoute进行暴露

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80

创建好之后，就可以通过域名直接访问了

3.2 配置https服务转发

配置https服务，其实就是在http的IngressRoute加上证书配置，一般配置证书有2种方式，一种是手动配置，还有一种是自动生成证书。

3.2.1 手动配置tls证书

可以在腾讯云上购买下免费的证书，然后下载下nginx的部署类型证书，通过secret保存下证书

kubectl create secret tls whoami-tls -n weixnie --cert=whoami.traefik.niewx.cn_bundle.crt --key=whoami.traefik.niewx.cn.key

然后创建IngressRoute配置下证书

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route-tls
  namespace: weixnie
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80 
  tls:
    secretName: whoami-tls

配置好证书后，就可以通过https来访问了，注意https的entryPoints是用websecure

3.2.2 自动配置tls证书

要使用Let's Encrypt自动生成证书，需要使用ACME。需要在静态配置中定义 "证书解析器"，Traefik负责从ACME服务器中检索证书。

然后，每个 "路由器 "被配置为启用TLS，并通过tls.certresolver配置选项与一个证书解析器关联。

Traefik的ACME验证方式主要有以下三种：

  • tlsChallenge
  • httpChallenge
  • dnsChallenge

如果使用tlsChallenge，则要求Let's Encrypt到 Traefik 443 端口必须是可达的。如果使用httpChallenge，则要求Let's Encrypt到 Traefik 80端口必须是可达的。如果使用dnsChallenge，则需要对应的providers[7]。

但是我们上面部署Traefik的时候并没有把80和443端口暴露出来，要测试tlsChallenge和httpChallenge的话就必须暴露，下面我们暴露下

service:
  type: NodePort

ingressRoute:
  dashboard:
    enabled: false
ports:
  traefik:
    port: 9000
    expose: true
  web:
    port: 8000
    hostPort: 80 #traefik暴露80端口
    expose: true
  websecure:
    port: 8443
    hostPort: 443 #traefik暴露443端口
    expose: true
persistence:
  enabled: true
  name: data
  accessMode: ReadWriteOnce
  size: 10Gi
  path: /data
additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--api.insecure=true"
  - "--api.dashboard=true"

修改后，更新下即可

helm upgrade traefik -n weixnie -f my-value.yaml .
  • tlsChallenge

使用tlschallenge，需要加上如下参数，

deployment:
  initContainers:
    - name: volume-permissions
      image: busybox:1.31.1
      command: ["sh", "-c", "chmod -Rv 600 /data/*"]
      volumeMounts:
        - name: data
          mountPath: /data

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--api.insecure=true"
  - "--api.dashboard=true"
  - "--certificatesresolvers.niewxtls.acme.email=nwx_qdlg@163.com"
  - "--certificatesresolvers.niewxtls.acme.storage=/data/acme.json"
  - "--certificatesresolvers.niewxtls.acme.tlschallenge=true"

配置好，helm更新下traefik，然后配置下ingressroute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route-auto-tls
  namespace: weixnie
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`whoami-tlschallenge.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80 
  tls:
    certResolver: niewxtls
  • httpChallenge

使用httpchallenge需要加上如下配置

additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--api.insecure=true"
  - "--api.dashboard=true"
  - "--certificatesresolvers.niewxhttp.acme.email=coolops@163.com"
  - "--certificatesresolvers.niewxhttp.acme.storage=/data/acme.json"
  - "--certificatesresolvers.niewxhttp.acme.httpchallenge=true"
  - "--certificatesresolvers.niewxhttp.acme.httpchallenge.entrypoint=web"

然后创建下ingressroute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route-auto-tls-http
  namespace: weixnie
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`whoami-httpchallenge.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
  tls:
    certResolver: niewxhttp
  • dnsChallenge

使用dnschallenge，需要用provider，traefik支持的provide可以参考文档https://doc.traefik.io/traefik/https/acme/ 这里我们用dnspod，首先用secret挂载下dnspod的token

kubectl create secret generic traefik-dnspod --from-literal=DNSPOD_API_KEY=d5a1ba6d0cxxxxxxxx91a -n weixnie

然后修改下配置，开启下dnschallenge

......
additionalArguments:
  - "--serversTransport.insecureSkipVerify=true"
  - "--api.insecure=true"
  - "--api.dashboard=true"
  - "--certificatesresolvers.niewxdns.acme.email=coolops@163.com"
  - "--certificatesresolvers.niewxdns.acme.storage=/data/acme.json"
  - "--certificatesresolvers.niewxdns.acme.dnschallenge=true"
  - "--certificatesResolvers.niewxdns.acme.dnsChallenge.provider=alidns"
envFrom:
  - secretRef: 
      name: traefik-dnspod

添加之后，更新下traefik，创建下ingressroute

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route-auto-tls-dns
  namespace: weixnie
spec:
  entryPoints:
  - websecure
  routes:
  - match: Host(`whoami-dnschallenge.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80 
  tls:
    certResolver: niewxdns
    domains:
    - main: "*.niewx.cn"

3.3 配置tcp转发

traefik2.x是支持tcp的转发，这里我们测试通过转发redis服务，首先部署一个redis服务

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redis
  namespace: weixnie
spec:
  selector:
    matchLabels:
      app: redis
  template:
    metadata:
      labels:
        app: redis
    spec:
      containers:
      - name: redis
        image: redis:5.0.14
        ports:
        - containerPort: 6379
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: weixnie
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    app: redis

暴露TCP端口使用的是SNI【10】，而SNI又是依赖TLS的，所以我们需要配置证书才行，但是如果没有证书的话，我们可以使用通配符 * 进行配置

ports:
  traefik:
    port: 9000
    expose: true
  web:
    port: 8000
    hostPort: 80
    expose: true
  websecure:
    port: 8443
    hostPort: 443
    expose: true
  redis:
    port: 6379
    containerPort: 6379
    hostPort: 6379

在启动参数中添加--entryPoints.redis.address=:6379用来指定entrypoint，然后创建ingressrout暴露路由，traefik有提供IngressRouteTCP这个crd，我们直接创建就行

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteTCP
metadata:
  name: redis-traefik-tcp
  namespace: weixnie
spec:
  entryPoints:
    - redis
  routes:
  - match: HostSNI(`*`)
    services:
    - name: redis
      port: 6379

创建好之后，我们直接测试下

$ telnet redis.traefik.niewx.cn 6379
Trying 172.16.55.14...
Connected to redis.traefik.niewx.cn.
Escape character is '^]'.
$ redis-cli -h redis.traefik.niewx.cn
redis.traefik.niewx.cn:6379> ping
pong

直接通过域名访问redis服务是正常的，说明traefik转发tcp服务成功。

3.4 配置udp转发

traefik同样提供了udp服务的转发，首先我们部署一个udp的服务

apiVersion: v1
kind: Service
metadata:
  name: whoamiudp
  namespace: weixnie
spec:
  ports:
  - protocol: UDP
    name: udp
    port: 8080
  selector:
    app: whoamiudp
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: whoamiudp
  namespace: weixnie
  labels:
    app: whoamiudp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: whoamiudp
  template:
    metadata:
      labels:
        app: whoamiudp
    spec:
      containers:
        - name: whoamiudp
          image: containous/whoamiudp
          ports:
            - name: udp
              containerPort: 8080

我们在控制台这里定义了一个名为 udpep 的入口点，但是 protocol 协议是 UDP

ports:
  traefik:
    port: 9000
    expose: true
  web:
    port: 8000
    hostPort: 80
    expose: true
  websecure:
    port: 8443
    hostPort: 443
    expose: true
  redis:
    port: 6379
    containerPort: 6379
    hostPort: 6379
  udpep:
    port: 18080
    hostPort: 18080
    protocol: UDP

然后部署下对应的转发路由

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRouteUDP
metadata:
  name: whoamiudp
  namespace: weixnie
spec:
  entryPoints:
  - udpep
  routes:
  - services:
    - name: whoamiudp
      port: 8080

whoamiudp这个应用当我们输入 WHO 的时候，就会打印出访问的 Pod 的 Hostname 这些信息，如果不是则打印接收到字符串,下面我们测试下

$ echo "WHO" | socat - udp4-datagram:udp.traefik.niewx.cn:18080
Hostname: whoamiudp-69485d4699-5fxlm
IP: 127.0.0.1
IP: 10.55.0.41

$ k get pod -o wide -n weixnie | grep trae
traefik-5f7c9f6f4-vdgll      1/1     Running   0          12m     10.55.0.40   172.16.55.14   <none>           <none>

这里打印出来的ip是traefik的pod ip，说明这里转发udp协议成功了。

4. traefik中间件的配置使用

附加到路由器的中间件是一种在请求发送到您的服务之前（或在服务的答案发送到客户端之前）调整请求的方法。Traefik 中有几个可用的中间件，有的可以修改请求、headers，有的负责重定向，有的添加认证等等。使用相同协议的中间件可以组合成链以适应各种场景。具体介绍可以参考文档https://doc.traefik.io/traefik/middlewares/overview/ ，下面来讲讲常用的中间件如何使用。

4.1 Add Prefix

AddPrefix 中间件在转发请求之前更新请求的路径。

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: add-foo
  namespace: weixnie
spec:
  addPrefix:
    prefix: /foo

然后我们将这个中间件加ingressroute里面

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
    middlewares:
    - name: add-foo

然后访问域名可以发现，请求路径会默认加上/foo

4.2 BasicAuth

BasicAuth 中间件将您的服务的访问权限限制为已知用户。

首先我们用httpd创建下账号密码，然后通过secret挂载

htpasswd -nb admin 123456 | openssl base64
apiVersion: v1
kind: Secret
metadata:
  name: authsecret
  namespace: weixnie
data:
  users: |2
    YWRtaW46JGFwcjEkZWR4S083Y1ckZEh1eXdWaFlLdEd2eWpOREM3aFdXLwoK

配置secret可以直接配置用户密码，不通过httpd，里面的账号密码都经过了base64编码了。

apiVersion: v1
kind: Secret
metadata:
  name: authsecret
  namespace: weixnie
type: kubernetes.io/basic-auth
data:
  username: dXNlcg== # username: user
  password: cGFzc3dvcmQ= # password: password

创建下basicauth的中间件

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-auth
  namespace: weixnie
spec:
  basicAuth:
    secret: authsecret

然后我们将中间件配置到ingressroute上

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
    middlewares:
    - name: test-auth
    - name: add-foo

浏览器输入域名就会提示我们输入账号和密码

4.3 Buffering

缓冲中间件限制可以转发到服务的请求的大小。通过缓冲，Traefik 将整个请求读入内存（可能将大请求缓冲到磁盘中），并拒绝超过指定大小限制的请求。这可以帮助服务避免大量数据（例如多部分/表单数据），并且可以最大限度地减少向服务发送数据所花费的时间。

当你上传大文件的时候，可能会出现413 (Request Entity Too Large) response，这个时候我们就可以配置下maxResponseBodyBytes来解决这个问题。下面我们测试下，首先部署一个上传文件应用

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: up-and-down
    qcloud-app: up-and-down
  name: up-and-down
  namespace: weixnie
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: up-and-down
      qcloud-app: up-and-down
  template:
    metadata:
      labels:
        k8s-app: up-and-down
        qcloud-app: up-and-down
    spec:
      containers:
      - image: ccr.ccs.tencentyun.com/nwx_registry/up-and-down
        imagePullPolicy: Always
        name: up-and-down
        resources:
          limits:
            cpu: 500m
            memory: 1Gi
        securityContext:
          privileged: false
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: qcloudregistrykey
      restartPolicy: Always

---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: up-and-down
    qcloud-app: up-and-down
  name: up-and-down
  namespace: weixnie
spec:
  ports:
  - name: 5000-5000-tcp
    port: 8000
    protocol: TCP
    targetPort: 8000
  selector:
    k8s-app: up-and-down
    qcloud-app: up-and-down
  sessionAffinity: None
  type: ClusterIP

部署好之后，我们配置一个ingressroute来转发下

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: up-and-down-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`up-and-down.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: up-and-down
        port: 8000

然后我们配置一个缓存中间件

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: limit
  namespace: weixnie
spec:
  buffering:
    memRequestBodyBytes: 2000000

创建好中间件后，我们将中间件加到IngressRoute，这样我们上传文件大小最大就可以到2000000字节了

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: up-and-down-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`up-and-down.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: up-and-down
        port: 8000
    middlewares:
    - name: limit

4.4 Chain

Chain 中间件使您能够定义其他中间件的可重用组合。它使重用相同的组更容易。

上面我们开始给whoami定义了basic-auth和AddPrefix 这2个中间件，我们可以用chain来组合这2个中间成一个新的，然后ingressrout引用新的中间件即可。

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: my-middleware
  namespace: weixnie
spec:
  chain:
    middlewares:
    - name: add-foo
    - name: test-auth

然后将whoami的中间件配置改成my-middleware

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
    middlewares:
    - name: my-middleware

4.5 IPWhiteList

IPWhitelist 根据客户端 IP 接受/拒绝请求。

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-ipwhitelist
spec:
  ipWhiteList:
    sourceRange:
      - 127.0.0.1/32
      - 192.168.1.7

然后在ingressroute配置下这个test-ipwhitelist中间件

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
    middlewares:
    - name: my-middleware
    - name: test-ipwhitelist

4.6 RedirectScheme

RedirectScheme 将请求从一个方案/端口重定向到另一个，比如场景的http重定向到https

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-redirectscheme
  namespace: weixnie
spec:
  redirectScheme:
    scheme: https
    permanent: true

也可以重定向到某个端口

apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: test-redirectscheme
  namespace: weixnie
spec:
  redirectScheme:
    scheme: https
    port: "443"
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: whoami-route
  namespace: weixnie
spec:
  entryPoints:
  - web
  routes:
  - match: Host(`whoami.traefik.niewx.cn`)
    kind: Rule
    services:
      - name: whoami
        port: 80
    middlewares:
    - name: my-middleware
    - name: test-ipwhitelist
    - name: test-redirectscheme

浏览器输入whoami.traefik.niewx.cn会自动跳转到https。

5. traefik的金丝雀发布

traefik中提供了TraefikService这个对象，我们可以配置灰度发布和流量镜像等功能，具体介绍参考文档https://doc.traefik.io/traefik/routing/providers/kubernetes-crd/ ，首先我们部署下2个nginx版本

apiVersion: apps/v1
kind: Deployment
metadata:
  name: appv1
  namespace: weixnie
spec:
  selector:
    matchLabels:
      app: appv1
  template:
    metadata:
      labels:
        use: test
        app: appv1
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        lifecycle:
          postStart:
            exec:
              command:  ["/bin/sh", "-c", "echo Hello v1 > /usr/share/nginx/html/index.html"]
        ports:
        - containerPort: 80
          name: portv1

---

apiVersion: v1
kind: Service
metadata:
  name: appv1
  namespace: weixnie
spec:
  selector:
    app: appv1
  ports:
  - name: http
    port: 80
    targetPort: portv1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: appv2
  namespace: weixnie
spec:
  selector:
    matchLabels:
      app: appv2
  template:
    metadata:
      labels:
        use: test
        app: appv2
    spec:
      containers:
      - name: nginx
        image: nginx
        imagePullPolicy: IfNotPresent
        lifecycle:
          postStart:
            exec:
              command:  ["/bin/sh", "-c", "echo Hello v2 > /usr/share/nginx/html/index.html"]
        ports:
        - containerPort: 80
          name: portv2

---

apiVersion: v1
kind: Service
metadata:
  name: appv2
  namespace: weixnie
spec:
  selector:
    app: appv2
  ports:
  - name: http
    port: 80
    targetPort: portv2

部署好之后，我们通过TraefikService配置下灰度的比例

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: app-wrr
  namespace: weixnie
spec:
  weighted:
    services:
      - name: appv1
        weight: 3
        port: 80
        kind: Service
      - name: appv2
        weight: 1
        port: 80
        kind: Service

然后配置下域名来引用这个service

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-ingressroute-canary 
  namespace: weixnie
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`app.traefik.niewx.cn`)
    kind: Rule
    services:
    - name: app-wrr
      kind: TraefikService

然后我们来测试下，可以发现，v1:v2的请求比例是3:1，说明配置成功了

[niewx@VM-0-4-centos yaml]$ for i in {1..10} ; do curl app.traefik.niewx.cn; done
Hello v1
Hello v2
Hello v1
Hello v1
Hello v1
Hello v1
Hello v2
Hello v1
Hello v1
Hello v2

6. traefik的流量镜像功能

流量镜像可以将请求的流量按规则复制一份发送给其他服务，并且会忽略这部分请求的响应，下面我们还是以上面app为例，我们直接访问访问v1版本，然后将一定比例流量复制给v2版本，首先通过TraefikService定义下流量复制规则。

apiVersion: traefik.containo.us/v1alpha1
kind: TraefikService
metadata:
  name: app-mirror
  namespace: weixnie
spec:
  mirroring:
    name: appv1
    port: 80
    mirrors:
    - name: appv2
      percent: 50
      port: 80

然后我们给v1版本定义一个ingressroute，引用下这个TraefikService

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: app-ingressroute-canary 
  namespace: weixnie
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`mirror.traefik.niewx.cn`)
    kind: Rule
    services:
    - name: app-mirror
      kind: TraefikService

然后我们来测试访问下，可以发现，我们访问10次v1，有5次请求复制给了v2，这里说明流量复制成功。

7. prometheus监控traefik

traefik默认是有提供监控数据接口，我们可以通过prometheus获取监控数据，然后将数据展示到grafana上。

首先traefik默认的metrics的端口是9100，我们创建一个svc来提供下metrics接口。

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  name: traefik-metrics
  namespace: weixnie
spec:
  ports:
  - name: 9100-9100-tcp
    port: 9100
    protocol: TCP
    targetPort: 9100
  selector:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
  sessionAffinity: None
  type: ClusterIP

然后我们通过ServiceMonitor来采集下数据

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: traefik-metrics
  namespace: weixnie
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/name: traefik
spec:
  endpoints:
  - port: 9100-9100-tcp
    interval: 10s
  namespaceSelector:
    matchNames:
    - weixnie
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik
      app.kubernetes.io/name: traefik

rawjob的采集配置如下

scrape_configs:
- job_name: traefik-metrics
  honor_labels: true
  honor_timestamps: true
  scrape_interval: 15s
  metrics_path: /metrics
  scheme: http
  kubernetes_sd_configs:
  - role: endpoints
    namespaces:
      names:
      - weixnie
  relabel_configs:
  - source_labels: [__config_type]
    separator: ;
    regex: service
    target_label: __config_type
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: 9100-9100-tcp
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: traefik-metrics
    replacement: $1
    action: keep
  - source_labels: [__meta_kubernetes_pod_node_name]
    separator: ;
    regex: (.*)
    target_label: node
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_namespace]
    separator: ;
    regex: (.*)
    target_label: namespace
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_service_name]
    separator: ;
    regex: (.*)
    target_label: service
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_pod_name]
    separator: ;
    regex: (.*)
    target_label: pod
    replacement: $1
    action: replace
  - source_labels: [__meta_kubernetes_endpoint_port_name]
    separator: ;
    regex: (.*)
    target_label: endpoint
    replacement: $1
    action: replace

配置好之后，我们可以引用11462这个dashboard来进行数据展示。

8. 开启access访问日志

--accesslog=true
--accesslog.filepath=/data/access.log
--accesslog.format=json

开启access日志，只需要在启动参数加上这个既可。开启后，就可以在/data下查看到access日志

9. traefik工作负载完整的yaml

最后我这里提供下我的traefik完整的yaml，里面的参数配置可以供大家参考

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    meta.helm.sh/release-name: traefik
    meta.helm.sh/release-namespace: weixnie
  labels:
    app.kubernetes.io/instance: traefik
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: traefik
    helm.sh/chart: traefik-10.15.0
  name: traefik
  namespace: weixnie
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: traefik
      app.kubernetes.io/name: traefik
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9100"
        prometheus.io/scrape: "true"
      labels:
        app.kubernetes.io/instance: traefik
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: traefik
        helm.sh/chart: traefik-10.15.0
    spec:
      containers:
      - args:
        - --global.checknewversion
        - --global.sendanonymoususage
        - --entrypoints.metrics.address=:9100/tcp
        - --entrypoints.redis.address=:6379/tcp
        - --entrypoints.traefik.address=:9000/tcp
        - --entrypoints.udpep.address=:18080/udp
        - --entrypoints.web.address=:8000/tcp
        - --entrypoints.websecure.address=:8443/tcp
        - --api.dashboard=true
        - --ping=true
        - --metrics.prometheus=true
        - --metrics.prometheus.entrypoint=metrics
        - --providers.kubernetescrd
        - --providers.kubernetesingress
        - --serversTransport.insecureSkipVerify=true
        - --api.insecure=true
        - --api.dashboard=true
        - --certificatesresolvers.niewxtls.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.niewxtls.acme.email=nwx_qdlg@163.com
        - --certificatesresolvers.niewxtls.acme.storage=/data/acme.json
        - --certificatesresolvers.niewxtls.acme.tlschallenge=true
        - --certificatesresolvers.niewxtls.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.niewxhttp.acme.email=nwx_qdlg@163.com
        - --certificatesresolvers.niewxhttp.acme.storage=/data/acme.json
        - --certificatesresolvers.niewxhttp.acme.httpchallenge=true
        - --certificatesresolvers.niewxtls.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory
        - --certificatesresolvers.niewxhttp.acme.httpchallenge.entrypoint=web
        - --certificatesresolvers.niewxdns.acme.email=nwx_qdlg@163.com
        - --certificatesresolvers.niewxdns.acme.storage=/data/acme.json
        - --certificatesresolvers.niewxdns.acme.dnschallenge=true
        - --certificatesResolvers.niewxdns.acme.dnsChallenge.provider=dnspod
        - --accesslog=true
        - --accesslog.filepath=/data/access.log
        - --accesslog.format=json
        envFrom:
        - secretRef:
            name: traefik-dnspod
        image: traefik:2.6.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        name: traefik
        ports:
        - containerPort: 10053
          hostPort: 10053
          name: dns
          protocol: UDP
        - containerPort: 9100
          name: metrics
          protocol: TCP
        - containerPort: 6379
          hostPort: 6379
          name: redis
          protocol: TCP
        - containerPort: 9000
          name: traefik
          protocol: TCP
        - containerPort: 18080
          hostPort: 18080
          name: udpep
          protocol: UDP
        - containerPort: 8000
          hostPort: 80
          name: web
          protocol: TCP
        - containerPort: 8443
          hostPort: 443
          name: websecure
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /ping
            port: 9000
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources: {}
        securityContext:
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsGroup: 65532
          runAsNonRoot: true
          runAsUser: 65532
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /tmp
          name: tmp
      dnsPolicy: None
      initContainers:
      - command:
        - sh
        - -c
        - chmod -Rv 600 /data/*
        image: busybox:1.31.1
        imagePullPolicy: IfNotPresent
        name: volume-permissions
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 65532
      serviceAccount: traefik
      serviceAccountName: traefik
      terminationGracePeriodSeconds: 60
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: traefik
      - emptyDir: {}
        name: tmp
本站文章资源均来源自网络，除非特别声明，否则均不代表站方观点，并仅供查阅，不作为任何参考依据！
如有侵权请及时跟我们联系，本站将及时删除！
如遇版权问题，请查看 本站版权声明
THE END
分享
二维码
海报
TKE之Traefik最佳实践
k8s的接入层有很多种，常见的7层负载均衡有nginx-ingress、traefik、kong等，还有每个云厂商为了对接自己的负载均衡产品所开发的控制器，tk...
<<上一篇
下一篇>>