金庆的专栏

  C++博客 :: 首页 :: 新随笔 :: 联系 :: 聚合  :: 管理 ::
  355 随笔 :: 0 文章 :: 454 评论 :: 0 Trackbacks
kubernetes导出有状态服务

(金庆的专栏 2018.7)

网游服务器中的房间服务器是有状态服务器,可以用 kubernetes statefulset 开启多个实例。

为了让客户端能够直连房间服务器,除了 statefulset 要求的 headless 服务,
还须为每个实例创建 NodePort 类型的服务, 并且选择Pod和禁止转发。

下面 bootcamp.yml 先创建了 bootcamp headless 服务(clusterIP: None),
又创建了 bootcamp StatefulSet, 实例个数为 2.
然后创建 bootcamp-0,1,2 服务,分别对应 bootcamp-0,1,2 pod.

服务个数大于实例个数,是想测试下服务没有对应的实例时的表现。

网游中的匹配服务器将分配一个房间给客户端,列举 bootcamp-0,1,2 pod 所在节点的外网 IP,
连同对应服务的端口,发送给客户端,让客户端直连。

[jinqing@host-10-240-79-10 statefulset]$ cat bootcamp.yml
apiVersion: v1
kind: Service
metadata:
  name: bootcamp
  namespace: jq
  labels:
    name: bootcamp
spec:
  ports:
    - port: 8080
  clusterIP: None  # StatefulSet要求Headless服务
  selector:
    app: bootcamp  # 选择 bootcamp 应用

---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: bootcamp
  namespace: jq
spec:
  serviceName: bootcamp  # 上面的 Headless 服务名
  replicas: 2
  template:
    metadata:
      labels:
        app: bootcamp  # 应用名,与服务中的 selector 对应
    spec:
      containers:
      - name: bootcamp
        image: docker.io/jocatalin/kubernetes-bootcamp:v1

---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-0
  namespace: jq
spec:
  type: NodePort  # 对外服务
  externalTrafficPolicy: Local  # 不要转发到其他节点
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-0  # 选择 pod
  ports:
  - protocol: TCP
    nodePort: 30880  # 对外端口
    port: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-1
  namespace: jq
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-1
  ports:
  - protocol: TCP
    nodePort: 30881
    port: 8080
---
kind: Service
apiVersion: v1
metadata:
  name: bootcamp-2
  namespace: jq
spec:
  type: NodePort
  externalTrafficPolicy: Local
  selector:
    app: bootcamp
    statefulset.kubernetes.io/pod-name: bootcamp-2
  ports:
  - protocol: TCP
    nodePort: 30882
    port: 8080

因为 statefulset 的每个实例有不同的标签,所以可以为服务选择一个实例。

利用 externalTrafficPolicy: Local 设置来禁止转发。
参考 service.spec.externalTrafficPolicy 的说明:

https://kubernetes.io/docs/tutorials/services/source-ip/#source-ip-for-services-with-type-nodeport

Setting service.spec.externalTrafficPolicy to the value Local will only proxy requests to local endpoints, never forwarding traffic to other nodes and thereby preserving the original source IP address. If there are no local endpoints, packets sent to the node are dropped, ...

因为有可能多个Pod开在同一节点上,所以对外端口设成了不同的 30880-30882。
如果限制每个节点只开一个实例,则对外端口可以设成同一个。

创建服务:

[jinqing@host-10-240-79-10 statefulset]$ kubectl apply -f bootcamp.yml
service "bootcamp" created
statefulset.apps "bootcamp" created
service "bootcamp-0" created
service "bootcamp-1" created
service "bootcamp-2" created

服务如下:

[jinqing@host-10-240-79-10 statefulset]$ kubectl get service -n jq
NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
bootcamp     ClusterIP   None             <none>        8080/TCP         3m
bootcamp-0   NodePort    10.96.128.137    <none>        8080:30880/TCP   3m
bootcamp-1   NodePort    10.109.2.56      <none>        8080:30881/TCP   3m
bootcamp-2   NodePort    10.102.181.193   <none>        8080:30882/TCP   3m

2个实例:

[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME         READY     STATUS    RESTARTS   AGE       IP            NODE
bootcamp-0   1/1       Running   0          4m        10.244.0.42   host-10-240-79-10
bootcamp-1   1/1       Running   0          4m        10.244.1.63   host-10-240-79-11

访问服务必须指定节点,不会自动转发:

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30880
Hello Kubernetes bootcamp! | Running on: bootcamp-0 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30881
curl: (7) Failed connect to 10.240.79.10:30881; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30880
curl: (7) Failed connect to 10.240.79.11:30880; Connection timed out
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1

没有负载均衡,30881端口总是访问 bootcamp-1:

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30881
Hello Kubernetes bootcamp! | Running on: bootcamp-1 | v=1

也可以从外网访问.

30882 端口无法连接:

[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.11:30882
curl: (7) Failed connect to 10.240.79.11:30882; Connection refused
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.10:30882
curl: (7) Failed connect to 10.240.79.10:30882; Connection refused

3个端口都有监听:

[root@host-10-240-79-11 tmp]# netstat -ntl | grep 3088
tcp6       0      0 :::30881                :::*                    LISTEN     
tcp6       0      0 :::30882                :::*                    LISTEN     
tcp6       0      0 :::30880                :::*                    LISTEN     

iptables-save 输出如下, 其中 10.244是Pod的网段。

没有实例运行的节点上,会丢弃请求:

-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH

-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "jq/bootcamp-1: has no local endpoints" -j KUBE-MARK-DROP
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP

有实例运行的节点上会转发给 Pod 8080:

-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-0:" -m tcp --dport 30880 -j KUBE-XLB-U5NEOQT6R5WSBVOH
-A KUBE-NODEPORTS -s 127.0.0.0/8 -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-MARK-MASQ
-A KUBE-NODEPORTS -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp --dport 30881 -j KUBE-XLB-LJXDQ4W47M42IZBH

-A KUBE-XLB-LJXDQ4W47M42IZBH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-LJXDQ4W47M42IZBH
-A KUBE-XLB-LJXDQ4W47M42IZBH -m comment --comment "Balancing rule 0 for jq/bootcamp-1:" -j KUBE-SEP-LJQA4WUIKJUQ5ALU
-A KUBE-XLB-U5NEOQT6R5WSBVOH -s 10.244.0.0/16 -m comment --comment "Redirect pods trying to reach external loadbalancer VIP to clusterIP" -j KUBE-SVC-U5NEOQT6R5WSBVOH
-A KUBE-XLB-U5NEOQT6R5WSBVOH -m comment --comment "jq/bootcamp-0: has no local endpoints" -j KUBE-MARK-DROP

-A KUBE-SEP-LJQA4WUIKJUQ5ALU -s 10.244.1.63/32 -m comment --comment "jq/bootcamp-1:" -j KUBE-MARK-MASQ
-A KUBE-SEP-LJQA4WUIKJUQ5ALU -p tcp -m comment --comment "jq/bootcamp-1:" -m tcp -j DNAT --to-destination 10.244.1.63:8080

30882 端口无法连接
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment "jq/bootcamp-2: has no endpoints" -m addrtype --dst-type LOCAL -m tcp --dport 30882 -j REJECT --reject-with icmp-port-unreachable

测试下扩容:

[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME       DESIRED   CURRENT   AGE
bootcamp   2         2         45m
[jinqing@host-10-240-79-10 statefulset]$ kubectl scale --replicas=3 statefulset/bootcamp -n jq
statefulset.apps "bootcamp" scaled
[jinqing@host-10-240-79-10 statefulset]$ kubectl get statefulset -n jq
NAME       DESIRED   CURRENT   AGE
bootcamp   3         3         47m
[jinqing@host-10-240-79-10 statefulset]$ kubectl get pod -n jq -o wide
NAME         READY     STATUS    RESTARTS   AGE       IP            NODE
bootcamp-0   1/1       Running   0          48m       10.244.0.42   host-10-240-79-10
bootcamp-1   1/1       Running   0          48m       10.244.1.63   host-10-240-79-11
bootcamp-2   1/1       Running   0          45s       10.244.2.60   host-10-240-79-12
[jinqing@host-10-240-79-10 statefulset]$ curl 10.240.79.12:30882
Hello Kubernetes bootcamp! | Running on: bootcamp-2 | v=1
posted on 2018-07-14 11:43 金庆 阅读(99) 评论(0)  编辑 收藏 引用 所属分类: 8. 其它2. 网游开发

只有注册用户登录后才能发表评论。
【推荐】超50万行VC++源码: 大型组态工控、电力仿真CAD与GIS源码库
网站导航: 博客园   IT新闻   BlogJava   知识库   博问   管理