「KUBERNETES-ADDONS」- MetalLB(裸机集群的负载均衡器实现)

  CREATED BY JENKINSBOT

问题描述

该笔记将记录:在 Kuberntes Cluster 中,我们部署 MetalLB 组件的过程,以及常见问题的解决办法。

解决方案

参考 MetalLB, bare metal load-balancer for Kubernetes 文档,以获取官方详细说明。

该部分的笔记仅是我们阅读官方文档而产生的阅读摘录(MetalLB v0.12.1),并记录下安装过程需要完成的工作。

Concepts

目前(04/28/2022,MetalLB v0.12.1)为止,看起来 MetalLB 并不那么依赖于 Kubernetes API 版本,其主要工作是在 Worker 完成地址绑定及通告、管理流量转发规则。

MetalLB 的 Speaker 使用 Host Network,所以 BGP Peer 使用 Worker 的网络地址(单物理网卡)。

Requirements

A Kubernetes cluster, running Kubernetes 1.13.0 or later, that does not already have network load-balancing functionality.

A cluster network configuration that can coexist with MetalLB.(我们使用 Calico,默认部署,所以问题不大)
https://metallb.universe.tf/installation/network-addons/
https://metallb.universe.tf/installation/clouds/

Some IPv4 addresses for MetalLB to hand out.(我们是内网开发环境使用,地址有很多,但是我们使用额外地址空间)

When using the BGP operating mode, you will need one or more routers capable of speaking BGP.(鉴于需要使用额外地址空间,所以我们使用 BGP 模式,但需要配置 BGP Peer 设备,通常是路由器,某些交换机也支持 BGP 路由协议)

When using the L2 operating mode, traffic on port 7946 (TCP & UDP, other port can be configured) must be allowed between nodes, as required by memberlist.

Installation

如果使用 kube-proxy IPVS 模式,需要开启 strict ARP 选项:

# kubectl edit configmap -n kube-system kube-proxy
...
    ipvs:
...
      strictARP: true 
...

通过 HELM 安装:

# helm repo add metallb https://metallb.github.io/metallb
# helm repo update

# helm show values metallb/metallb > helm-value-v0.12.1.yaml
# vim helm-value-v0.12.1.yaml
...
configInline:
  peers:
  - peer-address: 192.168.10.1
    peer-asn: 65000
    my-asn: 65130
    node-selectors:
    - match-expressions:
      - key: node-role.kubernetes.io/control-plane
        operator: DoesNotExist
  address-pools:
  - name: default
    protocol: bgp
    addresses:
    - 198.168.130.0/24
    avoid-buggy-ips: true
    bgp-advertisements:
    - aggregation-length: 24
...

# helm -n metallb-system \
    install metallb metallb/metallb -f helm-value-v0.12.1.yaml \
    --create-namespace

// 当调整参数后,启用更新操作

# helm -n metallb-system \
    upgrade metallb metallb/metallb -f helm-value-v0.12.1.yaml 

配置路由器(我们使用华为交换机,具体配置取决于网络环境,如下仅供参考):

bgp 65000

group develop-130-worker external
peer develop-130-worker as-number 65130

peer 192.168.10.134 group develop-130-worker
...
peer 192.168.10.139 group develop-130-worker

peer develop-130-worker connect-interface Vlanif10

[bgp] display bgp group develop-130-worker
...
 Peer Members:
  Peer            V          AS  MsgRcvd  MsgSent  OutQ  Up/Down       State PrefRcv

  192.168.10.134  4       65130       13       20     0 00:05:32 Established       0
  192.168.10.135  4       65130       16       19     0 00:07:30 Established       0
  192.168.10.136  4       65130       16       19     0 00:07:05 Established       0
  192.168.10.137  4       65130       13       20     0 00:05:34 Established       0
  192.168.10.138  4       65130       13       20     0 00:05:32 Established       0
  192.168.10.139  4       65130       13       20     0 00:05:34 Established       0

Testing

部署 LoadBalancer 类型的 Service 资源进行验证:

cat > svc-metallb-testing.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
  name: metallb-testing
spec:
  selector:
    app.kubernetes.io/name: MyApp
  ports:
    - protocol: TCP
      port: 80
      targetPort: 9376
  type: LoadBalancer
EOF

kubectl apply -f svc-metallb-testing.yaml

参考文献

Helm | Helm Upgrade
MetalLB/Installation
MetalLB/Configuration