dtcms 模板下载 node.js ssh命令 firebase sdk cassandra testng vue源码下载 vue钩子函数 vue特点 jquery去空格 leach算法 安卓虚拟机运行windows java解析pdf python3文件操作 python3删除文件 python高级 python零基础教程 python读文件 python返回函数 python匹配字符串 java入门教程 配置java环境 java课程学习 java获取ip地址 java终止线程 java将数据写入文件 java的date linux命令 linuxtar命令 路由器有辐射吗 微信助力软件 ezcad2 战斗的召唤 js刷新页面 今日头条邀请码 字典生成器 cad多段线线宽设置 苹果x怎么用 画图橡皮擦怎么放大
当前位置: 首页 > 学习教程  > 编程语言

Kubeadm方式搭建K8S集群

2020/8/11 18:51:59 文章标签:

Kubeadm方式搭建集群优缺点:

优点:
    简单优雅,支持高可用,升级方便
    
缺点:
    不易维护,文档不够细致

将master作为deploy节点,未指定节点时默认在master上进行操作。

建议deploy节点与其它节点配置ssh免密登录,配置过程参考:批量实现SSH免密登录 。


环境准备

环境准备工作请在所有节点进行。

  • 主机说明:
系统 ip 角色 cpu 内存 hostname
CentOS 7.8 192.168.30.128 master、deploy >=2 >=2G master
CentOS 7.8 192.168.30.129 node >=2 >=2G node1
CentOS 7.8 192.168.30.130 node >=2 >=2G node2
CentOS 7.8 192.168.30.131 node >=2 >=2G node3
  • 设置主机名:

以master为例,

hostnamectl set-hostname master
  • 安装依赖包:
yum update -y

yum install -y curl git iptables conntrack ipvsadm ipset jq sysstat libseccomp
  • 关闭防火墙、selinux和swap,重置iptables:
systemctl stop firewalld && systemctl disable firewalld

sed -i 's/=enforcing/=disabled/g' /etc/selinux/config && setenforce 0

iptables -F && iptables -X && iptables -F -t nat && iptables -X -t nat && iptables -P FORWARD ACCEPT

swapoff -a

sed -i '/swap/s/^\(.*\)$/#\1/g' /etc/fstab
  • 系统参数设置:
cat > /etc/sysctl.d/kubernetes.conf <<EOF
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
net.ipv4.ip_forward=1
vm.swappiness=0
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
EOF

modprobe br_netfilter

sysctl -p /etc/sysctl.d/kubernetes.conf
  • 安装docker:
curl http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo -o /etc/yum.repos.d/docker.repo

yum makecache fast

yum install -y docker-ce

systemctl enable docker && systemctl start docker

cat > /etc/docker/daemon.json <<EOF
{  
    "registry-mirrors": ["http://f1361db2.m.daocloud.io"],
    "exec-opts":["native.cgroupdriver=systemd"]
}
EOF

systemctl restart docker
  • 安装必要工具:
kubeadm     用于部署集群

bukelet     集群中各节点需要运行的组件,负责管理pod、容器的生命周期

kubectl     集群管理工具(master节点安装即可)
cat > /etc/yum.repos.d/kubernetes.repo <<EOF
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
       http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

yum install -y kubeadm-1.18.3 kubelet-1.18.3 kubectl-1.18.3 --disableexcludes=kubernetes

systemctl enable kubelet && systemctl start kubelet

集群初始化

  • 集群初始化:
mkdir /software

vim /software/kubeadm-config.yaml
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.18.3
controlPlaneEndpoint: 192.168.30.128:6443
networking:
    podSubnet: 172.10.0.0/16
imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
kubeadm config images pull --kubernetes-version=v1.18.3 --image-repository=registry.cn-hangzhou.aliyuncs.com/google_containers

kubeadm init --config=/software/kubeadm-config.yaml --upload-certs

初始化这一步如果报错:

error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition

解决:

swapoff -a

kubeadm reset -f

systemctl daemon-reload

systemctl restart kubelet

iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
mkdir ~/.kube

\cp /etc/kubernetes/admin.conf ~/.kube/config

kubectl get pods -n kube-system

NAME                             READY   STATUS    RESTARTS   AGE
coredns-546565776c-srxkq         0/1     Pending   0          60s
coredns-546565776c-w9fbs         0/1     Pending   0          60s
etcd-master                      1/1     Running   0          75s
kube-apiserver-master            1/1     Running   0          75s
kube-controller-manager-master   1/1     Running   0          75s
kube-proxy-qb7d5                 1/1     Running   0          60s
kube-scheduler-master            1/1     Running   0          75s
kubectl completion bash > ~/.kube/completion.bash.inc
 
echo 'source ~/.kube/completion.bash.inc' >> ~/.bash_profile

source ~/.bash_profile

注意备份上面初始化之后打印的join命令,这里分别是以master、node节点加入集群。

kubeadm join 192.168.30.128:6443 --token 1ndel7.xb623vep9pl5o6vl \
    --discovery-token-ca-cert-hash sha256:0e41f6020955c36970bf504cbfc0047941240dda57ebb9d85086706da14dcd1f \
    --control-plane --certificate-key 6518fe9f3eca5cb4a5860170d18c03109f54c94fba8ca7e5408a9aab5e598663

kubeadm join 192.168.30.128:6443 --token 1ndel7.xb623vep9pl5o6vl \
    --discovery-token-ca-cert-hash sha256:0e41f6020955c36970bf504cbfc0047941240dda57ebb9d85086706da14dcd1f

部署calico

  • 部署calico:
mkdir /etc/kubernetes/addons

vim /etc/kubernetes/addons/calico-rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
rules:
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - watch
      - list
      - get
  - apiGroups: [""]
    resources:
      - pods
    verbs:
      - get
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
    verbs:
      - list
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - hostendpoints
    verbs:
      - get
      - list
      - create
      - update
      - delete
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - clusterinformations
    verbs:
      - get
      - create
      - update
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - kubecontrollersconfigurations
    verbs:
      - get
      - create
      - update
      - watch
      
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-kube-controllers
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-kube-controllers
subjects:
- kind: ServiceAccount
  name: calico-kube-controllers
  namespace: kube-system

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-node
  namespace: kube-system

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: calico-node
rules:
  - apiGroups: [""]
    resources:
      - pods
      - nodes
      - namespaces
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - endpoints
      - services
    verbs:
      - watch
      - list
      - get
  - apiGroups: [""]
    resources:
      - configmaps
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - nodes/status
    verbs:
      - patch
      - update
  - apiGroups: ["networking.k8s.io"]
    resources:
      - networkpolicies
    verbs:
      - watch
      - list
  - apiGroups: [""]
    resources:
      - pods
      - namespaces
      - serviceaccounts
    verbs:
      - list
      - watch
  - apiGroups: [""]
    resources:
      - pods/status
    verbs:
      - patch
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - globalfelixconfigs
      - felixconfigurations
      - bgppeers
      - globalbgpconfigs
      - bgpconfigurations
      - ippools
      - ipamblocks
      - globalnetworkpolicies
      - globalnetworksets
      - networkpolicies
      - networksets
      - clusterinformations
      - hostendpoints
      - blockaffinities
    verbs:
      - get
      - list
      - watch
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ippools
      - felixconfigurations
      - clusterinformations
    verbs:
      - create
      - update
  - apiGroups: [""]
    resources:
      - nodes
    verbs:
      - get
      - list
      - watch
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - bgpconfigurations
      - bgppeers
    verbs:
      - create
      - update
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
      - ipamblocks
      - ipamhandles
    verbs:
      - get
      - list
      - create
      - update
      - delete
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - ipamconfigs
    verbs:
      - get
  - apiGroups: ["crd.projectcalico.org"]
    resources:
      - blockaffinities
    verbs:
      - watch
  - apiGroups: ["apps"]
    resources:
      - daemonsets
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-node
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-node
subjects:
- kind: ServiceAccount
  name: calico-node
  namespace: kube-system
vim /etc/kubernetes/addons/calico.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: calico-config
  namespace: kube-system
data:
  typha_service_name: "none"
  calico_backend: "bird"
  veth_mtu: "1440"
  cni_network_config: |-
    {
      "name": "k8s-pod-network",
      "cniVersion": "0.3.1",
      "plugins": [
        {
          "type": "calico",
          "log_level": "info",
          "datastore_type": "kubernetes",
          "nodename": "__KUBERNETES_NODE_NAME__",
          "mtu": __CNI_MTU__,
          "ipam": {
              "type": "calico-ipam"
          },
          "policy": {
              "type": "k8s"
          },
          "kubernetes": {
              "kubeconfig": "__KUBECONFIG_FILEPATH__"
          }
        },
        {
          "type": "portmap",
          "snat": true,
          "capabilities": {"portMappings": true}
        },
        {
          "type": "bandwidth",
          "capabilities": {"bandwidth": true}
        }
      ]
    }
  
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgpconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPConfiguration
    listKind: BGPConfigurationList
    plural: bgpconfigurations
    singular: bgpconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              asNumber:
                format: int32
                type: integer
              logSeverityScreen:
                type: string
              nodeToNodeMeshEnabled:
                type: boolean
              serviceClusterIPs:
                items:
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
              serviceExternalIPs:
                items:
                  properties:
                    cidr:
                      type: string
                  type: object
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: bgppeers.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BGPPeer
    listKind: BGPPeerList
    plural: bgppeers
    singular: bgppeer
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              asNumber:
                format: int32
                type: integer
              node:
                type: string
              nodeSelector:
                type: string
              peerIP:
                type: string
              peerSelector:
                type: string
            required:
            - asNumber
            - peerIP
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: blockaffinities.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: BlockAffinity
    listKind: BlockAffinityList
    plural: blockaffinities
    singular: blockaffinity
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              cidr:
                type: string
              deleted:
                type: string
              node:
                type: string
              state:
                type: string
            required:
            - cidr
            - deleted
            - node
            - state
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
  
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: clusterinformations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: ClusterInformation
    listKind: ClusterInformationList
    plural: clusterinformations
    singular: clusterinformation
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              calicoVersion:
                type: string
              clusterGUID:
                type: string
              clusterType:
                type: string
              datastoreReady:
                type: boolean
              variant:
                type: string
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: felixconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: FelixConfiguration
    listKind: FelixConfigurationList
    plural: felixconfigurations
    singular: felixconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              bpfConnectTimeLoadBalancingEnabled:
                type: boolean
              bpfDataIfacePattern:
                type: string
              bpfDisableUnprivileged:
                type: boolean
              bpfEnabled:
                type: boolean
              bpfExternalServiceMode:
                type: string
              bpfKubeProxyEndpointSlicesEnabled:
                type: boolean
              bpfKubeProxyIptablesCleanupEnabled:
                type: boolean
              bpfKubeProxyMinSyncPeriod:
                type: string
              bpfLogLevel:
                type: string
              chainInsertMode:
                type: string
              dataplaneDriver:
                type: string
              debugDisableLogDropping:
                type: boolean
              debugMemoryProfilePath:
                type: string
              debugSimulateCalcGraphHangAfter:
                type: string
              debugSimulateDataplaneHangAfter:
                type: string
              defaultEndpointToHostAction:
                type: string
              deviceRouteProtocol:
                type: integer
              deviceRouteSourceAddress:
                type: string
              disableConntrackInvalidCheck:
                type: boolean
              endpointReportingDelay:
                type: string
              endpointReportingEnabled:
                type: boolean
              externalNodesList:
                items:
                  type: string
                type: array
              failsafeInboundHostPorts:
                items:
                  properties:
                    port:
                      type: integer
                    protocol:
                      type: string
                  required:
                  - port
                  - protocol
                  type: object
                type: array
              failsafeOutboundHostPorts:
                items:
                  properties:
                    port:
                      type: integer
                    protocol:
                      type: string
                  required:
                  - port
                  - protocol
                  type: object
                type: array
              genericXDPEnabled:
                type: boolean
              healthEnabled:
                type: boolean
              healthHost:
                type: string
              healthPort:
                type: integer
              interfaceExclude:
                type: string
              interfacePrefix:
                type: string
              ipipEnabled:
                type: boolean
              ipipMTU:
                type: integer
              ipsetsRefreshInterval:
                type: string
              iptablesBackend:
                type: string
              iptablesFilterAllowAction:
                type: string
              iptablesLockFilePath:
                type: string
              iptablesLockProbeInterval:
                type: string
              iptablesLockTimeout:
                type: string
              iptablesMangleAllowAction:
                type: string
              iptablesMarkMask:
                format: int32
                type: integer
              iptablesNATOutgoingInterfaceFilter:
                type: string
              iptablesPostWriteCheckInterval:
                type: string
              iptablesRefreshInterval:
                type: string
              ipv6Support:
                type: boolean
              kubeNodePortRanges:
                items:
                  anyOf:
                  - type: integer
                  - type: string
                  pattern: ^.*
                  x-kubernetes-int-or-string: true
                type: array
              logFilePath:
                type: string
              logPrefix:
                type: string
              logSeverityFile:
                type: string
              logSeverityScreen:
                type: string
              logSeveritySys:
                type: string
              maxIpsetSize:
                type: integer
              metadataAddr:
                type: string
              metadataPort:
                type: integer
              natOutgoingAddress:
                type: string
              natPortRange:
                anyOf:
                - type: integer
                - type: string
                pattern: ^.*
                x-kubernetes-int-or-string: true
              netlinkTimeout:
                type: string
              openstackRegion:
                type: string
              policySyncPathPrefix:
                type: string
              prometheusGoMetricsEnabled:
                type: boolean
              prometheusMetricsEnabled:
                type: boolean
              prometheusMetricsHost:
                type: string
              prometheusMetricsPort:
                type: integer
              prometheusProcessMetricsEnabled:
                type: boolean
              removeExternalRoutes:
                type: boolean
              reportingInterval:
                type: string
              reportingTTL:
                type: string
              routeRefreshInterval:
                type: string
              routeSource:
                type: string
              routeTableRange:
                properties:
                  max:
                    type: integer
                  min:
                    type: integer
                required:
                - max
                - min
                type: object
              sidecarAccelerationEnabled:
                type: boolean
              usageReportingEnabled:
                type: boolean
              usageReportingInitialDelay:
                type: string
              usageReportingInterval:
                type: string
              useInternalDataplaneDriver:
                type: boolean
              vxlanEnabled:
                type: boolean
              vxlanMTU:
                type: integer
              vxlanPort:
                type: integer
              vxlanVNI:
                type: integer
              wireguardEnabled:
                type: boolean
              wireguardInterfaceName:
                type: string
              wireguardListeningPort:
                type: integer
              wireguardMTU:
                type: integer
              wireguardRoutingRulePriority:
                type: integer
              xdpEnabled:
                type: boolean
              xdpRefreshInterval:
                type: string
            required:
            - bpfLogLevel
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: globalnetworkpolicies.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: GlobalNetworkPolicy
    listKind: GlobalNetworkPolicyList
    plural: globalnetworkpolicies
    singular: globalnetworkpolicy
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              applyOnForward:
                type: boolean
              doNotTrack:
                type: boolean
              egress:
                items:
                  properties:
                    action:
                      type: string
                    destination:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                    http:
                      properties:
                        methods:
                          items:
                            type: string
                          type: array
                        paths:
                          items:
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    ipVersion:
                      type: integer
                    metadata:
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    notICMP:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              ingress:
                items:
                  properties:
                    action:
                      type: string
                    destination:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                    http:
                      properties:
                        methods:
                          items:
                            type: string
                          type: array
                        paths:
                          items:
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    ipVersion:
                      type: integer
                    metadata:
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    notICMP:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              namespaceSelector:
                type: string
              order:
                type: number
              preDNAT:
                type: boolean
              selector:
                type: string
              serviceAccountSelector:
                type: string
              types:
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: globalnetworksets.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: GlobalNetworkSet
    listKind: GlobalNetworkSetList
    plural: globalnetworksets
    singular: globalnetworkset
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              nets:
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: hostendpoints.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: HostEndpoint
    listKind: HostEndpointList
    plural: hostendpoints
    singular: hostendpoint
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              expectedIPs:
                items:
                  type: string
                type: array
              interfaceName:
                type: string
              node:
                type: string
              ports:
                items:
                  properties:
                    name:
                      type: string
                    port:
                      type: integer
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                  required:
                  - name
                  - port
                  - protocol
                  type: object
                type: array
              profiles:
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamblocks.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMBlock
    listKind: IPAMBlockList
    plural: ipamblocks
    singular: ipamblock
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              affinity:
                type: string
              allocations:
                items:
                  type: integer
                  nullable: true
                type: array
              attributes:
                items:
                  properties:
                    handle_id:
                      type: string
                    secondary:
                      additionalProperties:
                        type: string
                      type: object
                  type: object
                type: array
              cidr:
                type: string
              deleted:
                type: boolean
              strictAffinity:
                type: boolean
              unallocated:
                items:
                  type: integer
                type: array
            required:
            - allocations
            - attributes
            - cidr
            - deleted
            - strictAffinity
            - unallocated
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamconfigs.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMConfig
    listKind: IPAMConfigList
    plural: ipamconfigs
    singular: ipamconfig
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              autoAllocateBlocks:
                type: boolean
              strictAffinity:
                type: boolean
            required:
            - autoAllocateBlocks
            - strictAffinity
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ipamhandles.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPAMHandle
    listKind: IPAMHandleList
    plural: ipamhandles
    singular: ipamhandle
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              block:
                additionalProperties:
                  type: integer
                type: object
              handleID:
                type: string
            required:
            - block
            - handleID
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: ippools.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: IPPool
    listKind: IPPoolList
    plural: ippools
    singular: ippool
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              blockSize:
                type: integer
              cidr:
                type: string
              disabled:
                type: boolean
              ipip:
                properties:
                  enabled:
                    type: boolean
                  mode:
                    type: string
                type: object
              ipipMode:
                type: string
              nat-outgoing:
                type: boolean
              natOutgoing:
                type: boolean
              nodeSelector:
                type: string
              vxlanMode:
                type: string
            required:
            - cidr
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: kubecontrollersconfigurations.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: KubeControllersConfiguration
    listKind: KubeControllersConfigurationList
    plural: kubecontrollersconfigurations
    singular: kubecontrollersconfiguration
  scope: Cluster
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              controllers:
                properties:
                  namespace:
                    properties:
                      reconcilerPeriod:
                        type: string
                    type: object
                  node:
                    properties:
                      hostEndpoint:
                        properties:
                          autoCreate:
                            type: string
                        type: object
                      reconcilerPeriod:
                        type: string
                      syncLabels:
                        type: string
                    type: object
                  policy:
                    properties:
                      reconcilerPeriod:
                        type: string
                    type: object
                  serviceAccount:
                    properties:
                      reconcilerPeriod:
                        type: string
                    type: object
                  workloadEndpoint:
                    properties:
                      reconcilerPeriod:
                        type: string
                    type: object
                type: object
              etcdV3CompactionPeriod:
                type: string
              healthChecks:
                type: string
              logSeverityScreen:
                type: string
            required:
            - controllers
            type: object
          status:
            properties:
              environmentVars:
                additionalProperties:
                  type: string
                type: object
              runningConfig:
                properties:
                  controllers:
                    properties:
                      namespace:
                        properties:
                          reconcilerPeriod:
                            type: string
                        type: object
                      node:
                        properties:
                          hostEndpoint:
                            properties:
                              autoCreate:
                                type: string
                            type: object
                          reconcilerPeriod:
                            type: string
                          syncLabels:
                            type: string
                        type: object
                      policy:
                        properties:
                          reconcilerPeriod:
                            type: string
                        type: object
                      serviceAccount:
                        properties:
                          reconcilerPeriod:
                            type: string
                        type: object
                      workloadEndpoint:
                        properties:
                          reconcilerPeriod:
                            type: string
                        type: object
                    type: object
                  etcdV3CompactionPeriod:
                    type: string
                  healthChecks:
                    type: string
                  logSeverityScreen:
                    type: string
                required:
                - controllers
                type: object
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: networkpolicies.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: NetworkPolicy
    listKind: NetworkPolicyList
    plural: networkpolicies
    singular: networkpolicy
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              egress:
                items:
                  properties:
                    action:
                      type: string
                    destination:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                    http:
                      properties:
                        methods:
                          items:
                            type: string
                          type: array
                        paths:
                          items:
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    ipVersion:
                      type: integer
                    metadata:
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    notICMP:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              ingress:
                items:
                  properties:
                    action:
                      type: string
                    destination:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                    http:
                      properties:
                        methods:
                          items:
                            type: string
                          type: array
                        paths:
                          items:
                            properties:
                              exact:
                                type: string
                              prefix:
                                type: string
                            type: object
                          type: array
                      type: object
                    icmp:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    ipVersion:
                      type: integer
                    metadata:
                      properties:
                        annotations:
                          additionalProperties:
                            type: string
                          type: object
                      type: object
                    notICMP:
                      properties:
                        code:
                          type: integer
                        type:
                          type: integer
                      type: object
                    notProtocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    protocol:
                      anyOf:
                      - type: integer
                      - type: string
                      pattern: ^.*
                      x-kubernetes-int-or-string: true
                    source:
                      properties:
                        namespaceSelector:
                          type: string
                        nets:
                          items:
                            type: string
                          type: array
                        notNets:
                          items:
                            type: string
                          type: array
                        notPorts:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        notSelector:
                          type: string
                        ports:
                          items:
                            anyOf:
                            - type: integer
                            - type: string
                            pattern: ^.*
                            x-kubernetes-int-or-string: true
                          type: array
                        selector:
                          type: string
                        serviceAccounts:
                          properties:
                            names:
                              items:
                                type: string
                              type: array
                            selector:
                              type: string
                          type: object
                      type: object
                  required:
                  - action
                  type: object
                type: array
              order:
                type: number
              selector:
                type: string
              serviceAccountSelector:
                type: string
              types:
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []

---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: networksets.crd.projectcalico.org
spec:
  group: crd.projectcalico.org
  names:
    kind: NetworkSet
    listKind: NetworkSetList
    plural: networksets
    singular: networkset
  scope: Namespaced
  versions:
  - name: v1
    schema:
      openAPIV3Schema:
        properties:
          apiVersion:
            type: string
          kind:
            type: string
          metadata:
            type: object
          spec:
            properties:
              nets:
                items:
                  type: string
                type: array
            type: object
        type: object
    served: true
    storage: true
status:
  acceptedNames:
    kind: ""
    plural: ""
  conditions: []
  storedVersions: []
              
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
  name: calico-node
  namespace: kube-system
  labels:
    k8s-app: calico-node
spec:
  selector:
    matchLabels:
      k8s-app: calico-node
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        k8s-app: calico-node
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      hostNetwork: true
      tolerations:
        - effect: NoSchedule
          operator: Exists
        - key: CriticalAddonsOnly
          operator: Exists
        - effect: NoExecute
          operator: Exists
      serviceAccountName: calico-node
      terminationGracePeriodSeconds: 0
      priorityClassName: system-node-critical
      initContainers:
        - name: upgrade-ipam
          image: calico/cni:v3.15.1
          command: ["/opt/cni/bin/calico-ipam", "-upgrade"]
          env:
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
          volumeMounts:
            - mountPath: /var/lib/cni/networks
              name: host-local-net-dir
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
          securityContext:
            privileged: true
        - name: install-cni
          image: calico/cni:v3.15.1
          command: ["/install-cni.sh"]
          env:
            - name: CNI_CONF_NAME
              value: "10-calico.conflist"
            - name: CNI_NETWORK_CONFIG
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: cni_network_config
            - name: KUBERNETES_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CNI_MTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: SLEEP
              value: "false"
          volumeMounts:
            - mountPath: /host/opt/cni/bin
              name: cni-bin-dir
            - mountPath: /host/etc/cni/net.d
              name: cni-net-dir
          securityContext:
            privileged: true
        - name: flexvol-driver
          image: calico/pod2daemon-flexvol:v3.15.1
          volumeMounts:
          - name: flexvol-driver-host
            mountPath: /host/driver
          securityContext:
            privileged: true
      containers:
        - name: calico-node
          image: calico/node:v3.15.1
          env:
            - name: DATASTORE_TYPE
              value: "kubernetes"
            - name: WAIT_FOR_DATASTORE
              value: "true"
            - name: NODENAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: CALICO_NETWORKING_BACKEND
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: calico_backend
            - name: CLUSTER_TYPE
              value: "k8s,bgp"
            - name: IP
              value: "autodetect"
            - name: CALICO_IPV4POOL_IPIP
              value: "Always"
            - name: CALICO_IPV4POOL_VXLAN
              value: "Never"
            - name: FELIX_IPINIPMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: FELIX_VXLANMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: FELIX_WIREGUARDMTU
              valueFrom:
                configMapKeyRef:
                  name: calico-config
                  key: veth_mtu
            - name: CALICO_IPV4POOL_CIDR
              value: "172.10.0.0/16"                #与前面定义的pod的CIDR保持一致
            - name: CALICO_DISABLE_FILE_LOGGING
              value: "true"
            - name: FELIX_DEFAULTENDPOINTTOHOSTACTION
              value: "ACCEPT"
            - name: FELIX_IPV6SUPPORT
              value: "false"
            - name: FELIX_LOGSEVERITYSCREEN
              value: "info"
            - name: FELIX_HEALTHENABLED
              value: "true"
          securityContext:
            privileged: true
          resources:
            requests:
              cpu: 250m
          livenessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-live
              - -bird-live
            periodSeconds: 10
            initialDelaySeconds: 10
            failureThreshold: 6
          readinessProbe:
            exec:
              command:
              - /bin/calico-node
              - -felix-ready
              - -bird-ready
            periodSeconds: 10
          volumeMounts:
            - mountPath: /lib/modules
              name: lib-modules
              readOnly: true
            - mountPath: /run/xtables.lock
              name: xtables-lock
              readOnly: false
            - mountPath: /var/run/calico
              name: var-run-calico
              readOnly: false
            - mountPath: /var/lib/calico
              name: var-lib-calico
              readOnly: false
            - name: policysync
              mountPath: /var/run/nodeagent
      volumes:
        - name: lib-modules
          hostPath:
            path: /lib/modules
        - name: var-run-calico
          hostPath:
            path: /var/run/calico
        - name: var-lib-calico
          hostPath:
            path: /var/lib/calico
        - name: xtables-lock
          hostPath:
            path: /run/xtables.lock
            type: FileOrCreate
        - name: cni-bin-dir
          hostPath:
            path: /opt/cni/bin
        - name: cni-net-dir
          hostPath:
            path: /etc/cni/net.d
        - name: host-local-net-dir
          hostPath:
            path: /var/lib/cni/networks
        - name: policysync
          hostPath:
            type: DirectoryOrCreate
            path: /var/run/nodeagent
        - name: flexvol-driver-host
          hostPath:
            type: DirectoryOrCreate
            path: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/nodeagent~uds
            
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: calico-kube-controllers
  namespace: kube-system
  labels:
    k8s-app: calico-kube-controllers
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: calico-kube-controllers
  strategy:
    type: Recreate
  template:
    metadata:
      name: calico-kube-controllers
      namespace: kube-system
      labels:
        k8s-app: calico-kube-controllers
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      tolerations:
        - key: CriticalAddonsOnly
          operator: Exists
        - key: node-role.kubernetes.io/master
          effect: NoSchedule
      serviceAccountName: calico-kube-controllers
      priorityClassName: system-cluster-critical
      containers:
        - name: calico-kube-controllers
          image: calico/kube-controllers:v3.15.1
          env:
            - name: ENABLED_CONTROLLERS
              value: node
            - name: DATASTORE_TYPE
              value: kubernetes
          readinessProbe:
            exec:
              command:
              - /usr/bin/check-status
              - -r
#所有节点拉取镜像

docker pull calico/kube-controllers:v3.15.1

docker pull calico/cni:v3.15.1

docker pull calico/pod2daemon-flexvol:v3.15.1

docker pull calico/node:v3.15.1
kubectl apply -f /etc/kubernetes/addons/calico-rbac.yaml

kubectl apply -f /etc/kubernetes/addons/calico.yaml

node加入集群

  • node加入(所有node节点执行):
kubeadm join 192.168.30.128:6443 --token 1ndel7.xb623vep9pl5o6vl \
    --discovery-token-ca-cert-hash sha256:0e41f6020955c36970bf504cbfc0047941240dda57ebb9d85086706da14dcd1f
  • 查看集群节点:
kubectl edit cm kube-proxy -n kube-system               #修改mode为ipvs

kubectl get pod -n kube-system

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-578894d4cd-vtv6z   1/1     Running   0          68m
calico-node-6qlsv                          1/1     Running   0          68m
calico-node-c5nz9                          1/1     Running   0          68m
calico-node-j2b2q                          1/1     Running   0          68m
calico-node-sphbr                          1/1     Running   0          68m
coredns-546565776c-fdzsv                   1/1     Running   0          4m27s
coredns-546565776c-z7tzl                   1/1     Running   0          4m1s
etcd-master                                1/1     Running   0          29h
kube-apiserver-master                      1/1     Running   0          29h
kube-controller-manager-master             1/1     Running   0          29h
kube-proxy-k4tj5                           1/1     Running   0          12m
kube-proxy-nl5wd                           1/1     Running   0          11m
kube-proxy-rzz9c                           1/1     Running   0          12m
kube-proxy-ssgc5                           1/1     Running   0          12m
kube-scheduler-master                      1/1     Running   0          29h

kubectl get nodes

NAME     STATUS   ROLES    AGE   VERSION
master   Ready    master   28h   v1.18.3
node1    Ready    <none>   26h   v1.18.3
node2    Ready    <none>   26h   v1.18.3
node3    Ready    <none>   26h   v1.18.3

可以看到,整个集群有1个master节点和3个node节点,都处于ready状态。


可用性测试

集群已经初步搭建起来,下面进行集群的可用性测试。

一个DaemonSet对象能确保其创建的Pod在集群中的每一台(或指定)Node上都运行一个副本。如果集群中动态加入了新的Node,DaemonSet中的Pod也会被添加在新加入的Node上运行。删除一个DaemonSet也会级联删除所有其创建的Pod。

因此,创建一个DaemonSet对象来测试可用性比较合适。

  • 创建nginx daemonset:
vim /software/nginx-ds.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx-ds
  labels:
    app: nginx-ds
spec:
  type: NodePort
  selector:
    app: nginx-ds
  ports:
  - name: http
    port: 80
    targetPort: 80

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: nginx-ds
spec:
  selector:
    matchLabels:
      app: nginx-ds
  template:
    metadata:
      labels:
        app: nginx-ds
    spec:
      containers:
      - name: my-nginx
        image: jbednarik/nginx-ping:latest
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80
kubectl apply -f /software/nginx-ds.yaml
  • 检查ip连通性:
kubectl get pods -o wide

NAME             READY   STATUS    RESTARTS   AGE   IP               NODE    NOMINATED NODE   READINESS GATES
nginx-ds-5mm88   1/1     Running   0          68m   172.10.135.1     node3   <none>           <none>
nginx-ds-db2wt   1/1     Running   0          68m   172.10.166.129   node1   <none>           <none>
nginx-ds-zwfrh   1/1     Running   0          68m   172.10.104.2     node2   <none>           <none>

kubectl get svc

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        19m
nginx-ds     NodePort    10.104.212.162   <none>        80:32329/TCP   2m22s

在每个节点上ping pod ip,同时访问服务ip及其端口,在每个节点检查node-port可用性。

  • 检查dns可用性:
vim /software/nginx-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: jbednarik/nginx-ping:latest
    imagePullPolicy: IfNotPresent
    ports:
    - containerPort: 80
kubectl apply -f /software/nginx-pod.yaml

kubectl exec -it nginx ping nginx-ds

PING nginx-ds.default.svc.cluster.local (10.104.212.162) 56(84) bytes of data.
64 bytes from nginx-ds.default.svc.cluster.local (10.104.212.162): icmp_seq=1 ttl=64 time=0.042 ms
64 bytes from nginx-ds.default.svc.cluster.local (10.104.212.162): icmp_seq=2 ttl=64 time=0.066 ms
64 bytes from nginx-ds.default.svc.cluster.local (10.104.212.162): icmp_seq=3 ttl=64 time=0.075 ms
64 bytes from nginx-ds.default.svc.cluster.local (10.104.212.162): icmp_seq=4 ttl=64 time=0.092 ms
64 bytes from nginx-ds.default.svc.cluster.local (10.104.212.162): icmp_seq=5 ttl=64 time=0.073 ms

kubectl get svc

NAME         TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)        AGE
kubernetes   ClusterIP   10.96.0.1        <none>        443/TCP        19m
nginx-ds     NodePort    10.104.212.162   <none>        80:32329/TCP   2m22s

可以看到,在nginx pod中ping nginx-ds时dns解析没问题,返回的是nginx-ds的cluster-ip。这说明之前搭建的集群正常可用。


部署ingress-nginx

  • 部署ingress-nginx:
vim /etc/kubernetes/addons/ingress-nginx.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx

---
apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: default-http-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app.kubernetes.io/name: default-http-backend
    app.kubernetes.io/part-of: ingress-nginx
    
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: default-http-backend
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: default-http-backend
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: default-http-backend
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: default-http-backend
        app.kubernetes.io/part-of: ingress-nginx
    spec:
      terminationGracePeriodSeconds: 60
      containers:
        - name: default-http-backend
          image: k8s.gcr.io/defaultbackend-amd64:1.5
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 10m
              memory: 20Mi
            requests:
              cpu: 10m
              memory: 20Mi
          livenessProbe:
            httpGet:
              path: /healthz
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx
    
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: https
      port: 443
      targetPort: https
      
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  revisionHistoryLimit: 2147483647
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      containers:
      - name: nginx-ingress-controller
        image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
        imagePullPolicy: IfNotPresent
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --configmap=$(POD_NAMESPACE)/nginx-configuration
        - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
        - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
        - --publish-service=$(POD_NAMESPACE)/ingress-nginx
        - --annotations-prefix=nginx.ingress.kubernetes.io
        env:
        - name: POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
        ports:
        - containerPort: 80
          hostPort: 80
          name: http
          protocol: TCP
        - containerPort: 443
          hostPort: 443
          name: https
          protocol: TCP
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext:
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - ALL
          procMount: Default
          runAsUser: 33
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      hostNetwork: true
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: nginx-ingress-serviceaccount
      serviceAccountName: nginx-ingress-serviceaccount
      terminationGracePeriodSeconds: 30
#node节点拉取镜像

docker pull k8s.gcr.io/defaultbackend-amd64:1.5

docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
kubectl apply -f /etc/kubernetes/addons/ingress-nginx.yaml

kubectl get all -n ingress-nginx

NAME                                        READY   STATUS    RESTARTS   AGE
pod/default-http-backend-6bf4c44778-n5b55   1/1     Running   0          3m35s
pod/ingress-nginx-2fkhl                     1/1     Running   0          3m34s
pod/ingress-nginx-lbvrv                     1/1     Running   0          3m34s
pod/ingress-nginx-sz89b                     1/1     Running   0          3m34s

NAME                           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
service/default-http-backend   ClusterIP   10.106.154.51    <none>        80/TCP           3m35s
service/ingress-nginx          ClusterIP   10.109.151.107   <none>        80/TCP,443/TCP   3m34s

NAME                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/ingress-nginx   3         3         3       3            3           <none>          3m34s

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/default-http-backend   1/1     1            1           3m35s

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/default-http-backend-6bf4c44778   1         1         1       3m35s
  • 使用测试:
vim /software/tomcat-demo.yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: tomcat-demo
spec:
  rules:
  - host: tomcat.lzxlinux.cn
    http:
      paths:
      - path: /
        backend:
          serviceName: tomcat-demo
          servicePort: 80
          
---
apiVersion: v1
kind: Service
metadata:
  name: tomcat-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: tomcat-demo
          
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: tomcat-demo
spec:
  selector:
    matchLabels:
      app: tomcat-demo
  replicas: 1
  template:
    metadata:
      labels:
        app: tomcat-demo
    spec:
      containers:
      - name: tomcat-demo
        image: registry.cn-hangzhou.aliyuncs.com/liuyi01/tomcat:8.0.51-alpine
        ports:
        - containerPort: 8080
kubectl apply -f /software/tomcat-demo.yaml

kubectl get ing

NAME          CLASS    HOSTS                ADDRESS   PORTS   AGE
tomcat-demo   <none>   tomcat.lzxlinux.cn             80      2m46s

任选一node节点ip,在Windows电脑hosts文件中添加本地dns:

192.168.30.129 tomcat.lzxlinux.cn
192.168.30.129 api.lzxlinux.cn

在这里插入图片描述

在这里插入图片描述

可以看到,因为在yaml文件中有定义,所以通过域名可以访问到集群内的名为tomcat-demo(端口为80)的服务,否则返回默认后端404。

ingres-nginx部署完成,kubernetes集群搭建完成。



本文链接: http://www.dtmao.cc/news_show_100038.shtml

附件下载

相关教程

    暂无相关的数据...

共有条评论 网友评论

验证码: 看不清楚?