Skip to content
Snippets Groups Projects
Select Git revision
  • master default protected
1 result
Compare
Name Last commit Last update
externalDNS
ingressDNSRecord
resource
README.md
README.md

Kubernetes Cluster Federation (KubeFed)

Documentation

Below some useful links for the main tools of a Kubernetes federation:

If you want create the Kubernetes clusters on Openstack and you use Juju tool to deploy your applications, you can use this Juju bundle: https://jaas.ai/u/csd-garr/kubernetes-openstack-integrator/bundle.

Environment configuration

Before starting it is necessary create several Kubernetes clusters to make the federation. One cluster will be the Host (Host Cluster), in which will be install the KubeFed (Control Plane), and the remaining clusters will be the Members (Member Cluster). To create the clusters on Openstack, it is possible to use the Juju bundle in the Documention section.

Once all the clusters for federation have been created, it is useful to collect all cluster access configurations (.kube/config) in a single config file. In this way you can easily switch from one cluster to another. Below an example:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-host
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-member1
- cluster:
    certificate-authority-data: LS0tLS1CRUdJT...
    server: https://<ip>:443
  name: k8s-cluster-member2
contexts:
- context:
    cluster: k8s-cluster-host
    user: admin-host
  name: k8s-context-host
- context:
    cluster: k8s-cluster-member1
    user: admin-member1
  name: k8s-context-member1
- context:
    cluster: k8s-cluster-member2
    user: admin-member2
  name: k8s-context-member2
current-context: k8s-context-host
kind: Config
preferences: {}
users:
- name: admin-host
  user:
    password: 8Rvl5qW...
    username: admin
- name: admin-member1
  user:
    password: 0JefxZq...
    username: admin
- name: admin-member2
  user:
    password: hJLiVLI...
    username: admin

In this way you have access to all the clusters by the context switch, below are the commands for the context switch:

list contexts:

$ kubectl config get-contexts

switch context:

$ kubectl config use-context <context-name>

Installation

KubeFed client (kubefedctl)

To install the KubeFed client run the following commands in your local machine:

$ VERSION=0.2.0-alpha.1
$ OS=linux
$ ARCH=amd64
$ curl -LO https://github.com/kubernetes-sigs/kubefed/releases/download/v${VERSION}/kubefedctl-${VERSION}-${OS}-${ARCH}.tgz
$ tar -zxvf kubefedctl-*.tgz
$ chmod u+x kubefedctl
$ sudo mv kubefedctl /usr/local/bin/ # make sure the location is in the PATH

N.B.: the versions we tested are: 0.1.0-rc6 and 0.2.0-alpha.1, you can try the latest version.

KubeFed server (Control Plane)

To install the Controll Plane, switch to the host context and configuring the RBAC role for Tiller (in this case we use Helm v2):

$ cat << EOF | kubectl apply -f -
	apiVersion: v1
	kind: ServiceAccount
	metadata:
	  name: tiller
	  namespace: kube-system
	---
	apiVersion: rbac.authorization.k8s.io/v1
	kind: ClusterRoleBinding
	metadata:
	  name: tiller
	roleRef:
	  apiGroup: rbac.authorization.k8s.io
	  kind: ClusterRole
	  name: cluster-admin
	subjects:
	  - kind: ServiceAccount
	    name: tiller
	    namespace: kube-system
	EOF

helm init command:

$ helm init --service-account tiller

Now it is possible install the Control Plane on the Host Cluster:

$ helm repo add kubefed-charts https://raw.githubusercontent.com/kubernetes-sigs/kubefed/master/charts
$ helm install kubefed-charts/kubefed --name kubefed --version=0.2.0-alpha.1 --namespace kube-federation-system

N.B.: use the same version for client and server.

N.B.: in this way the Control Plane can find Federated resources in each namespace, to limit this action in a specific namespace add the parameter --set global.scope=Namespaced.

Create the federation

To create a Kubernetes Federation it is necessary federate Member Clusters. Use the KubeFed join command:

$ kubefedctl join fed-cluster-member1 --cluster-context k8s-context-member1 --host-cluster-context k8s-context-host --v=2
$ kubefedctl join fed-cluster-member2 --cluster-context k8s-context-member2 --host-cluster-context k8s-context-host --v=2

Remove cluster to the federation, use the unjoin command:

$ kubefedctl unjoin fed-cluster-member1 --cluster-context k8s-context-member1 --host-cluster-context k8s-context-host --v=2
$ kubefedctl unjoin fed-cluster-member2 --cluster-context k8s-context-member2 --host-cluster-context k8s-context-host --v=2

Command to show the federated clusters:

$ kubectl -n kube-federation-system get kubefedclusters

Enable API

To enable a type of resource to be federated it is necessary to use the enable command of kubefedctl:

$ kubefedctl enable <api-resources-name.api-group> --kubefed-namespace kube-federation-system

To know the list of resourses:

$ kubectl api-resources -o wide

Some examples:

$ kubefedctl enable deployments.apps --kubefed-namespace kube-federation-system
$ kubefedctl enable services --kubefed-namespace kube-federation-system
$ kubefedctl enable ingresses.extensions --kubefed-namespace kube-federation-system

Command to disable federated API:

$ kubefedctl disable services --kubefed-namespace kube-federation-system

N.B.: Do not federate the “kind: pod” resource. In this case, federating a whole namespace will federate pods and deployments at the same time. Deployments will create replicasets which in turn create pods. It will result a duplication of the pods resources.

Federate an application

Below the procedure to deploy and federate an application and enable ExternalDNS.

N.B.: all commands must be run on Host Cluster.

Create namespace

The first step is create a namespace in the host cluster:

command line:

$ kubectl create namespace <namespace-name>

or create it by yaml file (resource/namespace.yaml):

# namespace.yaml file
apiVersion: v1
kind: Namespace
metadata:
  name: fed-namespace
$ kubectl create -f resource/namespace.yaml

Federate namespace

Once the namespace is created in the Host Cluster, it can be federate on the Member Cluster:

# federated_namespace.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedNamespace
metadata:
  name: fed-namespace
  namespace: fed-namespace
spec:
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2
$ kubectl --context=<host-cluster-context> create -f resource/federated_namespace.yaml

N.B.: the option --context is not necessary but make sure that the right context is selected.

Create RBAC for ExternalDNS

Now it is possible to deploy ExternalDNS in the federated namespace. Create the RBAC for the ExternalDNS:

# rbac_externaldns.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-dns
  namespace: fed-namespace
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: external-dns
  namespace: fed-namespace
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: external-dns-viewer
  namespace: fed-namespace
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: external-dns
subjects:
- kind: ServiceAccount
  name: external-dns
  namespace: fed-namespace
$ kubectl --context=<host-cluster-context> create -f resource/rbac_externaldns.yaml

Deploy ExternalDNS for PowerDNS

Create the ExternalDNS deployment and configure it for PowerDNS (in the our case):

# externaldns.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: external-dns
  namespace: fed-namespace
spec:
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: external-dns
  template:
    metadata:
      labels:
        app: external-dns
    spec:
      # Only use if you're also using RBAC
      serviceAccountName: external-dns
      containers:
      - name: external-dns
        image: registry.opensource.zalan.do/teapot/external-dns:latest
        args:
        - --source=crd # or service or ingress
        - --crd-source-apiversion=multiclusterdns.kubefed.io/v1alpha1
        - --crd-source-kind=DNSEndpoint
        - --provider=pdns
        - --pdns-server=http://<ip>:<port>
        - --pdns-api-key=<api-key>
        - --registry=txt
        - --txt-prefix=cname
        - --domain-filter=<domain> # will make ExternalDNS see only the zones matching provided domain; omit to process all available zones in PowerDNS 
        - --policy=upsert-only # would prevent ExternalDNS from deleting any records, omit to enable full synchronization

$ kubectl --context=<host-cluster-context> create -f resource/externaldns.yaml

The previous configuration is specific to integrate ExternalDNS with the Multi-Cluster Ingress DNS , in this way it will be possible to obtain the ingress information through the "DNSEndpoint" object. The DNSEndpoint object will be created when the Multi-Cluster Ingress DNS is configured (see below).

The main configuration are:

  • source: the field represents the type of resources for which it is necessary create a DNS record. For example: service or ingress. In this case the source value is crd for Multi-Cluster Ingress DNS integration;
  • crd-source-apiversion: the field specifies the api for the crd source (required to Multi-Cluster Ingress DNS);
  • crd-source-kind: specifies the type of resource which will contain the service or ingress endpoint of the member clusters (required to Multi-Cluster Ingress DNS);
  • provider: represents the DNS Provider (pdns for PowerDNS);
  • pdns-server: it is the DNS server endpoint;
  • pdns-api-key: the API key for server authentication;
  • registry: specifies the type of the mechanism for safeguarding records that do not depend on ExternalDNS (required to Multi-Cluster Ingress DNS);
  • txt-prefix: required to Multi-Cluster Ingress DNS;
  • domain-filter: will make ExternalDNS see only the zones matching provided domain;
  • policy: are the permissions with which External DNS can operate on PowerDNS.

For more details see: https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/pdns.md.

Federate deployment

Now create the federated deployment (for this example we used a simple "Hello World" image):

# federated_deployment.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: fed-helloworld
  namespace: fed-namespace
spec:
  template:
    metadata:
      name: helloworld
    spec:
      replicas: 2
      selector:
        matchLabels:
          app: helloworld
      template:
        metadata:
          labels:
            app: helloworld
        spec:
          containers:
          - image: docker.io/csdgarr/hello-world:v1
            name: helloworld
            imagePullPolicy: IfNotPresent
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2
  overrides:
    - clusterName: member-cluster-2
      clusterOverrides:
      - path: "/spec/replicas"
        value: 3

$ kubectl --context=<host-cluster-context> create -f resource/federated_deployment.yaml

In this way the deployment resource will create inside the member clusters and it will create the pods for your application or image. In the previous example we add the overrides field to increse the number of pods in the member-cluster-2.

Federate service

Create the service for the "fed-helloworld" deployment:

# federated_service.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedService
metadata:
  name: fed-helloworld-service
  namespace: fed-namespace
spec:
  template:
    metadata:
      name: fed-helloworld-service
      labels:
        app: helloworld
    spec:
      selector:
        app: helloworld
      type: ClusterIP
      ports:
      - name: http
        port: 8080
        targetPort: 8080
        protocol: TCP
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2

$ kubectl --context=<host-cluster-context> create -f resource/federated_service.yaml

This is a ClusterIP service, in this way the your application is not exposed and the service have a private cluster IP. Now it is necessary that the service is exposed by an ingress resource.

Federate ingress

Create the ingress resource to expose the fed-helloworld-service:

# federated_ingress.yaml
apiVersion: types.kubefed.io/v1beta1
kind: FederatedIngress
metadata:
  name: fed-helloworld-ingress
  namespace: fed-namespace
spec:
  template:
    metadata:
      name: helloworld-ingress
      labels:
        app: helloworld
    spec:
      rules:
      - host: helloworld.<domain>
        http:
          paths:
          - path: / 
            backend:
              serviceName: fed-helloworld-service
              servicePort: 8080
  placement:
    clusters:
    - name: member-cluster-1
    - name: member-cluster-2

$ kubectl --context=<host-cluster-context> create -f resource/federated_ingress.yaml

In the "host" field the "domain" string must be the same that you insert in the "domain-filter" field for ExternalDNS configuration.

List federated resources

If you want to show or describe the federated resource that have been propagated on the Member Cluster, you can run the follow command:

$ kubectl describe federated<resource-type>.types.kubefed.io/<resource-name> --namespace <namespace>

Multi-Cluster Ingress DNS

One the service and the ingress are deployed the Multi-Cluster Ingress DNS can be run. To run this meccanism and use the ExternalDNS to configure your DNS Provider, it is necessary create an IngressDNSRecord.

Create IngressDNSRecord

The IngressDNSRecord resource can be created in the follow way:

# ingressdnsrecord.yaml
apiVersion: multiclusterdns.kubefed.io/v1alpha1
kind: IngressDNSRecord
metadata:
  name: fed-helloworld-ingress
  namespace: fed-namespace
spec:
  hosts:
  - helloworld.<domain>
  recordTTL: 300
$ kubectl --context=<host-cluster-context> create -f resource/ingressdnsrecord.yaml

The name of IngressDNSRecord must be the same of the ingress resource and the value in "hosts" field must matches the "host" field in the ingress configuration.

Now the Multi-Cluster Ingress DNS mechanism can start to retrive the ingress endpoints and the ExternalDNS can configure your DNS Provider.