Labels

Kubernetes Cluster using RKE2 HA and Rancher

I will show how to install kubernetes cluster with High Availibility using RKE2 and install Rancher to manage the cluster under ubuntu. To simplify access to the application deployed in the cluster, I set up nginx as load balancer to forward domain name request to the application. Here, I use simple ansible script to install the cluster.

Server Architecture

Ansible host configuration

In this example, my environment is behind the proxy, so you will see proxy configuration in the playbook. I assume, you already know how to use ansible, so this tutorial will not explain about that.

Create file rke2-hosts with this content
[allnode:children]
rke2-firstnode
rke2-nextnode

[rke2-firstnode]
rke2-node1 ansible_host=192.168.1.11

[rke2-nextnode]
rke2-node2 ansible_host=192.168.1.12
rke2-node3 ansible_host=192.168.1.13
Create ansible playbook, rke2-ha.yml
---
- hosts: allnode
  become: true
  tasks:
    - name: Copy Proxy Config
      copy:
        src: proxy
        dest: /etc/default/proxy
        mode: '0644'
      tags: [ 'envproxy', 'proxy' ]
    - name: Copy APT Proxy Config
      copy:
        src: proxy.conf
        dest: /etc/apt/apt.conf.d/proxy.conf
        mode: '0644'
      tags: [ 'aptproxy', 'proxy' ]
    - name: Insert Proxy Env
      lineinfile:
        path: /etc/environment
        line: '{{ item }}'
      with_items:
        - 'HTTP_PROXY="http://192.168.1.1:3128"'
        - 'HTTPS_PROXY="http://192.168.1.1:3128"'
        - 'NO_PROXY="127.0.0.0/8,10.0.0.0/8,192.168.1.0/16,*.mydomain.com"'
      tags: [ 'envproxy', 'proxy' ]

- hosts: rke2-firstnode
  become: true
  tasks:
    - name: create directory with parent directories
      file:
        path: /etc/rancher/rke2
        state: directory
        recurse: yes
      tags: [ 'rke-config']
    - name: create directory with parent directories
      file:
        path: /var/lib/rancher/rke2/server/manifests
        state: directory
        recurse: yes
      tags: [ 'rke-config']
    - name: Copy Master Config
      copy:
        src: config-master.yaml
        dest: /etc/rancher/rke2/config.yaml
        mode: '0644'
      tags: [ 'master','rke-config']
    - name: Run RKE2 script
      shell:
        cmd: curl -sfL https://get.rke2.io | sh -
        warn: False
      tags: [ 'download']
    - name: Add proxy in rke init script
      lineinfile:
        path: /usr/local/lib/systemd/system/rke2-server.service
        insertbefore: 'EnvironmentFile=-/etc/default/%N'
        line: EnvironmentFile=/etc/default/proxy
    - name: Copy Nginx Config
      copy:
        src: rke2-ingress-nginx-custom.yaml
        dest: /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-custom.yaml
        mode: '0644'
      tags: [ 'master','rke-config']
    - name: Enable RKE2 script
      shell:
        cmd: systemctl enable rke2-server.service
        warn: False
    - name: Reload systemd
      shell:
        cmd: systemctl daemon-reload
        warn: False
    - name: Start RKE2 script
      shell:
        cmd: systemctl start rke2-server.service
        warn: False

- hosts: rke2-nextnode
  become: true
  tasks:
    - name: create directory with parent directories
      file:
        path: /etc/rancher/rke2
        state: directory
        recurse: yes
    - name: create directory with parent directories
      file:
        path: /var/lib/rancher/rke2/server/manifests
        state: directory
        recurse: yes
      tags: [ 'rke-config']
    - name: Copy Slave Config
      copy:
        src: config-slave.yaml
        dest: /etc/rancher/rke2/config.yaml
        mode: '0644'
      tags: [ 'slave','rke-config']
    - name: Run RKE2 script
      shell:
        cmd: curl -sfL https://get.rke2.io | sh -
        warn: False
      tags: [ 'download']
    - name: Add proxy in rke init script
      lineinfile:
        path: /usr/local/lib/systemd/system/rke2-server.service
        insertbefore: 'EnvironmentFile=-/etc/default/%N'
        line: EnvironmentFile=/etc/default/proxy
    - name: Copy Nginx Config
      copy:
        src: rke2-ingress-nginx-custom.yaml
        dest: /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx-custom.yaml
        mode: '0644'
      tags: [ 'master','rke-config']
    - name: Enable RKE2 script
      shell:
        cmd: systemctl enable rke2-server.service
        warn: False
    - name: Reload systemd
      shell:
        cmd: systemctl daemon-reload
        warn: False
    - name: Start RKE2 script
      shell:
        cmd: systemctl start rke2-server.service
        warn: False
You need to create these following file and put in the same directory with ansible playbook:
  • proxy
  • proxy.conf
  • config-master.yaml
  • rke2-ingress-nginx-custom.yaml
Content of proxy file:
HTTP_PROXY=http://192.168.1.1:3128
HTTPS_PROXY=http://192.168.1.1:3128
NO_PROXY=127.0.0.0/8,10.0.0.0/8,192.168.1.0/16,*.mydomain.com
Content of proxy.conf file:
Acquire::http::Proxy "http://192.168.1.1:3128/";
Acquire::https::Proxy "https://192.168.1.1:3128/";
Content of config-master.yaml file:
tls-san:
  - "mydomain.com"
  - "192.168.1.11"
  - "192.168.1.12"
  - "192.168.1.13"
Content of rke2-ingress-nginx-custom.yaml file:
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      kind: DaemonSet
      daemonset:
        useHostPort: true

Install the First Node

To install the first node, do this following command from ansible host
ansible-playbook rke2-ha.yml -i rke2-hosts -l rke2-node1
When the playbook finished, login to the first node and get the rke2 token from /var/lib/rancher/rke2/server/node-token, then create file config-slave.yaml in ansible folder and put the token like in this following example: 
server: https://master.mydomain.com:9345
token: [token from /var/lib/rancher/rke2/server/node-token]
tls-san:
  - "mydomain.com"
  - "192.168.1.11"
  - "192.168.1.12"
  - "192.168.1.13"

Install the rest of Node

Now, you ready to install the rest of node using this command:
ansible-playbook rke2-ha.yml -i rke2-hosts -l rke2-nextnode
Wait a couple minutes, until all node are connected to each other. To check the status of the cluster, use this command from any rke2 nodes:
/var/lib/rancher/rke2/bin/kubectl \
        --kubeconfig /etc/rancher/rke2/rke2.yaml get nodes
        
NAME           STATUS   ROLES                       AGE   VERSION
rke2-node1     Ready    control-plane,etcd,master   1m    v1.21.4+rke2r2
rke2-node2     Ready    control-plane,etcd,master   1m    v1.21.4+rke2r2
rke2-node3     Ready    control-plane,etcd,master   1m    v1.21.4+rke2r2

Installing Rancher (Kubernetes Dashboard Management)

After your cluster is up, you can already manage and deploy application using kubectl tools, but it'll be much more easy to manage the cluster from web interface. We use Rancher for that. To install rancher, you must do the following step from any rke2 nodes

Install helm

To install helm you can download the script from their website:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Add Latest Helm Repo

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

Create a Namespace for Rancher

kubectl create namespace cattle-system

Install cert-manager

I use Rancher Generated Certificates, so the command will be:

# If you have installed the CRDs manually instead of with the `--set installCRDs=true` option added to your Helm install command, you should upgrade your CRD resources before upgrading the Helm chart:
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io

# Update your local Helm chart repository cache
helm repo update

# Install the cert-manager Helm chart
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.5.1

Wait until the cert-manager installed successfully and check using this command:

kubectl get pods --namespace cert-manager

NAME                                       READY   STATUS    RESTARTS   AGE
cert-manager-5c6866597-zw7kh               1/1     Running   0          2m
cert-manager-cainjector-577f6d9fd7-tr77l   1/1     Running   0          2m
cert-manager-webhook-787858fcdb-nlzsq      1/1     Running   0          2m

Install Rancher using helm

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --set hostname=rancher.mydomain.com \
  --set replicas=3 \
  --set proxy="http://192.168.1.1:3128/" \
  --set noProxy=".mydomain.com\,127.0.0.0/8\,192.168.1.0/16\,10.0.0.0/8"

Wait until deployment is finish, check the deployment status

kubectl -n cattle-system get deploy rancher
NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
rancher   3         3         3            3           3m

To access rancher, you must configure your DNS and Nginx because it uses domain to resolve the application

Configure DNS

Add this following entry into your DNS configuration

*.mydomain.com. IN A 192.168.1.1

Configure Nginx

Below is Nginx configuration that can be used to forward the request to the RKE2 cluster, put this config inside http section:

upstream kube {
  server 192.168.1.11:80;
  server 192.168.1.12:80;
  server 192.168.1.13:80;
}

server {
	listen 443 ssl;
	error_log   /var/log/nginx/kubes-error.log;

	ssl_certificate     /etc/ssl/private/kube.crt;
	ssl_certificate_key /etc/ssl/private/kube.key;

	# Support for wildcard domains
	server_name *.mydomain.com;

	location / {
		access_log      off;
		proxy_pass http://kube;
		proxy_set_header Host $host;
		proxy_set_header X-Real-IP  $remote_addr;
		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
	}
}

server {
	listen 80;
	error_log   /var/log/nginx/kubes-error.log;

	# Support for wildcard domains
	server_name *.mydomain.com;

	return 301 https://$host$request_uri;
}

Inside Main section:

stream {
    map $ssl_preread_server_name $name {
        rancher.mydomain.com rancher_servers_https;
    }

    upstream rancher_servers_https {
        least_conn;
        server 192.168.1.11:443 max_fails=3 fail_timeout=5s;
        server 192.168.1.12:443 max_fails=3 fail_timeout=5s;
        server 192.168.1.13:443 max_fails=3 fail_timeout=5s;
    }

    server {
        listen     8443;
        proxy_pass $name;
        ssl_preread on;
    }
}

Please remember to pointing the ip to the agent node, not master node if you add more agent node. If everything are fine, you can access rancher in this url https://rancher.mydomain.com:8443

Using rancher you can deploy application using marketplace or using custom yaml, you can assign domain to application for example: grafana.mydomain.com without needed to configure DNS again, because all domain using mydomain.com will be forwarded to rke2 cluster.

References:

Email Post
Labels:

Comments

Post a Comment

Popular Posts