Tuesday, February 28, 2023

Rancher Suite K8S Adventure - Chapter 012 - A first log in and tour of Rancher

Rancher Suite K8S Adventure - Chapter 012 - A first log in and tour of Rancher

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Today is the first log in and tour of Rancher.

Where we left off yesterday, a 'rollout status' kubectl shows Rancher is up and running.

Go to https://YourClusterHostname where YourClusterHostname is the round robin DNS name not one specific host.  Because you're using a self signed cert you'll have to click "ok" or "proceed" or whatever your web browser requires.

Log in with your bootstrap password from yesterday.

Rancher will want to verify the cluster URL, unless you messed something up the default should be correct.  Also you'll have to click the EULA acceptance checkbox.

This will drop you in the default dashboard page.

One cool feature of Rancher is centralized authorization for cluster control.  I'm not going to configure Active Directory auth.  Partially because I could never get it working; AD has near infinite flexibility therefore if the only feedback is "authentication failed" it can take a near infinite amount of time to configure.  The other problem is my DCs are hosted as VMs and I don't like the idea of being locked out of Rancher due to a cluster problem therefore being unable to log into Rancher to fix the cluster problem.  Kind of like the old joke about hosting your DHCP controllers as VMware images then making your ESXi hosts configure their networks using DHCP.

Anyway, add at least one user for daily use, much as most sysadmins do not use root on a Linux box all day, its probably good to not use admin on Rancher:

Left Hamburger menu, "Configuration" "Users and Authentication"

"Users" "Create" and pay close attention to Global Permissions, Administrator vs Standard User, etc.

To work around the "Password must be at least 12 characters" error:

Left Hamburger menu, "Global Settings" change password-min-length to something that doesn't force people to use post it notes as a password manager.

Log out as admin and log in as a normal-ish user.

Set your preferences in the right icon "Preferences".  The default color theme changes at night which I find incredible disturbing when it happens, so I always force it to "Light" theme.  This is also where you can change the "Login Landing Page" from home to a specific cluster.

Time for a quick tour.  This tour provides a high level view of the rest of the series.

Home

At login you will be dropped in "Home" which you can reach from the left Hamburger menu "Home".  Gives you a list of your clusters, and we will look at clusters later.

List of Clusters

Next hamburger menu entry is a list of your clusters, just local right now.  Again, we will look at clusters later.

Continuous Delivery

After the cluster list in the Hamburger menu is "Continuous Delivery" that Git Ops stuff where committing code results, optimistically, in passed tests and successful deployments.  You will get a "You don't have any Git Repositories in your Workspaces" and we will return to this cool feature another day.  Its an undermarketed feature, very cool...

Cluster Management

"Cluster Management" is the next entry in the Hamburger menu.  The right side is yet another view of your clusters, but the left side is where you enter your cloud credentials, select drivers for Rancher clusters, etc.  AWS is far too expensive to permanently use compared to cheap onsite cluster hardware, but its fun and cool to experiment with.

The Harvester component of the Rancher Suite allows HCI integration of clusters and virtual machines.  This is the page where you import your Harvester Cluster into Rancher.

Users & Authentication

You previously visited "Users & Authentication" when creating a non-admin user for daily Rancher use.  This is also where you configure Authorization Providers.

Extensions

By default, the "Extensions" menu does nothing because the Extension Operator is not enabled.  Clicking the button to enable it adds a new repo full of cool Rancher "stuff".  Obviously you can not install stuff from the internet if your install is air-gapped from the internet.  Anyway after enabling the Extension Operator, note that no extensions are installed by default, click "Available" and as of the time of writing this there are exactly two extensions available, one for elemental OS and one for Kubewarden.

Global Settings

"Global Settings" was where you reconfigured the minimum password length but there are all kinds of cool settings here.  "Home Links" lets you add or change the links on the home page, which is pretty cool.

Examine a typical cluster

Time for a quick glance at a cluster.  The only cluster we have right now is "local", the one running Rancher.  There are at least three different ways to access cluster "local", the home page, the entry in the hamburger menu, and looking at cluster management and clicking local.  Note that cluster management provides more "control" features and fewer "monitoring" features so just use the home page for now, because first stop is the event log.

If you enabled the Extensions Operator that will result in about two dozen events.  I selected them all and deleted them.  This is a much more exciting page when you're troubleshooting a problem.

Looking at the main cluster page, there's an option to add a cluster badge, which AFAIK is purely decorative.  I like to add the URL of my cluster, so a cluster named "local" for Rancher has, in my case, a badge named "rancher.cedar.mulhollon.com"

Note that next to the badge setting there's an option to enable monitoring via the cluster tools charts page.  This is a long story for another day.  Aside from monitoring there's plenty of other cluster scale tools Rancher can install, including the backup system and Longhorn for data storage and various scanning and alerting systems.

Something you'll rapidly discover when looking at the Rancher cluster's workload, etc, is there's a drop down menu at the top of the screen, probably by default set to "Only User Namespaces" and there is no user workload on the Rancher cluster so, for example, the deployment list will be empty.  Simply change the dropdown to something like "All Namspaces" and the list will fill with rancher and cert-manager and so forth.  Its instructive to click thru on the deployment for Rancher, then perhaps selec tthe services or ingresses for that deployment and see there's there is the https port you're using to access Rancher being examined in rancher.  Or click thru on the Rancher deployment to the pod for rancher then the container, then click the three dots to look at the container logs for Rancher.

That was a whirlwind tour of Rancher.  Here's a great operational resource for new Rancher users:

https://ranchermanager.docs.rancher.com/pages-for-subheaders/new-user-guides

At this point you have a great manager of clusters, but no clusters to manage.  Next we work on adding a small Harvester cluster.

Monday, February 27, 2023

Rancher Suite K8S Adventure - Chapter 011 - Install Rancher on RKE2 cluster

Rancher Suite K8S Adventure - Chapter 011 - Install Rancher on RKE2 cluster

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

The next step is to install Rancher on the RKE2 cluster.  Happily, this is the simplest step of the entire series, so far.

The references are:

https://www.rancher.com/products/rancher

https://ranchermanager.docs.rancher.com/

https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli

https://ranchermanager.docs.rancher.com/pages-for-subheaders/install-upgrade-on-a-kubernetes-cluster

There's really only one line to run, admittedly a very long line:

helm install rancher rancher-latest/rancher \
  --namespace cattle-system \
  --version 2.7.1 \
  --set hostname=rancher.cedar.mulhollon.com \
  --set replicas=1 \
  --set bootstrapPassword=ThisIsNotMyRealPassword

Obviously your hostname and password and maybe even requested version will be different than the above.  Don't use a real password, I will explain later.

Lets watch the process of the install:

kubectl -n cattle-system rollout status deploy/rancher

Note this will take awhile... at least five minutes in my experience.

Remember when I noted that you should not use a "real" password for the bootstrap password?  Try this command line:

kubectl get secret --namespace cattle-system bootstrap-secret -o go-template='{{.data.bootstrapPassword|base64decode}}{{"\n"}}'

Oh.  That's why.  Although I suppose anyone who has root or kubectl access on your cluster pretty much owns the cluster and everything on it anyway so not much loss in having the password in there.

Tomorrow we're done with the CLI and installation tasks; its time to tour Rancher!

Friday, February 24, 2023

Rancher Suite K8S Adventure - Chapter 010 - Cert-Manager

Rancher Suite K8S Adventure - Chapter 010 - Cert-Manager

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

The next step is to install cert-manager on the new Rancher RKE2 cluster.

The references are:

https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli

https://www.jetstack.io/open-source/cert-manager/

https://cert-manager.io/docs/

Add the repo for jetstack and rancher:

helm repo add rancher-latest https://releases.rancher.com/server-charts/latest
helm repo add jetstack https://charts.jetstack.io
helm repo update

then verify:

vince@ubuntu:~$ helm repo list
NAME                 URL                                              
rancher-latest       https://releases.rancher.com/server-charts/latest
jetstack                https://charts.jetstack.io   
vince@ubuntu:~$ 

Create the namespace for rancher, we'll create the ns for jetstack as part of the install:

kubectl create namespace cattle-system

then verify

vince@ubuntu:~$ kubectl get namespaces | grep cattle-system
cattle-system     Active   30s
vince@ubuntu:~$ 

Next install the CRDs (Custom Resource Definitions) used by cert-manager:

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.crds.yaml

Finally have helm create the cert-manager namespace (could have made it above... whatever) and install cert-manager:

helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.7.1

Lets take a look at the cert-manager namespace:

vince@ubuntu:~$ kubectl get all --namespace=cert-manager
NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-646c67487-kmrml               1/1     Running   0          112s
pod/cert-manager-cainjector-7cb8669d6b-wjdcz   1/1     Running   0          112s
pod/cert-manager-webhook-696c5db7ff-slrsv      1/1     Running   0          112s

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   10.43.61.195    <none>        9402/TCP   112s
service/cert-manager-webhook   ClusterIP   10.43.207.185   <none>        443/TCP    112s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           112s
deployment.apps/cert-manager-cainjector   1/1     1            1           112s
deployment.apps/cert-manager-webhook      1/1     1            1           112s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-646c67487               1         1         1       112s
replicaset.apps/cert-manager-cainjector-7cb8669d6b   1         1         1       112s
replicaset.apps/cert-manager-webhook-696c5db7ff      1         1         1       112s
vince@ubuntu:~$ 

Obviously you'll have different IP addresses and times above, but it should look similar, plus or minus obvious blogging platform formatting issues.

Now its time to verify cert-manager works.  You can go thru the steps listed here, but its tedious to cut and paste:

https://cert-manager.io/docs/installation/verify/

The verification process has you install cmctl which requires brew which I don't have on Ubuntu (long story) so that's tedious.  Next the verification process has you look at the pods in the namespace (see above when we did a get all, its the first 'paragraph' above.  After that is a long process to create a YAML cert request and submit that to cert manager, then see if cert manager issues you a self signed cert per the YAML, finally delete it.  There's also a cert-manager-verifier tool:

https://github.com/alenkacz/cert-manager-verifier

However, unless it looks like something broken, the simplest way to test cert-manager would be to install Rancher, and as the plan is to install Rancher tomorrow, its probably OK to skip extensive testing.


Thursday, February 23, 2023

Rancher Suite K8S Adventure - Chapter 009 - Rancher Cluster Additional Nodes Install

Rancher Suite K8S Adventure - Chapter 009 - Rancher Cluster Additional Nodes Install

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Now its time to connect the remaining nodes to the first node, which was configured yesterday.  Server nodes are schedulable on RKE2, and it takes three RKE2 server nodes for HA, and coincidentally I have exactly three mini servers allocated to this project, so my cluster design is three servers, zero agents.

Second and subsequent members of a cluster install process:

# curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION="v1.24.10+rke2r1" sh -

Now, create a micro-mini config just enough to bootstrap:

mkdir -p /etc/rancher/rke2/

cat node-token >> /etc/rancher/rke2/config.yaml

vi /etc/rancher/rke2/config.yaml

server: https://<server>:9345

token: move that first line here

# systemctl enable rke2-server.service

# systemctl start rke2-server.service

(Insert an extremely dramatic very long pause here, about nine minutes?)

Eventually these two commands will settle down and look normal:

systemctl status rke2-server

journalctl -u rke2-server -f

(Repeat above for all future additional nodes in your cluster)

Some fun commands to try:

kubectl get node

kubectl get pods --all-namespaces

At this point you should have a stable three (or more?) node RKE2 cluster ready to run K8S workloads.

Tomorrow, "cert-manager"

Wednesday, February 22, 2023

Rancher Suite K8S Adventure - Chapter 008 - Rancher Cluster First RKE2 Install

Rancher Suite K8S Adventure - Chapter 008 - Rancher Cluster First RKE2 Install

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

I chose not to automate this install as RKE2 doesn't support anything more modern than downloading shell scripts from the internet; I suppose the entire point of installing Rancher as a cluster orchestrator is to avoid this kind of weirdness in the future.

The strange looking RKE2_VERSION line below more or less comes from:

https://update.rke2.io/v1-release/channels

First member of a cluster install process:

# curl -sfL https://get.rke2.io | INSTALL_RKE2_VERSION="v1.24.10+rke2r1" sh -

# systemctl enable rke2-server.service

# systemctl start rke2-server.service

(Insert an extremely dramatic very long pause here)

Eventually these two commands will settle down and look normal.

systemctl status rke2-server

journalctl -u rke2-server -f

Note that a kubectl file will be written to /etc/rancher/rke2/rke2.yaml

and a token file will be here /var/lib/rancher/rke2/server/node-token

scp /var/lib/rancher/rke2/server/node-token root@otherHosts:

(Repeat above for all future additional nodes in your cluster)

scp /etc/rancher/rke2/rke2.yaml vince@ubuntu:

(Or whatever your "local" experimenting system)

Some fun commands to try as root on your first node:

export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

kubectl get node

kubectl get pods --all-namespaces

helm ls --all-namespaces

Log into your experimental machine, for me that's vince@ubuntu: and recall you scp'd over the kubectl config, filename rke2.yaml.  That yaml specifies the server address as 127.0.0.1, that's not going to do.

First, mkdir ~/.kube then cp rke2.yaml ~/.kube/config

vi ~/.kube/config and change the "server: https://127.0.0.1:6443" to something probably reminiscent of "server: https://rancher1.cedar.mulhollon.com:6443" obviously your DNS name will be different.

At this point you should be able to run "kubectl get node" from a remote machine, or try "helm ls --all-namespaces".  Cool.

Tomorrow, we add the rest of the nodes to the cluster.

Tuesday, February 21, 2023

Rancher Suite K8S Adventure - Chapter 007 - Helm

Rancher Suite K8S Adventure - Chapter 007 - Helm

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Helm version 3.11 is installed on all members of the Rancher RKE2 cluster and on my Ubuntu experimentation box using Ansible.  Honestly this is almost identical to the process for installing kubectl yesterday, it's just a different repo and different package.

https://helm.sh/docs/intro/install/

The exact version of the Ubuntu package I'm installing is 1.24.10-00 as seen at

https://helm.baltorepo.com/stable/debian/packages/helm/releases/

And I'm doing an "apt hold" on it to make sure its not accidentally upgraded.

Here is a link to the gitlab repo directory for the Ansible helm role:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/tree/master/roles/helm

If you look at the Ansible task named packages.yml, the task installs some boring required packages first, then deletes the repo key if its too old, then downloads a new copy of the repo key if its not already present, add the local copy of the repo key to apt's list of known good keys, installs the sources.list file for the repo, does an apt-get update, takes helm out of "hold" state, installs the latest package for helm version 3.11, finally places helm back on "hold" state so its not magically upgraded to the latest version (3.12 or 3.13 or something by now).  Glad I don't have to do that manually by hand on every machine.

Simply add "- helm" to a machine's Ansible playbook, then run "ansible-playbook --tags helm playbooks/someHostname.yml" and it works.

As of the time this blog was written, "helm version" looks like this:

vince@ubuntu:~$ helm version
version.BuildInfo{Version:"v3.11.1", GitCommit:"293b50c65d4d56187cd4e2f390f0ada46b4c4737", GitTreeState:"clean", GoVersion:"go1.18.10"}
vince@ubuntu:~$ 

Monday, February 20, 2023

Rancher Suite K8S Adventure - Chapter 006 - Kubectl

Rancher Suite K8S Adventure - Chapter 006 - Kubectl

A travelogue of converting from OpenStack to Suse's Rancher Suite for K8S including RKE2, Harvester, kubectl, helm.

Kubectl version 1.24 is installed on all members of the Rancher RKE2 cluster and on my Ubuntu experimentation box using Ansible.

https://kubernetes.io/docs/reference/kubectl/

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/

The exact version of the Ubuntu package I'm installing is 1.24.10-00 as seen at

https://packages.cloud.google.com/apt/dists/kubernetes-xenial/main/binary-amd64/Packages

And I'm doing an "apt hold" on it to make sure its not accidentally upgraded.

Here is a link to the gitlab repo directory for the Ansible kubectl role:

https://gitlab.com/SpringCitySolutionsLLC/ansible/-/tree/master/roles/kubectl

If you look at the Ansible task named packages.yml, the task installs some boring required packages first, then deletes the Google K8S repo key if its too old, then downloads a new copy of the Google K8S repo key if its not already present, add the local copy of the Google K8S repo key to apt's list of known good keys, installs the sources.list file for Google's K8S repo, does an apt-get update, takes kubectl out of "hold" state, installs the latest package for kubectl version 1.24, finally places kubectl back on "hold" state so its not magically upgraded to the latest version (1.26 or 1.27 or something by now).  Glad I don't have to do that manually by hand on every machine, LOL!

Ansible makes life easy, all I need to do to have the most recent kubectl installed on a Ubuntu system is add "- kubectl" to that system's playbook, then run "ansible-playbook --tags kubectl playbooks/someHostname.yml" and like magic in seconds it'll work.

As of the time this blog was written, "kubectl version --short" looks like this:

vince@ubuntu:~$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.24.10
Kustomize Version: v4.5.4
The connection to the server localhost:8080 was refused - did you specify the right host or port?
vince@ubuntu:~$ 

Note that the last step is you probably want to enable bash autocompletion for kubectl in .bashrc for whatever username you log in as. My .bashrc file has a line like this:

source <(kubectl completion bash)

Mine is actually wrapped by some if $HOSTNAME lines, but whatever.

After you do this and log back in, you can type "kubectl" and hit tab a couple times and autocompletion will work. Pretty cool!