Note: This guide was written for Kubernetes 1.9 for Docker on Ubuntu 16.04

This guide is written by a beginner in both Linux, Docker and Kubernetes and is aimed as a guide to assist others who are interested in trying out Kubernetes without using VMs and MiniKube.

This is a basic guide to installing Kubernetes on a clean Docker on Ubuntu 16.04 system.
The guide is geared towards setting up a single node Kubernetes cluster with Traefik as the ingress controller. It will serve both the Traefik and Kubernetes dashboards on sub-domains reachable from the internet with both protected by basic auth.

Prerequisites

This guide assume that you have a bare metal or VPS server running somewhere as well as a domain name poiting to the machine's IP. It also assumes that you have a fresh install of Docker on Ubuntu 16.04 (The guide was written on a VPS hosted by OVH and they provide a Docker on Ubuntu image) and have already set up a user with sudo access. All commands in this guide is execute on a regular user account unless otherwise noted.

Getting Started

First update your Ubuntu install by SSH-ing into your machine as a regular user and then running the following command

sudo apt-get update

followed by

sudo apt-get upgrade

This will update the package definitions and then upgrade the packages on your machine.

Getting the Kubernetes bits

The first step is to grab the key for the Kubernetes install. Do so by issueing the following command

sudo curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add

Next we need the Kubernetes repository. Execute the following commands as root, as sudo does not have the required priviledges to make the changes.
First create the file /etc/apt/sources.list.d/kubernetes.list. It can be created by issueing

touch /etc/apt/sources.list.d/kubernetes.list

To edit the file enter

vi /etc/apt/sources.list.d/kubernetes.list

and then enter the line below into the file

deb http://apt.kubernetes.io/ kubernetes-xenial main

Once the line has been added press Esc followed by Shift + : then type x! and press Enter. This will close and save the file.

We are now ready to install the parts we need for Kubernetes. Issue the following commands as your regular user

sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl kubernetes-cni

Once it is done, the parts for Kubernetes have been installed.

Initialize the Node

Kubernetes requires a Pod Network for the pods to communicate. For this guide we will use Flannel although there are several other Pod Networks available. You can take a look at other neworks here but installing them is outside the scope of this guide.

First we need to set /proc/sys/net/bridge/bridge-nf-call-iptables to 1 to pass bridged IPv4 traffic to iptables` chains which is required by certain CNI networks (in this case Flannel). Do this by issueing

sudo sysctl net.bridge.bridge-nf-call-iptables=1

We can now initialize Kubernetes by running the initialization command and passing --pod-network-cidr which is required for Flannel to work correctly

kubeadm init --pod-network-cidr=10.244.0.0/16

Once Kubernetes has been initialized we then install the Flannel Pod Network by running

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.9.1/Documentation/kube-flannel.yml

We can check that the pod is up by running

kubectl get pods --all-namespaces

which will display all the Pods.

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   etcd-vps520050                                1/1       Running   0          1d
kube-system   kube-apiserver-vps520050                      1/1       Running   0          1d
kube-system   kube-controller-manager-vps520050             1/1       Running   0          1d
kube-system   kube-dns-6f4fd4bdf-zpwjh                      3/3       Running   0          1d
kube-system   kube-flannel-ds-9szb9                         1/1       Running   0          1d
kube-system   kube-proxy-mgvg4                              1/1       Running   0          1d
kube-system   kube-scheduler-vps520050                      1/1       Running   0          1d

If there are pods that are not running, take a look at the Kubernetes troubleshooting guide

Because we are running only a single Kubernetes node we want to be able to run Pods on the master node. To do this we need to untaint the master node so it can run regular pods. To do so run

kubectl taint nodes --all node-role.kubernetes.io/master-

Installing Traefik

We will use Traefik as an Ingress Controller.
Traefik will be installed as a Pod on Kubernetes.
Our first step is to set up the Role Based Access Control(RBAC) configuration.
The provided configuration is not overly fine-grained, but will serve our purposes for now.
The configuration looks as follows

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

Save this as traefik-rbac.yml

Our next step is to set up the configuration for Traefik. The configuration below will redirect all http traffic to https and also ensure that Let's Encrypt autoconfiguration will work correctly.

apiVersion: v1
kind: ConfigMap
metadata:
  name: traefik-conf
  namespace: kube-system
data:
  traefik.toml: |
    # traefik.toml
    defaultEntryPoints = ["http","https"]
    [entryPoints]
      [entryPoints.http]
      address = ":80"
      [entryPoints.http.redirect]
        regex = "^http://(.*)"
        replacement = "https://$1"
      [entryPoints.https]
      address = ":443"
        [entryPoints.https.tls]
    [acme]
      email = "<email address>"
      storage = "/acme/acme.json"
      entryPoint = "https"
        [acme.httpChallenge]
        entryPoint = "http"
    [[acme.domains]]
    main = "<domain for https>"

Save this file as traefik-configmap.yml

Next we create our Traefik deploy. Because Let's Encrypt has a limited amount of certificates it can issue per domain we make use of local storage to ensure we do not have to recreate the certificates as the Pod gets restarted. The deployement creates the required Service Account for Traefik to run under. It then sets up our Pod on the appropriate ports as well as configuring the volumes where we will store our certificates on the local volume (under /srv/configs/).

# Service account for traefik
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      volumes:
        - name: config
          configMap:
            name: traefik-conf
        - name: acme
          hostPath:
            path: /srv/configs/
      containers:
        - image: traefik
          name: traefik-ingress-lb
          volumeMounts:
            - mountPath: "/config"
              name: "config"
            - mountPath: "/acme"
              name: "acme"
          ports:
            - containerPort: 80
              hostPort: 80
            - containerPort: 443
              hostPort: 443
            - containerPort: 8080
          args:
            - --configfile=/config/traefik.toml
            - --web
            - --kubernetes
            - --logLevel=INFO
---
apiVersion: v1
kind: Service
metadata:
  name: traefik-ingress-service
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - port: 80
      name: http
    - port: 443
      name: https
  externalIPs:
    - <Node's IP address>

Make sure to replace <Node's IP address> with your own machine's IP address, then save the file as traefik-deploy.yml

Finally we configure our Ingress, the deployment provides a service for the Traefik Web UI as well as name-based routing for our domain to the Traefik Dashboard.

apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: <domain to route to dashboard>
    http:
      paths:
      - backend:
          serviceName: traefik-web-ui
          servicePort: 80

Simply replace <domain to route to dashboard> with your own domain that points to your server and save the file as traefik-ingress.yml

Now it is time to actually run the deployment. Do so by executing the following commands

kubect create -f traefik-rbac.yml
kubect create -f traefik-configmap.yml
kubect create -f traefik-deploy.yml
kubect create -f traefik-ingress.yml

You can view the deployed items as follows

ClusterRole

kubectl get clusterrole --all-namespaces

Which will produce output similiar to

NAME                                                                   AGE
admin                                                                  5d
cluster-admin                                                          5d
flannel                                                                5d
system:aws-cloud-provider                                              5d
system:basic-user                                                      5d
traefik-ingress-controller                                             3d

ConfigMap

kubectl get configmap --all-namespaces

which will produce output similiar to

NAMESPACE     NAME                                 DATA      AGE
kube-public   cluster-info                         1         5d
kube-system   extension-apiserver-authentication   6         5d
kube-system   kube-flannel-cfg                     2         5d
kube-system   kube-proxy                           2         5d
kube-system   kubeadm-config                       1         5d
kube-system   traefik-conf                         1         3d

Service Accounts

kubectl get serviceaccounts --all-namespaces

which will produce output similiar to

NAMESPACE     NAME                                 SECRETS   AGE
default       default                              1         5d
kube-public   default                              1         5d
kube-system   admin-user                           1         3d
kube-system   attachdetach-controller              1         5d
kube-system   bootstrap-signer                     1         5d
kube-system   certificate-controller               1         5d
kube-system   clusterrole-aggregation-controller   1         5d
kube-system   cronjob-controller                   1         5d
kube-system   daemon-set-controller                1         5d
kube-system   default                              1         5d
kube-system   deployment-controller                1         5d
kube-system   disruption-controller                1         5d
kube-system   endpoint-controller                  1         5d
kube-system   flannel                              1         5d
kube-system   generic-garbage-collector            1         5d
kube-system   heapster                             1         3d
kube-system   horizontal-pod-autoscaler            1         5d
kube-system   job-controller                       1         5d
kube-system   kube-dns                             1         5d
kube-system   kube-proxy                           1         5d
kube-system   namespace-controller                 1         5d
kube-system   node-controller                      1         5d
kube-system   persistent-volume-binder             1         5d
kube-system   pod-garbage-collector                1         5d
kube-system   replicaset-controller                1         5d
kube-system   replication-controller               1         5d
kube-system   resourcequota-controller             1         5d
kube-system   service-account-controller           1         5d
kube-system   service-controller                   1         5d
kube-system   statefulset-controller               1         5d
kube-system   token-cleaner                        1         5d
kube-system   traefik-ingress-controller           1         3d
kube-system   ttl-controller                       1         5d

Deployments

kubectl get deployments --all-namespaces

which will produce output similiar to

NAMESPACE     NAME                         DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   heapster                     1         1         1            1           3d
kube-system   kube-dns                     1         1         1            1           5d
kube-system   monitoring-influxdb          1         1         1            1           3d
kube-system   traefik-ingress-controller   1         1         1            1           3d

Services

kubectl get services --all-namespaces

which produces output similiar to

NAMESPACE     NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP     PORT(S)          AGE
default       kubernetes                ClusterIP   [redacted]       <none>          443/TCP          5d
kube-system   heapster                  ClusterIP   [redacted]       <none>          80/TCP           3d
kube-system   kube-dns                  ClusterIP   [redacted]       <none>          53/UDP,53/TCP    5d
kube-system   monitoring-influxdb       ClusterIP   [redacted]       <none>          8086/TCP         3d
kube-system   traefik-ingress-service   ClusterIP   [redacted]       [your IP]       80/TCP,443/TCP   3d
kube-system   traefik-web-ui            ClusterIP   [redacted]       <none>          80/TCP           3d

Ingress

kubectl get ingress --all-namespaces

which will produce output similiar to

NAMESPACE     NAME                   HOSTS                                 ADDRESS   PORTS     AGE
kube-system   traefik-web-ui         [your domain]                         80        3d

Pods

kubectl get pods --all-namespaces

which will produce output similiar to

NAMESPACE     NAME                                          READY     STATUS    RESTARTS   AGE
kube-system   etcd-[redacted]                               1/1       Running   0          5d
kube-system   kube-apiserver-[redacted]                     1/1       Running   0          5d
kube-system   kube-controller-manager-[redacted]            1/1       Running   0          5d
kube-system   kube-dns-6f4fd4bdf-zpwjh                      3/3       Running   0          5d
kube-system   kube-flannel-ds-9szb9                         1/1       Running   0          5d
kube-system   kube-proxy-mgvg4                              1/1       Running   0          5d
kube-system   kube-scheduler-[redacted]                     1/1       Running   0          5d
kube-system   traefik-ingress-controller-7b7866b8fc-jpw94   1/1       Running   0          3d

Traefik Logs
You can view the startup logs for the Traefik Pod by running

kubectl logs <Pod name> -n Kube-System

This should show something like

time="2018-03-05T05:16:27Z" level=info msg="Using TOML configuration file /config/traefik.toml"
time="2018-03-05T05:16:27Z" level=warning msg="web provider configuration is deprecated, you should use these options : api, rest provider, ping and metrics"
time="2018-03-05T05:16:27Z" level=info msg="Traefik version v1.5.3 built on 2018-02-27_02:47:04PM"
time="2018-03-05T05:16:27Z" level=info msg="
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://docs.traefik.io/basics/#collected-data
"
time="2018-03-05T05:16:27Z" level=info msg="Preparing server traefik &{Network: Address::8080 TLS:<nil> Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420631d80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-03-05T05:16:27Z" level=info msg="Preparing server http &{Network: Address::80 TLS:<nil> Redirect:0xc4201cb290 Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420631d40} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-03-05T05:16:27Z" level=info msg="Preparing server https &{Network: Address::443 TLS:0xc4206b4e00 Redirect:<nil> Auth:<nil> WhitelistSourceRange:[] Compress:false ProxyProtocol:<nil> ForwardedHeaders:0xc420631d60} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s"
time="2018-03-05T05:16:27Z" level=info msg="Starting server on :80"
time="2018-03-05T05:16:27Z" level=info msg="Starting server on :8080"

You should now be able to access the Traefik dashboard by navigating to your domain
Traefik Dashboard

Note that the Dashboard is not currently protected and sits open on the internet, basic auth will be covered in the next section.

Basic auth for Dashboard

Install the apache2-utils in order to make use of the htpasswd command. You can install them by running

sudo apt install apache2-utils

Once complete set up a new password file by running

htpasswd -c ./auth <username>

and entering the desired password.

We will now create a secret in Kubernetes by issueing

kubectl create secret generic mysecret --from-file auth --namespace=kube-system

Note The secret must be in the same namespace as the Ingress object.

Next we can attach Basic Authentication to our Ingress by editing the Ingress configuration and adding

ingress.kubernetes.io/auth-type: "basic"
ingress.kubernetes.io/auth-secret: "mysecret"

The new ingress file will look as follows

apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
    ingress.kubernetes.io/auth-type: "basic"
    ingress.kubernetes.io/auth-secret: "mysecret"
spec:
  rules:
  - host: <domain to route to dashboard>
    http:
      paths:
      - backend:
          serviceName: traefik-web-ui
          servicePort: 80

Update the current Ingress by first deleting the old ingress

kubectl delete ingress traefik-web-ui -n kube-system

and then we create it again

kubectl create -f traefik-ingress.yml

The dashboard should now prompt for Basic auth when you try to access it
Basic Auth

Kubernetes Dashboard

Install Heapster

For the dashboard to be able to display all of its data it requires Heapster running in the Cluster.
The heapster setup is fairly straight-forward and won't be covered in much detail. For this guide we will be using InfluxDb to store the data. While the provided configurations also cater for setting up Grafana we will not be using it.

First grab the Heapster RBAC file, then execute

kubectl create -f heapster-rbac.yaml

This will set up the required RBAC for Heapster to run.

Next we need to deploy the InfluxDB and Heapster pods. Start off by grabbing the InfluxDb deployment and
Heapster deployment files.

Next, execute the deployment by using

kubectl create -f influxdb.yaml

followed by

kubectl create -f heapster.yaml

Verify that both pods have been deployed and are running by executing

kubectl get pods --all-namespaces

Installing the Kubernetes Dashboard

For the dashboard we will use the Deployment file provided by Kubernetes with some modifications to fit into our use case. In this section we will explore the differences and deploy the dashboard to our cluster.

First we will skip the Dashboard secret as our dashboard will be hosted inside the cluster and be securely exposed via the Traefik Ingress Controller. We will set up the RBAC exactly as it appears in the file as it fullfils our requirements as is.

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-minimal
  namespace: kube-system
rules:
  # Allow Dashboard to create 'kubernetes-dashboard-key-holder' secret.
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["create"]
  # Allow Dashboard to create 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["create"]
  # Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
  resources: ["secrets"]
  resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs"]
  verbs: ["get", "update", "delete"]
  # Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
  resources: ["configmaps"]
  resourceNames: ["kubernetes-dashboard-settings"]
  verbs: ["get", "update"]
  # Allow Dashboard to get metrics from heapster.
- apiGroups: [""]
  resources: ["services"]
  resourceNames: ["heapster"]
  verbs: ["proxy"]
- apiGroups: [""]
  resources: ["services/proxy"]
  resourceNames: ["heapster", "http:heapster:", "https:heapster:"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: kubernetes-dashboard-minimal
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: kubernetes-dashboard-minimal
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

Save the file as kubedashboard-rbac.yaml

Create the role by running

kubectl create -f kubedashboard-rbac.yaml

We need to make some changes to the Dashboard deployment as we do not need all of the parts

apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
---
kind: Deployment
apiVersion: apps/v1beta2
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      k8s-app: kubernetes-dashboard
  template:
    metadata:
      labels:
        k8s-app: kubernetes-dashboard
    spec:
      containers:
      - name: kubernetes-dashboard
        image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
        ports:
        - containerPort: 9090
          protocol: TCP
        args:
          - --insecure-bind-address=0.0.0.0
          - --insecure-port=9090
          - --enable-insecure-login
        volumeMounts:
        - name: kubernetes-dashboard-certs
          mountPath: /certs
          # Create on-disk volume to store exec logs
        - mountPath: /tmp
          name: tmp-volume
        livenessProbe:
          httpGet:
            scheme: HTTP
            path: /
            port: 9090
          initialDelaySeconds: 30
          timeoutSeconds: 30
      serviceAccountName: kubernetes-dashboard
      # Comment the following tolerations if Dashboard must not be deployed on master
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

Save the file as kubedashboard-deploy.yaml

First, we expose port 9090 on the container as we will be hosting it without SSL on the internal cube network the reason being that there is no way to deploy Let's Encrypt certificates inside the cluster network. The next change is a modification to the arguments

Bind to all available addresses

--insecure-bind-address=0.0.0.0

Bind to port 9090

--insecure-port=9090

Enable the sign in page on HTTP, not just HTTPS

--enable-insecure-login

Adjust the livenessProbe to run over HTTP and port 9090 and we remove the volume information as we will not be storing any certificates.

Deploy the dashboard by executing

kubectl create -f kubedashboard-deploy.yaml

Finally we need to set up the service and ingress to allow us to access the Dashboard

kind: Service
apiVersion: v1
metadata:
  labels:
    k8s-app: kubernetes-dashboard
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  ports:
    - port: 80
      targetPort: 9090
  selector:
    k8s-app: kubernetes-dashboard
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  rules:
  - host: <dashboard url>
    http:
      paths:
      - backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

Save the file as kubedashboard-ingress.yaml

This simply route traffic arriving at our dashboard url to the dashboard pod on port 9090.
Deploy the ingress by running

kubectl create -f kubedashboard-ingress.yaml

Finally we need a service account token to be able to access the dashboard's functionality. The deployment file will create a service account admin-user

apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system

Save the file as kubedashboard-serviceaccount.yaml

Create the account by running

kubectl create -f kubedashboard-serviceaccount.yaml

Now we can grab the token by running

kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

and then copying the token

Name:         admin-user-token-wfm5v
Namespace:    kube-system
Labels:       <none>
Annotations:  kubernetes.io/service-account.name=admin-user
              kubernetes.io/service-account.uid=a7d8dcd6-2092-11e8-a78e-fa163e7f0433

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1025 bytes
namespace:  11 bytes
token:      [redacted]

Access the dashboard on your dashboard url, choose token and log in to see the state of your cluster
Kubernetes Dashboard