Configure and access to the Kubernetes Dashboard
If you have deployed Kubernetes on Amazon Web Services (AWS), Google Compute Platform (GCP), Azure or any Cloud Provider where you don’t have local access to the server running the master, you may have run into issues trying to access the Dashboard. More likely you are unable to login to the Dashboard with a Forbidden Message. On this post we will explain what is the recommended way of fixing this following Best Practices and keeping our environment secure.
On version 1.7 of Kubernetes the RBAC service was introduced, this is the reason we are not able to connect and many applications and add-ons started to crash. This post will walk you through the process to deploy, configure and access to the Kubernetes Dashboard.
Kubernetes Dashboard is forbidden
Before we get into the solution, we will detail the exact error you may get after installing Kubernetes Cluster, on a machine and trying to access the Dashboard remotely. The error message when trying to access the Dashabord looks like this:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:kubernetes-dashboard:\" is forbidden: User \"system:anonymous\" cannot get resource \"services/proxy\" in API group \"\" in the namespace \"kube-system\"",
"reason": "Forbidden",
"details": {
"name": "https:kubernetes-dashboard:",
"kind": "services"
},
"code": 403
}Checking Kubernetes Dashboard is Running
If you are getting this message, is because Kubernetes Dashboard is actually running, we can check that running the command:
kubectl -n kube-system get podAnd verifying we have an entry for kubernetes dashboard in running state, as shown below:
NAME READY STATUS RESTARTS AGE
coredns-5b969f4c88-8vrkj 1/1 Running 2 4d18h
coredns-5b969f4c88-cgjhp 1/1 Running 2 4d18h
etcd-empty-dir-cleanup-kubernetes-master 1/1 Running 1 4d18h
etcd-server-events-kubernetes-master 1/1 Running 1 4d18h
etcd-server-kubernetes-master 1/1 Running 1 4d18h
event-exporter-v0.2.4-565cfffc57-wwchd 1/1 Running 1 4d18h
fluentd-gcp-scaler-7db4984bf4-snsnb 1/1 Running 1 4d18h
fluentd-gcp-v3.2.0-d6mqd 1/1 Running 1 4d18h
fluentd-gcp-v3.2.0-hgv5r 1/1 Running 1 4d18h
fluentd-gcp-v3.2.0-lxjbk 1/1 Running 1 4d18h
fluentd-gcp-v3.2.0-swvkk 1/1 Running 1 4d18h
heapster-v1.6.0-beta.1-7b59fcb475-xf6qf 2/2 Running 2 4d18h
kube-addon-manager-kubernetes-master 1/1 Running 1 4d18h
kube-apiserver-kubernetes-master 1/1 Running 1 4d18h
kube-controller-manager-kubernetes-master 1/1 Running 1 4d18h
kube-dns-autoscaler-97df449df-2855w 1/1 Running 1 4d18h
kube-proxy-kubernetes-minion-group-9xrz 1/1 Running 1 4d18h
kube-proxy-kubernetes-minion-group-ck7h 1/1 Running 1 4d18h
kube-proxy-kubernetes-minion-group-hmqb 1/1 Running 1 4d18h
kube-scheduler-kubernetes-master 1/1 Running 1 4d18h
kubernetes-dashboard-85bcf5dbf8-rl8kt 1/1 Running 1 4d18h
l7-default-backend-8f479dd9-c9sx4 1/1 Running 1 4d18h
l7-lb-controller-v1.2.3-kubernetes-master 1/1 Running 1 4d18h
metrics-server-v0.3.1-7d9cf58c5c-wmzbg 2/2 Running 2 4d18h
kubectl cluster-info
If you are not able to connect to the Dashboard without any error message, but the service is running. First, check that your firewall is allowing the communication on port 443 and you are targeting the correct IP address and URL. This information can be gathered with kubectl cluster-info
kubectl cluster-info
Kubernetes master is running at https://104.198.209.88
GLBCDefaultBackend is running at https://104.198.209.88/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy
Heapster is running at https://104.198.209.88/api/v1/namespaces/kube-system/services/heapster/proxy
CoreDNS is running at https://104.198.209.88/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
kubernetes-dashboard is running at https://104.198.209.88/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy
Metrics-server is running at https://104.198.209.88/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
Now that we have confirmed the dashboard is running, and we are getting an error related to authentication, we can assume is related to RBAC and the new securities features in Kubernetes 1.7, so lets continue to check our options to fix this issue.
Possible Solutions to Dashboard forbidden in Kubernetes
Checking the official documentation for accessing Dashboard on Kubernetes 1.7, we can see there are three possible ways of accessing the Dashboard:
- kubectl proxy
- NodePort
- API Server
kubectl proxy
Works locally, it can be tweaked to proxy remotely but, we would be exposing the dashboard to HTTP instead of HTTPS, so it is a really bad solution security-wise.
NodePort
Only works for a single master node deployment, which is usually not the case for production systems.
API Server
Is the preferred solution and the one we will describe. This way of accessing Dashboard is only possible if you choose to install your user certificates in the browser. In example certificates used by kubeconfig file to contact API Server can be used.
To extract the certificates from the kubeconfig file follow these steps:
Locate your kubeconfig or config file which you use to run kubectl commands. If you have used Vagrant, you can find it on /etc/kubernetes/admin.conf. On my example, I have used the kube-setup script for GCP and I run it using my ubuntu user, so the path is /home/ubuntu/.kube
You need to export a single file (.p12) with the following two files: the client-certificate-data, and the client-key-data. If you run this command on macOS, be sure to change the base64 -d to base64 -D
grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt
grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key
openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12Use the config file to login to the Dashboard
What these commands have done, is to extract the certificate and key from the Kubernetes config file and use them to create a P12 Certificate, which we will import to our Browser. At this point, we will also use the config file to login to the Dashboard, so if you are managing your Kubernetes cluster remotely, what we need to do now is to transfer these files to our local machine, so we can access the Dashboard. You can use SCP for this, as I am managing my Kubernetes cluster from an AWS instance, I can copy the files as shown below:
scp -i "east-key.pem" ubuntu@ec2-54-82-212-18.compute-1.amazonaws.com:/home/ubuntu/.kube/kubecfg.p12 .
scp -i "east-key.pem" ubuntu@ec2-54-82-212-18.compute-1.amazonaws.com:/home/ubuntu/.kube/config .Now that we have downlaod the files locally, we can import the kubecfg.p12 certificate. To do this on Chrome, go to Settings and type certificates. The manage certificates option where you can import the certificate will appear.
After importing the certificate, reopen your browser, and visit the Kubernetes Dashboard URL. Accept any warning and you should see the authentication page. You may need to select the certificate to use for authentication as shown below
After that the Authentication screen from Kubernetes will appear
To clarify, there are basically two ways of authenticating, using the kubeconfig file or generating a Token. As we already downloaded the kube config file to our local machine, we will use it for authenticating. Select your config file and click on ‘Sign-in’ . After that, you will then be able to access the Dashboard as shown below:
We hope this post helps you to save some time, but if you get stuck, don’t hesitate to contact our Kubernetes experts.