07. Authentication¶

Secure Hosts¶

Root access disabled
Password base authentication disabled
Only ssh-key based authentication to be made available

Kube API-Server¶

Kube API-Server in the Frontline of Defense¶

Controlling access to the API server itself

We need two types of decision¶

who can access? (Authentication)
what can they do? (Authorization)

Authentication - who can access¶

File: username and password
File: username and token
Certificates
External Authentication Provider - LDAP
Service Accounts

Authorization - what can they do¶

RBAC authorization
ABAC authorization
Node authorization
Webhook Mode

Authentication¶

Admins
Developers
End Users
Bots (such as gitlab runner)

Admins¶

Developers¶

Bots¶

Service Accounts

Types¶

All kubernetes authentication types

Note: Kubernetes official recommendation method is RoleBaseAuthentication

Certificates¶

cert-manager third party kubernetes certificate manager

CA authority¶

ca.key
ca.crt

Server¶

ETCD¶

etcdserver.key
etcdserver.crt

API-Server¶

apiserver.key
apiserver.crt

Kubelet¶

kubelet.key
kubelet.crt

Client¶

Admin User¶

admin.key
admin.crt

Kube-APIServer¶

apiserver-etcd-client.key
apiserver-etcd-client.crt
apiserver-kubelet-client.key
apiserver-kubelet-client.crt

Kubelet¶

kubelet-client.key
kubelet-client.crt

Kube-Scheduler¶

scheduler.key
scheduler.crt

Kube-Controller¶

control-manager.key
control-manager.crt

Kube-Proxy¶

kubeproxy.key
kubeproxy.crt

Update Exists Certificates¶

Update the old apiserver cert file¶

Get the cert file context¶

Full view

openssl x509 -in apiserver.crt -noout -text

Key filter view

openssl x509 -in apiserver.crt -noout -ext subjectAltName

Generate Certificates¶

CA¶

1: generate a new private key file ca.key

openssl genrsa -out ca.key 2048

2: create a new CSR file ca.csr by ca.key

openssl req -new -key ca.key -subj "/CN=KUBERNETES-CA" -out ca.csr

3: sign the created ca.csr and create a new ca.crt files

openssl x509 -req -in ca.csr -signkey ca.key -out ca.crt

Admin¶

1: generate a new private key file admin.key

openssl genrsa -out admin.key 2048

2: create a new CSR file admin.csr by admin.key

openssl req -new -key admin.key -subj "/CN=kube-admin \O=system:masters" -out admin.csr

3: sign the created admin.csr and create a new admin.crt file

openssl x509 -req -in admin.csr -CA ca.crt -CAkey ca.key -out admin.crt

KubeAPIServer¶

1: go to the kubernetes ssl directory

cd /etc/kubernetes/ssl/

2: generate a new private key file apiserver.key

openssl genrsa -out apiserver.key 2048

3: create a new CSR apiserver.csr by apiserver.key

openssl req -new -key apiserver.key \
-out apiserver.csr -config cert.cfg

4: sign the created apiserver.csr and create a new apiserver.crt file

openssl req -x509 \
-days 3650 \
-key apiserver.key \
-CA ca.crt -CAkey ca.key \
-extensions v3_req \
-config cert.conf \
-out apiserver.crt

Note: -subj "/CN=kubernetes" puts in the config file.

tls.conf

[ req ]
default_bits       = 2048
default_md         = sha256
distinguished_name = req_distinguished_name
req_extensions     = v3_req
prompt             = no
default_days       = 3651

[ req_distinguished_name ]
CN = kubernetes

[ v3_req ]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
basicConstraints = critical, CA:FALSE
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = kubernetes.default
DNS.2 = kubernetes.default.svc
DNS.3 = kubernetes.default.svc.cluster.local
DNS.4 = lb-apiserver.kubernetes.local
DNS.5 = localhost
DNS.6 = master
DNS.7 = master.cluster.local
IP.1  = 10.233.0.1
IP.2  = 192.168.197.12
IP.3  = 127.0.0.1
IP.4  = 192.168.196.12
IP.5  = 192.168.197.11

Note: You had better to check the old DNS and IP addresses exact the verify section.

5: Verify the new certificate apiserver.crt

openssl x509 -in apiserver.crt -text -noout | grep -A 1 "Subject Alternative Name"

6: Check the apiserver certificates

Kube-Scheduler¶

1: generate a new key file scheduler.key

openssl genrsa -out scheduler.key 2048

2: create a new CSR scheduler.csr by scheduler.key

openssl req -new -key scheduler.key -subj "/CN=kube-scheduler \O=system:kube-scheduler" -out scheduler.csr

3: sign the created scheduler.csr and create a new `scheduler.crt file

openssl x509 -req -in scheduler.csr -CA ca.crt -CAkey ca.key -out scheduler.crt

Kube Controller-Manager¶

1: generate a new key file controller-manager.key

openssl genrsa -out controller-manager.key 2048

2: create a new CSR controller-manager.csr by controller-manager.key

openssl req -new -key controller-manager.key -subj "/CN=kube-controller-manager \O=system:kube-controller-manager" -out controller-manager.csr

3: sign the created controller-manager.csr and create a new controller-manager.crt file

openssl x509 -req -in controller-manager.csr -CA ca.crt -CAkey ca.key -out controller-manager.crt

ETCd Servers (does not test it)¶

1: generate a new key file etcdserver.key

openssl genrsa -out etcdserver.key 2048

2: create a new CSR etcdserver.csr by etcdserver.key

openssl req -new -key etcdserver.key -subj "/CN=etcdserver \O=etcd-server" -out etcdserver.csr

3: sign the created etcdserver.csr and create a new etcdserver.crt

openssl x509 -req -in etcdserver.csr -CA ca.crt -CAkey ca.key -out etcdserver.crt

Kubelet¶

1: generate a new key file kubelet.key

openssl genrsa -out kubelet.key 2048

2: generate a new key file kubelet.csr by kubelet.key

openssl req -new -key kubelet.key -subj "/CN=kubelet \O=system:kubelet" -out kubelet.csr

3: sign the created kubelet.csr and create a new kubelet.crt

openssl x509 -req -in kubelet.csr -CA ca.crt -CAkey ca.key -out kubelet.crt

Kubelet client role certificate

Kube Proxy¶

1: generate a new key file kube-proxy.key

openssl genrsa -out kube-proxy.key 2048

2: generate a new key file kube-proxy.csr by kube-proxy.key

openssl req -new -key kube-proxy.key -subj "/CN=kube-proxy \O=system:kube-proxy" -out kube-proxy.csr

3: sign the created kube-proxy.csr and create a new kube-proxy.crt

openssl x509 -req -in kube-proxy.csr -CA ca.crt -CAkey ca.key -out kube-proxy.crt

kubeadm¶

Check the expiration date of all certs

kubeadm certs -h
kubeadm certs check-expiration

Renew all available certs

kubeadm certs renew -h
kubeadm certs renew all

User Certification Creation Progress¶

Certificate API Flow¶

1: User Create Certificate Signing Request Object

2: Review Requests by Admin

3: Approve Request by Admin

4: Share Certs to Users by Admin

Flow Example¶

1: create a PrivateKey

openssl genrsa -out mehrdad.key 2048

modify to correct permission

chmod 600 mehrdad.key others

2: create a CSR

openssl req -new -key mehrdad.key -out mehrdad.csr

pass CertificateSigningRequest to admin for approvement

3: put csr context to one line

cat mehrdad.csr | base64 | tr -d "\n" | pbcopy

4: create manifest file mehrdad-csr.yaml

apiVersion: certificates.k8s.io/v1
kind: CertificateSigningRequest
metadata:
    name: mehrdad
    namespace: collector
spec:
    groups:
        - system:authenticated
    request: "LS0tLS1CRUdJTiBDRVJUSUZ==..." # output of step #3
    signerName: kubernetes.io/kube-apiserver-client
    usages:
        - client auth

5: apply user's csr file

kubectl apply -f mehrdad-csr.yaml

6: get the list of csr

kubectl get csr

7: check csr context

kubectl describe csr mehrdad

8: approve or deny user csr

if you decided to approve

kubectl certificate approve mehrdad

if you decided to deny

kubectl certificate deny mehrdad

9: check final user csr status

kubectl get csr

10: fetch the certificate id

print in yaml format

kubectl get csr mehrdad -o yaml

print in json format

kubectl get csr mehrdad -o jsonpath='{.status.certificate}'
kubectl get csr mehrdad -o jsonpath='{.status.certificate}' | base64 -d

write to cert file

kubectl get csr mehrdad -o jsonpath='{.status.certificate}' | base64 -d > mehrdad-signed-csr.crt

11: fetch ca.crt cert file from master node

[cp/scp] /etc/kubernetes/ssl/ca.crt /path/to/ca/dir/ca.crt

12: we should have these 3 files

I. client-private.key
II. client-signed-csr.crt
III. ca.crt

13: check the final value of signed cert

openssl x509 -in mehrdad-signed-csr.crt -text -noout

14: check which Certificate Authority (CA) signed your certificate

openssl x509 -in mehrdad.crt -noout -issuer

15: check CA validity sign

openssl verify -CAfile ca.crt user.crt

check the important fields like, issuer, common-name and etc

CA Authority Module in Kubernetes¶

Controller-Manger¶

CSR-APPROVING and CSR-SIGNING modules are responsible for CA Authority in kubernetes cluster

Getting Request to API Server¶

using CURL¶

curl https://192.168.1.101:6443/api/v1/nodes \
    --key mehrdad.key \
    --cert mehrdad-signed-csr.crt \
    --cacert ca.crt

using Kubectl¶

 kubectl get nodes \
        --server https://node1:6443 \
        --client-key mehrdad.key \
        --client-certificate mehrdad-signed-csr.crt \
        --certificate-authority ca.crt

using kubectl-config file¶

Before you get request via kubectl, you need to put all configs into a specific file kubectl-config.yaml with absolute path of certs files.

apiVersion: v1
clusters:
    - cluster:
          certificate-authority: /home/sre/k8s/manifests/certificate/ca.crt
          server: https://192.168.1.101:6443
      name: cka
contexts:
    - context:
          cluster: cka
          user: mehrdad
      name: mehrdad@cka
current-context: mehrdad@cka
kind: Config
preferences: {}
users:
    - name: mehrdad
      user:
          client-certificate: /home/sre/k8s/manifests/certificate/mehrdad.crt
          client-key: /home/sre/k8s/manifests/certificate/mehrdad.key

you can also put values instead of file path

first you need to encode context to base64

cat ca.crt | base64 | tr -d "\n" | pbcopy

encode and copy client.crt

cat client.crt | base64 | tr -d "\n" | pbcopy

encode and copy client.key

cat client.key | base64 | tr -d "\n" | pbcopy

change some fields, append data to end of the keys

apiVersion: v1

clusters:
    - cluster:
          certificate-authority-data: "<ca.crt base64 encoded value>"
          server: https://192.168.1.101:6443
      name: cka

contexts:
    - context:
          cluster: cka
          user: mehrdad
      name: mehrdad@cka
    - context:
          cluster: cka
          user: babak
      name: babak@cka

current-context: mehrdad@cka
kind: Config
preferences: {}

users:
    - name: mehrdad
      user:
          client-certificate-data: "<mehrdad.crt base64 encoded value>"
          client-key-data: "<mehrdad.key base64 encoded value>"
    - name: babak
      user:
          client-certificate-data: "<babak.crt base64 encoded value>"
          client-key-data: "<babak.key base64 encoded value>"

check it after doing above step

kubectl --kubeconfig kube-config-mehrdad.yaml config view
kubectl --kubeconfig kube-config-mehrdad.yaml config get-users
kubectl --kubeconfig kube-config-mehrdad.yaml config get-clusters
kubectl --kubeconfig kube-config-mehrdad.yaml config get-contexts

finally you should have config file like it

apiVersion: v1

clusters:
    - cluster:
          certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0....
          server: https://192.168.1.101:6443
      name: cluster.local

contexts:
    - context:
          cluster: cluster.local
          user: kubernetes-admin
      name: kubernetes-admin@cluster.local
    - context:
          cluster: cluster.local
          user: mehrdad-admin
      name: mehrdad-admin@cluster.local
    - context:
          cluster: cluster.local
          user: babak-admin
          namespace: collector
      name: babak-admin@cluster.local

current-context: kubernetes-admin@cluster.local
kind: Config
preferences: {}

users:
    - name: kubernetes-admin
      user:
          client-certificate-data: LS0tLS1CRUdJTiBDRVJ...
          client-key-data: LS0tLS1CRUdJTiBSU0EgUFJJ...
    - name: babak-admin
      user:
          client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0...
          client-key-data: LS0tLS1CRUdJTiBQUklWQVR...
    - name: mehrdad-admin
      user:
          client-certificate-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0...
          client-key-data: LS0tLS1CRUdJTiBQUklWQV...

Note: All the hashed values have been encoded to base64

get config context

kubectl --kubeconfig <new-config> config get-contexts

set default context

kubectl --kubeconfig <new-config> config use-context <context-name>

set default config

copy the new-config to ~/.kube/config
or set the KUBECONFIG environment variable

export KUBECONFIG=/path/to/your/new/config