Deploying Secure Pods – Kubernetes Security Hardening

We created secure docker images in blog post https://idlbuzz.com/kubernetes-security-hardening-secure-docker-images/

Now we will look into how to deploy secure Pods to a Kubernetes cluster.

What is a secure Pod?

We need to make sure certain steps are taken to ensure the Pod we are deploying are secure. If we do not then Pods we are deploying may be vulnerable for exploitation.

These steps are as follows:

1. Set security context while creating pods: This will ensure any malicious code won’t be able to execute and cause any harm to your Kubernetes cluster.
   • Set user id for the container so that container run as non root user.
   • Disable privileges so that containers do not have access to host devices or able to change kernel level configurations. The containers are by default non privileged. But Kubernetes allows privilege Escalation. We need to disable it as well.
   • We also need to drop all Linux capabilities as even with non privileged permissions containers have some linux capabilities which can enable containers to perform some privileged actions.
2. Use sha256 digest instead of image name: Instead of using image name like ldsingh/demo-app-hello-world:3797977240 use ldsingh/demo-app-hello-world@sha256:df286eb78dec2f0cefb684a4592f2a16b15af844785ef68b18f45cef2e50a4f3. This will ensure Image immutability. This means if someone add some malicious code and re-create image with same tag as ldsingh/demo-app-hello-world:3797977240, the sha256 digest cannot be same and will be different and hence your POD will still use malicious free image as you used sha256 digest and not image tag.
3. Disable service account mounting for application pods: Service accounts provide an identity to pods. This means with Service Accounts Pods have ability to communicate with K8s API. Even if you do not specify service account, K8s will map default service account (which is created by default in any namespace) and mount the token to Pod. So your each of your POD can connect to K8s API. Now application pods do not need access to K8s API. Only pods like Prometheus POD as example, will need access to K8s API for monitoring purpose. So it makes sense to disable secrets mounting. This can be done as follows:
   • Create a new Service Account and set auto mounting to false with automountServiceAccountToken: false
   • Set the pod to use newly created Service Account. Now your Service Account will be mapped but secret won’t be mounted and hence POD won’t have K8s API access.
   • Service Account can still be used to access K8s API but we can restrict access using RBAC.

Create secure POD

Now let’s try to create a secure pod.

Let’s checkout the code from: https://github.com/ld-singh/demo-app-hello-world.git

git clone https://github.com/ld-singh/demo-app-hello-world.git

Change directory to directory

cd demo-app-hello-world

Now let’s follow this blog post to set up our lab: https://idlbuzz.com/how-to-do-quick-kubernetes-pocs-using-kind/

Let’s change directory to yaml file

/work # cd kubernetes/yaml/

/work/kubernetes/yaml # pwd
/work/kubernetes/yaml

The yaml file to create pod uses image from my docker hub account which is publicly available and we will use this image

We have pod.yaml file which follows all the best practices mentioned above

---
apiVersion: v1
kind: Namespace
metadata:
  name: demo-app-ns
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: demo-app-sa
  namespace: demo-app-ns
automountServiceAccountToken: false

---
apiVersion: v1
kind: Pod
metadata:
  name: demo-app-pod
  namespace: demo-app-ns
spec:
  serviceAccountName: demo-app-sa
  automountServiceAccountToken: false
  containers:
  - name: demo-app-container
    image: ldsingh/demo-app-hello-world@sha256:df286eb78dec2f0cefb684a4592f2a16b15af844785ef68b18f45cef2e50a4f3
    ports:
    - containerPort: 3000
    securityContext:
       privileged: false
       allowPrivilegeEscalation: false
       runAsUser: 65532
       capabilities:
          drop:
            - ALL

You can see automountServiceAccountToken mentioned in Pod manifest as well. This is optional if mentioned in Service Account manifest. You can explicitly avoid secret token of Service Account’s mounting by add automountServiceAccountToken flag to Pod manifest if Service account is created without automountServiceAccountToken.

NOTE: User ID we used above is id of nonroot user of distroless base image used for building app image.

Now let’s deploy this

kubectl apply -f pod.yaml

We can see pod is created

/work/kubernetes/yaml # kubectl get pods -n demo-app-ns
NAME           READY   STATUS             RESTARTS        AGE
demo-app-pod   0/1     CrashLoopBackOff   6 (3m48s ago)   10m

Now we will expose this from another terminal ( not inside docker container lab)

% kubectl port-forward pod/demo-app-pod 3000:3000 -n demo-app-ns 

Let’s check browser

Now we have secure pod running and is accessible as well. POD only have access which it needs to run the app and hence it is very secure.

Since we are using a distroless base image for our image it does not have a shell so we cannot login to POD. But in case we deploy debug version of distroless base image, then too we will only have fully non-privileged access.

Conclusion

We can have a secure docker image and then further make POD even more secure by using these methodology and hence adding another layer of security. These methods can be made standard across all microservices of your applications.