Immutable Runtime Containers – Kubernetes Security Hardening

In my other posts under Kubernetes Security Hardening (Guides > Kubernetes Security Hardening), we have talked about securing Kubernetes environment by securing the supply chain i.e. docker images, pods, etc. In this post, we will talk about runtime security.

What is Immutable Container?

By Immutable container, we mean that container cannot be modified during its lifetime.

If some get shell access to container and modify some file or install a package, then container becomes mutable and hence it drifts from its desired state during its lifetime. This means this is not the secure container anymore that we had deployed.

Hence it is important to enforce Immutability to a container running in Kubernetes Cluster.

Ways of Enforcing Immutability

There are few ways to enforce Immutability for containers. Mainly we make root file-system read only and remove shell or remove package manager or remove utilities like touch, nano editor, vim editor etc. We can do this at two levels:

1. Enforce at Image Level: This part we have already done in our blog post on docker image hardening when we used distroless images as base images. To enforce immutability at Image level you can remove shell or make file system read-only (in Dockerfile using chmod to make root system read-only by removing write permissions), etc.
   
2. Enforce at Runtime: In some scenario if you are for some reason not able to enforce immutability at image level, then we do this at POD level as well by using following:
   • startUpProbe: You can use startUpProbe to run commands to remove shell or utilities and also make filesystem read only.
   • initContainer: The initContainer run before application container. So you can use initContainer to perform actions that read-write permissions like updating a cache that your application container will use and then making file system of application readonly. This can be very specific and applicable to not many scenarios but is one way of running Immutable Containers.
   • securityContext: You can use securityContext to set flag readOnlyRootFilesystem to true

Let’s do some hands on examples for enforcing Immutability at runtime for containers.

Set up Test Lab

Before we start hands on, let’s set up a test lab using kind and docker. Please follow below for setting up test Lab

How to do quick Kubernetes POCs using kind?

Now we will start hands on.

Enforce Immutability with startUpProbe

startUpProbe is a feature provided by Kubernetes which is used to finish application start before other probes can run and application comes into READY state.

So we can use startUpProbe to make our container Immutable

Copy and paste the below code into apache-httpd.yaml file

apiVersion: v1
kind: Pod
metadata:
  name: apache
spec:
  containers:
  - name: httpd
    image: httpd
    ports:
    - containerPort: 80
    startupProbe:
      exec:
        command:
        - rm
        - /bin/touch
      initialDelaySeconds: 5
      periodSeconds: 5

Deploy the Pod

# kubectl apply -f apache-httpd.yaml 

Now we will try to create a file inside apache container using touch command

# kubectl get pod
NAME     READY   STATUS    RESTARTS   AGE
apache   1/1     Running   0          26s
# kubectl exec -it apache -- bash
root@apache:/usr/local/apache2# touch file
bash: touch: command not found

As you can see we could not create a file using touch. In this way you can remove any other utilities from stopping users or attackers to make any modifications.

Let’s try removing bash to stop users from getting shell access

Copy and paste this code to apache-httpd.yaml

apiVersion: v1
kind: Pod
metadata:
  name: apache
spec:
  containers:
  - name: httpd
    image: httpd
    ports:
    - containerPort: 80
    startupProbe:   
      exec:
        command:
        - rm 
        - /bin/bash
      initialDelaySeconds: 5
      periodSeconds: 5

Let’s recreate the pod

# kubectl delete -f apache-httpd.yaml 
pod "apache" deleted

# kubectl apply -f apache-httpd.yaml 
pod/apache created

We will try accessing shell and see if it allows us

# kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
apache   1/1     Running   0          48s
# kubectl exec -it apache -- bash
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "d7559d0db5d99a44618cc467c4b59094364949502f81286a1976a2889114bf06": OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown

Awesome! We could not get shell access to container.

Enforce Immutability with securityContext

Like mentioned before we can use securityContext to make root filesystem secure

Copy and paste this yaml to apache-httpd.yaml

apiVersion: v1
kind: Pod
metadata:
  name: apache
spec:
  containers:
  - name: httpd
    image: httpd
    ports:
    - containerPort: 80
    securityContext:        
      readOnlyRootFilesystem: true

Recreate the pod

# kubectl delete -f apache-httpd.yaml 
pod "apache" deleted

# kubectl apply -f apache-httpd.yaml 
pod/apache created

This will create the pod but it will fail

# kubectl get pods
NAME     READY   STATUS             RESTARTS      AGE
apache   0/1     CrashLoopBackOff   2 (17s ago)   41s

Why this happened?

You may have guessed it right. Because filesystem is read only. This will fail almost all of the applications.

How to fix Read only file system issue?

In order to fix read only filesystem issue we can mount emptyDir{} volume to provide read/write mounts to applications and hence enabling us to keep the root file system read only.

Let’s check the logs of above created apache pod

# kubectl logs apache

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.33. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.33. Set the 'ServerName' directive globally to suppress this message
[Wed Jan 04 02:01:17.289092 2023] [core:error] [pid 1:tid 281473095491600] (30)Read-only file system: AH00099: could not create /usr/local/apache2/logs/httpd.pid.InrxVg
[Wed Jan 04 02:01:17.289152 2023] [core:error] [pid 1:tid 281473095491600] AH00100: httpd: could not log pid to file /usr/local/apache2/logs/httpd.pid

As you can see above it failed due to /usr/local/apache2/logs being read only. Let’s fix this now

Copy and paste below yaml to apache-httpd.yaml

apiVersion: v1
kind: Pod
metadata:
  name: apache
spec:
  containers:
  - name: httpd
    image: httpd
    ports:
    - containerPort: 80
    securityContext:
      readOnlyRootFilesystem: true
    volumeMounts:
    - mountPath: /usr/local/apache2/logs
      name: apache-volume
  volumes:
  - name: apache-volume
    emptyDir:
      sizeLimit: 500Mi

Recreate the pod

# kubectl delete -f apache-httpd.yaml 
pod "apache" deleted
# kubectl apply -f apache-httpd.yaml 
pod/apache created

Let’s check the pod status

# kubectl get pods
NAME     READY   STATUS    RESTARTS   AGE
apache   1/1     Running   0          9s

# kubectl logs apache
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.34. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 10.244.0.34. Set the 'ServerName' directive globally to suppress this message
[Wed Jan 04 02:04:54.633015 2023] [mpm_event:notice] [pid 1:tid 281473599520784] AH00489: Apache/2.4.54 (Unix) configured -- resuming normal operations
[Wed Jan 04 02:04:54.633208 2023] [core:notice] [pid 1:tid 281473599520784] AH00094: Command line: 'httpd -D FOREGROUND'

Nice!! It is working now

Let’s check if filesystem other than /usr/local/apache2/logs read only or not.

# kubectl exec -it apache -- bash
root@apache:/usr/local/apache2# pwd
/usr/local/apache2
root@apache:/usr/local/apache2# touch test
touch: cannot touch 'test': Read-only file system

Awesome!! We have read only Filesystem.

Conclusion

In this post we learnt how we can further improve our security. This will definitely additional layer of security and will make your system more difficult to crack. Happy learning!!