Securing Kubernetes with OPA and Gatekeeper

Why?

The first question should be come in your mind is, why do we need to enforce policies to our kubernetes cluster?

Answer is It will help us to make our cluster more secure, compliant, and we can operate cluster smoothly. Here are some real world examples where it will be fruitful to enforce policy with OPA.

1 ) Security Compliance : let's assume you are working in finance and healthcare sector. In such case , such policy can help to check that all the containers have the latest security updates and are not using any special permission like root user(Sudo). This helps us to prevents hackers from breaking in and stealing important information.

Resource management: Let's assume you are part of big team. You are working on small application or small feature of application and your created container are consuming too much resources like memory and cpus. By enforcing policies, we can ensure that resource limits and requests are properly set for each application, preventing resource contention and ensuring fair usage across the cluster.
Operation Consistency : To main consistency in deployments and operations, we can enforce policies that can ensure that all deployments have proper liveness and readliness probes configured. This helps in maintaining the reliability of applications by ensuring they are monitored correctly and can recover from failures promptly.

Now let's see How?

Prerequisites : Kubernetes cluster

Installing OPA Gatekeeper

Run Following commands to install gatekeeper.

helm repo add gatekeeper https://open-policy-agent.github.io/gatekeeper/charts
helm install gatekeeper/gatekeeper --name-template=gatekeeper --namespace gatekeeper-system --create-namespace

Now let's check

Let's create and apply policies.

Enforcing Security Policies

Scenario: Ensuring all containers run with non-root users for enhanced security.

create constraint template:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sdisallowrootuser
spec:
  crd:
    spec:
      names:
        kind: K8sDisallowRootUser
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sdisallowrootuser

        violation[{"msg": msg}] {
          input.review.object.spec.containers[_].securityContext.runAsUser == 0
          msg := "Running containers as root is not allowed"
        }

Let's create constraint:

constraint.yml (we will use same name as created CRD in Kind)

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: :
metadata:
  name: disallow-root-user
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

Now let's test our created policy.

create one pod using manifest, which will violate the policy.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-root
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        runAsUser: 0

When we try to create pod using above manifest them we will get error.

Let's try one more pod, which will follow policy.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-non-root
spec:
  containers:
    - name: nginx
      image: nginx
      securityContext:
        runAsUser: 1000

Now create one pod using above manifest.

Compliance and Governance

Scenario: Enforcing resource limits and requests to ensure fair usage and prevent resource exhaustion.

create constraint template:

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredresourcelimits
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredResourceLimits
      validation:
        openAPIV3Schema:
          properties:
            message:
              type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredresourcelimits

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not container.resources.requests.cpu
          not container.resources.requests.memory
          msg := "CPU and memory limits must be set for all containers"
        }

create constarint.yml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredResourceLimits
metadata:
  name: require-resource-limits
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

Now, Let's try it:

we will create 2 pods , one which will violate the policy and one which follow the policy.

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-no-limits
spec:
  containers:
    - name: nginx
      image: nginx

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-with-limits
spec:
  containers:
    - name: nginx
      image: nginx
      resources:
        limits:
          cpu: "500m"
          memory: "256Mi"

Let's Last admission policy which will restrict to use docker images from unknown registry.

Operational Policies

Usecase: If you want to whitelist image to ensure only approved container images are deployed.

Template.yml

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8sallowedimages
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedImages
      validation:
        openAPIV3Schema:
          properties:
            message:
              type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedimages

        violation[{"msg": msg}] {
          container := input.review.object.spec.containers[_]
          not startswith(container.image, "docker.io/")
          msg := "Only images from 'docker.io' are allowed"
        }

constraint.yml

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedImages
metadata:
  name: allow-approved-images
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]

Let's try two pods.

vpod.yml

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-unapproved-image
spec:
  containers:
    - name: nginx
      image: gcr.io/example.com/my-project/nginx

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-approved-image
spec:
  containers:
    - name: nginx
      image: docker.io/nginx

The policies we implemented fall under the category of Validating Admission Controllers.These controllers check requests to the Kubernetes API server against predefined policies before they are saved to the cluster.

Other Types of Admission Controllers

Mutating Admission Controllers: These controllers can change the objects being added to the cluster. They can add or modify fields in resource requests before they are saved.
Validating Admission Controllers: As we used in our examples, these controllers check requests against policies and can reject those that do not comply.

Conclusion and Next Steps

By using OPA and Gatekeeper with Kubernetes, we can enforce various policies to enhance security and compliance. The examples we covered demonstrate how powerful and flexible policy enforcement can be.

In the next article, I will cover Mutating Admission Controllers in more detail. We will explore.