Defining Kubernetes native policies with Kyverno – full tutorial

Anais Urlichs -

There are multiple ways for controlling the resources that get deployed to your Kubernetes environment including GitOps and automating deployments through CI/CD pipelines so that only a few people require the kubeconfig. However, what if all measures set in place fail?

This is exactly where Kubernetes-native policies can provide additional security.

In this blog post, we will show you how to

  • Use Kyverno in your Kubernetes cluster,
  • Deploy policies that ensure only container images with a signature get deployed
  • And provide an overview of the Kyverno CLI

The Video Tutorial:

What is Kyverno

Kyverno makes it possible to set up Kubernetes-native policies inside your cloud native infrastructure to validate the resources that get deployed. The deployed resources can then be used to validate, mutate or generate Kubernetes resources. At the time of writing this blog post, Kyverno is a CNCF incubating project with 2.8k stars on GitHub. (Give it a star if you like the project ⭐)

The benefit of defining policies through Kubernetes manifests are

  1. If everything is a Kubernetes resource, you can use the same tools and processes to manage every aspect of your application. For instance, Kyverno can be used through the `krew` Kubernetes-plugin tool in `kubectl` and the Kyverno CLI connects directly to kubectl and your kubeconfig.
  2. Policies can be enforced from within the cluster, making enforcement more reliable.

Feature List

A full feature list is provided below:

  • policies as Kubernetes resources (no new language to learn!)
  • validate, mutate, or generate any resource
  • verify container images for software supply chain security
  • inspect image metadata
  • match resources using label selectors and wildcards
  • validate and mutate using overlays (like Kustomize!)
  • synchronize configurations across Namespaces
  • block non-conformant resources using admission controls, or report policy violations
  • test policies and validate resources using the Kyverno CLI, in your CI/CD pipeline, before applying to your cluster
  • manage policies as code using familiar tools like git and kustomize

Taken from the official documentation. Source

Hands-On tutorial

The resources used in the following section can be found in the following GitHub repository.

Prerequisites

  • Helm Installed
  • A running Kubernetes cluster & kubectl connected to the cluster

Ensure that your environment is working by running the following commands:

❯ kubectl get nodes
NAME                       STATUS   ROLES                  AGE   VERSION
kyverno-control-plane   Ready    control-plane,master   46d   v1.21.1

Note that I am running a one node KinD Kubernetes cluster. 

helm version
version.BuildInfo{Version:"v3.9.1", GitCommit:"a7c043acb5ff905c261cfdc923a35776ba5e66e4", GitTreeState:"clean", GoVersion:"go1.17.5"}


Installing the Kyverno CLI

This tutorial will use the Kyverno CLI in the last section. If you would like to follow along, please make sure to install the Kyverno CLI. The CLI can either be used locally on your machine or through your CI/CD platform.

The Kyverno documentation provides several different installation options.

In our case, we will use Homebrew:

brew install kyverno

Alternatively, the CLI can also be used through the kubectl krew plugin manager:

# Install Kyverno CLI using kubectl krew plugin manager 
kubectl krew install kyverno  

# test the Kyverno CLI kubectl 
kyverno version

Installing the Kyverno Helm Chart in your Kubernetes Cluster

In this tutorial, we are going to use the Kyverno Helm Chart to deploy Kyverno inside our Kubernetes cluster.

# Add the Kyverno Helm repository to your repository list helm
repo add kyverno https://kyverno.github.io/kyverno/  

# Scan your Helm repositories to fetch the latest available charts 
helm repo update  

# Install the Kyverno Helm chart into a new namespace called "kyverno" 
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Note that if you have a cluster with more than two nodes, or you need to have your deployments more reliable, then you probably want to install Kyverno with a replica count of 3. In case one of the pods is down, the others will still be able to respond.

helm install kyverno kyverno/kyverno -n kyverno --create-namespace --set replicaCount=3
NAME: kyverno
LAST DEPLOYED: Tue Aug 30 14:59:15 2022
NAMESPACE: kyverno
STATUS: deployed
REVISION: 1
NOTES:
Chart version: v2.5.3
Kyverno version: v1.7.3
Thank you for installing kyverno! Your release is named kyverno.
⚠️  WARNING: Setting replicas count below 3 means Kyverno is not running in high availability mode.

💡 Note: There is a trade-off when deciding which approach to take regarding Namespace exclusions. Please see the documentation at https://kyverno.io/docs/installation/#security-vs-operability to understand the risks.

You can find further configuration options of the Helm Chart in the values.yaml file.

The following is important: If all your Kyverno replicas are down and cannot be reached to validate your Kubernetes manifests, by default, Kyverno will let the policy checks and thus, the deployment of those manifests fail.

Next, ensure that Kyverno has started its pods and is running in your cluster:

❯ kubectl get deployment -n kyverno
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
kyverno   1/1     1            1           37s

Defining Policies

The Kyverno documentation provides a vast list of Kubernetes policies that you can deploy to Kyverno as YAML manifests. These include application-specific policies, such as policies for your ArgoCD deployment, and many more.

If you are new to Kubernetes policies, a policy basically defines which fields should or should not be present in your Kubernetes manifests, what values are or are not valid for the field and how the policy should be enforced. For instance, the following policy will ensure that the container image used in my pods has an attestation:

enforce-attestation.yaml

apiVersion: kyverno.io/v1 
kind: ClusterPolicy 
metadata: 
  name: check-image 
spec: 
  validationFailureAction: enforce 
  background: false 
  webhookTimeoutSeconds: 10 
  failurePolicy: Fail 
  rules: 
    - name: check-image 
      match: 
        any: 
        - resources: 
            kinds: 
              - Pod 
      verifyImages: 
      - imageReferences: 
        - "*:*" 
        attestors: 
        - count: 1 
          entries: 
          - keys: 
              publicKeys: |- 
                -----BEGIN PUBLIC KEY-----
                MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEvcLByTJSlRc6dSsBdZct3UrQQUu7
                /Q78ncxSQDu821I933Q3lVEXrd14km1CuWM7sHZdFNzVHM5sPMiY6Rj1NA==
                -----END PUBLIC KEY-----

The Kyverno verifyImages rule uses Cosign to verify container image signatures and in-toto attestations stored in an OCI registry. If you are new to attestations, Cosign or in-toto, have a look at the previous video on the Aqua Open Source YouTube channel:

Creating an SBOM Attestation with Trivy and Cosign from Sigstore

The policy rule check fails if the signature is not found in the OCI registry, or if the image was not signed using the specified key.

Let’s look at the core parts of the policy in more detail:

  1. kind: ClusterPolicy – if it is a `Policy` it will be namespace specific but if it is a `ClusterPolicy`, the policy will be applied across the cluster
  2. validationFailureAction: enforce – `enforce` means that when the policy is not met by the resource, an error will be thrown and the deployment of the resource in the cluster will fail. Alternatively, `audit` can be used to get a report on the resource violation
  3. resources – specifies the resource on which the policy should be enforced
  4. The last part first checks that the image has a signature that matches the public key provided.

A lot of times malicious images can be deployed because a malicious container registry `sounds like` or is `written like` the official registry. With this policy, we can ensure that we only use images that have been created by a trusted party of whom we know the public key.

We can then apply the YAML manifest like any other Kubernetes resource:

kubectl apply -f example-policies/enforce-attestation.yaml
clusterpolicy.kyverno.io/check-image configured

Lastly, we want to ensure that our policy works. First, we are going to apply a deployment that is not from our container registry:

kubectl apply -f https://k8s.io/examples/application/deployment.yaml

Error from server: error when creating "https://k8s.io/examples/application/deployment.yaml": admission webhook "mutate.kyverno.svc-fail" denied the request:

resource Deployment/default/nginx-deployment was blocked due to the following policies

check-image:
autogen-check-image: |
failed to verify signature for docker.io/nginx:1.14.2: .attestors[0].entries[0].keys: no matching signatures:

As you can see, the deployment failed.

Only when we deploy a container image from the registry specified that contains a signature that matches our public key, the deployment will work:

kubectl apply -f deployment.yaml
deployment.apps/react-application created

Using the Kyverno CLI

The Kyverno CLI has three main commands, namely:

  1. Apply
  2. Test
  3. JP
  4. Version

This section will demonstrate at a high level how to use the apply and the test command.

1. Apply

  • Dry-run resources before applying them to the cluster
  • Determine the effectiveness of a policy
  • Can show any mutated resources
kyverno apply example-policies/enforce-attestation.yaml --resource ./deployment.yaml

Applying 1 policy to 1 resource...
(Total number of result count may vary as the policy is mutated by Kyverno. To check the mutated policy please try with log level 5)
pass: 1, fail: 0, warn: 0, error: 0, skip: 2

The `apply` command can also be used to retrieve a report of your Kyverno policy checks:

2. Test

The test command scans a Git repository or local folder and runs the tests within.

The command recursively looks for YAML files with policy test declarations (described below) with a specified file name and then executes those tests.

The following can be defined in the test manifest:

  1. pass: The resource passes the policy definition. To validate rules which are written with a deny statement, this will not be a possible result. mutate rules can declare a pass.
  2. skip: The resource does not meet either the match or exclude block, or does not pass the preconditions statements. To validate rules which are written with a deny statement, this is a possible result. If a rule contains certain conditional anchors which are not satisfied, the result may also be scored as a skip.
  3. fail: The resource does not pass the policy definition. Typically used to validate rules with pattern-style policy definitions.
  4. warn: Setting the annotation policies.kyverno.io/scored to "false" on a resource or policy which would otherwise fail will be considered a warning.

Taken from the official documentation. Source

To use the Kyverno test command, we will need two things, a policy YAML file and then a test YAML file. The test YAML file will reference the policy and the resource (YAML manifest such as a deployment or pod) that you want to verify.

The following policy YAML file ensures that we are not using the latest tag in our image and that we specify a tag:

disallow-latest-tag.yaml

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-latest-tag
  annotations:
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/description: >-
      The ':latest' tag is mutable and can lead to unexpected errors if the 
      image changes. A best practice is to use an immutable tag that maps to 
      a specific version of an application pod.      
spec:
  validationFailureAction: audit
  rules:
  - name: require-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
      clusterRoles:
      - cluster-admin
    validate:
      message: "An image tag is required."  
      pattern:
        spec:
          containers:
          - image: "*:*"
  - name: validate-image-tag
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using a mutable image tag e.g. 'latest' is not allowed."
      pattern:
        spec:
          containers:
          - image: "!*:latest"

The disallow-latest-tag.yaml and our deployment.yaml are referenced within the kyverno-test.yaml manifest:

name: disallow_latest_tag
policies:
  - disallow-latest-tag.yaml
resources:
  - deployment.yaml
results:
  - policy: disallow-latest-tag
    rule: require-image-tag
    resource: cns-website
    kind: Deployment
    result: pass
  - policy: disallow-latest-tag
    rule: validate-image-tag
    resource: cns-website
    kind: Deployment
    result: pass

Next, we can use the kyverno-test.yaml to check our Kubernetes manifest; specifically that we defined an image tag, otherwise the latest tag would be used, and we have not defined the `latest tag` as the image tag.

kyverno test ./

With the kyverno test command, you can ensure that your manifests meet the requirements and standards of your organisation before trying to deploy them.

What’s next

This blog post covered:

  • An overview of Kyverno
  • Deploying Kyverno inside your Kubernetes cluster with Helm
  • Defining a policy, deploying the policy and testing it
  • Using the Kyverno CLI

If you enjoyed this blog post, make sure to subscribe to the following channels for more content:

  1. My YouTube Channel
  2. The Aqua Open Source YouTube Channel