Trivy Operator Overview

Trivy, the all in one, cloud native security scanner enables users to scan your Kubernetes cluster for security issues right through the CLI. This is great for scanning static resources before deployment. However, to provide continuous scanning of already deployed resources inside your Kubernetes cluster, the Trivy Operator provides a better alternative to automate scans.

This blog post provides an overview of the difference between the Trivy CLI and the Trivy Operator and then details how you can get started with the Trivy Operator.

Using the Trivy Operator for in-cluster scanning vs the Trivy CLI

If you are already a Trivy user, you might be familiar with the trivy k8s command in the Trivy CLI that allows you to scan a Kubernetes cluster.

Using the Trivy CLI to perform Kubernetes cluster scans is a great option to perform ad-hoc scans if you have direct access to the cluster’s kubeconfig. However, in production environments, most cluster-admins and SREs might not have the necessary permissions to scan the cluster through the CLI.

For more information on the differences between the Trivy Operator and the Trivy CLI, take a look at the following blog post on the Aqua Security blog.

As an alternative to performing scans on the developer machine, it is also possible to scan the in-cluster resources with the Trivy CLI through CI/CD pipelines. The Trivy documentation provides several examples for different providers. However, that might have the following disadvantages if used for running workloads:

1. You will need to connect your cluster to your CI/CD pipeline. In some cases, those environments might already be connected. There is a difference between creating new services in specific namespaces and accessing your entire cluster for different types of security scans. The latter will provide your CI/CD pipeline with lots of privileges. Generally, it is bad practice to access infrastructure resources from outside the environment. It is much better to push information to external platforms from inside your infrastructure.

2. Inflating the purpose of your CI/CD pipeline. Most of the scans that you would perform with Trivy are focused on scanning your deployment resources. Trivy treats those resources as static. Meaning, those resources are scanned as they are at a single point in time. 
In order to prevent inflating your CI/CD pipeline, you would likely set up a separate pipeline for in-cluster scanning.

3. A lot of times, CI/CD pipelines will only run if there are changes to any of the workloads that are managed by the pipeline. However, new vulnerabilities might be disclosed in the meantime that affect your running workloads.

Overview of the Trivy Operator

The Trivy Operator is installed like any other Kubernetes Operator inside the cluster, following Kubernetes best practices. Users have the option to either install it through a Helm Chart or as Kubernetes Manifests through kubectl. Additionally, the following repository provides several other installation options, including:

• ArgoCD
• Terraform
• Pulumi (to be added)

You can find a video tutorial on the different installation options here.

Once installed, the Operator is responsible for monitoring the Kubernetes infrastructure and installed resources. First, the Trivy Operator will scan all existing resources. If the state of the cluster changes, either because existing resources are modified or new resources are installed, new scans will be triggered from the Trivy Operator on the resources that changed.

Features

The Trivy Operator can perform several different scans inside your cluster. Below, you will first find the name of the scan report that the Operator will generate and then the description of the report. *Taken from the README on GitHub

• VulnerabilityReports: Automated vulnerability scanning for Kubernetes workloads.
• ConfigAuditReport: Automated configuration audits for Kubernetes resources with predefined rules or custom Open Policy Agent (OPA) policies.
• ExposedSecretReport: Automated secret scans which find and detail the location of exposed Secrets within your cluster.
• RBACAssessmentReport: Role Based Access Control scans provide detailed information on the access rights of the different resources installed.
• InfraAssessment: K8s core component Infrastructure Assessment scan Kubernetes infra core components (etcd,apiserver,scheduler,controller-manager and etc) setting and configuration.
• Compliance reports:

* NSA, CISA Kubernetes Hardening Guidance v1.1 cybersecurity technical report is produced.

* CIS Kubernetes Benchmark v1.23 cybersecurity technical report is produced.

* Kubernetes pss-baseline, Pod Security Standards

* Kubernetes pss-restricted, Pod Security Standards

GitHub link: https://github.com/aquasecurity/trivy-operator/
Docs https://aquasecurity.github.io/trivy-operator/v0.14.1/

Installation

The Trivy Operator Documentation details different installation options. In this blog post, we will use the Helm Chart installation.

First, add the Aqua Security Helm registry to your Helm Charts:

helm repo add aqua https://aquasecurity.github.io/helm-charts/

Next, update your Chart registry:

helm repo update

And lastly, install the Chart*:

helm install trivy-operator aqua/trivy-operator \
  --namespace trivy-system \
  --create-namespace \
  --set="trivy.ignoreUnfixed=true" \
  --version 0.14.1

Note, please make sure that you are installing the latest version of the Trivy Operator Helm Chart.

There are multiple different ways in which you can customise the Helm Chart. The most common way is by providing a new values.yaml manifest.

To customise the Trivy Operator Helm Chart through the Values.yaml manifest, follow these steps:

1. Select the values that you would like to modify through the Helm Chart in the Trivy Operator GitHub repository.
2. Create a new values.yaml file in the directory from where you would like to install the Trivy Operator Helm Chart.
3. Add the values that you want to modify to that new values.yaml file.

For instance, the following content of the values.yaml manifest will specify that Trivy should only display vulnerabilities that already have a fix available:

trivy:
	ignoreUnfixed: false

Next, if you have not installed the operator already, install the operator but specify that you want to overwrite some of the default values with the values specified in the values.yaml file:

helm install trivy-operator aqua/trivy-operator \
--namespace trivy-system \
--create-namespace \
--values values.yaml  \
--version 0.14.1

Alternatively, modify the existing installation in your cluster:

helm upgrade --install trivy-operator aqua/trivy-operator \
--namespace trivy-system \
--create-namespace \
--values values.yaml  \
--version 0.14.1

Once you have installed Trivy, you want to ensure that it is running correctly inside your cluster:

kubectl get all -n trivy-system

Additionally, you could use either of the other installation options provided in the following GitHub repository:

Querying Reports

As soon as the Triviy Operator Helm Chart is installed inside of the cluster, it will launch the scans of your existing resources and infrastructure. The Trivy Operator will need access to the internet to update the Trivy Vulnerability Database. By default, the Operator will run scans every 6 hours. Meaning, if there is a new vulnerability for one of your container images, the Trivy Operator will report the new vulnerability even though you have not updated the image or related resources.

You can query each report produced by the Operator by the type provided in the “Feature” Section above. For instance, to look at the Vulnerability Reports, use the following command:

kubectl get vulnerabilityreports --all-namespaces -o wide

In the case of Vulnerability scans, each scan corresponds to one ReplicaSet inside of the cluster. Thus, if you run the same container image in multiple ReplicaSet, Trivy will scan each container image separately. This is by design.

Inspect created ConfigAuditReports, these are the misconfiguration scans, through the following command:

kubectl get configauditreports --all-namespaces -o wide

Inspect the work log of trivy-operator by:

kubectl logs -n trivy-system deployment/trivy-operator

Infrastructure scans:

kubectl get infraassessmentreports --all-namespaces -o wide

RBAC Assessment report:

kubectl get rbacassessmentreports --all-namespaces -o wide

Exposed Secrets Report:

kubectl get exposedsecretreport --all-namespaces -o wide

Cluster Compliance Report:

kubectl get clustercompliancereport --all-namespaces -o wide

This scan will produce 4 different types of compliance reports: NSA, CIS, Pod Security Standard Baseline and Restricted. You can read more in the documentation.

What’s next?

You might wonder how you can make it easier to access the scan reports from within the cluster. We have a tutorial in the Trivy Operator documentation that shows you how you can set up Grafana and Prometheus to access the metrics from the Trivy Operator scans inside your Grafana Dashboards. Do check that out for further information on managing security reports.

Additionally, we have lots of tutorials on the Aqua Security Open Source YouTube channel:

If you have any questions, either join the Slack community and ask your questions in the #trivy or #trivy-operator channel OR create a discussion on GitHub:

It would mean a lot to us if you could give the Trivy Operator and Trivy a Star on GitHub:
Trivy Operator: https://github.com/aquasecurity/trivy-operator/
Trivy: https://github.com/aquasecurity/trivy

Thank you!! ✨