Getting started with Notary V2

The software supply chain is composed of all the resources that contribute to the development, deployment and management of software projects. Basically, anything that is used anywhere of your project, whether as a direct or indirect dependency, is contributing to the software supply chain. Each resource provides an attack vector in the software supply chain.

Notary is one of the projects that enable users to verify the origin of container images before resources are accessed from container registries. This way, users can ensure only resources that have been labelled as trusted are used.

This blog post provides an overview of Notary, specifically,

The background of the project
Some of the theory on how Notary works
How to get started using the Notation (Notary V2) CLI
My personal thoughts about using Notary

Disclaimer Notary V2

I think it is worth noting that there is a lot of discussion going on at the moment around Notary V2. It is important to differentiate between Notary V1 and Notary V2. Notary V1 became a CNCF sandbox project in 2017 and is now an incubating project. Not only is the naming convention between both quite different & one would not necessarily find Notary V2 from Notary V1 in the GitHub repositories, but the user experience is also very different between the two projects.

Since Notary V1 does not seem actively maintained, I am focusing this blog post on Notary V2. “Not maintained” refers hereby to it having an active community with a public governance structure within the CNCF. There still seem to be lots of projects that integrate and build upon Notary V1 instead of Notary V2, so tech-wise, it still seems to be alive but just not maintained the way you would expect it from a CNCF project.

Overview Notary V2

Notary -- “it allows you to sign stuff” -- Source

From the official documentation:

> “Notation is a CLI project to add signatures as standard items in the registry ecosystem, and to build a set of simple tooling for signing and verifying these signatures. This should be viewed as similar security to checking git commit signatures, although the signatures are generic and can be used for additional purposes. Notation is an implementation of the Notary v2 specifications.”

In my words:

Notary can be used to sign container images and verify that container images are from a trusted entity. The idea is that instead of looking through all the resources used in your software supply chain to identify any resources that should not be there, you only allow verified resources to be used.

With Notary, users can generate signatures and use these signatures to sign their container image. The signature is then directly attached to the container image, and stored in the container registry. Anyone wanting to use that container image can then verify that the image they want to use does indeed belong to the entity whom they think has generated the container image. This explanation is a very simplified version of the key management that Notary uses to make this happen.

The screenshot below is taken from a KubeCon presentation that highlights this process. The Docker Daemon is first requesting the signatures of the ubuntu:latest image. One of the signatures that have been added to the image, was created by Bob. Given that I trust Bob, I will trust the image source and request the actual resources from the container registry.

Trusting Bob can mean in this context that I have used other images from Bob before and I have Bob's public signature, which makes it possible to verify resources are indeed from Bob.

Each signature consists of a public & private key pair. The private key of other people should always be unknown. However, Bob can distribute their public key and announce that they are the owner of the container image X. A user can then verify that Bob’s public key matches the signature on container image X.

KubeCon Presentation on Notary V1 and TUF

You might assume that you could just be trusting Bob. However, what If I manage to pretend to be Bob, or I look like Bob and start distributing malicious images that look similar to Bob's valid images? In that case, you would have no way of verifying which image belongs to whom. Thus, Notary can help with the following:

Visualise supply chain security
Ensure secure access to resources
Remove assumptions across the supply chain

Getting started

Assumptions that I think are true

Notary V2 only works with OCI-compatible registries. Docker Hub is not compatible with the latest OCI standards.
Notary V2 is a CLI tool. It is not meant to be installed e.g. in your Kubernetes cluster. However, you could probably install a policy in your cluster that verifies container images have been signed through Notary/are linked to a known truststore. This might be content for a follow-up blog post.

Installation

Notary V2 comes as a CLI tool. The documentation provides an installation guide

Thoughts:

It is great that they provide detailed installation instructions. However, being on e.g. Mac I would rather use Homebrew to install any CLI than to use the binary.

At this point, please follow the installation guide and then ensure that you have access to the ‘notation’ CLI. Here are the steps I followed on MacOS:

Download the binary from the release page -- In my case, it is notation_1.0.0-rc.3_darwin_arm64.tar.gz
Next, unpack the binary: tar -zxvf notation_1.0.0-rc.3_darwin_arm64.tar.gz
And move the binary into your bin directory: sudo mv ./notation /usr/local/bin/notation
Lastly, verify that you have access to the CLI by running the version command:

notation version
Notation: Notary v2, A tool to sign, store, and verify artifacts.
Version: 	1.0.0-rc.3
Go version:  go1.20.1
Git commit:  233c0ea71edfb68b951eb54a739a101e2a05cd26

Specify a trust policy

Policies allow us to define rules for our resources. These rules could check whether certain conditions are true or not. In this section, we will specify a “trust” policy that defines which entities are allowed to deploy resources.

First, we will create and build a container image for the following application: https://github.com/Cloud-Native-Security/website

The Dockerfile is specified here.

Build the container image for multiple platforms by running the following command at the root of the application:

docker buildx build --tag anaisurlichs/cns-website:0.0.9 --platform=linux/arm64,linux/amd64 .

You can find further details on creating multi-platform builds with buildx in the following blog post. Otherwise, use the docker build command that you are already familiar with.

If you built your image differently, make sure to push it to your container registry before carrying on with the next steps. In this case, we are going to use the GitHub container registry. DockerHub does not properly work with Notary/the Notation CLI since it is not fully OCI compliant. Thus, it will not allow you to sign images.

To use the GitHub container registry, follow these steps:

Find the image tag for the image that you would like to use: docker image ls
Tag your container image with the GitHub registry: docker tag <image tag> <image> e.g. docker tag eea985412114 ghcr.io/cloud-native-security/cns-website:0.0.9
Log into the GitHub container registry: notation login ghcr.io --username <your account username in all lowercase> --password <your personal access token> e.g. notation login ghcr.io --username cloud-native-security --password ghp_EG6Dg2o6rHt1wdakjhfsfghiusdgfisdugfisfbv34hebrhwef

To generate a personal access token, follow the GitHub documentation.

Moving on to actually using Notary

Before we can sign the container image and push the signature to our container registry, we need to log into our registry through the notation CLI. The command used is structured like this:

notation login <registry URL> --password <your password OR account token> --username <your registry username>

For example:

notation login ghcr.io --username cloud-native-security --password dhajsdhasdjaasfjhaskjfbsdfbsdhfbksdhbfksd

Make sure that you are using the access token to your registry, not actually a password. (LOL, I know!)

<registry URL> will be `ghcr.io `
<your password OR account token> will be your GitHub Personal Access token
<your registry username> will be your GitHub username or organisation name in lowercase letters e.g. `cloud-native-security`

With the following command, you can list the signatures of the container image:

IMAGE=<your image name>
notation ls docker.io/anaisurlichs/cns-website:0.0.9

At this point, we do not expect any signatures to be shown for our container image since we did not create any signatures yet.

notation cert generate-test --default "anaisurl.com"

anaisurl.com -- the name of the trust store, this can be the repository name that your docker image is based on
The `--default` flag sets this key as the default signing key

Note that this is a self-signed certificate and is for development purposes only.

Once we have created the key, we can verify we have it as part of our key

notation key ls

Note that if you have multiple keys, the key that is marked with a `*` is going to be the default key used if you don’t specify a key.

Then we can also check that the certificate is stored in a trust store:

notation cert ls

And now, we can sign our container image:

notation sign <image name>

For example

notation sign ghcr.io/cloud-native-security/cns-website@sha256:eda2a6ba1bc4fae703bfc78d947178a9e61b1d2418bb206e36efb105bc7d2ce4

Make sure to use the sha256 identifier as this is the only unique number; the image tag itself is mutable, meaning that you can push another image with the same tag to the container registry. The image tag might then be called the same but the SHA will be different.

By default, Notary supports two signature formats, JWS and COSE. If you do not specify the format, it will use JWS. However, you can specify COSE through the following flag:

notation sign --signature-format cose $IMAGE

Now that we have added a signature, we can list all of the signatures for our container image again:

notation ls $IMAGE

For example:

notation ls ghcr.io/cloud-native-security/cns-website:0.0.9
ghcr.io/cloud-native-security/cns-website@sha256:eda2a6ba1bc4fae703bfc78d947178a9e61b1d2418bb206e36efb105bc7d2ce4
└── application/vnd.cncf.notary.signature
└── sha256:8093cf0e5d4d287711954779ce937df5ea28afe23a5e13bb607946f1982e555b

Creating a trustStore to verify images

Lastly, we can verify our container image through a `trustStore`. More details are provided in the official Notation documentation. For this, we first need to create a turstpolicy.json file, similar to the following:

{
    "version": "1.0",
    "trustPolicies": [
        {
            "name": "trust-policy-example",
            "registryScopes": [ "*" ],
            "signatureVerification": {
                "level" : "strict" 
            },
            "trustStores": [ "ca:anaisurl.com" ],
            "trustedIdentities": [
                "*"
            ]
        }
    ]
}

Make sure to replace the information in the `trustStores` with the trustStore that you defined earlier. The `ca` is the trustStore type.

Next, we have to move this file into the respective directories, here is the snippet from the docs:

NOTE: For Linux, the notation configuration directory is ${HOME}/.config/notation/. For macOS, the notation configuration directory is ${HOME}/Library/Application\ Support/notation/. For Windows, the notation configuration folder is %USERPROFILE%\AppData\Roaming\notation\.

So we move it

mv trustpolicy.json ${HOME}/Library/"Application Support"/Notation

And then we can verify our container image against the trustPolicy that we know of:

notation verify $IMAGE

For example:

notation verify ghcr.io/cloud-native-security/cns-website@sha256:eda2a6ba1bc4fae703bfc78d947178a9e61b1d2418bb206e36efb105bc7d2ce4

Thoughts on Notary

Getting started with Notary is not easy due to several reasons. First of all, when you are looking for the Notary project via Google, it is difficult to find the actual project and to understand what the latest version is. Furthermore, when you look at ways to get started with Notary, the examples fall into either of two categories:

Overly simplistic example demos. These demos showcase how to spin up a local container registry and notary server locally to sign and verify container images.
Very complex examples that incorporate enterprise tools into the demo.

Furthermore, the documentation seems to need some work from the user-perspective as opposed to developers enhancing it further. So in that way, it would be a great project to contribute to.

Lastly, the documentation also does not provide adequate examples yet on integrating Notary into your existing workflow. If you have use cases and ways on how you are using Notary, please do contribute back to the documentation.

What’s next?

I hope that this blog post was useful. My goal was to try out Notary, share a getting-started guide and provide my thoughts on the project and the process.

If you enjoyed this article, please give my YouTube video a Thumbs-Up and give the Notary project a Star on GitHub. I am sure that it would mean a lot to them.

Also, if you enjoyed this end-to-end tutorial, do let me know in the YouTube comments of the video what tutorials you would like to see next.

Getting started with Notary V2

Introduction to Notary V2, a CNCF incubating project -- FULL TUTORIAL

Disclaimer Notary V2

Overview Notary V2

Getting started

Installation

Specify a trust policy

Moving on to actually using Notary

Creating a trustStore to verify images

Thoughts on Notary

What’s next?

Additional Resources

Getting started with Notary V2

Introduction to Notary V2, a CNCF incubating project -- FULL TUTORIAL

Disclaimer Notary V2

Overview Notary V2

Getting started

Installation

Specify a trust policy

Moving on to actually using Notary

Creating a trustStore to verify images

Thoughts on Notary

What’s next?

Additional Resources

💌 Subscribe to Anais Urlichs