For a while, we’ve been exploring the idea of using OCI annotations to track the lifecycle of container images. The problem we are trying to solve is as follows. Container images are immutable and cannot be dynamically patched like virtual machines. To apply the latest updates to a containerized application, teams must produce a new image with the patches. Once the new image is produced, the old one should be considered outdated and vulnerable, and all workloads using the old image should be redeployed with the new one.

The problem is how to mark the old image as outdated. Why? Because teams may have pinned their deployments to a digest or an immutable tag, and we want them to move to the patched version. We also want to create policies that outdated images should not be deployed. Finally, we want an automated way to point the teams to the latest patched image. Unfortunately, using digests and tags prevents us from achieving those goals.

Container Image Lifecycle Example

Here is a concrete example. I will use semantic versioning for the tags to make the example more relatable.

  • First Revision: Build the application image ghcr.io/toddysm/flasksample:1.0
    1.0 in this example is a rolling tag. To differentiate between images, I may want to use an immutable tag; for example 1.0-20230707 using the YYYYMMDD format for the date. This image has a digest sha256@1234567890.
  • Second Revision: New vulnerabilities are found in ghcr.io/toddysm/flasksample:1.0 and I rebuild the image and tag it using the same 1.0 rolling tag.
    I also tag it with an immutable tag like 1.0-20230710. This image has a different digest sha256@5647382910.

At this moment, I have two images for the same application from the same lineage 1.0. Here is the relation between tags and digests:

  • Tags 1.0 and 1.0-20230710 point to digest sha256@5647382910
  • Tag 1.0-20230707 points to digest sha256@1234567890

If I have pinned my deployment to the tag 1.0-20230707 or the digest sha256@1234567890 I do not know whether the image is still fresh or has a newer patched version. I could try interpreting the tags, but this will be very custom to my tagging scheme. For example, my tagging scheme 1.0-YYYYMMDD differs from Python’s, NodeJS’, Alpine’s, or Ubuntu’s scheme (well, Ubuntu’s is very similar :)). Also, obtaining the tag from the image digest is not possible.

The idea is to use OCI annotations to add additional information to the images. This information can help us communicate the deprecation of images and preserve vital information lost when retagging. We are not the only ones thinking in this direction – the folks from Ubuntu also want to use annotations to deprecate images, although their goal is a bit different.

OCI Annotations for Image Lifecycle

OCI annotations are key-value pairs that you can add to the manifest of any OCI artifact, including container images. The problem is that annotations cannot be changed once the manifest is created. Hence, if you add the OCI annotations to the image manifest, you cannot update them anymore. The workaround to that is to use the OCI referrer capability and add a new artifact with annotations that is linked to the image. In essence, whenever you need to update an annotation, you must push a new artifact with the full set of annotations and link it to the image. The consumer of the annotations needs to list all referrer artifacts with the “annotation” type and take the latest one.

The other question that comes to mind is: “What annotations will help you track the image lifecycle?” OCI already specifies some pre-defined keys that you can leverage. There are three important ones that will help you manage the lifecycle:

  • org.opencontainers.image.created can be used to specify the date at which the image was created. For example, when the image is built.
  • org.opencontainers.image.version can be used to specify the version of the software. Think of this as the lineage of the software (i.e. Python 3.10 or Ubuntu Jammy).
  • org.opencontainers.image.revision can be used to sperify the patch version of the software. For example, Python 3.10.12 or Ubuntu Jammy 20230624.

One thing that OCI does not specify is an annotation for end-of-life of the image. For that, you can use custom annotation like vnd.myorganization.image.end-of-life.

How to Use Annotations for Image Lifecycle?

Let’s look at how the above annotations can be used to manage the lifecycle of a series of images. I will use concrete dates for the example.

First Revision of the Image: Build the application image ghcr.io/toddysm/flasksample:1.0. As part of the build process, add the following annotations to the image:

{
    "org.opencontainers.image.created" : "2023-07-07T00:00:00-08:00",
    "org.opencontainers.image.version" : "1.0",
    "org.opencontainers.image.revision" : "20230707"
}

Second Revision of the Image: Vulnerabilities are discovered in the ghcr.io/toddysm/flasksample:1.0 image. Rebuild the image with the fixes and add the following annotations to it:

{
    "org.opencontainers.image.created" : "2023-07-10T00:00:00-08:00",
    "org.opencontainers.image.version" : "1.0",
    "org.opencontainers.image.revision" : "20230710"
}

Find the previous image in the registry ghcr.io/toddysm/flasksample:1.0-20230707 and update the annotations to the following:

{
    "org.opencontainers.image.created" : "2023-07-07T00:00:00-08:00",
    "org.opencontainers.image.version" : "1.0",
    "org.opencontainers.image.revision" : "20230707",
    "vnd.myorganization.image.end-of-life" : "2023-07-10T00:00:00-08:00"
}

With those annotations in place, you can track the lifecycle of each image. But not only that! You can always determine the latest and most up-to-date image in the lineage by just pulling the 1.0 tag (which is available in the org.opencontainers.image.version annotation for every image in the lineage). Both the mutable and the immutable tags are preserved with the image. New annotations can always be added to the image by adding “empty” referrer artifacts with just the annotations. Also, annotations for the images are available even if the image is pulled by its digest.

Here are a couple of scenarios that you can implement with this information.

  • Block deployment of images that are end-of-life
    You can implement a policy to block the deployment of images that are end-of-life on your Kubernetes clusters. Such a policy can be implemented in admission controllers like Kyverno or Gatekeeper.
  • Suggest updated image in action items in vulnerability reports
    Current vulnerability reports for container images are hardly actionable because they do not provide an update path for the reported images. Development teams are not interested in how many vulnerabilities are discovered but in the quickest way to fix those.
  • Automate the process for rebuilding dependent images
    Tools like Dependabot can use the image lifecycle information to create pull requests for dependent images. This can speed up the process of fixing vulnerabilities and improving the vulnerability posture for the application.

All that sounds great, but the problem is the tooling support. While OCI specifies how you can store artifacts in registries and defines some standard annotations, very few, if any, tools allow you to easily achieve the above experience. I took it upon myself to give it a try!

Implementing Image Lifecycle Annotations with Existing Tools

Note: The below experience uses Docker buildx, ORAS, and GitHub Container Registry (GHCR) for the experience. The experience is as of July 10th, 2023, and may (or most certainly will) change. Other tools like regctl can also be used instead of ORAS. As always, I will use my cssc-pipeline repository for storing any code for this blog post.

First, I will set some environment variables to avoid retyping and make the commands easier to follow.

export TEMP_LOCATION=temp
export IMAGE_VERSION=1.0
export FIRST_REVISION=20230707
export SECOND_REVISION=20230710
export REGISTRY=ghcr.io/toddysm/cssc-pipeline
export REPOSITORY=flasksample
mkdir -p $TEMP_LOCATION

Building the First Revision of the Image

The first step is to build the first revision of the image. Docker buildx can build the image, but the default option is to save the image in Docker’s proprietary format, which doesn’t allow the use of annotations. To use annotations, you must use the OCI exporter and save the image as a tarball. Here is the command that will allow you to do that:

docker buildx build . -f Dockerfile \
  -t ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  -o "type=oci,dest=${TEMP_LOCATION}/flasksample-${IMAGE_VERSION}-${FIRST_REVISION}.tar,annotation.org.opencontainers.image.created=20230707T00:00-08:00,annotation.org.opencontainers.image.version=${IMAGE_VERSION},annotation.org.opencontainers.image.revision=${FIRST_REVISION}" \
  --metadata-file ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}-metadata.json

The command above creates an image in OCI format and saves it as a tarball. We can use the generated metadata file to obtain the image’s digest.

export FIRST_REVISION_DIGEST=`cat ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}-metadata.json \
  | jq -r '."containerimage.descriptor".digest'`
echo $FIRST_REVISION_DIGEST

In my case, the digest is sha256:1446094f076dcbc2b7e7943ae3806bb44003ee9e6c94efd3208b8f04159aa8c0.

Next, I will use the following ORAS command to push the OCI image to the GHCR registry:

oras cp --from-oci-layout ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}.tar:${IMAGE_VERSION} \
  ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION}

I can verify that the annotations are set on the image by pulling the manifest and checking the annotations field:

oras manifest fetch ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  | jq .annotations

At this point, I have the first revision of the image built and published to GHCR under ghcr.io/toddysm/cssc-pipeline/flasksample:1.0.

Building the Second Revision of the Image

A few days later, if vulnerabilities are discovered in the image, I need to update the image with the latest patches. As part of the build process, I also need to update the annotations of the previous image revision.

The first thing I need to do is to obtain the digest of the first revision. Because the first revision is still tagged with 1.0, I can quickly get the digest using the following command:

export OLD_IMAGE_DIGEST=`oras manifest fetch --descriptor ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  | jq .digest | tr -d '"'`
echo $OLD_IMAGE_DIGEST

That command returns the same digest as before sha256:1446094f076dcbc2b7e7943ae3806bb44003ee9e6c94efd3208b8f04159aa8c0. Now, I have a unique reference to the first revision of the image. I can build the second revision of the image using the same commands as before.

# Build the second revision of the container image with annotations...
docker buildx build . -f Dockerfile \
  -t ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  -o "type=oci,dest=${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}.tar,annotation.org.opencontainers.image.created=20230710T00:00-08:00,annotation.org.opencontainers.image.version=${IMAGE_VERSION},annotation.org.opencontainers.image.revision=${SECOND_REVISION}" \
  --metadata-file ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}-metadata.json

# Get the digest for the second revision...
export SECOND_REVISION_DIGEST=`cat ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}-metadata.json \
  | jq -r '."containerimage.descriptor".digest'`
echo $SECOND_REVISION_DIGEST

# Use ORAS to push the second revision to the registry...
oras cp --from-oci-layout ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}.tar:${IMAGE_VERSION} \
  ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION}

# Use ORAS to verify the annotations are set on the image...
oras manifest fetch ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  | jq .annotations

You can check that the second revision digest is different from the first revision using the following commands:

export IMAGE_DIGEST=`oras manifest fetch --descriptor ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
  | jq .digest | tr -d '"'`
echo $IMAGE_DIGEST

For me, the digest of the second revision (or the most up-to-date image) is sha256:4ee61e3d9d28fe15cffc33854a2b851e2c87929a99f0c71bfc7a689ad372894d.

Updating the Lifecycle Annotations for the First Revision of the Image

This is the most important step of the process – I need to go back and update the lifecycle annotations for the first revision of the image and mark it as end-of-life. This is a bit trickier process because the manifest of the original image cannot be modified. It is immutable! I need to create a referrer artifact and store the lifecycle annotations in the manifest of this referrer artifact. However, the referrer artifact should be empty (well, you can put a cat picture there, but it is irrelevant πŸ™‚ ) ORAS already supports push with an empty artifact. In the future, OCI-compliant registries will support empty layers for artifacts too. Here are the steps for that.

First, I will fetch the annotations for the first revision and update them with the end-of-life annotation.

oras manifest fetch ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} \
  | jq .annotations \
  | jq '. += {"vnd.myorganization.image.end-of-life":"20230710T00:00-08:00"}' \
  | jq '{"$manifest":.}' \
  > ${TEMP_LOCATION}/annotations.json

Note that ORAS uses a special JSON schema for annotation files. Hence, I needed to convert the annotations that I retrieved from the image manifest to a new JSON object expected by ORAS. Here is what the resulting file looks like:

jq . ${TEMP_LOCATION}/annotations.json                                                                         

{
  "$manifest": {
    "org.opencontainers.image.created": "20230707T00:00-08:00",
    "org.opencontainers.image.revision": "20230707",
    "org.opencontainers.image.version": "1.0",
    "vnd.myorganization.image.end-of-life": "20230710T00:00-08:00"
  }
}

Now, I need to push the empty artifact and refer to the first revision of the image. To do that, I will also need to use an artifact type (or mediaType in OCI language) to find my lifecycle annotations later on easily. There is no standard mediaType for that, so I have to invent my own – I will use application/vnd.myorganization.image.lifecycle.metadata. Here is the ORAS command that you can use to update the annotations:

oras attach --artifact-type application/vnd.myorganization.image.lifecycle.metadata \
  --annotation-file ${TEMP_LOCATION}/annotations.json \
  ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} \
  ${TEMP_LOCATION}/empty.layer

OK, I am done with setting the lifecycle annotations for the images.

Fetching the Lifecycle Annotations for Each Image

I can fetch the annotations for each image by simply using the following commands:

oras manifest fetch ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} | jq .annotations

oras manifest fetch ${REGISTRY}/${REPOSITORY}@${NEW_IMAGE_DIGEST} | jq .annotations

There is a problem, though! Those commands fetch the annotations that are set in the image manifest. For the outdated image (i.e. $OLD_IMAGE), the command will not return the end-of-life annotation. To get that annotation, I will need to fetch the referrer with the particular type application/vnd.myorganization.image.lifecycle.metadata. Here is how to do that.

First, I need to get the digest of the referrer artifact.

export ANNOTATIONS_ARTIFACT_DIGEST=`oras discover --artifact-type "application/vnd.myorganization.image.lifecycle.metadata" \
  ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} -o json \
  | jq '.manifests[0].digest' \
  | tr -d '"'`
echo $ANNOTATIONS_ARTIFACT_DIGEST

And then retrieve the annotations set in the referrer’s artifact manifest.

oras manifest fetch ${REGISTRY}/${REPOSITORY}@${ANNOTATIONS_ARTIFACT_DIGEST} | jq .annotations

As a logic for implementation, I would always check if the image has a referrer lifecycle artifact. If so, I will ignore the lifecycle annotations in the image manifest.

Closing Thoughts

Lifecycle annotations enable interesting scenarios for securing container supply chains and improving containerized applications’ vulnerability posture. Though the tooling can undoubtedly be improved – I had to do a lot of JSON conversions to get it working. The lack of standard annotation for end-of-life and mediaType for the referrer artifact makes the above solution very proprietary.

One issue that can arise is if multiple application/vnd.myorganization.image.lifecycle.metadata referrers are created. OCI doesn’t specify how to retrieve the latest artifact from registries by type. If multiple lifecycle annotations artifacts are pushed for the image, the client must pull and inspect each. This logic can be quite complex and can impact the performance on the client’s side.

An improvement that can be made to the process above is to sign each artifact (the image and the lifecycle annotations artifact). This will ensure that the annotations are trustable and not tampered with. Of course, an attacker can always remove the referrer artifact from the registry and leave the client with the impression that the image is still fresh.

Those are all food for thought and good topics for future posts. Here is also a video of the whole experience described above.

[UPDATE: 2023-03-26] When I wrote this post, the expectation was that OCI will release version 1.1 of the specification with artifact manifest included. This release was supposed to happen by end of Jan 2023 or mid Feb 2023. Unfortunately, the OCI 1.1 Image Spec PR 999 put a hold on that and as of today, the spec is not released. Although I promised to have a Part 2, due to the changes in the spec, continuing the investigation in the original direction may not be fruitful and helpful to anyone. Most of the functionality described below is removed from many registries and the steps and the information may be incorrect. The concepts are still relevant but their actual implementation may not be as described in this post. Consider the relevance of the information applicable only between Jan 5th 2023 and Jan 24th 2023 – the date the above PR was submitted. There will be no other updates to this post or Part 2 of the series. Instead of Part 2, folks may find the Registry & client support for Image Manifest type artifacts issue relevant to what they are looking for.

If you are deep into containers and software supply chain security, you may have heard of OCI referrers API and OCI artifacts. If not, but you are interested in the containers’ secure supply chain topic, this post will give you enough details to start exploring new registry capabilities that can significantly improve your software supply chain architecture.

This will be a two-part series. In the first part, I will examine the differences between OCI 1.0 and OCI 1.1 and their support across registries. In the second part, I will look at more advanced scenarios like deep hierarchies, deleting artifacts, and migrating content between registries with different support.

But before we start…

What is OCI?

The Open Container Initiative (OCI) is the governance organization responsible for creating open industry standards for container formats and runtimes. OCI develops and maintains three essential specifications:

  1. The OCI Image Format Specification defines the structure and the layout of an image or artifact. If you are interested in reading more about the OCI image layout, I recommend the No More Additional Network Requests – Enter: OCI Image Layout post from @developerguy. It will give you a good background on how the image is structured. I will mainly discuss the OCI Artifract Manifest in this post.
  2. The OCI Distribution Specification defines the APIs that registries should implement to enable the distribution of artifacts. The OCI Referrers API is part of this specification and will be discussed in this post.
  3. The OCI Runtime Specification specifies the configuration, the execution environment, and the lifecycle of a container.Β  I will not discuss the runtime specification in this post.

One additional note. You may have heard of the term OCI reference types in the past. This was the name of the working group (WG) responsible for driving the changes in the image format and distribution specification. The prototype implementation of reference types was first implemented in ORAS. Its usefulness was the reason it was brought to the attention of the OCI group and resulted in the new changes.

Disclaimer: One last thing I have to mention is that, at the time of this writing, the OCI specifications (OCI 1.1) that support the new artifact manifest changes and the referrers API is in release candidate 2 (RC.2). The release of the OCI 1.1 specifications is planned for February 2023. Keep in mind that not many registries support the new artifact manifest and the referrers API due to this fact. This post aims to describe the scenarios it enables and discuss the backward compatibilities with registries that support the current OCI 1.0 specifications. I will also test several registries and point out their current capabilities.

What Scenarios Do OCI Artifact Manifest and Referrers API Enable?

As always, I would like to start with the scenarios and what are the benefits of using those new capabilities. As part of the ongoing software secure supply chain efforts, every vendor must produce metadata in addition to the actual software. Vendors need to add human and machine-readable metadata describing the software, whether this is a binary executable or a container image. The most common metadata discussed nowadays is software bills of materials (SBOMs) and signatures. SBOMs list the packages and binaries used in the individual piece of software (aka the software “ingredients”). The signature is intended to testify about the authenticity of the software and prevent tampering with the bits.

In the past, container registries were intended to store only container images. With the introduction of OCI artifacts, container registries can store other artifacts like SBOMs, signatures, plain text files, and even videos. The OCI referrers API goes even further and allows you to establish relationships between artifacts. This is a compelling functionality that allows you to create structures like this:

+ Container Image
    - Signature of the Container Image
    + SBOM for the Container Image
        - Signature of the SBOM
    + Vulnerability Report for the Container Image
        - Signature of the Vulnerability Report
    + Additional Container Image metadata
        - Signature of the additional metadata
    - ...

Now, the container registry is not just a storage place for images but a generic artifacts storage that can also define relations between the artifacts. As you may have noticed the trend in the industries, the registries are not referred to as container registries anymore but as artifact registries.

There are many benefits that the new capabilities offer in addition to storing various artifacts:

  • Relevant artifacts can be stored and managed together with the subject (or primary) artifact.
    Querying and visualizing the related artifacts is much easier than storing them unrelated. This can result not only in more manageable implementations but also in better performance.
  • Relevant artifacts are easily discoverable.
    Pulling an image from a registry may require additional artifacts for verification. An example is signature verification before allowing deployment. Using the OCI referrers API to get an image’s signature will be a trivial and standardized operation.
  • Relevant artifacts can be copied together between registries.
    Content promotion between registries is a common scenario in container supply chains. Now, the image can be promoted to the target registry with all relevant artifacts instead of making many calls to the registry to discover them before promotion.

Because the capabilities are still new, how to standardize the implementations is still in discussion. You can look at my request for guidance for OCI artifacts for more variations of the above scenario and the possible implementations.

For this post, though, I will concentrate on a straightforward scenario using BOMs. I want to attach three different SBOMs to an image and test with a few major registries to understand the current capabilities. I will build the following content structure:

+ Container image
  artifactType: "application/vnd.docker.container.image.v1+json"
    - SPDX SBOM in JSON format
      artifactType: "application/spdx+json"
    - SPDX SBOM in TEXT format
      artifactType: "text/spdx"
    - CycloneDX SBOM in JSON format
      artifactType: "application/vnd.cyclonedx+json"

I also chose the following registries to test with:

You may not be familiar with the Zot and the ORAS registries listed above, but they are Open Source registries that you can run locally. Those registries are on top of any new OCI capabilities and one of the first registries to implement those. They make it a good option for testing new OCI capabilities.

Now, let’s dive into the registry capabilities.

Creating the Artifacts

All artifacts and results can be found in my container secure supply chain playground repository on GitHub. I have created the usual flasksample image and generated the SBOMs using Syft. Here are all the commands for that:

# Buld and push the image
docker build -t toddysm/flasksample:oci1.1-tests .
docker login -u toddysm
docker push toddysm/flasksample:oci1.1-tests

# Generate the SBOM in various formats
syft packages toddysm/flasksample:oci1.1-tests -o spdx-json > oci1.1-tests.spdx.json
syft packages toddysm/flasksample:oci1.1-tests -o spdx > oci1.1-tests.spdx
syft packages toddysm/flasksample:oci1.1-tests -o cyclonedx-json > oci1.1-tests.cyclonedx.json

I will use the above image and the generated SBOMs to push to various registries and test their behavior. Note that ORAS CLI can handle registries that support the new OCI 1.1 specifications and registries that support only OCI 1.0 specifications. ORAS CLI automatically converts the manifest to the most appropriate manifest based on the registry support.

Referring to Artifacts in Registries with OCI 1.0 Support

Docker Hub recently announced support for OCI Artifacts. Note, though, that this is support for OCI 1.0. Here are the commands to push the SBOMs to Docker Hub and reference the image as a subject:

oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json
# Command reponse
Uploading e6011f4dd3fa oci1.1-tests.spdx.json
Uploaded  e6011f4dd3fa oci1.1-tests.spdx.json
Attached to docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4

oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx
# Command response
Uploading d9c2135fe4b9 oci1.1-tests.spdx
Uploaded  d9c2135fe4b9 oci1.1-tests.spdx
Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:16a58d1ed78402935d61e524f5609087334b164861618373d7b96a7b7c612f1a": response status code 405: unsupported: The operation is unsupported.

oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json
# Command response
Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Uploaded  c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:37ebfdebe499bcec8e5a5ce04ae4526d3e560c199c16a85a97f80a91fbf1d2c3": response status code 405: unsupported: The operation is unsupported.

Checking Docker Hub, I can see that the image digest is sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 as returned by the ORAS CLI above.

I expected to see another artifact with sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 (again returned by the ORAS CLI above). However, such an artifact is not shown in the Docker Hub UI. There is another artifact tagged with the digest of the image.

However, the digest of that artifact (sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd) is not the one from the ORAS CLI output.

Of course, the question in my mind is: “What is the digest that ORAS CLI returned above?” The sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 one. Using ORAS CLI or crane, I can explore the various manifests.

What Manifests Are Created When Referring Between Artifacts in OCI 1.0 Registries?

The oras discover command helps visualize the hierarchy of artifacts that reference a subject.

oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree                      
docker.io/toddysm/flasksample:oci1.1-tests
β”œβ”€β”€ application/spdx+json
β”‚Β Β  └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
β”œβ”€β”€ text/spdx
β”‚Β Β  └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
    └── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b

According to the output above, I should be able to see four artifacts. Also, as you can see, the digest sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 is the one for the first SBOM I attached to the image. To understand what is happening, let’s look at the different manifests. I will use the oras manifest command to pull the manifests of all artifacts by referencing them by digest:

# Pull the manifest for the image
oras manifest fetch docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json

# Pull the manifest for the SPDX SBOM in JSON format
oras manifest fetch docker.io/toddysm/flasksample@sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 > manifest-sha256-0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4.json

# Pull the manifest for the SPDX SBOM in TEXT format
oras manifest fetch docker.io/toddysm/flasksample@sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67 > manifest-sha256-6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67.json

# Pull the manifest for the CycloneDX SBOM in JSON format
oras manifest fetch docker.io/toddysm/flasksample@sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b > manifest-sha256-047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b.json

# Pull the manifest of the artifact tagged with the image digest
oras manifest fetch docker.io/toddysm/flasksample@sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd > manifest-sha256-c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd.json

All manifests are available in the dockerhub folder in my container secure supply chain playground repository on GitHub. The image manifest is self-explanatory and I will not dig into it. The other four are more interesting. Opening the manifest for the SPDX SBOM in JSON format, I can see that it is an artifact manifest "mediaType": "application/vnd.oci.artifact.manifest.v1+json" of type "artifactType": "application/spdx+json". It has a blob annotated with the name of the file I pushed. It also has a subject field referring to the image. The manifest for the SPDX SBOM in TEXT format and the CycloneDX SBOM in JSON format have the same structure. The hierarchy represented by the oras discover command above shows exactly those manifests. I believe the output of oras discover could be improved to show also the image digest for completeness:

oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree                      
docker.io/toddysm/flasksample:oci1.1-tests
│   └── sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 
β”œβ”€β”€ application/spdx+json
β”‚Β Β  └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
β”œβ”€β”€ text/spdx
β”‚Β Β  └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
    └── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b

The question remains how the manifest tagged with the image digest plays a role here. Looking at it, I can see that it is an index manifest "mediaType": "application/vnd.oci.image.index.v1+json" that lists the three SBOM artifacts I pushed. Remember, this index manifest is tagged with the image digest. This is similar to the structure Sigstore creates that I described in Implementing Containers’ Secure Supply Chain with Sigstore Part 2 – The Magic Behind. Here is a visual of how the manifests are related:

The SBOM artifacts are not visible in the Docker Hub UI because they are not tagged, and Docker Hub has no UI to show untagged artifacts. A few? questions remain:

  • What happens if I delete the image?
  • What happens if I delete the index manifest?
  • Can I create deeper hierarchical structures in registries that support OCI 1.0?
  • What happens when I copy related artifacts from OCI 1.0 registry to OCI 1.1 registry?

I will come back to those in the second part of the series. Before that, I would like to examine how registries with OCI 1.1 support storing the manifests for the referred artifacts.

Referring to Artifacts in Registries with OCI 1.1 Support

Azure Container Registry (ACR) just announced support for OCI 1.1. It is in Public Preview and supports the OCI 1.1 RC spec at the moment of this writing. After retagging the image, the commands for pushing the SBOMs are similar to the ones used for Docker Hub.

# Re-tag and push the image
docker image tag toddysm/flasksample:oci1.1-tests tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests
docker push tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests

oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json
# Command response
Uploading e6011f4dd3fa oci1.1-tests.spdx.json
Uploaded  e6011f4dd3fa oci1.1-tests.spdx.json
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a

oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx
# Command response
Uploading d9c2135fe4b9 oci1.1-tests.spdx
Uploaded  d9c2135fe4b9 oci1.1-tests.spdx
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766

oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json
# Command response
Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Uploaded  c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954

Azure portal does not show any additional artifacts and manifests, as shown in this screenshot:

This is a bit confusing, as I would at least expect to see a few more manifests. The distribution specification does not define functionality for listing untagged manifests; figuring out those dependencies without additional information will be hard. One noticeable thing is that no additional index manifest is tagged with the image digest.

oras discover command returns the following tree:

oras discover tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests -o tree
tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests
β”œβ”€β”€ application/vnd.cyclonedx+json
β”‚Β Β  └── sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954
β”œβ”€β”€ text/spdx
β”‚Β Β  └── sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766
└── application/spdx+json
    └── sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a

This is the same structure I saw above when using the command on the Docker Hub image. Pulling the manifests reveals that they are precisely the same as the ones from Docker Hub.

# Pull the manifest for the image
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json

# Pull the manifest for the SPDX SBOM in JSON format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a > manifest-sha-71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a.json

# Pull the manifest for the SPDX SBOM in TEXT format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766 > manifest-sha-0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766.json

# Pull the manifest for the CycloneDX SBOM in JSON format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954 > manifest-sha256-198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954.json

All manifests are available in the acr folder in my container secure supply chain playground repository on GitHub.

Luckily, ACR has CLI commands to list the manifests. Those commands call ACR’s proprietary APIs to gather the information. There are two ACR CLI commands I can use to list the manifests for a repository: acr manifest list and acr manifest metadata list . At the time of this writing acr manifest list had a bug and couldn’t list the OCI artifact manifests. acr manifest list-metadata worked fine and I could list all manifests in the repository. The output from the acr manifest list-metadata command is available here. From the output, I can see that only four manifests were created. There is no manifest index that points to the three artifacts. Here is a visual of how the manifests are related in an OCI 1.1 compliant registry:

To summarize the differences between the OCI 1.0 and OCI 1.1 referrers’ implementation:

  • In OCI 1.0 compliant registries, you will see an additional index manifest that is tagged with the image digest
  • In OCI 1.0 compliant registries, the index manifest lists the artifacts related to the image
  • In OCI 1.0 compliant registries, the artifact manifests still refer to the image using the subject field

I will look at how this impacts the content in your registry in the second part of this series.

Referrers Support Across Registries

Here is a table that shows the current (as of Jan 5th, 2023) support in registries.

The manifests and the debug logs are available in the corresponding registry folders in the cssc-pipeline repository on GitHub. You can refer to those for details.

Note: The investigation is done using the ORAS tool – the only one I am aware of that can create references between artifacts at the time of this writing. It may be possible to craft manifests manually and push them to the registries failing with ORAS.

Learnings

In addition to the above, I learned a few more things while experimenting with different registries.

  • As far as I know, OCI does not specify an API to list untagged manifests in a registry. This can be a problem because the attached artifacts do not have tags but only digests. I am pretty sure I ended up with some orphaned artifacts in the registries that do not support artifact referrers. Unfortunately, I cannot be sure due to the lack of such an API.
  • Registries are inconsistent in their responses when the capabilities are not supported. In my opinion, there is a lack of feedback on what capabilities each registry supports, which makes it hard for the clients. An easy way to check the capabilities of a registry would be beneficial.

In the next post of the series, I will go over more advanced scenarios like promotion between registries and building deeper hierarchies.

Photo by Petrebels on Unsplash