software supply chain artifacts Archives

OCI Artifact Manifests, OCI Referrers API and Their Support Across Registries (Part 1)

[UPDATE: 2023-03-26] When I wrote this post, the expectation was that OCI will release version 1.1 of the specification with artifact manifest included. This release was supposed to happen by end of Jan 2023 or mid Feb 2023. Unfortunately, the OCI 1.1 Image Spec PR 999 put a hold on that and as of today, the spec is not released. Although I promised to have a Part 2, due to the changes in the spec, continuing the investigation in the original direction may not be fruitful and helpful to anyone. Most of the functionality described below is removed from many registries and the steps and the information may be incorrect. The concepts are still relevant but their actual implementation may not be as described in this post. Consider the relevance of the information applicable only between Jan 5th 2023 and Jan 24th 2023 – the date the above PR was submitted. There will be no other updates to this post or Part 2 of the series. Instead of Part 2, folks may find the Registry & client support for Image Manifest type artifacts issue relevant to what they are looking for.

If you are deep into containers and software supply chain security, you may have heard of OCI referrers API and OCI artifacts. If not, but you are interested in the containers’ secure supply chain topic, this post will give you enough details to start exploring new registry capabilities that can significantly improve your software supply chain architecture.

This will be a two-part series. In the first part, I will examine the differences between OCI 1.0 and OCI 1.1 and their support across registries. In the second part, I will look at more advanced scenarios like deep hierarchies, deleting artifacts, and migrating content between registries with different support.

But before we start…

What is OCI?

The Open Container Initiative (OCI) is the governance organization responsible for creating open industry standards for container formats and runtimes. OCI develops and maintains three essential specifications:

The OCI Image Format Specification defines the structure and the layout of an image or artifact. If you are interested in reading more about the OCI image layout, I recommend the No More Additional Network Requests – Enter: OCI Image Layout post from @developerguy. It will give you a good background on how the image is structured. I will mainly discuss the OCI Artifract Manifest in this post.
The OCI Distribution Specification defines the APIs that registries should implement to enable the distribution of artifacts. The OCI Referrers API is part of this specification and will be discussed in this post.
The OCI Runtime Specification specifies the configuration, the execution environment, and the lifecycle of a container. I will not discuss the runtime specification in this post.

One additional note. You may have heard of the term OCI reference types in the past. This was the name of the working group (WG) responsible for driving the changes in the image format and distribution specification. The prototype implementation of reference types was first implemented in ORAS. Its usefulness was the reason it was brought to the attention of the OCI group and resulted in the new changes.

Disclaimer: One last thing I have to mention is that, at the time of this writing, the OCI specifications (OCI 1.1) that support the new artifact manifest changes and the referrers API is in release candidate 2 (RC.2). The release of the OCI 1.1 specifications is planned for February 2023. Keep in mind that not many registries support the new artifact manifest and the referrers API due to this fact. This post aims to describe the scenarios it enables and discuss the backward compatibilities with registries that support the current OCI 1.0 specifications. I will also test several registries and point out their current capabilities.

What Scenarios Do OCI Artifact Manifest and Referrers API Enable?

As always, I would like to start with the scenarios and what are the benefits of using those new capabilities. As part of the ongoing software secure supply chain efforts, every vendor must produce metadata in addition to the actual software. Vendors need to add human and machine-readable metadata describing the software, whether this is a binary executable or a container image. The most common metadata discussed nowadays is software bills of materials (SBOMs) and signatures. SBOMs list the packages and binaries used in the individual piece of software (aka the software “ingredients”). The signature is intended to testify about the authenticity of the software and prevent tampering with the bits.

In the past, container registries were intended to store only container images. With the introduction of OCI artifacts, container registries can store other artifacts like SBOMs, signatures, plain text files, and even videos. The OCI referrers API goes even further and allows you to establish relationships between artifacts. This is a compelling functionality that allows you to create structures like this:

+ Container Image
    - Signature of the Container Image
    + SBOM for the Container Image
        - Signature of the SBOM
    + Vulnerability Report for the Container Image
        - Signature of the Vulnerability Report
    + Additional Container Image metadata
        - Signature of the additional metadata
    - ...

Now, the container registry is not just a storage place for images but a generic artifacts storage that can also define relations between the artifacts. As you may have noticed the trend in the industries, the registries are not referred to as container registries anymore but as artifact registries.

There are many benefits that the new capabilities offer in addition to storing various artifacts:

Relevant artifacts can be stored and managed together with the subject (or primary) artifact.
Querying and visualizing the related artifacts is much easier than storing them unrelated. This can result not only in more manageable implementations but also in better performance.
Relevant artifacts are easily discoverable.
Pulling an image from a registry may require additional artifacts for verification. An example is signature verification before allowing deployment. Using the OCI referrers API to get an image’s signature will be a trivial and standardized operation.
Relevant artifacts can be copied together between registries.
Content promotion between registries is a common scenario in container supply chains. Now, the image can be promoted to the target registry with all relevant artifacts instead of making many calls to the registry to discover them before promotion.

Because the capabilities are still new, how to standardize the implementations is still in discussion. You can look at my request for guidance for OCI artifacts for more variations of the above scenario and the possible implementations.

For this post, though, I will concentrate on a straightforward scenario using BOMs. I want to attach three different SBOMs to an image and test with a few major registries to understand the current capabilities. I will build the following content structure:

+ Container image
  artifactType: "application/vnd.docker.container.image.v1+json"
    - SPDX SBOM in JSON format
      artifactType: "application/spdx+json"
    - SPDX SBOM in TEXT format
      artifactType: "text/spdx"
    - CycloneDX SBOM in JSON format
      artifactType: "application/vnd.cyclonedx+json"

I also chose the following registries to test with:

You may not be familiar with the Zot and the ORAS registries listed above, but they are Open Source registries that you can run locally. Those registries are on top of any new OCI capabilities and one of the first registries to implement those. They make it a good option for testing new OCI capabilities.

Now, let’s dive into the registry capabilities.

Creating the Artifacts

All artifacts and results can be found in my container secure supply chain playground repository on GitHub. I have created the usual flasksample image and generated the SBOMs using Syft. Here are all the commands for that:

# Buld and push the image
docker build -t toddysm/flasksample:oci1.1-tests .
docker login -u toddysm
docker push toddysm/flasksample:oci1.1-tests

# Generate the SBOM in various formats
syft packages toddysm/flasksample:oci1.1-tests -o spdx-json > oci1.1-tests.spdx.json
syft packages toddysm/flasksample:oci1.1-tests -o spdx > oci1.1-tests.spdx
syft packages toddysm/flasksample:oci1.1-tests -o cyclonedx-json > oci1.1-tests.cyclonedx.json

I will use the above image and the generated SBOMs to push to various registries and test their behavior. Note that ORAS CLI can handle registries that support the new OCI 1.1 specifications and registries that support only OCI 1.0 specifications. ORAS CLI automatically converts the manifest to the most appropriate manifest based on the registry support.

Referring to Artifacts in Registries with OCI 1.0 Support

Docker Hub recently announced support for OCI Artifacts. Note, though, that this is support for OCI 1.0. Here are the commands to push the SBOMs to Docker Hub and reference the image as a subject:

oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json
# Command reponse
Uploading e6011f4dd3fa oci1.1-tests.spdx.json
Uploaded  e6011f4dd3fa oci1.1-tests.spdx.json
Attached to docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4

oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx
# Command response
Uploading d9c2135fe4b9 oci1.1-tests.spdx
Uploaded  d9c2135fe4b9 oci1.1-tests.spdx
Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:16a58d1ed78402935d61e524f5609087334b164861618373d7b96a7b7c612f1a": response status code 405: unsupported: The operation is unsupported.

oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json
# Command response
Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Uploaded  c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:37ebfdebe499bcec8e5a5ce04ae4526d3e560c199c16a85a97f80a91fbf1d2c3": response status code 405: unsupported: The operation is unsupported.

Checking Docker Hub, I can see that the image digest is sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 as returned by the ORAS CLI above.

I expected to see another artifact with sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 (again returned by the ORAS CLI above). However, such an artifact is not shown in the Docker Hub UI. There is another artifact tagged with the digest of the image.

However, the digest of that artifact (sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd) is not the one from the ORAS CLI output.

Of course, the question in my mind is: “What is the digest that ORAS CLI returned above?” The sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 one. Using ORAS CLI or crane, I can explore the various manifests.

What Manifests Are Created When Referring Between Artifacts in OCI 1.0 Registries?

The oras discover command helps visualize the hierarchy of artifacts that reference a subject.

oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree                      
docker.io/toddysm/flasksample:oci1.1-tests
├── application/spdx+json
│   └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
├── text/spdx
│   └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
    └── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b

According to the output above, I should be able to see four artifacts. Also, as you can see, the digest sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 is the one for the first SBOM I attached to the image. To understand what is happening, let’s look at the different manifests. I will use the oras manifest command to pull the manifests of all artifacts by referencing them by digest:

# Pull the manifest for the image
oras manifest fetch docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json

# Pull the manifest for the SPDX SBOM in JSON format
oras manifest fetch docker.io/toddysm/flasksample@sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 > manifest-sha256-0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4.json

# Pull the manifest for the SPDX SBOM in TEXT format
oras manifest fetch docker.io/toddysm/flasksample@sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67 > manifest-sha256-6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67.json

# Pull the manifest for the CycloneDX SBOM in JSON format
oras manifest fetch docker.io/toddysm/flasksample@sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b > manifest-sha256-047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b.json

# Pull the manifest of the artifact tagged with the image digest
oras manifest fetch docker.io/toddysm/flasksample@sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd > manifest-sha256-c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd.json

All manifests are available in the dockerhub folder in my container secure supply chain playground repository on GitHub. The image manifest is self-explanatory and I will not dig into it. The other four are more interesting. Opening the manifest for the SPDX SBOM in JSON format, I can see that it is an artifact manifest "mediaType": "application/vnd.oci.artifact.manifest.v1+json" of type "artifactType": "application/spdx+json". It has a blob annotated with the name of the file I pushed. It also has a subject field referring to the image. The manifest for the SPDX SBOM in TEXT format and the CycloneDX SBOM in JSON format have the same structure. The hierarchy represented by the oras discover command above shows exactly those manifests. I believe the output of oras discover could be improved to show also the image digest for completeness:

oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree                      
docker.io/toddysm/flasksample:oci1.1-tests
│   └── sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 
├── application/spdx+json
│   └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
├── text/spdx
│   └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
    └── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b

The question remains how the manifest tagged with the image digest plays a role here. Looking at it, I can see that it is an index manifest "mediaType": "application/vnd.oci.image.index.v1+json" that lists the three SBOM artifacts I pushed. Remember, this index manifest is tagged with the image digest. This is similar to the structure Sigstore creates that I described in Implementing Containers’ Secure Supply Chain with Sigstore Part 2 – The Magic Behind. Here is a visual of how the manifests are related:

The SBOM artifacts are not visible in the Docker Hub UI because they are not tagged, and Docker Hub has no UI to show untagged artifacts. A few? questions remain:

What happens if I delete the image?
What happens if I delete the index manifest?
Can I create deeper hierarchical structures in registries that support OCI 1.0?
What happens when I copy related artifacts from OCI 1.0 registry to OCI 1.1 registry?

I will come back to those in the second part of the series. Before that, I would like to examine how registries with OCI 1.1 support storing the manifests for the referred artifacts.

Referring to Artifacts in Registries with OCI 1.1 Support

Azure Container Registry (ACR) just announced support for OCI 1.1. It is in Public Preview and supports the OCI 1.1 RC spec at the moment of this writing. After retagging the image, the commands for pushing the SBOMs are similar to the ones used for Docker Hub.

# Re-tag and push the image
docker image tag toddysm/flasksample:oci1.1-tests tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests
docker push tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests

oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json
# Command response
Uploading e6011f4dd3fa oci1.1-tests.spdx.json
Uploaded  e6011f4dd3fa oci1.1-tests.spdx.json
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a

oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx
# Command response
Uploading d9c2135fe4b9 oci1.1-tests.spdx
Uploaded  d9c2135fe4b9 oci1.1-tests.spdx
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766

oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json
# Command response
Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Uploaded  c0ddc2a5ea78 oci1.1-tests.cyclonedx.json
Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
Digest: sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954

Azure portal does not show any additional artifacts and manifests, as shown in this screenshot:

This is a bit confusing, as I would at least expect to see a few more manifests. The distribution specification does not define functionality for listing untagged manifests; figuring out those dependencies without additional information will be hard. One noticeable thing is that no additional index manifest is tagged with the image digest.

oras discover command returns the following tree:

oras discover tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests -o tree
tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests
├── application/vnd.cyclonedx+json
│   └── sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954
├── text/spdx
│   └── sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766
└── application/spdx+json
    └── sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a

This is the same structure I saw above when using the command on the Docker Hub image. Pulling the manifests reveals that they are precisely the same as the ones from Docker Hub.

# Pull the manifest for the image
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json

# Pull the manifest for the SPDX SBOM in JSON format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a > manifest-sha-71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a.json

# Pull the manifest for the SPDX SBOM in TEXT format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766 > manifest-sha-0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766.json

# Pull the manifest for the CycloneDX SBOM in JSON format
oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954 > manifest-sha256-198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954.json

All manifests are available in the acr folder in my container secure supply chain playground repository on GitHub.

Luckily, ACR has CLI commands to list the manifests. Those commands call ACR’s proprietary APIs to gather the information. There are two ACR CLI commands I can use to list the manifests for a repository: acr manifest list and acr manifest metadata list . At the time of this writing acr manifest list had a bug and couldn’t list the OCI artifact manifests. acr manifest list-metadata worked fine and I could list all manifests in the repository. The output from the acr manifest list-metadata command is available here. From the output, I can see that only four manifests were created. There is no manifest index that points to the three artifacts. Here is a visual of how the manifests are related in an OCI 1.1 compliant registry:

To summarize the differences between the OCI 1.0 and OCI 1.1 referrers’ implementation:

In OCI 1.0 compliant registries, you will see an additional index manifest that is tagged with the image digest
In OCI 1.0 compliant registries, the index manifest lists the artifacts related to the image
In OCI 1.0 compliant registries, the artifact manifests still refer to the image using the subject field

I will look at how this impacts the content in your registry in the second part of this series.

Referrers Support Across Registries

Here is a table that shows the current (as of Jan 5th, 2023) support in registries.

wdt_ID	Registry	Artifact References Support	OCI Specification Support	Image Manifest mediaType	Artifact Manifest mediaType	Notes
1	Azure Container Registry (ACR)	Yes	1.1	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.artifact.manifest.v1+json
2	Docker Hub	Yes	1.0	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.artifact.manifest.v1+json	When attaching multiple artifacts to an image the registry responds with "Error: DELETE ...: response status code 405: unsupported: The operation is unsupported." The attachments succeed.
3	Elastic Container Registry (ECR)	No	-	-	-	When attaching an artifact to an image the registry responds with "status code 405: unsupported: Invalid parameter at 'ImageManifest' failed to satisfy constraint: 'Invalid JSON syntax'"
10	GitHub Container Registry	Yes	1.0	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.image.manifest.v1+json	When attaching multiple artifacts to an image the registry responds with "Error: DELETE ...: response status code 405: unsupported: The operation is unsupported." The attachments succeed.
11	Google Artifact Registry	Yes	1.0	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.image.manifest.v1+json
12	Harbor	Yes	1.0	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.image.manifest.v1+json
13	JFrog	No	-	-	-	When attaching an artifact to an image the registry responds with "Error: PUT...: response status code 403: unauthorized: Pushing Docker images with manifest v2 schema 1 to this repository is blocked.
14	ORAS Registry	Yes	1.1	application/vnd.docker.distribution.manifest.v2+json	application/vnd.oci.artifact.manifest.v1+json
15	Quay	No	-	-	-	When attaching an artifact to an image the registry responds with "Error: PUT...: response status code 415: manifest invalid: map[message:manifest schema version not supported]"
16	Zot	Yes	1.1	application/vnd.oci.image.manifest.v1+json	application/vnd.oci.artifact.manifest.v1+json	The registry does not support the legacy Docker image manifest.

The manifests and the debug logs are available in the corresponding registry folders in the cssc-pipeline repository on GitHub. You can refer to those for details.

Note: The investigation is done using the ORAS tool – the only one I am aware of that can create references between artifacts at the time of this writing. It may be possible to craft manifests manually and push them to the registries failing with ORAS.

Learnings

In addition to the above, I learned a few more things while experimenting with different registries.

As far as I know, OCI does not specify an API to list untagged manifests in a registry. This can be a problem because the attached artifacts do not have tags but only digests. I am pretty sure I ended up with some orphaned artifacts in the registries that do not support artifact referrers. Unfortunately, I cannot be sure due to the lack of such an API.
Registries are inconsistent in their responses when the capabilities are not supported. In my opinion, there is a lack of feedback on what capabilities each registry supports, which makes it hard for the clients. An easy way to check the capabilities of a registry would be beneficial.

In the next post of the series, I will go over more advanced scenarios like promotion between registries and building deeper hierarchies.

Photo by Petrebels on Unsplash