OCI Artifact Manifests, OCI Referrers API and Their Support Across Registries (Part 1)
[UPDATE: 2023-03-26] When I wrote this post, the expectation was that OCI will release version 1.1 of the specification with artifact manifest included. This release was supposed to happen by end of Jan 2023 or mid Feb 2023. Unfortunately, the OCI 1.1 Image Spec PR 999 put a hold on that and as of today, the spec is not released. Although I promised to have a Part 2, due to the changes in the spec, continuing the investigation in the original direction may not be fruitful and helpful to anyone. Most of the functionality described below is removed from many registries and the steps and the information may be incorrect. The concepts are still relevant but their actual implementation may not be as described in this post. Consider the relevance of the information applicable only between Jan 5th 2023 and Jan 24th 2023 – the date the above PR was submitted. There will be no other updates to this post or Part 2 of the series. Instead of Part 2, folks may find the Registry & client support for Image Manifest type artifacts issue relevant to what they are looking for.
If you are deep into containers and software supply chain security, you may have heard of OCI referrers API and OCI artifacts. If not, but you are interested in the containers’ secure supply chain topic, this post will give you enough details to start exploring new registry capabilities that can significantly improve your software supply chain architecture.
This will be a two-part series. In the first part, I will examine the differences between OCI 1.0 and OCI 1.1 and their support across registries. In the second part, I will look at more advanced scenarios like deep hierarchies, deleting artifacts, and migrating content between registries with different support.
But before we start…
What is OCI?
The Open Container Initiative (OCI) is the governance organization responsible for creating open industry standards for container formats and runtimes. OCI develops and maintains three essential specifications:
- The OCI Image Format Specification defines the structure and the layout of an image or artifact. If you are interested in reading more about the OCI image layout, I recommend the No More Additional Network Requests – Enter: OCI Image Layout post from @developerguy. It will give you a good background on how the image is structured. I will mainly discuss the OCI Artifract Manifest in this post.
- The OCI Distribution Specification defines the APIs that registries should implement to enable the distribution of artifacts. The OCI Referrers API is part of this specification and will be discussed in this post.
- The OCI Runtime Specification specifies the configuration, the execution environment, and the lifecycle of a container. I will not discuss the runtime specification in this post.
One additional note. You may have heard of the term OCI reference types in the past. This was the name of the working group (WG) responsible for driving the changes in the image format and distribution specification. The prototype implementation of reference types was first implemented in ORAS. Its usefulness was the reason it was brought to the attention of the OCI group and resulted in the new changes.
Disclaimer: One last thing I have to mention is that, at the time of this writing, the OCI specifications (OCI 1.1) that support the new artifact manifest changes and the referrers API is in release candidate 2 (RC.2). The release of the OCI 1.1 specifications is planned for February 2023. Keep in mind that not many registries support the new artifact manifest and the referrers API due to this fact. This post aims to describe the scenarios it enables and discuss the backward compatibilities with registries that support the current OCI 1.0 specifications. I will also test several registries and point out their current capabilities.
What Scenarios Do OCI Artifact Manifest and Referrers API Enable?
As always, I would like to start with the scenarios and what are the benefits of using those new capabilities. As part of the ongoing software secure supply chain efforts, every vendor must produce metadata in addition to the actual software. Vendors need to add human and machine-readable metadata describing the software, whether this is a binary executable or a container image. The most common metadata discussed nowadays is software bills of materials (SBOMs) and signatures. SBOMs list the packages and binaries used in the individual piece of software (aka the software “ingredients”). The signature is intended to testify about the authenticity of the software and prevent tampering with the bits.
In the past, container registries were intended to store only container images. With the introduction of OCI artifacts, container registries can store other artifacts like SBOMs, signatures, plain text files, and even videos. The OCI referrers API goes even further and allows you to establish relationships between artifacts. This is a compelling functionality that allows you to create structures like this:
+ Container Image
- Signature of the Container Image
+ SBOM for the Container Image
- Signature of the SBOM
+ Vulnerability Report for the Container Image
- Signature of the Vulnerability Report
+ Additional Container Image metadata
- Signature of the additional metadata
- ...
Now, the container registry is not just a storage place for images but a generic artifacts storage that can also define relations between the artifacts. As you may have noticed the trend in the industries, the registries are not referred to as container registries anymore but as artifact registries.
There are many benefits that the new capabilities offer in addition to storing various artifacts:
- Relevant artifacts can be stored and managed together with the subject (or primary) artifact.
Querying and visualizing the related artifacts is much easier than storing them unrelated. This can result not only in more manageable implementations but also in better performance. - Relevant artifacts are easily discoverable.
Pulling an image from a registry may require additional artifacts for verification. An example is signature verification before allowing deployment. Using the OCI referrers API to get an image’s signature will be a trivial and standardized operation. - Relevant artifacts can be copied together between registries.
Content promotion between registries is a common scenario in container supply chains. Now, the image can be promoted to the target registry with all relevant artifacts instead of making many calls to the registry to discover them before promotion.
Because the capabilities are still new, how to standardize the implementations is still in discussion. You can look at my request for guidance for OCI artifacts for more variations of the above scenario and the possible implementations.
For this post, though, I will concentrate on a straightforward scenario using BOMs. I want to attach three different SBOMs to an image and test with a few major registries to understand the current capabilities. I will build the following content structure:
+ Container image
artifactType: "application/vnd.docker.container.image.v1+json"
- SPDX SBOM in JSON format
artifactType: "application/spdx+json"
- SPDX SBOM in TEXT format
artifactType: "text/spdx"
- CycloneDX SBOM in JSON format
artifactType: "application/vnd.cyclonedx+json"
I also chose the following registries to test with:
You may not be familiar with the Zot and the ORAS registries listed above, but they are Open Source registries that you can run locally. Those registries are on top of any new OCI capabilities and one of the first registries to implement those. They make it a good option for testing new OCI capabilities.
Now, let’s dive into the registry capabilities.
Creating the Artifacts
All artifacts and results can be found in my container secure supply chain playground repository on GitHub. I have created the usual flasksample image and generated the SBOMs using Syft. Here are all the commands for that:
# Buld and push the image docker build -t toddysm/flasksample:oci1.1-tests . docker login -u toddysm docker push toddysm/flasksample:oci1.1-tests # Generate the SBOM in various formats syft packages toddysm/flasksample:oci1.1-tests -o spdx-json > oci1.1-tests.spdx.json syft packages toddysm/flasksample:oci1.1-tests -o spdx > oci1.1-tests.spdx syft packages toddysm/flasksample:oci1.1-tests -o cyclonedx-json > oci1.1-tests.cyclonedx.json
I will use the above image and the generated SBOMs to push to various registries and test their behavior. Note that ORAS CLI can handle registries that support the new OCI 1.1 specifications and registries that support only OCI 1.0 specifications. ORAS CLI automatically converts the manifest to the most appropriate manifest based on the registry support.
Referring to Artifacts in Registries with OCI 1.0 Support
Docker Hub recently announced support for OCI Artifacts. Note, though, that this is support for OCI 1.0. Here are the commands to push the SBOMs to Docker Hub and reference the image as a subject:
oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json # Command reponse Uploading e6011f4dd3fa oci1.1-tests.spdx.json Uploaded e6011f4dd3fa oci1.1-tests.spdx.json Attached to docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 Digest: sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.spdx # Command response Uploading d9c2135fe4b9 oci1.1-tests.spdx Uploaded d9c2135fe4b9 oci1.1-tests.spdx Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:16a58d1ed78402935d61e524f5609087334b164861618373d7b96a7b7c612f1a": response status code 405: unsupported: The operation is unsupported. oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" docker.io/toddysm/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json # Command response Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json Uploaded c0ddc2a5ea78 oci1.1-tests.cyclonedx.json Error: DELETE "https://registry-1.docker.io/v2/toddysm/flasksample/manifests/sha256:37ebfdebe499bcec8e5a5ce04ae4526d3e560c199c16a85a97f80a91fbf1d2c3": response status code 405: unsupported: The operation is unsupported.
Checking Docker Hub, I can see that the image digest is sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 as returned by the ORAS CLI above.
I expected to see another artifact with sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 (again returned by the ORAS CLI above). However, such an artifact is not shown in the Docker Hub UI. There is another artifact tagged with the digest of the image.
However, the digest of that artifact (sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd) is not the one from the ORAS CLI output.
Of course, the question in my mind is: “What is the digest that ORAS CLI returned above?” The sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 one. Using ORAS CLI or crane, I can explore the various manifests.
What Manifests Are Created When Referring Between Artifacts in OCI 1.0 Registries?
The oras discover command helps visualize the hierarchy of artifacts that reference a subject.
oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree
docker.io/toddysm/flasksample:oci1.1-tests
├── application/spdx+json
│ └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
├── text/spdx
│ └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
└── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b
According to the output above, I should be able to see four artifacts. Also, as you can see, the digest sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 is the one for the first SBOM I attached to the image. To understand what is happening, let’s look at the different manifests. I will use the oras manifest command to pull the manifests of all artifacts by referencing them by digest:
# Pull the manifest for the image oras manifest fetch docker.io/toddysm/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json # Pull the manifest for the SPDX SBOM in JSON format oras manifest fetch docker.io/toddysm/flasksample@sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4 > manifest-sha256-0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4.json # Pull the manifest for the SPDX SBOM in TEXT format oras manifest fetch docker.io/toddysm/flasksample@sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67 > manifest-sha256-6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67.json # Pull the manifest for the CycloneDX SBOM in JSON format oras manifest fetch docker.io/toddysm/flasksample@sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b > manifest-sha256-047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b.json # Pull the manifest of the artifact tagged with the image digest oras manifest fetch docker.io/toddysm/flasksample@sha256:c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd > manifest-sha256-c8c7d53f0e1ed5553a815c7b5ccf40c09801f7636a3c64940eafeb7bfab728cd.json
All manifests are available in the dockerhub folder in my container secure supply chain playground repository on GitHub. The image manifest is self-explanatory and I will not dig into it. The other four are more interesting. Opening the manifest for the SPDX SBOM in JSON format, I can see that it is an artifact manifest "mediaType": "application/vnd.oci.artifact.manifest.v1+json" of type "artifactType": "application/spdx+json". It has a blob annotated with the name of the file I pushed. It also has a subject field referring to the image. The manifest for the SPDX SBOM in TEXT format and the CycloneDX SBOM in JSON format have the same structure. The hierarchy represented by the oras discover command above shows exactly those manifests. I believe the output of oras discover could be improved to show also the image digest for completeness:
oras discover docker.io/toddysm/flasksample:oci1.1-tests -o tree
docker.io/toddysm/flasksample:oci1.1-tests
│ └── sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0
├── application/spdx+json
│ └── sha256:0a1dd8fcdef54eb489aaa99978e19cffd7f6ae11595322ab5af694913da177d4
├── text/spdx
│ └── sha256:6f6c9260247ad876626f742508550665ad20c75ac7e4469782d18e47d40cac67
└── application/vnd.cyclonedx+json
└── sha256:047054894cbe7c9e57532f4e01d03f631e92c3aec48b4a06485296aee1374b3b
The question remains how the manifest tagged with the image digest plays a role here. Looking at it, I can see that it is an index manifest "mediaType": "application/vnd.oci.image.index.v1+json" that lists the three SBOM artifacts I pushed. Remember, this index manifest is tagged with the image digest. This is similar to the structure Sigstore creates that I described in Implementing Containers’ Secure Supply Chain with Sigstore Part 2 – The Magic Behind. Here is a visual of how the manifests are related:
The SBOM artifacts are not visible in the Docker Hub UI because they are not tagged, and Docker Hub has no UI to show untagged artifacts. A few? questions remain:
- What happens if I delete the image?
- What happens if I delete the index manifest?
- Can I create deeper hierarchical structures in registries that support OCI 1.0?
- What happens when I copy related artifacts from OCI 1.0 registry to OCI 1.1 registry?
I will come back to those in the second part of the series. Before that, I would like to examine how registries with OCI 1.1 support storing the manifests for the referred artifacts.
Referring to Artifacts in Registries with OCI 1.1 Support
Azure Container Registry (ACR) just announced support for OCI 1.1. It is in Public Preview and supports the OCI 1.1 RC spec at the moment of this writing. After retagging the image, the commands for pushing the SBOMs are similar to the ones used for Docker Hub.
# Re-tag and push the image docker image tag toddysm/flasksample:oci1.1-tests tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests docker push tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests oras attach --artifact-type "application/spdx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx.json # Command response Uploading e6011f4dd3fa oci1.1-tests.spdx.json Uploaded e6011f4dd3fa oci1.1-tests.spdx.json Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 Digest: sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a oras attach --artifact-type "text/spdx" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.spdx # Command response Uploading d9c2135fe4b9 oci1.1-tests.spdx Uploaded d9c2135fe4b9 oci1.1-tests.spdx Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 Digest: sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766 oras attach --artifact-type "application/vnd.cyclonedx+json" --annotation "producer=syft 0.63.0" tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests ./oci1.1-tests.cyclonedx.json # Command response Uploading c0ddc2a5ea78 oci1.1-tests.cyclonedx.json Uploaded c0ddc2a5ea78 oci1.1-tests.cyclonedx.json Attached to tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 Digest: sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954
Azure portal does not show any additional artifacts and manifests, as shown in this screenshot:
This is a bit confusing, as I would at least expect to see a few more manifests. The distribution specification does not define functionality for listing untagged manifests; figuring out those dependencies without additional information will be hard. One noticeable thing is that no additional index manifest is tagged with the image digest.
oras discover command returns the following tree:
oras discover tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests -o tree
tsmacrwcusocitest.azurecr.io/flasksample:oci1.1-tests
├── application/vnd.cyclonedx+json
│ └── sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954
├── text/spdx
│ └── sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766
└── application/spdx+json
└── sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a
This is the same structure I saw above when using the command on the Docker Hub image. Pulling the manifests reveals that they are precisely the same as the ones from Docker Hub.
# Pull the manifest for the image oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0 > manifest-sha256-b89e2098603bead4f07e318e1a4e11b4a4ef1f3614725c88b3fcdd469d55c0e0.json # Pull the manifest for the SPDX SBOM in JSON format oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a > manifest-sha-71e90130cb912fbcff6556c0395878a8e7a0c7244eb8e8ee9001e84f9cba804a.json # Pull the manifest for the SPDX SBOM in TEXT format oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766 > manifest-sha-0fbd0e611ec9fe620b72ebe130da680de9402e1e241b30c2aa4515610ed2d766.json # Pull the manifest for the CycloneDX SBOM in JSON format oras manifest fetch tsmacrwcusocitest.azurecr.io/flasksample@sha256:198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954 > manifest-sha256-198e405344b5fafd6127821970eb4a84129ae729402e8c2c71fc1bb80abf0954.json
All manifests are available in the acr folder in my container secure supply chain playground repository on GitHub.
Luckily, ACR has CLI commands to list the manifests. Those commands call ACR’s proprietary APIs to gather the information. There are two ACR CLI commands I can use to list the manifests for a repository: acr manifest list and acr manifest metadata list . At the time of this writing acr manifest list had a bug and couldn’t list the OCI artifact manifests. acr manifest list-metadata worked fine and I could list all manifests in the repository. The output from the acr manifest list-metadata command is available here. From the output, I can see that only four manifests were created. There is no manifest index that points to the three artifacts. Here is a visual of how the manifests are related in an OCI 1.1 compliant registry:
To summarize the differences between the OCI 1.0 and OCI 1.1 referrers’ implementation:
- In OCI 1.0 compliant registries, you will see an additional index manifest that is tagged with the image digest
- In OCI 1.0 compliant registries, the index manifest lists the artifacts related to the image
- In OCI 1.0 compliant registries, the artifact manifests still refer to the image using the subject field
I will look at how this impacts the content in your registry in the second part of this series.
Referrers Support Across Registries
Here is a table that shows the current (as of Jan 5th, 2023) support in registries.
The manifests and the debug logs are available in the corresponding registry folders in the cssc-pipeline repository on GitHub. You can refer to those for details.
Note: The investigation is done using the ORAS tool – the only one I am aware of that can create references between artifacts at the time of this writing. It may be possible to craft manifests manually and push them to the registries failing with ORAS.
Learnings
In addition to the above, I learned a few more things while experimenting with different registries.
- As far as I know, OCI does not specify an API to list untagged manifests in a registry. This can be a problem because the attached artifacts do not have tags but only digests. I am pretty sure I ended up with some orphaned artifacts in the registries that do not support artifact referrers. Unfortunately, I cannot be sure due to the lack of such an API.
- Registries are inconsistent in their responses when the capabilities are not supported. In my opinion, there is a lack of feedback on what capabilities each registry supports, which makes it hard for the clients. An easy way to check the capabilities of a registry would be beneficial.
In the next post of the series, I will go over more advanced scenarios like promotion between registries and building deeper hierarchies.
