Using OCI Annotation to Track Container Image Lifecycle
For a while, we’ve been exploring the idea of using OCI annotations to track the lifecycle of container images. The problem we are trying to solve is as follows. Container images are immutable and cannot be dynamically patched like virtual machines. To apply the latest updates to a containerized application, teams must produce a new image with the patches. Once the new image is produced, the old one should be considered outdated and vulnerable, and all workloads using the old image should be redeployed with the new one.
The problem is how to mark the old image as outdated. Why? Because teams may have pinned their deployments to a digest or an immutable tag, and we want them to move to the patched version. We also want to create policies that outdated images should not be deployed. Finally, we want an automated way to point the teams to the latest patched image. Unfortunately, using digests and tags prevents us from achieving those goals.
Container Image Lifecycle Example
Here is a concrete example. I will use semantic versioning for the tags to make the example more relatable.
- First Revision: Build the application image
ghcr.io/toddysm/flasksample:1.0
1.0in this example is a rolling tag. To differentiate between images, I may want to use an immutable tag; for example1.0-20230707using theYYYYMMDDformat for the date. This image has a digestsha256@1234567890. - Second Revision: New vulnerabilities are found in
ghcr.io/toddysm/flasksample:1.0and I rebuild the image and tag it using the same1.0rolling tag.
I also tag it with an immutable tag like1.0-20230710. This image has a different digestsha256@5647382910.
At this moment, I have two images for the same application from the same lineage 1.0. Here is the relation between tags and digests:
- Tags
1.0and1.0-20230710point to digestsha256@5647382910 - Tag
1.0-20230707points to digestsha256@1234567890
If I have pinned my deployment to the tag 1.0-20230707 or the digest sha256@1234567890 I do not know whether the image is still fresh or has a newer patched version. I could try interpreting the tags, but this will be very custom to my tagging scheme. For example, my tagging scheme 1.0-YYYYMMDD differs from Python’s, NodeJS’, Alpine’s, or Ubuntu’s scheme (well, Ubuntu’s is very similar :)). Also, obtaining the tag from the image digest is not possible.
The idea is to use OCI annotations to add additional information to the images. This information can help us communicate the deprecation of images and preserve vital information lost when retagging. We are not the only ones thinking in this direction – the folks from Ubuntu also want to use annotations to deprecate images, although their goal is a bit different.
OCI Annotations for Image Lifecycle
OCI annotations are key-value pairs that you can add to the manifest of any OCI artifact, including container images. The problem is that annotations cannot be changed once the manifest is created. Hence, if you add the OCI annotations to the image manifest, you cannot update them anymore. The workaround to that is to use the OCI referrer capability and add a new artifact with annotations that is linked to the image. In essence, whenever you need to update an annotation, you must push a new artifact with the full set of annotations and link it to the image. The consumer of the annotations needs to list all referrer artifacts with the “annotation” type and take the latest one.
The other question that comes to mind is: “What annotations will help you track the image lifecycle?” OCI already specifies some pre-defined keys that you can leverage. There are three important ones that will help you manage the lifecycle:
org.opencontainers.image.createdcan be used to specify the date at which the image was created. For example, when the image is built.org.opencontainers.image.versioncan be used to specify the version of the software. Think of this as the lineage of the software (i.e. Python 3.10 or Ubuntu Jammy).org.opencontainers.image.revisioncan be used to sperify the patch version of the software. For example, Python 3.10.12 or Ubuntu Jammy 20230624.
One thing that OCI does not specify is an annotation for end-of-life of the image. For that, you can use custom annotation like vnd.myorganization.image.end-of-life.
How to Use Annotations for Image Lifecycle?
Let’s look at how the above annotations can be used to manage the lifecycle of a series of images. I will use concrete dates for the example.
First Revision of the Image: Build the application image ghcr.io/toddysm/flasksample:1.0. As part of the build process, add the following annotations to the image:
{
"org.opencontainers.image.created" : "2023-07-07T00:00:00-08:00",
"org.opencontainers.image.version" : "1.0",
"org.opencontainers.image.revision" : "20230707"
}
Second Revision of the Image: Vulnerabilities are discovered in the ghcr.io/toddysm/flasksample:1.0 image. Rebuild the image with the fixes and add the following annotations to it:
{
"org.opencontainers.image.created" : "2023-07-10T00:00:00-08:00",
"org.opencontainers.image.version" : "1.0",
"org.opencontainers.image.revision" : "20230710"
}
Find the previous image in the registry ghcr.io/toddysm/flasksample:1.0-20230707 and update the annotations to the following:
{
"org.opencontainers.image.created" : "2023-07-07T00:00:00-08:00",
"org.opencontainers.image.version" : "1.0",
"org.opencontainers.image.revision" : "20230707",
"vnd.myorganization.image.end-of-life" : "2023-07-10T00:00:00-08:00"
}
With those annotations in place, you can track the lifecycle of each image. But not only that! You can always determine the latest and most up-to-date image in the lineage by just pulling the 1.0 tag (which is available in the org.opencontainers.image.version annotation for every image in the lineage). Both the mutable and the immutable tags are preserved with the image. New annotations can always be added to the image by adding “empty” referrer artifacts with just the annotations. Also, annotations for the images are available even if the image is pulled by its digest.
Here are a couple of scenarios that you can implement with this information.
- Block deployment of images that are end-of-life
You can implement a policy to block the deployment of images that are end-of-life on your Kubernetes clusters. Such a policy can be implemented in admission controllers like Kyverno or Gatekeeper. - Suggest updated image in action items in vulnerability reports
Current vulnerability reports for container images are hardly actionable because they do not provide an update path for the reported images. Development teams are not interested in how many vulnerabilities are discovered but in the quickest way to fix those. - Automate the process for rebuilding dependent images
Tools like Dependabot can use the image lifecycle information to create pull requests for dependent images. This can speed up the process of fixing vulnerabilities and improving the vulnerability posture for the application.
All that sounds great, but the problem is the tooling support. While OCI specifies how you can store artifacts in registries and defines some standard annotations, very few, if any, tools allow you to easily achieve the above experience. I took it upon myself to give it a try!
Implementing Image Lifecycle Annotations with Existing Tools
Note: The below experience uses Docker buildx, ORAS, and GitHub Container Registry (GHCR) for the experience. The experience is as of July 10th, 2023, and may (or most certainly will) change. Other tools like regctl can also be used instead of ORAS. As always, I will use my cssc-pipeline repository for storing any code for this blog post.
First, I will set some environment variables to avoid retyping and make the commands easier to follow.
export TEMP_LOCATION=temp export IMAGE_VERSION=1.0 export FIRST_REVISION=20230707 export SECOND_REVISION=20230710 export REGISTRY=ghcr.io/toddysm/cssc-pipeline export REPOSITORY=flasksample mkdir -p $TEMP_LOCATION
Building the First Revision of the Image
The first step is to build the first revision of the image. Docker buildx can build the image, but the default option is to save the image in Docker’s proprietary format, which doesn’t allow the use of annotations. To use annotations, you must use the OCI exporter and save the image as a tarball. Here is the command that will allow you to do that:
docker buildx build . -f Dockerfile \
-t ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
-o "type=oci,dest=${TEMP_LOCATION}/flasksample-${IMAGE_VERSION}-${FIRST_REVISION}.tar,annotation.org.opencontainers.image.created=20230707T00:00-08:00,annotation.org.opencontainers.image.version=${IMAGE_VERSION},annotation.org.opencontainers.image.revision=${FIRST_REVISION}" \
--metadata-file ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}-metadata.json
The command above creates an image in OCI format and saves it as a tarball. We can use the generated metadata file to obtain the image’s digest.
export FIRST_REVISION_DIGEST=`cat ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}-metadata.json \
| jq -r '."containerimage.descriptor".digest'`
echo $FIRST_REVISION_DIGEST
In my case, the digest is sha256:1446094f076dcbc2b7e7943ae3806bb44003ee9e6c94efd3208b8f04159aa8c0.
Next, I will use the following ORAS command to push the OCI image to the GHCR registry:
oras cp --from-oci-layout ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${FIRST_REVISION}.tar:${IMAGE_VERSION} \
${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION}
I can verify that the annotations are set on the image by pulling the manifest and checking the annotations field:
oras manifest fetch ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
| jq .annotations
At this point, I have the first revision of the image built and published to GHCR under ghcr.io/toddysm/cssc-pipeline/flasksample:1.0.
Building the Second Revision of the Image
A few days later, if vulnerabilities are discovered in the image, I need to update the image with the latest patches. As part of the build process, I also need to update the annotations of the previous image revision.
The first thing I need to do is to obtain the digest of the first revision. Because the first revision is still tagged with 1.0, I can quickly get the digest using the following command:
export OLD_IMAGE_DIGEST=`oras manifest fetch --descriptor ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
| jq .digest | tr -d '"'`
echo $OLD_IMAGE_DIGEST
That command returns the same digest as before sha256:1446094f076dcbc2b7e7943ae3806bb44003ee9e6c94efd3208b8f04159aa8c0. Now, I have a unique reference to the first revision of the image. I can build the second revision of the image using the same commands as before.
# Build the second revision of the container image with annotations...
docker buildx build . -f Dockerfile \
-t ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
-o "type=oci,dest=${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}.tar,annotation.org.opencontainers.image.created=20230710T00:00-08:00,annotation.org.opencontainers.image.version=${IMAGE_VERSION},annotation.org.opencontainers.image.revision=${SECOND_REVISION}" \
--metadata-file ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}-metadata.json
# Get the digest for the second revision...
export SECOND_REVISION_DIGEST=`cat ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}-metadata.json \
| jq -r '."containerimage.descriptor".digest'`
echo $SECOND_REVISION_DIGEST
# Use ORAS to push the second revision to the registry...
oras cp --from-oci-layout ${TEMP_LOCATION}/${REPOSITORY}-${IMAGE_VERSION}-${SECOND_REVISION}.tar:${IMAGE_VERSION} \
${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION}
# Use ORAS to verify the annotations are set on the image...
oras manifest fetch ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
| jq .annotations
You can check that the second revision digest is different from the first revision using the following commands:
export IMAGE_DIGEST=`oras manifest fetch --descriptor ${REGISTRY}/${REPOSITORY}:${IMAGE_VERSION} \
| jq .digest | tr -d '"'`
echo $IMAGE_DIGEST
For me, the digest of the second revision (or the most up-to-date image) is sha256:4ee61e3d9d28fe15cffc33854a2b851e2c87929a99f0c71bfc7a689ad372894d.
Updating the Lifecycle Annotations for the First Revision of the Image
This is the most important step of the process – I need to go back and update the lifecycle annotations for the first revision of the image and mark it as end-of-life. This is a bit trickier process because the manifest of the original image cannot be modified. It is immutable! I need to create a referrer artifact and store the lifecycle annotations in the manifest of this referrer artifact. However, the referrer artifact should be empty (well, you can put a cat picture there, but it is irrelevant 🙂 ) ORAS already supports push with an empty artifact. In the future, OCI-compliant registries will support empty layers for artifacts too. Here are the steps for that.
First, I will fetch the annotations for the first revision and update them with the end-of-life annotation.
oras manifest fetch ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} \
| jq .annotations \
| jq '. += {"vnd.myorganization.image.end-of-life":"20230710T00:00-08:00"}' \
| jq '{"$manifest":.}' \
> ${TEMP_LOCATION}/annotations.json
Note that ORAS uses a special JSON schema for annotation files. Hence, I needed to convert the annotations that I retrieved from the image manifest to a new JSON object expected by ORAS. Here is what the resulting file looks like:
jq . ${TEMP_LOCATION}/annotations.json
{
"$manifest": {
"org.opencontainers.image.created": "20230707T00:00-08:00",
"org.opencontainers.image.revision": "20230707",
"org.opencontainers.image.version": "1.0",
"vnd.myorganization.image.end-of-life": "20230710T00:00-08:00"
}
}
Now, I need to push the empty artifact and refer to the first revision of the image. To do that, I will also need to use an artifact type (or mediaType in OCI language) to find my lifecycle annotations later on easily. There is no standard mediaType for that, so I have to invent my own – I will use application/vnd.myorganization.image.lifecycle.metadata. Here is the ORAS command that you can use to update the annotations:
oras attach --artifact-type application/vnd.myorganization.image.lifecycle.metadata \
--annotation-file ${TEMP_LOCATION}/annotations.json \
${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} \
${TEMP_LOCATION}/empty.layer
OK, I am done with setting the lifecycle annotations for the images.
Fetching the Lifecycle Annotations for Each Image
I can fetch the annotations for each image by simply using the following commands:
oras manifest fetch ${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} | jq .annotations
oras manifest fetch ${REGISTRY}/${REPOSITORY}@${NEW_IMAGE_DIGEST} | jq .annotations
There is a problem, though! Those commands fetch the annotations that are set in the image manifest. For the outdated image (i.e. $OLD_IMAGE), the command will not return the end-of-life annotation. To get that annotation, I will need to fetch the referrer with the particular type application/vnd.myorganization.image.lifecycle.metadata. Here is how to do that.
First, I need to get the digest of the referrer artifact.
export ANNOTATIONS_ARTIFACT_DIGEST=`oras discover --artifact-type "application/vnd.myorganization.image.lifecycle.metadata" \
${REGISTRY}/${REPOSITORY}@${OLD_IMAGE_DIGEST} -o json \
| jq '.manifests[0].digest' \
| tr -d '"'`
echo $ANNOTATIONS_ARTIFACT_DIGEST
And then retrieve the annotations set in the referrer’s artifact manifest.
oras manifest fetch ${REGISTRY}/${REPOSITORY}@${ANNOTATIONS_ARTIFACT_DIGEST} | jq .annotations
As a logic for implementation, I would always check if the image has a referrer lifecycle artifact. If so, I will ignore the lifecycle annotations in the image manifest.
Closing Thoughts
Lifecycle annotations enable interesting scenarios for securing container supply chains and improving containerized applications’ vulnerability posture. Though the tooling can undoubtedly be improved – I had to do a lot of JSON conversions to get it working. The lack of standard annotation for end-of-life and mediaType for the referrer artifact makes the above solution very proprietary.
One issue that can arise is if multiple application/vnd.myorganization.image.lifecycle.metadata referrers are created. OCI doesn’t specify how to retrieve the latest artifact from registries by type. If multiple lifecycle annotations artifacts are pushed for the image, the client must pull and inspect each. This logic can be quite complex and can impact the performance on the client’s side.
An improvement that can be made to the process above is to sign each artifact (the image and the lifecycle annotations artifact). This will ensure that the annotations are trustable and not tampered with. Of course, an attacker can always remove the referrer artifact from the registry and leave the client with the impression that the image is still fresh.
Those are all food for thought and good topics for future posts. Here is also a video of the whole experience described above.
