In previous posts I described how you can implement a GitHub actions to mirror and quarantine images from Docker Hub, promote images from quarantine to golden registry if they meet the vulnerability threshold as well as how to break down the monolithic GitHub actions implementation into reusable components. However, the biggest problem with this approach is that most of the time the images you acquire from public registries like Docker Hub will have vulnerabilities in them and will be blocked from promotion to the golden registry. To get around this, you need to implement an approval workflow to enable manual review of the vulnerabilities and only promote images that meet internal compliance criteria.
I saw this happen with my implementation over the past few weeks. None of my base images got promoted to the golden repository because each one of them had at least one HIGH vulnerability and some of them even multiple. Unless you opt in to source your images from certified or hardened registries like Chainguard’s or DHI (Docker Hardened Images), you should always expect to see vulnerabilities reported even in the freshest images. With the approval workflow, though, you have at least the opportunity to manually review those vulnerabilities and accept the risk before they start proliferating into your internal applications. The caveat here is that this is a one time snapshot of the vulnerability state of an image that you will approve, and we all know, new vulnerabilities are discovered all the time. I will write about how to deal with this last issue in a later post.
To implement the approval step in my promotion GitHub action, I had to split the action in two pieces:
- Scan and generate the report
- Promote the image from quarantine to golden repository
The actions use a GitHub issue to manage the approval process. I have also configured Slack to get proactively notified for any issues in the repository and have the possibility to approve the promotion from there.
So, here is how the promotion workflow works in details:
- The promotion action is triggered on a daily schedule. Here is the permalink to the action for the
python:3.14-slimimage: https://github.com/toddysm/cssc-framework/blob/69deeecf15a9d8b9ed5968c1457c324f0ba0c119/.github/workflows/promote-from-quarantine-python.yml - If the vulnerability scan discovers CVEs in the image, the workflow will generate a GitHub issue and halt the promotion. Here is a link to an example issue https://github.com/toddysm/cssc-framework/issues/87 and the corresponding action run that generated that issue https://github.com/toddysm/cssc-framework/actions/runs/28318457640. In addition, an notification is sent to the Slack channel dedicated to this project. This is how the notification looks like:
- To approve the promotion, I need to add
/approveas a new comment to the issue. I can do the same from Slack by commenting on the issue. - After the approval, the workflow completes by copying the image from the quarantine to the golden repository.
This is how currently many enterprises are implementing their acquisition and quarantining of public images. The process is imperfect but it is the best that is possible right now due to the presence of vulnerabilities in almost every public image in Docker Hub and other public registries. At a minimum, you keep track of the acquired vulnerabilities and have an audit trail that can enable you to analyze and quantify the risk.
We all hope that the proliferation of hardened images will improve the situation and the more enterprises start using those, the vulnerability posture will get better. Note, though, that certified and hardened images are not a solver bullet – even the latest images from Chainguard and DHI can have some vulnerabilities – here is a screenshot of the latest Python development image from Chainguard:
As you can see, it has two HIGH severity vulnerabilities and if I want to use this one as a base development image and run it through the current acquisition and promotion workflow, it will be blocked. Doesn’t matter if you are pulling images from public registries like Docker Hub or from certified or hardened ones like Chianguard’s or DHI, my recommendation is to implement the workflows above to have a controlled acquisition of such images.
Having achieved some level of stability in my workflows including capabilities like:
- Componentization and reusability
- End-to-end experience for image acquisition/mirroring, quarantine, and promotion to golden repository
- Manual approval and notifications
I decided to tag the code base and cut a first minor release 0.1.0 of the CSSC workflows in my GitHub repository. Feel free to take a look, reuse and modify based on your own needs. And don’t hesitate to file issues and provide feedback.
I will follow up with steps to annotate images, build application images with relevant metadata and supply chain attestations and explore more of the best practices for containers supply chain.