CVE Scanning with zot¶

The CVE (Common Vulnerabilities and Exposures) scanning extension provides vulnerability detection for container images stored in zot. It uses Trivy, a comprehensive security scanner, to identify vulnerabilities in container images.

Overview¶

The CVE scanning feature enables zot to:

• Scan container images for known vulnerabilities
• Maintain an up-to-date vulnerability database
• Provide CVE information through the search API and CLI
• Cache scan results for improved performance
• Support both single manifest images and multi-arch image indexes

How It Works¶

Architecture¶

The CVE scanning extension integrates Trivy as a library to perform vulnerability scans:

1. Database Management: Trivy vulnerability databases are downloaded and stored locally in _trivy directories under each storage root. The databases are updated periodically based on the configured updateInterval.

2. Scanning Process: When an image is scanned:

3. The image layers are analyzed for installed packages (OS packages and application libraries)
4. The Trivy scanner matches detected packages against the vulnerability database
5. Results are cached to avoid redundant scans

6. CVE information is returned with severity levels, package details, and fix versions

7. Automatic Background Scanning: Images are automatically scanned in the background by a scheduler task that runs every 15 minutes (hardcoded, not configurable). This task starts as soon as zot starts and the Trivy database is downloaded. The scheduler checks for unscanned images and processes them in the background.

Scanning does not occur on image push; it happens asynchronously via the scheduler.

1. On-Demand Scanning: If a user requests CVE results for an image that doesn't have cached results, the image is scanned immediately. The request will block until the scan completes and results are available. This ensures users always get results, even for newly pushed images that haven't been processed by the background scanner yet.

Using Trivy¶

zot uses Trivy as an embedded library, not as an external binary. This means:

• No separate Trivy installation is required
• Trivy databases are downloaded from container registries (default: ghcr.io/aquasecurity/trivy-db)
• Scans run offline using the local database
• The Trivy cache is stored in _trivy directories under each storage root

Our Trivy integration supports:

• OS Packages: Vulnerabilities in operating system packages (e.g., apt, yum, apk packages)
• Application Libraries: Vulnerabilities in application dependencies (e.g., npm, pip, maven packages)
• Java Dependencies: Special support for Java applications via the Trivy Java database

The scan configuration is equivalent to running trivy image --scanners vuln --vuln-type os,library. zot uses Trivy as a Go library with additional options for database management, image input handling, and result processing.

Supported Image Formats¶

The CVE scanner supports:

• Single Manifest Images: Standard OCI/Docker images with a single manifest
• Multi-Arch Indexes: Images with multiple platform-specific manifests (e.g., linux/amd64, linux/arm64)
• Layer Types: Standard compressed and uncompressed image layers

Images with unsupported layer media types cannot be scanned and will return an error.

Prerequisites¶

Before enabling CVE scanning, ensure the following prerequisites are met:

1. Search Extension Required: The search extension is mandatory for CVE scanning. It is the only way to access scan results. zot must be built with the search extension enabled (this is typically the default in standard builds).

2. Storage Access: The zot process needs write access to the storage root directory to create the _trivy cache directories. Ensure zot has write permissions to the storage directory.

3. Temporary Directory Space and Permissions: The Trivy library requires sufficient space and write permissions on the temporary partition (/tmp by default). This space is needed for:

4. Database Downloads: Trivy databases are initially downloaded to /tmp before being moved to the final destination under the zot root directory
5. Layer Unpacking: Large image layers are unpacked under /tmp during scanning. Smaller layers are unpacked in-memory

Ensure your system has adequate /tmp space and write permissions, especially if scanning images with large layers.

1. Network Access: zot needs network access to download Trivy databases from container registries. The initial database download and periodic updates require connectivity to the configured registry (default: ghcr.io/aquasecurity/*).

Configuration¶

To enable CVE scanning, you need to:

1. Enable the search extension
2. Configure CVE scanning settings

Basic Configuration¶

The minimal configuration to enable CVE scanning:

{
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "24h"
            }
        }
    }
}

You can also omit updateInterval entirely, and it will default to 2 hours:

{
    "extensions": {
        "search": {
            "enable": true,
            "cve": {}
        }
    }
}

Configuration with Custom Trivy Databases¶

You can customize the Trivy database repositories:

{
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "24h",
                "trivy": {
                    "dbRepository": "ghcr.io/aquasecurity/trivy-db",
                    "javaDBRepository": "ghcr.io/aquasecurity/trivy-java-db"
                }
            }
        }
    }
}

Configuration Options¶

The following table lists the configurable attributes for CVE scanning.

The updateInterval is optional. If not specified, it defaults to 2 hours. If a value less than 2 hours is specified, it will be automatically adjusted to 2 hours.

The automatic image scanning interval is not configurable and is hardcoded to 15 minutes. This is separate from the database update interval.

Example Configurations¶

Minimal Configuration¶

{
    "distSpecVersion": "1.1.1",
    "storage": {
        "rootDirectory": "/tmp/zot"
    },
    "http": {
        "address": "127.0.0.1",
        "port": "8080"
    },
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "24h"
            }
        }
    }
}

Configuration with Multiple Storage Paths¶

When using sub-paths for storage, each storage location will have its own Trivy database:

{
    "distSpecVersion": "1.1.1",
    "storage": {
        "rootDirectory": "/tmp/zot",
        "subPaths": {
            "/infra": {
                "rootDirectory": "/tmp/zot1"
            },
            "/apps": {
                "rootDirectory": "/tmp/zot2"
            }
        }
    },
    "http": {
        "address": "127.0.0.1",
        "port": "5000"
    },
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "24h"
            }
        }
    }
}

Configuration with Custom Database Repositories¶

If you're using a private registry or mirror for Trivy databases:

{
    "distSpecVersion": "1.1.1",
    "storage": {
        "rootDirectory": "/tmp/zot"
    },
    "http": {
        "address": "127.0.0.1",
        "port": "8080"
    },
    "extensions": {
        "search": {
            "enable": true,
            "cve": {
                "updateInterval": "12h",
                "trivy": {
                    "dbRepository": "registry.example.com/trivy-db",
                    "javaDBRepository": "registry.example.com/trivy-java-db"
                }
            }
        }
    }
}

Usage¶

CLI Commands¶

The zot CLI (zli) provides several commands for querying CVE information. For detailed information, see the zli command reference.

List CVEs for an Image¶

List all CVEs found in a specific image:

zli cve list <repo>:<tag>
# or using digest
zli cve list <repo>@<digest>

Options:

• --cve-id: Filter results to a specific CVE ID

Examples:

# Basic usage - list all CVEs in an image by tag
zli cve list alpine:latest

# List CVEs using image digest
zli cve list myapp@sha256:abc123def456...

# Filter to show only a specific CVE
zli cve list alpine:latest --cve-id CVE-2021-44228

Find Images Affected by a CVE¶

List all images in a repository (or all repositories) that contain a specific CVE:

zli cve affected <CVE-ID>

Options:

• --repo: Filter results to a specific repository

Examples:

# List all images affected by a CVE across all repositories
zli cve affected CVE-2021-44228

# Filter to show only images in a specific repository
zli cve affected CVE-2021-44228 --repo myapp

Find Tags Where a CVE is Fixed¶

List image tags where a specific CVE has been fixed:

zli cve fixed <repo> <CVE-ID>

Examples:

# List all tags where a CVE is fixed
zli cve fixed myapp CVE-2021-44228

Compare CVEs Between Images¶

Find CVEs present in one image but not in another:

zli cve diff <image1> <image2>
# or with platform specifications
zli cve diff <image1> [platform1] <image2> [platform2]

Examples:

# Compare CVEs between two images by tag
zli cve diff myapp:v1.0 myapp:v2.0

# Compare CVEs between two images using digest
zli cve diff myapp@sha256:abc123... myapp@sha256:def456...

# Compare CVEs between images with platform specification for first image
zli cve diff myapp:v1.0 linux/amd64 myapp:v2.0

# Compare CVEs between images with platform specifications for both images
zli cve diff myapp:v1.0 linux/amd64 myapp:v2.0 linux/arm64

# Compare CVEs between different repositories
zli cve diff prod/app:v1.0 staging/app:v2.0

GraphQL API¶

The search extension exposes a GraphQL API for querying CVE information. The endpoint is available at /v2/_zot/ext/search.

For detailed GraphQL query examples, see Using GraphQL for Enhanced Searches.

Example Query - Get CVEs for an Image:

query {
  CVEListForImage(image: "alpine:latest") {
    Tag
    CVEList {
      Id
      Title
      Description
      Severity
      PackageList {
        Name
        InstalledVersion
        FixedVersion
        PackagePath
      }
    }
  }
}

Example Query - Get Images Affected by a CVE:

query {
  ImageListForCVE(id: "CVE-2021-44228", repo: "myapp") {
    Tag
    ManifestDigest
  }
}

How Scanning Works Internally¶

Database Updates¶

1. Initial Download: When CVE scanning is first enabled, zot downloads the Trivy databases. The databases are initially downloaded to /tmp (due to limitations in the ORAS download library used by Trivy) and then moved to the final destination at {storageRoot}/_trivy/db/

2. Periodic Updates: The databases are updated automatically in the background based on the updateInterval setting. The update task runs as soon as zot starts and continues periodically

3. Update Process:

4. Downloads the latest database from the configured repository (default: ghcr.io/aquasecurity/trivy-db)
5. Updates both the main database and Java database (if configured)

6. Purges the entire scan cache to ensure new scans use updated vulnerability data

7. Error Handling: If an update fails, the system will retry with exponential backoff

8. Database Sizes:

9. The main Trivy database is initially under 100 MB and may grow as new scan results are stored
10. The Trivy Java database is larger (several hundred MB) and typically does not grow

Image Scanning¶

1. Automatic Background Scanning: The scanner runs every 15 minutes (hardcoded, not configurable) to check for unscanned images

2. Cache Check: Before scanning, the system checks if results are already cached for the image digest

3. Format Validation: The image format is validated to ensure it's scannable

4. Trivy Execution: The Trivy scanner analyzes the image:

5. Extracts package information from image layers
6. Matches packages against the vulnerability database

7. Identifies CVEs with severity levels

8. Result Caching: Scan results are cached using manifest digests as keys. This means:

9. If the same manifest is stored in different repositories or with different tag names, the cached results are reused
10. If a new manifest is pushed with an existing tag name, it will be picked up on the next scheduled scan run

11. Cache entries are keyed by digest, not by repository or tag, enabling efficient result reuse

12. Result Formatting: CVE information is formatted and returned with:

13. CVE ID and title
14. Severity (Unknown, Low, Medium, High, Critical)
15. Affected packages with installed and fixed versions
16. Package paths

Multi-Arch Image Support¶

For multi-arch images (image indexes):

• Each platform-specific manifest is scanned individually
• Results are aggregated across all platforms
• The index is considered scannable if at least one manifest is scannable

Storage Structure¶

When CVE scanning is enabled, zot creates the following directory structure:

{storageRoot}/
├── _trivy/
│   ├── db/
│   │   ├── metadata.json
│   │   └── trivy.db
│   ├── fanal/
│   │   └── fanal.db
│   └── java-db/
│       ├── metadata.json
│       └── trivy-java.db
└── (image storage)

For configurations with multiple storage paths, each path will have its own _trivy directory.

Performance Considerations¶

1. First Scan: The initial scan of an image may take longer as it needs to analyze all layers. Large layers are unpacked under /tmp, so ensure adequate temporary space

2. Cached Results: Subsequent queries for the same image digest return cached results instantly

3. Database Updates: Database updates run in the background and don't block image operations

4. Concurrent Scans: Multiple images can be scanned concurrently, but database updates are serialized

5. Cache Size: The scan cache has a default size limit (1,000,000 entries) to manage memory usage

6. On-Demand Scanning: If a scan is triggered on-demand (when results aren't cached), the request will wait for the scan to complete. This ensures accurate results but may cause delays for large images

Limitations¶

1. Offline Scanning Only: zot uses Trivy in offline mode; it doesn't perform real-time vulnerability lookups

2. Database Updates: Vulnerability information is only as current as the last database update. Only use trusted sources for Trivy databases. The default repositories (ghcr.io/aquasecurity/*) are maintained by Aqua Security. If you configure custom database repositories, ensure they are from trusted sources.

3. Image Format Support: Some image formats or layer types may not be scannable

4. Java Database: The Java database is optional; Java vulnerabilities may not be detected if not configured

5. Cache Invalidation: The cache is purged on database updates; all images will need to be rescanned

6. Scale-Out Clustering: CVE scanning is not supported for cloud deployments or scale-out clustering scenarios. In these cases, we recommend implementing a CVE repository with a zot instance outside of the cluster using a local disk for storage.

Related Documentation¶

• Configuring zot - Configuration details
• Using the command line interface (zli) - CLI command reference
• Using GraphQL for Enhanced Searches - GraphQL API examples
• Trivy Documentation - Trivy scanner documentation