OCI Registry Mirroring With zot¶

A zot registry can mirror one or more upstream OCI registries, including popular cloud registries such as Docker Hub and Google Container Registry (gcr.io).

A key use case for zot is to act as a mirror for upstream registries. If an upstream registry is OCI distribution-spec conformant for pulling images, you can use zot's sync feature to implement a downstream mirror, synchronizing OCI images and corresponding artifacts. Because synchronized images are stored in zot's local storage, registry mirroring allows for a fully distributed disconnected container image build pipeline. Container image operations terminate in local zot storage, which may reduce network latency and costs.

Beginning with zot v2.1.22, sync preserves upstream media types and digests. To sync Docker-format images (application/vnd.docker.distribution.manifest.v2+json), enable http.compat as ["docker2s2"]. Without this compatibility setting, zot rejects Docker schema 2 content instead of converting it to OCI.

Mirroring modes¶

For mirroring an upstream registry, two common use cases are a fully mirrored or a pull through (on-demand) cache registry.

As with git, wherein every clone is a full repository, you can configure your local zot instance to be a fully mirrored OCI registry. For this mode, configure zot for synchronization by periodic polling, not on-demand. Zot copies and caches a full copy of every image on the upstream registry, updating the cache whenever polling discovers a change in content or image version at the upstream registry.

For a pull through cache mirrored registry, configure zot for on-demand synchronization. When an image is first requested from the local zot registry, the image is downloaded from the upstream registry and cached in local storage. Subsequent requests for the same image are served from zot's cache. Images that have not been requested are not downloaded. If a polling interval is also configured, zot periodically polls the upstream registry for changes, updating any cached images if changes are detected.

Because Docker Hub rate-limits pulls and does not support catalog listing, do not use polled mirroring with Docker Hub. Use only on-demand mirroring with Docker Hub.

When to enable compat¶

Enable http.compat: ["docker2s2"] when any of the following apply:

• Upstream images use Docker Image Manifest v2, Schema 2 and you need to store them unchanged (common on Docker Hub and operator/catalog images)
• Docker-format upstream images have cosign or notation signatures and referrers tied to their Docker manifest digests

You can omit compat only when all synchronized content is OCI-formatted. Sync preserves the upstream digest in either case.

preserveDigest is deprecated and ignored beginning with v2.1.22. Remove it from new configurations. Digest preservation is automatic, while http.compat controls whether Docker schema 2 content is accepted.

Migrating or updating a registry using mirroring¶

Mirroring zot using the sync feature allows you to easily migrate a registry. In situations such as the following, zot mirroring provides an easy solution.

• Migrating an existing zot or non-zot registry to a new location.

  Provided that the source registry is OCI-compliant for image pulls, you can mirror the registry to a new zot registry, delete the old registry, and reroute network traffic to the new registry.

• Updating (or downgrading) a zot registry.

  To minimize downtime during an update, or to avoid any incompatibilities between zot releases that would preclude an in-place update, you can bring up a new zot registry with the desired release and then migrate from the existing registry.

To ensure a complete migration of the registry contents, set a polling interval in the configuration of the new zot registry and set prefix to **, as shown in this example:

  {
    "urls": [
        "https://registry1:5000"
    ],
    "pollInterval": "12h",
    "onDemand": true,
    "content": [
        {
            "prefix": "**"
        }
    ]
  }

Basic configuration for mirroring with sync¶

The sync feature of zot is an extension of the OCI-compliant registry implementation. You can configure the sync feature under the extensions section of the zot configuration file, as shown in this example.

Configure http.compat: ["docker2s2"] in the same configuration file when an upstream can return Docker schema 2 content. See When to enable compat.

  "extensions": {
    "sync": {
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "platforms": ["linux/amd64", "linux/arm64"],
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m",
          "maxRetryDelay": "30m",
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo2/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              },
              "destination": "/repo2",
              "stripPrefix": true,
              "platforms": ["linux/amd64"]
            }
          ]
        }
      ]
    }
  }

The following table lists the configurable attributes for the sync feature:

{
  "127.0.0.1:8080": {
    "username": "user",
    "password": "pass"
  },
  "registry2:5000": {
    "username": "user2",
    "password": "pass2"
  }
}

Selecting platforms for periodic sync¶

Use platforms to create a sparse local copy of a multi-architecture image. The registry-level list supplies the default for periodic sync, and a content rule can override it:

{
  "urls": ["https://registry.example.com"],
  "pollInterval": "6h",
  "platforms": ["linux/amd64", "linux/arm64"],
  "content": [{
    "prefix": "edge/**",
    "platforms": ["linux/arm64/v8"]
  }]
}

The index retains its upstream digest even when only selected child manifests are stored. Storage validation, garbage collection, scanning, and scrub tolerate children that were intentionally omitted. On-demand sync remains demand-driven and ignores both platform lists.

Platform entries can use os/arch, os/arch/variant, or a bare architecture such as amd64. Include an empty string ("") in a nonempty allowlist to retain descriptors that do not declare a platform, including attestation-style entries.

On-demand sync in the background¶

Set both onDemand and onDemandInBackground to return immediately when a manifest is absent locally and populate it asynchronously:

{
  "urls": ["https://registry-1.docker.io"],
  "onDemand": true,
  "onDemandInBackground": true
}

The initial manifest request receives a real 404; later requests succeed after the background sync completes. Blob misses alone do not start background sync. Do not enable this mode when zot is the client's only registry, and do not combine it with manifestCheckInterval.

Controlling on-demand upstream requests¶

The following example revalidates a cached tag at most once every five minutes and limits each upstream host to 20 concurrent requests and 100 requests per second. It also uses a larger HTTP/1.1 connection pool for high-throughput transfers.

{
  "extensions": {
    "sync": {
      "registries": [
        {
          "urls": ["https://registry.example.com"],
          "onDemand": true,
          "manifestCheckInterval": "5m",
          "reqConcurrent": 20,
          "reqPerSec": 100,
          "disableHTTP2": true,
          "maxIdleConnsPerHost": 20
        }
      ]
    }
  }
}

These limits are applied independently for each upstream host. When concurrent clients request the same uncached image, zot coalesces those requests into one sync operation.

Configuring mirroring modes¶

Two mirroring modes were described in this document: - periodic - the registry syncs images matching specific patters from the upstream registries at a given polling interval - on demand - the registry reaches out to the upstream registries when the image is requested by the user

These two modes can be configured, separately or together, using specific settings. See the table below for details:

Configuration examples for mirroring¶

Example: Multiple repositories with polled mirroring¶

The following is an example of sync configuration for mirroring multiple repositories with polled mirroring. If the upstream can return Docker schema 2 content, the complete configuration file must also include http.compat: ["docker2s2"] (see When to enable compat).

"sync": {
  "enable": true,
  "credentialsFile": "./examples/sync-auth-filepath.json",
  "registries": [
    {
      "urls": [
        "https://registry1:5000"
      ],
      "onDemand": false,
      "pollInterval": "6h",
      "tlsVerify": true,
      "certDir": "/home/user/certs",
      "maxRetries": 3,
      "retryDelay": "5m",
      "onlySigned": true,
      "content": [
        {
          "prefix": "/repo1/repo",
          "tags": {
            "regex": "4.*",
            "semver": true,
            "tags": {
              "excludeRegex": ".*-(amd64|arm64)$"
            }
          }
        },
        {
          "prefix": "/repo2/repo",
          "destination": "/repo2",
          "stripPrefix": true
        },
        {
          "prefix": "/repo3/repo"
        }
      ]
    }
  ]
}

The configuration in this example will result in the following behavior:

• Only signed images (notation and cosign) are synchronized.
• The sync communication is secured using certificates in certDir.
• This registry synchronizes with upstream registry every 6 hours.
• Upstream digests are preserved. Enable http.compat: ["docker2s2"] in the full configuration file when the upstream can return Docker schema 2 content.
• On-demand mirroring is disabled.
• Based on the content filtering options, this registry synchronizes these images:
  • From /repo1/repo, images with tags that begin with "4." and are semver compliant but excluding some tag patterns
    Files are stored locally in /repo1/repo on localhost.
  • From /repo2/repo, images with all tags.
    Because stripPrefix is enabled, files are stored locally in /repo2. For example, docker://upstream/repo2/repo:v1 is stored as docker://local/repo2:v1.
  • From /repo3/repo, images with all tags.
    Files are stored locally in /repo3/repo.

Example: Multiple registries with on-demand mirroring¶

The following is an example of sync configuration for mirroring multiple registries with on-demand mirroring.

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "rootDirectory": "/tmp/zot",
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": ["https://k8s.gcr.io"],
          "content": [
            {
              "prefix": "**",
              "destination": "/k8s-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true,
          "syncTimeout": "10m"
        },
        {
          "urls": [
              "https://index.docker.io"
          ],
          "content": [
            {
              "prefix": "**",
              "destination": "/docker-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        }
      ]
    }
  }
}

With this zot configuration, the sync behavior is as follows:

1. This initial user request for content from the zot registry:
   skopeo copy --src-tls-verify=false docker://localhost:8080/docker-images/alpine <dest>
   causes zot to synchronize the content with the docker.io registry:
       docker.io/library/alpine:latest
   to the zot registry:
       localhost:8080/docker-images/alpine:latest
   before delivering the content to the requestor at <dest>.

2. This initial user request for content from the zot registry:
   skopeo copy --src-tls-verify=false docker://localhost:8080/k8s-images/kube-proxy:v1.19.2 <dest>
   causes zot to synchronize the content with the gcr.io registry:
       k8s.gcr.io/kube-proxy:v1.19.2
   to the zot registry:
       localhost:8080/k8s-images/kube-proxy:v1.19.2
   before delivering the content to the requestor at <dest>.

You can use this command:
     curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy"
    ]
  }

Example: Multiple registries with mixed mirroring modes¶

The following is an example of a zot configuration file for mirroring multiple upstream registries.

{
  "distSpecVersion": "1.1.0-dev",
  "storage": {
    "rootDirectory": "/tmp/zot"
  },
  "http": {
    "address": "127.0.0.1",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m",
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo1/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              }
            },
            {
              "prefix": "/repo1/repo",
              "destination": "/repo",
              "stripPrefix": true
            },
            {
              "prefix": "/repo2/repo"
            }
          ]
        },
        {
          "urls": [
            "https://registry2:5000",
            "https://registry3:5000"
          ],
          "pollInterval": "12h",
          "tlsVerify": false,
          "onDemand": false,
          "content": [
            {
              "prefix": "/repo2",
              "tags": {
                "semver": true
              }
            }
          ]
        },
        {
          "urls": [
              "https://index.docker.io"
          ],
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 6,
          "retryDelay": "5m"
        }
      ]
    }
  }
}

Example: Support for subpaths in local storage¶

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "subPaths":{
      "/kube-proxy":{
        "rootDirectory": "/tmp/kube-proxy",
        "dedupe": true,
        "gc": true
       }
     },
    "rootDirectory": "/tmp/zot",
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": ["https://k8s.gcr.io"],
          "content": [
            {
              "destination": "/kube-proxy",
              "prefix": "**"
            }
          ],
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 2,
          "retryDelay": "5m"
        }
      ]
    }
  }
}

• This user request for content from the zot registry:
  skopeo copy --src-tls-verify=false docker://localhost:8080/kube-proxy/kube-proxy:v1.19.2 <dest>
  causes zot to synchronize the content with this remote registry:
      k8s.gcr.io/kube-proxy:v1.19.2
  to the zot registry:
      localhost:8080/kube-proxy/kube-proxy:v1.19.2
  before delivering the content to the requestor at <dest>.

You can use this command:
     curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy",
      "kube-proxy/kube-proxy"
    ]
  }

In zot storage, the requested content is located here:
    /tmp/zot/kube-proxy/kube-proxy/kube-proxy/
This subpath is created from the following path components:

• /tmp/zot is the rootDirectory of the zot registry
• kube-proxy is the rootDirectory of the storage subpath
• kube-proxy is the sync destination parameter
• kube-proxy is the repository name

Example: Support for AWS ECR¶

This is an example configuration demonstrating how to use the sync extension with Amazon ECR (Elastic Container Registry) credential helper. The configuration enables zot to synchronize container images from an ECR registry.

"extensions": {
        "sync": {
            "credentialsFile": "",
            "DownloadDir": "/tmp/zot",
            "registries": [
                {
                    "urls": [
                        "https://ACCOUNTID.dkr.ecr.REGION.amazonaws.com"
                    ],
                    "onDemand": true,
                    "maxRetries": 5,
                    "retryDelay": "2m",
                    "credentialHelper": "ecr"
                }
            ]
        }
    }

Example: Google registry authentication¶

Use the gcp credential helper to obtain a read-only access token from Google Application Default Credentials. It supports GCE/GKE metadata credentials, GOOGLE_APPLICATION_CREDENTIALS, workload identity federation external-account files, and gcloud user credentials. The principal still needs permission to read the upstream repository, such as roles/artifactregistry.reader.

"extensions": {
  "sync": {
    "registries": [{
      "urls": ["https://REGION-docker.pkg.dev"],
      "onDemand": true,
      "credentialHelper": "gcp"
    }]
  }
}

The helper also works with Google Container Registry hostnames that accept Google access tokens. It requests the cloud-platform.read-only scope and refreshes the short-lived token through the configured ADC token source.

Example: OAuth2 credential helper¶

The oauth2 helper exchanges an assertion for a short-lived registry access token and refreshes it before expiry. Configure exactly one assertion source: assertionFile for an externally issued token that can rotate on disk, or signingFile for a JSON configuration that lets zot mint a fresh single-use JWT for each exchange.

This RFC 8693 example exchanges a rotating workload identity token with a security token service:

"extensions": {
  "sync": {
    "registries": [{
      "urls": ["https://registry.example.com"],
      "onDemand": true,
      "credentialHelper": "oauth2",
      "oauth2CredentialHelper": {
        "tokenURL": "https://sts.example.com/v1/token",
        "assertionFile": "/run/secrets/subject-token",
        "grantType": "urn:ietf:params:oauth:grant-type:token-exchange",
        "audience": "//sts.example.com/pools/example/providers/workload",
        "subjectTokenType": "urn:ietf:params:oauth:token-type:jwt",
        "requestedTokenType": "urn:ietf:params:oauth:token-type:access_token",
        "username": "oauth2accesstoken"
      }
    }]
  }
}

The oauth2CredentialHelper attributes are:

A signingFile contains privateKeyFile and algorithm, plus optional keyId, issuer, subject, and audience values. The issuer and subject default to clientID; the audience defaults to tokenURL.