OCI Registry Mirroring With zot¶

A zot registry can mirror one or more upstream OCI registries, including popular cloud registries such as Docker Hub and Google Container Registry (gcr.io).

A key use case for zot is to act as a mirror for upstream registries. If an upstream registry is OCI distribution-spec conformant for pulling images, you can use zot's sync feature to implement a downstream mirror, synchronizing OCI images and corresponding artifacts. Because synchronized images are stored in zot's local storage, registry mirroring allows for a fully distributed disconnected container image build pipeline. Container image operations terminate in local zot storage, which may reduce network latency and costs.

Because zot is a OCI-only registry, any upstream image stored in the Docker image format is converted to OCI format when downloading to zot. In the conversion, some non-OCI attributes may be lost. Signatures, for example, are removed due to the incompatibility between formats.

Mirroring modes¶

For mirroring an upstream registry, two common use cases are a fully mirrored or a pull through (on-demand) cache registry.

As with git, wherein every clone is a full repository, you can configure your local zot instance to be a fully mirrored OCI registry. For this mode, configure zot for synchronization by periodic polling, not on-demand. Zot copies and caches a full copy of every image on the upstream registry, updating the cache whenever polling discovers a change in content or image version at the upstream registry.

For a pull through cache mirrored registry, configure zot for on-demand synchronization. When an image is first requested from the local zot registry, the image is downloaded from the upstream registry and cached in local storage. Subsequent requests for the same image are served from zot's cache. Images that have not been requested are not downloaded. If a polling interval is also configured, zot periodically polls the upstream registry for changes, updating any cached images if changes are detected.

Because Docker Hub rate-limits pulls and does not support catalog listing, do not use polled mirroring with Docker Hub. Use only on-demand mirroring with Docker Hub.

Migrating or updating a registry using mirroring¶

Mirroring zot using the sync feature allows you to easily migrate a registry. In situations such as the following, zot mirroring provides an easy solution.

• Migrating an existing zot or non-zot registry to a new location.

  Provided that the source registry is OCI-compliant for image pulls, you can mirror the registry to a new zot registry, delete the old registry, and reroute network traffic to the new registry.

• Updating (or downgrading) a zot registry.

  To minimize downtime during an update, or to avoid any incompatibilities between zot releases that would preclude an in-place update, you can bring up a new zot registry with the desired release and then migrate from the existing registry.

To ensure a complete migration of the registry contents, set a polling interval in the configuration of the new zot registry and set prefix to **, as shown in this example:

  {
    "urls": [
        "https://registry1:5000"
    ],
    "pollInterval": "12h",
    "onDemand": true,
    "content": [
        {
            "prefix": "**"
        }
    ]
  }

Basic configuration for mirroring with sync¶

The sync feature of zot is an extension of the OCI-compliant registry implementation. You can configure the sync feature under the extensions section of the zot configuration file, as shown in this example:

  "extensions": {
    "sync": {
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m", 
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo2/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              }
              "destination": "/repo2",
              "stripPrefix": true
            }
          ]
        }
      ]
    }
  }

The following table lists the configurable attributes for the sync feature:

{
  "127.0.0.1:8080": {
    "username": "user",
    "password": "pass"
  },
    "registry2:5000": {
    "username": "user2",
    "password": "pass2"
  }
}

Configuration examples for mirroring¶

Example: Multiple repositories with polled mirroring¶

The following is an example of sync configuration for mirroring multiple repositories with polled mirroring.

"sync": {
  "enable": true,
  "credentialsFile": "./examples/sync-auth-filepath.json",
  "registries": [
    {
      "urls": ["https://registry1:5000"],
      "onDemand": false,
      "pollInterval": "6h",
      "tlsVerify": true,
      "certDir": "/home/user/certs",
      "maxRetries": 3,
      "retryDelay": "5m",
      "onlySigned": true,
      "content": [
        {
          "prefix": "/repo1/repo",
          "tags": {
            "regex": "4.*",
            "semver": true
          }
        },
        {
          "prefix": "/repo2/repo",
          "destination": "/repo2",
          "stripPrefix": true
        },
        {
          "prefix": "/repo3/repo"
        }
      ]
    }
  }

The configuration in this example will result in the following behavior:

• Only signed images (notation and cosign) are synchronized.
• The sync communication is secured using certificates in certDir.
• This registry synchronizes with upstream registry every 6 hours.
• On-demand mirroring is disabled.
• Based on the content filtering options, this registry synchronizes these images:
  • From /repo1/repo, images with tags that begin with "4." and are semver compliant.
    Files are stored locally in /repo1/repo on localhost.
  • From /repo2/repo, images with all tags.
    Because stripPrefix is enabled, files are stored locally in /repo2. For example, docker://upstream/repo2/repo:v1 is stored as docker://local/repo2:v1.
  • From /repo3/repo, images with all tags.
    Files are stored locally in /repo3/repo.

Example: Multiple registries with on-demand mirroring¶

The following is an example of sync configuration for mirroring multiple registries with on-demand mirroring.

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "rootDirectory": "/tmp/zot",
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": ["https://k8s.gcr.io"],
          "content": [
            {
              "prefix": "**", 
              "destination": "/k8s-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        },
        {
          "urls": ["https://docker.io/library"],
          "content": [
            {
              "prefix": "**", 
              "destination": "/docker-images"
            }
          ],
          "onDemand": true,
          "tlsVerify": true
        }
      ]
    }
  }
}

With this zot configuration, the sync behavior is as follows:

1. This initial user request for content from the zot registry:
   skopeo copy --src-tls-verify=false docker://localhost:8080/docker-images/alpine <dest>
   causes zot to synchronize the content with the docker.io registry:
       docker.io/library/alpine:latest
   to the zot registry:
       localhost:8080/docker-images/alpine:latest
   before delivering the content to the requestor at <dest>.

2. This initial user request for content from the zot registry:
   skopeo copy --src-tls-verify=false docker://localhost:8080/k8s-images/kube-proxy:v1.19.2 <dest>
   causes zot to synchronize the content with the gcr.io registry:
       k8s.gcr.io/kube-proxy:v1.19.2
   to the zot registry:
       localhost:8080/k8s-images/kube-proxy:v1.19.2
   before delivering the content to the requestor at <dest>.

You can use this command:
     curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy"
    ]
  }

Example: Multiple registries with mixed mirroring modes¶

The following is an example of a zot configuration file for mirroring multiple upstream registries.

{
  "distSpecVersion": "1.1.0-dev",
  "storage": {
    "rootDirectory": "/tmp/zot"
  },
  "http": {
    "address": "127.0.0.1",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "credentialsFile": "./examples/sync-auth-filepath.json",
      "registries": [
        {
          "urls": [
            "https://registry1:5000"
          ],
          "onDemand": false,
          "pollInterval": "6h",
          "tlsVerify": true,
          "certDir": "/home/user/certs",
          "maxRetries": 3,
          "retryDelay": "5m",
          "onlySigned": true,
          "content": [
            {
              "prefix": "/repo1/repo",
              "tags": {
                "regex": "4.*",
                "semver": true
              }
            },
            {
              "prefix": "/repo1/repo",
              "destination": "/repo",
              "stripPrefix": true
            },
            {
              "prefix": "/repo2/repo"
            }
          ]
        },
        {
          "urls": [
            "https://registry2:5000",
            "https://registry3:5000"
          ],
          "pollInterval": "12h",
          "tlsVerify": false,
          "onDemand": false,
          "content": [
            {
              "prefix": "/repo2",
              "tags": {
                "semver": true
              }
            }
          ]
        },
        {
          "urls": [
            "https://docker.io/library"
          ],
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 6,
          "retryDelay": "5m"
        }
      ]
    }
  }
}

Example: Support for subpaths in local storage¶

{
  "distSpecVersion": "1.0.1",
  "storage": {
    "subPaths":{
      "/kube-proxy":{
        "rootDirectory": "/tmp/kube-proxy",
        "dedupe": true,
        "gc": true
       }
     },
    "rootDirectory": "/tmp/zot",
    "gc": true
  },
  "http": {
    "address": "0.0.0.0",
    "port": "8080"
  },
  "log": {
    "level": "debug"
  },
  "extensions": {
    "sync": {
      "enable": true,
      "registries": [
        {
          "urls": ["https://k8s.gcr.io"],
          "content": [
            {
              "destination": "/kube-proxy", 
              "prefix": "**"
            }
          ],
          "onDemand": true,
          "tlsVerify": true,
          "maxRetries": 2,
          "retryDelay": "5m"
        }
      ]
    }
  }
}

• This user request for content from the zot registry:
  skopeo copy --src-tls-verify=false docker://localhost:8080/kube-proxy/kube-proxy:v1.19.2 <dest>
  causes zot to synchronize the content with this remote registry:
      k8s.gcr.io/kube-proxy:v1.19.2
  to the zot registry:
      localhost:8080/kube-proxy/kube-proxy:v1.19.2
  before delivering the content to the requestor at <dest>.

You can use this command:
     curl http://localhost:8080/v2/_catalog
to display the local repositories:

  {
    "repositories":[
      "docker-images/alpine",
      "k8s-images/kube-proxy",
      "kube-proxy/kube-proxy"
    ]
  }

In zot storage, the requested content is located here:
    /tmp/zot/kube-proxy/kube-proxy/kube-proxy/
This subpath is created from the following path components:

• /tmp/zot is the rootDirectory of the zot registry
• kube-proxy is the rootDirectory of the storage subpath
• kube-proxy is the sync destination parameter
• kube-proxy is the repository name