Container Image Cache Support

Enable a pull through cache for your Hybrid deployment

In the event that you can't pull container images directly from our public/private container image registries, we support caching images in customer owned image registries. This configuration is commonly known as a pull through cache and is supported by a number of different container registries. The steps to implement this solution are documented below.

Setting Up a Pull Through Cache

1. Reach out to the Customer Success Team via email to request a read only DockerHub token for our private DockerHub repository. Please reach out to your enterprise sales representative for help starting this conversation if you are not already engaged with the Customer Success Team.

2. Set up a Pull Through Cache within your customer owned container registry.

Amazon ECR

For Amazon ECR, follow the ECR Pull through cache rule docs for Docker Hub

Azure

For Azure Container Registry, follow the instructions to set up a Cache for Docker Hub

You'll want to set the upstream to docker.io/gretelai/* and your ACR repository to be gretelai/*

GCP

GCP's Artifact Registry does not provide a pull through cache configuration at this time. Please reach out to the Customer Success Team for further support.

Artifactory

For Artifactory, follow the instructions to configure a remote repository.

1. You can verify this works by running a pull to your registry for our images

docker pull <your_registry_url>/gretelai/gcc-controller:0.0.3

1. You will have to create a pull secret so the Gretel Hybrid controller knows it can authenticate with the image registry. You will need to create a username and password for the Gretel Hybrid controller to use within your image registry. Please consult your image registry's documentation to create a username/password identity with read access to the registry. Once that username and password have been created, use the below example kubectl command to create a pull secret in the Gretel Hybrid namespace.

Note that the secret name my-pull-secret will be referenced in the next step when setting Helm values for the Gretel Hybrid installation. It can be any name EXCEPT gretel-pull-secret, which is already used by the agent

kubectl create secret docker-registry my-pull-secret \
  --namespace gretel-hybrid \
  --docker-server=$registry_name \
  --docker-username=$username \
  --docker-password=$password

1. When executing the hybrid install, utilize the helm values below to pass the updated registry URL to the Gretel Hybrid helm deployment. Be sure to review each value and change them as necessary based on your environment.

imagePullSecrets: ["my-pull-secret"]
gretelController:
  image:
    repository: "<your_registry_url>/gretelai/gcc-controller"
gretelAgent:
  image:
    repository: "<your_registry_url>/gretelai/gcc-agent"

gretelWorkers:
  images:
    registry: "<your_registry_url>/gretelai/"

If you are deploying the helm chart using our gretel_hybrid Terraform module, the customized values may be passed to the module declaration using the Terraform variable extra_helm_values. An example is provided below.

module "full_deployment" {
    ...
   extra_helm_values = {
    imagePullSecrets = ["my-pull-secret"]
    gretelController = {
      image = {
        repository = "<your_registry_url>/gretelai/gcc-controller"
      }
    }
    gretelAgent = {
      image = {
        repository = "<your_registry_url>/gretelai/gcc-controller"
      }
    }

    gretelWorkers = {
      image = {
        registry = "<your_registry_url>/gretelai"
      }
    }
  }
}

Pulling images by tag

You can also pass a specific tag associated with each image. This is necessary in the case where your image repository is not a pull through cache but a manually updated mirror. In this case, you could specify the latest tag for each image to ensure your deployment pulls the latest image version that has been manually synced.

In addition to the Terraform example which uses extra_helm_values to configure the Helm deployment, you would add a tag section under each image property. An example is show below in YAML format.

imagePullSecrets: ["my-pull-secret"]
gretelController:
  image:
    repository: "<your_registry_url>/gretelai/gcc-controller"
    tag: "<your_helm_chart_version>"
gretelAgent:
  image:
    repository: "<your_registry_url>/gretelai/gcc-agent"
    tag: "<your_helm_chart_version>"

gretelWorkers:
  images:
    registry: "<your_registry_url>/gretelai/"
    tag: latest

module "full_deployment" {
    ...
   extra_helm_values = {
    imagePullSecrets = ["my-pull-secret"]
    gretelController = {
      image = {
        repository = "<your_registry_url>/gretelai/gcc-controller"
        tag        = "<your_helm_chart_version>
      }
    }
    gretelAgent = {
      image = {
        repository = "<your_registry_url>/gretelai/gcc-agent"
        tag        = "<your_helm_chart_version>
      }
    }

    gretelWorkers = {
      image = {
        registry = "<your_registry_url>/gretelai"
        tag      = "latest"
      }
    }
  }
}

Pulling argo images from another registry

If you have a requirement to pull the argoexec and workflow-controller, you'll first need to mirror those images (quay.io/argoproj/argoexec:v3.5.2 and quay.io/argoproj/workflow-controller:v3.5.2) to your repository, then you can update the yaml/ terraform as follows.

argoController:
  image:
    ## The repository of the image to use.
    repository: "<your_registry>/argoproj/workflow-controller"
    ## The tag of the image to use.
    tag: "v3.5.2"

argoConfig:
  executor:
    image: <your_registry_url>/argoproj/argoexec:v3.5.2

module "full_deployment" {
    ...
   extra_helm_values = {

    argoController = {
      image = {
        repository = "<your_registry>/argoproj/workflow-controller"
        tag: "v3.5.2"
      }
    }

    argoConfig = {
      executor = {
        image = "<your_registry_url>/argproj/argoexec:v3.5.2"
      }
    }

  }
}