Kubernetes operator for air-gapped instances

Deploy W&B Platform with Kubernetes Operator (Airgapped)

Introduction

This guide provides step-by-step instructions to deploy the W&B Platform in air-gapped customer-managed environments.

Use an internal repository or registry to host the Helm charts and container images. Run all commands in a shell console with proper access to the Kubernetes cluster.

You could utilize similar commands in any continuous delivery tooling that you use to deploy Kubernetes applications.

Step 1: Prerequisites

Before starting, make sure your environment meets the following requirements:

Version Requirements

Software	Minimum version
Kubernetes	v1.32 or newer (Supported Kubernetes versions)
Helm	v3.x
MySQL	v8.0.x is required, v8.0.32 or newer; v8.0.44 or newer is recommended. Aurora MySQL 3.x releases, must be v3.05.2 or newer
Redis	v7.x

SSL/TLS Requirements

W&B requires a valid signed SSL/TLS certificate for secure communication between clients and the server. SSL/TLS termination must occur on the ingress/load balancer. The W&B Server application does not terminate SSL or TLS connections.

Important: W&B does not support self-signed certificates and custom CAs. Using self-signed certificates will cause challenges for users and is not supported.

If possible, using a service like Let’s Encrypt is a great way to provide trusted certificates to your load balancer. Services like Caddy and Cloudflare manage SSL for you.

If your security policies require SSL communication within your trusted networks, consider using a tool like Istio and side car containers.

Hardware Requirements

CPU Architecture: W&B runs on Intel (x86) CPU architecture only. ARM is not supported.

Sizing: For CPU, memory, and disk sizing recommendations for Kubernetes nodes and MySQL, see the Sizing section in the reference architecture. Requirements vary based on whether you’re running Models, Weave, or both.

MySQL Database

W&B requires an external MySQL database.

For production, W&B strongly recommends using managed database services:

• AWS RDS Aurora MySQL
• Google Cloud SQL for MySQL
• Azure Database for MySQL

Managed database services provide automated backups, monitoring, high availability, patching, and reduce operational overhead.

See the reference architecture for complete MySQL requirements, including sizing recommendations and configuration parameters. For database creation SQL, see the bare-metal guide. For questions about your deployment’s database configuration, contact support or your AISE.

Redis

W&B depends on a single-node Redis 7.x deployment used by W&B’s components for job queuing and data caching. For convenience during testing and development of proofs of concept, W&B Self-Managed includes a local Redis deployment that is not appropriate for production deployments.

For production deployments, W&B can connect to a Redis instance in the following environments:

• AWS Elasticache
• Google Cloud Memory Store
• Azure Cache for Redis
• Redis deployment hosted in your cloud or on-premise infrastructure

Object storage

W&B requires object storage with pre-signed URL and CORS support.

For production deployments, W&B recommends using managed object storage services:

• Amazon S3: Object storage service offering industry-leading scalability, data availability, security, and performance.
• Google Cloud Storage: Managed service for storing unstructured data at scale.
• Azure Blob Storage: Cloud-based object storage solution for storing massive amounts of unstructured data.
• CoreWeave AI Object Storage: High-performance, S3-compatible object storage service optimized for AI workloads.

For self-hosted object storage options, see the bare-metal guide object storage section for detailed setup instructions including CORS configuration and enterprise alternatives.

MinIO Open Source is in maintenance mode with no active development or pre-compiled binaries. For production deployments, W&B recommends using managed object storage services or enterprise-grade S3-compatible solutions.

See the reference architecture object storage section for complete requirements.

Additional Requirements

• Access to an internal container registry with the required W&B images
• Access to an internal Helm repository for W&B Helm charts

For complete infrastructure requirements, including networking and load balancer configuration, see the reference architecture.

W&B can be deployed on air-gapped OpenShift Kubernetes clusters. See the reference architecture for details, and review the OpenShift section of the Operator guide for specific configuration instructions that you can adapt for your air-gapped OpenShift deployment.

Step 2: Prepare internal container registry

For a successful air-gapped deployment, the following container images must be available in your air-gapped container registry.

You are responsible for tracking the W&B Operator’s requirements and maintaining your container registry with updated images regularly. For the most current list of required container images and versions, refer to the Helm chart, or contact support or your AISE.

Core W&B component containers

• docker.io/wandb/controller
• docker.io/wandb/local
• docker.io/wandb/console
• docker.io/wandb/megabinary

Dependencies

• docker.io/bitnamilegacy/redis: Required for local Redis deployment during testing and development. To use the local Redis deployment, ensure that this image is available in your container registry. For production Redis requirements, see the Redis section in Prerequisites.
• docker.io/otel/opentelemetry-collector-contrib: W&B depends on the OpenTelemetry agent to collect metrics and logs from resources at the Kubernetes layer for display in W&B.
• quay.io/prometheus/prometheus: W&B depends on Prometheus to capture metrics from various components for display in W&B.
• quay.io/prometheus-operator/prometheus-config-reloader: A required dependency of Prometheus.

Get required images

To extract the complete list of required images and versions from the Helm chart values:

1. Download the W&B Operator and Platform Helm charts from the W&B Helm charts repository.

2. Inspect the values.yaml files to identify all container images and their versions:

   # Extract image references from the Helm chart
   helm show values wandb/operator \
     | awk -F': *' '/^[[:space:]]*repository:/{print $2}' \
     | grep -E '^wandb/' \
     | sort -u
   

   The list might look similar to the following. Image versions may vary.

   wandb/anaconda2
   wandb/console
   wandb/frontend-nginx
   wandb/local
   wandb/megabinary
   wandb/weave-python
   wandb/weave-trace
   

Step 3: Prepare internal Helm chart repository

Along with the container images, you also must ensure that the following Helm charts are available in your internal Helm Chart repository. Download them from:

• W&B Operator
• W&B Platform

The operator chart is used to deploy the W&B Operator, which is also referred to as the Controller Manager. The platform chart is used to deploy the W&B Platform using the values configured in the custom resource definition (CRD).

Step 4: Set up Helm repository

Now, configure the Helm repository to pull the W&B Helm charts from your internal repository. Run the following commands to add and update the Helm repository:

helm repo add local-repo https://charts.yourdomain.com
helm repo update

Step 5: Install the Kubernetes operator

The W&B Kubernetes operator, also known as the controller manager, is responsible for managing the W&B platform components. To install it in an air-gapped environment, you must configure it to use your internal container registry.

To do so, you must override the default image settings to use your internal container registry and set the key airgapped: true to indicate the expected deployment type. Update the values.yaml file as shown below:

image:
  repository: registry.yourdomain.com/library/controller
  tag: 1.13.3
airgapped: true

Replace the tag with the version that is available in your internal registry.

Install the operator and the CRD:

helm upgrade --install operator local-repo/operator -n wandb --create-namespace -f values.yaml

For full details about the supported values, refer to the Kubernetes operator GitHub repository.

Step 6: Set up MySQL database

Before configuring the W&B Custom Resource, you must set up an external MySQL database. For production deployments, W&B strongly recommends using managed database services. However, if you are running your own MySQL instance, create the database and user:

Create a database and a user with the following SQL commands. Replace SOME_PASSWORD with a secure password of your choice:

CREATE USER 'wandb_local'@'%' IDENTIFIED BY 'SOME_PASSWORD';
CREATE DATABASE wandb_local CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;
GRANT ALL ON wandb_local.* TO 'wandb_local'@'%' WITH GRANT OPTION;

For MySQL configuration parameters, see the reference architecture MySQL configuration section.

Step 7: Configure W&B Custom Resource

After installing the W&B Kubernetes operator, you must configure the Custom Resource (CR) to point to your internal Helm repository and container registry.

This configuration ensures that the Kubernetes operators uses your internal registry and repository are when it deploys the required components of the W&B platform.

Copy this example CR to a new file named wandb.yaml.

apiVersion: apps.wandb.com/v1
kind: WeightsAndBiases
metadata:
  labels:
    app.kubernetes.io/instance: wandb
    app.kubernetes.io/name: weightsandbiases
  name: wandb
  namespace: default

spec:
  chart:
    url: http://charts.yourdomain.com
    name: operator-wandb
    version: 0.18.0

  values:
    global:
      host: https://wandb.yourdomain.com
      license: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      bucket:
        accessKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        secretKey: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
        name: s3.yourdomain.com:port #Ex.: s3.yourdomain.com:9000
        path: bucket_name
        provider: s3
        region: us-east-1
      mysql:
        database: wandb
        host: mysql.home.lab
        password: password
        port: 3306
        user: wandb
      redis:
        host: redis.yourdomain.com
        port: 6379
        password: password
      api:
        enabled: true
      glue:
        enabled: true
      executor:
        enabled: true
      extraEnv:
        ENABLE_REGISTRY_UI: 'true'

    app:
      image:
        repository: registry.yourdomain.com/local
        tag: 0.59.2

    console:
      image:
        repository: registry.yourdomain.com/console
        tag: 2.12.2

    api:
      image:
        repository: registry.yourdomain.com/megabinary
        tag: 0.59.2

    executor:
      image:
        repository: registry.yourdomain.com/megabinary
        tag: 0.59.2

    glue:
      image:
        repository: registry.yourdomain.com/megabinary
        tag: 0.59.2

    parquet:
      image:
        repository: registry.yourdomain.com/megabinary
        tag: 0.59.2

    weave:
      image:
        repository: registry.yourdomain.com/weave-python
        tag: 0.59.2

    otel:
      image:
        repository: registry.yourdomain.com/otel/opentelemetry-collector-contrib
        tag: 0.97.0

    prometheus:
      server:
        image:
          repository: registry.yourdomain.com/prometheus/prometheus
          tag: v2.47.0
      configmapReload:
        prometheus:
          image:
            repository: registry.yourdomain.com/prometheus-operator/prometheus-config-reloader
            tag: v0.67.0

    ingress:
      annotations:
        nginx.ingress.kubernetes.io/proxy-body-size: 0
      class: nginx

    

To deploy the W&B platform, the Kubernetes Operator uses the values from your CR to configure the operator-wandb Helm chart from your internal repository.

Replace all tags and versions with the versions available in your internal registry. The example above shows the most commonly used components. Depending on your deployment needs, you may also need to configure image repositories for additional components such as settingsMigrationJob, weave-trace, filestream, and others. Refer to the W&B Helm repository values file for the complete list of configurable components.

Step 8: Deploy the W&B platform

Now that the Kubernetes operator and the CR are configured, apply the wandb.yaml configuration to deploy the W&B platform:

kubectl apply -f wandb.yaml

Step 9: Verify your installation

To verify the installation, W&B recommends using the W&B CLI. The verify command executes several tests that verify all components and configurations.

This step assumes that the first admin user account is created with the browser.

Follow these steps to verify the installation:

1. Install the W&B CLI:

pip install wandb

2. Log in to W&B:

wandb login --host=https://YOUR_DNS_DOMAIN

For example:

wandb login --host=https://wandb.company-name.com

3. Verify the installation:

wandb verify

A successful installation and fully working W&B deployment shows the following output:

Default host selected:  https://wandb.company-name.com
Find detailed logs for this test at: /var/folders/pn/b3g3gnc11_sbsykqkm3tx5rh0000gp/T/tmpdtdjbxua/wandb
Checking if logged in...................................................✅
Checking signed URL upload..............................................✅
Checking ability to send large payloads through proxy...................✅
Checking requests to base url...........................................✅
Checking requests made over signed URLs.................................✅
Checking CORs configuration of the bucket...............................✅
Checking wandb package version is up to date............................✅
Checking logged metrics, saving and downloading a file..................✅
Checking artifact save and download workflows...........................✅

Contact W&B Support if you encounter errors.

FAQ

Refer to the below frequently asked questions (FAQs) and troubleshooting tips during the deployment process:

There is another ingress class. Can that class be used?

Yes, you can configure your ingress class by modifying the ingress settings in values.yaml.

The certificate bundle has more than one certificate. Would that work?

You must split the certificates into multiple entries in the customCACerts section of values.yaml.

How do you prevent the Kubernetes operator from applying unattended updates. Is that possible?

You can turn off auto-updates from the W&B console. Reach out to your W&B team for any questions on the supported versions. W&B supports a major W&B Server release for 12 months from its initial release date. Customers with Self-Managed instances are responsible for upgrading in time to maintain support. Avoid staying on an unsupported version. Refer to Release policies and processes.

W&B strongly recommends customers with Self-Managed instances to update their deployments with the latest release at minimum once per quarter to maintain support and receive the latest features, performance improvements, and fixes.

Does the deployment work if the environment has no connection to public repositories?

If your configuration sets airgapped to true, the Kubernetes operator uses only your internal resources and does not attempt to connect to public repositories.