Running GPU Workloads

VastAI

Currently the only supported cloud provider. More are to come.

Networking

Since VastAI does not allow us to run Wireguard in kernel space the VPN is done by Wireproxy in user space. This means all traffic to the cluster needs to go either over HTTP proxy or through tunnel.

Proxy is available at localhost:3128

TCP Tunnels can be set by setting ENV variables in format GW_TUNNEL_<port-number> = private address to tunnel to. Set these variables in pod’s definition.

Machine Selection

You can control machine selection by adding annotations to your Pod specification. All annotations use the prefix gpu-provider.glami.cz/.

Boolean Filters

Region Filters

List Filters

GPU Filters

All numeric filters support three variants:

• Exact value: <field-name>
• Minimum value: <field-name>-min
• Maximum value: <field-name>-max

Note: If an exact value is specified, min/max values for that field are ignored.

CPU and RAM Filters

Price Filters

Network Speed Filters

VastAI-Specific Filters

Example Pod Specification

Here’s a complete example showing how to use these annotations:

apiVersion: v1
kind: Pod
metadata:
  name: my-gpu-workload
  annotations:
    # Only verified datacenter machines in Europe
    gpu-provider.glami.cz/verified-only: "true"
    gpu-provider.glami.cz/datacenter-only: "true"
    gpu-provider.glami.cz/region: "europe"
    
    # GPU requirements: 2x RTX 4090 or RTX 3090
    gpu-provider.glami.cz/gpu-names: "RTX 4090,RTX 3090"
    gpu-provider.glami.cz/gpu-count: "2"
    gpu-provider.glami.cz/vram-min: "20480"  # At least 20GB per GPU
    
    # CPU and RAM requirements
    gpu-provider.glami.cz/cpu-min: "8"
    gpu-provider.glami.cz/ram-min: "32768"  # At least 32GB RAM
    
    # Price constraint
    gpu-provider.glami.cz/price-max: "1.50"  # Maximum $1.50 per hour
    
    # Network requirements
    gpu-provider.glami.cz/download-speed-min: "1000"  # At least 1Gbps
spec:
  containers:
  - name: training-container
    image: pytorch/pytorch:latest
    command: ["python", "train.py"]

Filter Behavior

• Exact vs Range: When you specify an exact value (e.g., gpu-count: "2"), the corresponding min/max filters are ignored
• Multiple Filters: All specified filters must be satisfied (AND logic)
• List Filters: For list filters like gpu-names, any value in the list is acceptable (OR logic)
• Default Ordering: Machines are ordered by price (ascending) by default
• Machine Bans: Machines that fail during startup are temporarily banned and excluded from future selections