Why I Ditched Kubernetes KIND for Podman Quadlets (And You Should Too)

Last week, I migrated Colosseum from Kubernetes (KIND) to Podman Quadlets. The result? Simpler deployment, better localhost networking, and native Linux integration. Here’s why and how I did it.

The Problem with KIND

Kubernetes in Docker (KIND) is great for learning Kubernetes, but it has a notorious problem: localhost networking is broken.

My agents needed to:

• Connect to PostgreSQL on the host
• Access MCP servers running locally
• Call OpenAI APIs without complex networking

KIND makes this unnecessarily painful:

# What I wanted:
postgresql://localhost:5432/colosseum

# What KIND required:
postgresql://host.docker.internal:5432/colosseum
# ...which doesn't work reliably on Linux

# Or this mess:
postgresql://$(docker network inspect kind | jq -r '.[0].IPAM.Config[0].Gateway'):5432/colosseum

Worse, KIND adds Kubernetes complexity where I didn’t need it:

• Deployments, Services, Ingress for a single-node setup
• kubectl for managing what should be simple containers
• YAML sprawl for basic container operations

Enter Podman Quadlets

Podman Quadlets are systemd unit files for containers. They bring:

1. Native systemd integration - Manage containers like any other service
2. Simple networking - Bridge mode with proper DNS resolution
3. No daemon required - Rootless containers by default
4. Fedora-native - Perfect for my Fedora-based development

What Are Quadlets?

Quadlets extend systemd with container-specific units:

• .container - Container definitions (like docker run)
• .volume - Persistent storage
• .network - Container networking
• .kube - Kubernetes YAML support (if needed)

They live in:

• /etc/containers/systemd/ (system-wide)
• ~/.config/containers/systemd/ (user-specific)

systemd automatically converts them to service units at daemon-reload.

The Migration

Before: Kubernetes Manifests

My old KIND setup required multiple YAML files:

# deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: colosseum-agent
spec:
  replicas: 1
  selector:
    matchLabels:
      app: colosseum-agent
  template:
    metadata:
      labels:
        app: colosseum-agent
    spec:
      containers:
      - name: agent
        image: colosseum-agent:latest
        env:
        - name: DATABASE_URL
          value: postgresql://postgres:password@colosseum-db:5432/colosseum
---
# service.yaml
apiVersion: v1
kind: Service
metadata:
  name: colosseum-db
spec:
  ports:
  - port: 5432
  selector:
    app: postgres

Plus ConfigMaps, Secrets, and Ingress configurations.

After: Podman Quadlets

Here’s the complete Quadlet setup:

1. Network Configuration

colosseum-network.network:

[Network]
NetworkName=colosseum
Driver=bridge
Subnet=10.88.0.0/16
Gateway=10.88.0.1

2. PostgreSQL Container

colosseum-db.container:

[Unit]
Description=Colosseum PostgreSQL Database
After=network.target colosseum-network.service
Requires=colosseum-network.service

[Container]
Image=docker.io/library/postgres:16-alpine
ContainerName=colosseum-db
Network=colosseum-network.network

# Persistent storage
Volume=colosseum-db-data.volume:/var/lib/postgresql/data:Z

# Environment
Environment=POSTGRES_DB=colosseum
Environment=POSTGRES_USER=colosseum
Environment=POSTGRES_PASSWORD=colosseum_dev

# Port mapping (optional - mainly for dev access)
PublishPort=5432:5432

# Health check
HealthCmd=/usr/local/bin/pg_isready -U colosseum -d colosseum
HealthInterval=10s
HealthRetries=5
HealthStartPeriod=30s

[Service]
Restart=always
TimeoutStartSec=300

[Install]
WantedBy=default.target

3. Database Volume

colosseum-db-data.volume:

[Volume]
VolumeName=colosseum-db-data

4. Agent Container

colosseum-agent.container:

[Unit]
Description=Colosseum Multi-Agent Service
After=colosseum-db.service
Requires=colosseum-db.service colosseum-network.service

[Container]
Image=docker.io/library/fedora:latest
ContainerName=colosseum-agent
Network=colosseum-network.network

# Application state
Volume=colosseum-state.volume:/var/lib/colosseum:Z

# Configuration
Volume=/etc/colosseum:/etc/colosseum:ro,Z

# Environment
Environment=DATABASE_URL=postgresql://colosseum:colosseum_dev@colosseum-db:5432/colosseum
Environment=OPENAI_API_KEY_FILE=/etc/colosseum/openai_api_key

# Port for future API
PublishPort=8000:8000

# Keep running with Python command
Exec=/usr/bin/python3 -m colosseum.agent_supervisor

[Service]
Restart=always
TimeoutStartSec=60

[Install]
WantedBy=default.target

Deployment Automation

I wrote quadlet_deploy.py to manage the entire lifecycle:

class QuadletDeployment:
    def __init__(self, user_mode: bool = False):
        self.user_mode = user_mode
        self.quadlet_dir = (
            Path.home() / '.config/containers/systemd'
            if user_mode else
            Path('/etc/containers/systemd')
        )

    def install(self):
        """Install quadlet files and reload systemd"""
        self.quadlet_dir.mkdir(parents=True, exist_ok=True)

        # Copy .container, .network, .volume files
        for quadlet in QUADLET_FILES:
            shutil.copy(quadlet, self.quadlet_dir)

        # Reload systemd to process quadlets
        self._systemctl('daemon-reload')

    def enable_services(self):
        """Enable all colosseum services"""
        services = [
            'colosseum-network',
            'colosseum-db',
            'colosseum-agent',
        ]

        for service in services:
            self._systemctl('enable', f'{service}.service')

    def start(self):
        """Start all services in order"""
        self._systemctl('start', 'colosseum-network')
        self._systemctl('start', 'colosseum-db')

        # Wait for DB to be healthy
        self._wait_for_db()

        self._systemctl('start', 'colosseum-agent')

    def logs(self, service: str, follow: bool = False):
        """View service logs via journalctl"""
        cmd = ['journalctl', '-u', f'{service}.service']
        if follow:
            cmd.append('-f')
        subprocess.run(cmd)

Usage is simple:

# Install and start everything
python3 quadlet_deploy.py install
python3 quadlet_deploy.py enable
python3 quadlet_deploy.py start

# View logs
python3 quadlet_deploy.py logs colosseum-agent --follow

# Access database
python3 quadlet_deploy.py db-shell

# Get container shell
python3 quadlet_deploy.py shell colosseum-agent

Networking That Just Works

The biggest win? Networking actually works:

# In the agent container, this just works:
DATABASE_URL = "postgresql://colosseum:password@colosseum-db:5432/colosseum"

# DNS resolution via the colosseum-network bridge:
# colosseum-db -> 10.88.0.2
# colosseum-agent -> 10.88.0.3

No special Docker Desktop tricks. No host.docker.internal. Just normal networking.

systemd Integration Benefits

Managing Colosseum is now identical to any other Linux service:

# Standard systemd commands
systemctl status colosseum-agent
systemctl restart colosseum-db
systemctl stop colosseum-agent

# View logs
journalctl -u colosseum-agent -f
journalctl -u colosseum-db --since "1 hour ago"

# Boot persistence
systemctl enable colosseum-agent
systemctl disable colosseum-db

System administrators already know these commands. No new tools to learn.

Resource Usage Comparison

Metric	KIND	Podman Quadlets
Memory (idle)	~2.5 GB	~500 MB
Startup time	45s	8s
Components	Docker + KIND + kubectl + containerd	Podman + systemd
Config files	8 YAML files	5 quadlet files

Gotchas and Lessons Learned

1. SELinux Labels

Always use :Z flag for volumes:

Volume=colosseum-db-data.volume:/var/lib/postgresql/data:Z

Without :Z, containers can’t write to volumes on SELinux-enabled systems (like Fedora).

2. Service Dependencies

Use After= and Requires=:

[Unit]
After=colosseum-db.service
Requires=colosseum-db.service

This ensures services start in the correct order.

3. Health Checks

PostgreSQL needs time to initialize:

HealthStartPeriod=30s
HealthInterval=10s
HealthRetries=5

Start-period gives the initial schema setup time to complete.

4. User vs System Mode

User mode (~/.config/containers/systemd/):

• No root required
• Survives user logout if linger is enabled: loginctl enable-linger
• Perfect for development

System mode (/etc/containers/systemd/):

• Requires root
• Runs at boot
• Better for production

When to Use Quadlets vs Kubernetes

Use Quadlets when:

• Single-node deployment (dev, small prod)
• You need simple container orchestration
• Native Linux integration matters
• You’re already using systemd

Use Kubernetes when:

• Multi-node clusters
• Complex scaling requirements
• Need Kubernetes-specific features (operators, CRDs)
• Industry-standard orchestration is required

The Results

After migration:

• ✅ Localhost networking works perfectly
• ✅ Deployment complexity reduced by 60%
• ✅ Memory usage down from 2.5GB to 500MB
• ✅ Startup time from 45s to 8s
• ✅ Native systemd integration
• ✅ Simpler debugging (journalctl, systemctl)

For single-node AI agent deployments, Podman Quadlets are a perfect fit.

---

Running into KIND networking issues? Consider Podman Quadlets. Questions? Find me on GitHub.