Skip to main content

Runbook: Docker Proxy Network Access (403 Forbidden)

Purpose​

Resolve 403 Forbidden errors when Vidar/Bifrost attempts to access Docker network information through the docker-socket-proxy.

Symptoms​

  • Vidar logs show: Client error '403 Forbidden' for url 'http://dockerproxy:2375/networks'
  • Docker network relationships not being discovered
  • Entity network topology incomplete in CMDB

Prerequisites​

  • SSH access to odin as ravenhelm
  • Access to modify homepage docker-compose.yml (where dockerproxy is defined)

Diagnosis​

Step 1: Confirm the 403 Error​

# Check vidar-api logs for network access errors
ssh ravenhelm@100.115.101.81 "docker logs vidar-api 2>&1 | grep -i 'networks.*403\|403.*networks' | tail -5"

Example output:

bifrost.services.discovery.docker - WARNING - Failed to discover network relationships: Client error '403 Forbidden' for url 'http://dockerproxy:2375/networks'

Step 2: Check Docker Proxy Configuration​

# Check current dockerproxy environment variables
ssh ravenhelm@100.115.101.81 "docker inspect dockerproxy --format '{{json .Config.Env}}' | jq '.'"

Look for NETWORKS=0 or missing NETWORKS setting.

Step 3: Verify Which Endpoints Are Enabled​

The docker-socket-proxy uses environment variables to control access:

  • CONTAINERS=1 - Allow container listing
  • NETWORKS=1 - Allow network listing
  • SERVICES=1 - Allow service listing
  • etc.

Procedure​

Step 1: Locate dockerproxy Configuration​

# Find the docker-compose.yml containing dockerproxy
ssh ravenhelm@100.115.101.81 "grep -r 'dockerproxy' /Users/ravenhelm/ravenhelm --include='*.yml' | head -5"

Typically located at: /Users/ravenhelm/ravenhelm/services/homepage/docker-compose.yml

Step 2: Add NETWORKS=1 Permission​

Edit the docker-compose.yml to add the NETWORKS environment variable:

# View current configuration
ssh ravenhelm@100.115.101.81 "grep -A10 'dockerproxy:' /Users/ravenhelm/ravenhelm/services/homepage/docker-compose.yml"

Before:

dockerproxy:
  image: ghcr.io/tecnativa/docker-socket-proxy:latest
  environment:
    - CONTAINERS=1
    - SERVICES=1
    - TASKS=1
    - POST=1

After:

dockerproxy:
  image: ghcr.io/tecnativa/docker-socket-proxy:latest
  environment:
    - CONTAINERS=1
    - SERVICES=1
    - TASKS=1
    - POST=1
    - NETWORKS=1   # Added for Vidar network discovery

Add the setting:

ssh ravenhelm@100.115.101.81 'python3 << "EOF"
with open("/Users/ravenhelm/ravenhelm/services/homepage/docker-compose.yml", "r") as f:
    content = f.read()

if "NETWORKS=1" not in content:
    content = content.replace(
        "      - POST=1\n    volumes:",
        "      - POST=1\n      - NETWORKS=1\n    volumes:"
    )
    with open("/Users/ravenhelm/ravenhelm/services/homepage/docker-compose.yml", "w") as f:
        f.write(content)
    print("Added NETWORKS=1")
else:
    print("NETWORKS=1 already configured")
EOF'

Step 3: Restart dockerproxy​

ssh ravenhelm@100.115.101.81 "cd /Users/ravenhelm/ravenhelm/services/homepage && docker compose up -d dockerproxy"

Step 4: Verification​

# Test network endpoint directly
ssh ravenhelm@100.115.101.81 "curl -s http://localhost:2375/networks | jq '.[0].Name'"

# Wait for next discovery run
sleep 30

# Check vidar logs for successful network access
ssh ravenhelm@100.115.101.81 "docker logs vidar-api --tail 20 2>&1 | grep networks"

Success criteria:

  • Network endpoint returns 200 OK with network data
  • No more 403 errors in vidar logs
  • Docker discovery completes without warnings

Available Docker Proxy Permissions​

VariablePurposeDefault
CONTAINERSList/inspect containers0
NETWORKSList/inspect networks0
SERVICESList services (Swarm)0
TASKSList tasks (Swarm)0
IMAGESList/pull images0
VOLUMESList volumes0
POSTAllow POST requests0
EVENTSStream Docker events0

Security Considerations​

The docker-socket-proxy is a security layer between applications and the Docker socket. Only enable permissions that are actually needed:

  • NETWORKS=1 is read-only and safe for network topology discovery
  • Avoid enabling dangerous permissions like ALLOW_RESTARTS, ALLOW_STOP, or BUILD
  • The proxy prevents unauthorized container manipulation

Rollback​

# Remove NETWORKS=1 if causing issues
ssh ravenhelm@100.115.101.81 "sed -i '' '/NETWORKS=1/d' /Users/ravenhelm/ravenhelm/services/homepage/docker-compose.yml"

# Restart proxy
ssh ravenhelm@100.115.101.81 "cd /Users/ravenhelm/ravenhelm/services/homepage && docker compose up -d dockerproxy"

Escalation​

If network discovery still fails after enabling NETWORKS:

  1. Check if dockerproxy container is healthy
  2. Verify network connectivity between vidar-api and dockerproxy
  3. Check Docker daemon logs for permission issues
  4. Contact: Platform team