OpenBao Operations Runbook

Day-to-day operations and maintenance procedures for OpenBao.

Quick Reference

Essential Commands

# Set up root token
ROOT_TOKEN=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Root Token" --reveal)

# Check status
ssh ravenhelm@100.115.101.81 "docker exec openbao bao status"

# View logs
ssh ravenhelm@100.115.101.81 "docker logs -f openbao"

# Restart service
ssh ravenhelm@100.115.101.81 "cd ~/ravenhelm/services/openbao && docker compose restart"

Important URLs

Resource	URL
OpenBao UI	https://vault.ravenhelm.dev
Health Check	https://vault.ravenhelm.dev/v1/sys/health

1Password Items

Item	Contents
`OpenBao Root Keys`	Root token, recovery keys
`OpenBao OIDC - Zitadel`	OIDC client credentials
`OpenBao AppRole - *`	Service AppRole credentials
`GCP Service Account - ravenmask`	GCP KMS credentials

Daily Operations

Check OpenBao Health

# Quick health check
curl -s https://vault.ravenhelm.dev/v1/sys/health | jq .

# Full status
ssh ravenhelm@100.115.101.81 "docker exec openbao bao status"

Expected healthy response:

{
  "initialized": true,
  "sealed": false,
  "standby": false,
  "performance_standby": false,
  "version": "2.4.4"
}

View Recent Activity

ssh ravenhelm@100.115.101.81 "docker logs openbao --since 1h 2>&1 | tail -50"

Secret Management

List All Secrets

ROOT_TOKEN=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Root Token" --reveal)

# List top-level paths
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv list secret/"

# List specific path
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv list secret/api-keys/"

Read a Secret

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv get secret/api-keys/anthropic"

Create/Update a Secret

# Single field
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv put secret/api-keys/newservice api_key=xxx"

# Multiple fields
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv put secret/integrations/myservice \
  client_id=xxx \
  client_secret=yyy"

Delete a Secret

# Soft delete (can be recovered)
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv delete secret/api-keys/oldservice"

# Permanent delete (all versions)
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv metadata delete secret/api-keys/oldservice"

View Secret Versions

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao kv metadata get secret/api-keys/anthropic"

Policy Management

List Policies

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao policy list"

Read a Policy

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao policy read norns"

Create/Update a Policy

# Create policy file
cat > /tmp/newpolicy.hcl << 'EOF'
path "secret/data/mypath/*" {
  capabilities = ["read"]
}
EOF

# Upload and apply
scp /tmp/newpolicy.hcl ravenhelm@100.115.101.81:/tmp/
ssh ravenhelm@100.115.101.81 "docker cp /tmp/newpolicy.hcl openbao:/tmp/"
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao policy write newpolicy /tmp/newpolicy.hcl"

Delete a Policy

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao policy delete oldpolicy"

AppRole Management

List AppRoles

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao list auth/approle/role"

View AppRole Details

ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao read auth/approle/role/norns"

Create New AppRole

# 1. Create policy first (see above)

# 2. Create role
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao write auth/approle/role/newservice \
  token_policies=newservice \
  token_ttl=1h \
  token_max_ttl=4h"

# 3. Get role_id
ROLE_ID=$(ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao read -field=role_id auth/approle/role/newservice/role-id")

# 4. Generate secret_id
SECRET_ID=$(ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao write -field=secret_id -f auth/approle/role/newservice/secret-id")

# 5. Store in 1Password
op item create --vault ravenmask --category "API Credential" --title "OpenBao AppRole - NewService" \
  "role_id=$ROLE_ID" \
  "secret_id[password]=$SECRET_ID" \
  "vault_addr=https://vault.ravenhelm.dev"

Rotate Secret ID

# Generate new secret
NEW_SECRET=$(ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao write -field=secret_id -f auth/approle/role/norns/secret-id")

# Update 1Password
op item edit "OpenBao AppRole - Norns" --vault ravenmask "secret_id=$NEW_SECRET"

# Update service .env and restart

Backup & Restore

Create Backup

ROOT_TOKEN=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Root Token" --reveal)

# Create snapshot
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao operator raft snapshot save /openbao/data/backup.snap"

# Copy to backup location
ssh ravenhelm@100.115.101.81 "docker cp openbao:/openbao/data/backup.snap ~/ravenhelm/backups/openbao-$(date +%Y%m%d).snap"

# Verify backup
ssh ravenhelm@100.115.101.81 "ls -la ~/ravenhelm/backups/openbao-*.snap"

Restore from Backup

# Copy snapshot to container
ssh ravenhelm@100.115.101.81 "docker cp ~/ravenhelm/backups/openbao-YYYYMMDD.snap openbao:/openbao/data/restore.snap"

# Restore (WARNING: overwrites current state)
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao operator raft snapshot restore /openbao/data/restore.snap"

Troubleshooting

OpenBao is Sealed

With auto-unseal, this should not happen. If it does:

Check GCP KMS connectivity
Check GCP credentials file
Review logs for errors
If GCP KMS unavailable, use recovery keys

# Check seal status
ssh ravenhelm@100.115.101.81 "docker exec openbao bao status"

# If sealed and GCP KMS unavailable, use recovery keys
KEY1=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Unseal Key 1" --reveal)
KEY2=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Unseal Key 2" --reveal)
KEY3=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Unseal Key 3" --reveal)

ssh ravenhelm@100.115.101.81 "docker exec openbao bao operator unseal -recovery $KEY1"
ssh ravenhelm@100.115.101.81 "docker exec openbao bao operator unseal -recovery $KEY2"
ssh ravenhelm@100.115.101.81 "docker exec openbao bao operator unseal -recovery $KEY3"

Container Not Starting

# Check container status
ssh ravenhelm@100.115.101.81 "docker ps -a | grep openbao"

# View logs
ssh ravenhelm@100.115.101.81 "docker logs openbao 2>&1 | tail -50"

# Common issues:
# - GCP credentials missing/invalid
# - Config syntax error
# - Port conflict

Lost Root Token

Generate a new root token using recovery keys:

# Start generation
ssh ravenhelm@100.115.101.81 "docker exec openbao bao operator generate-root -init"

# Get OTP from output, then provide recovery keys
KEY1=$(op item get "OpenBao Root Keys" --vault ravenmask --fields "Unseal Key 1" --reveal)
ssh ravenhelm@100.115.101.81 "docker exec openbao bao operator generate-root -nonce=<NONCE> $KEY1"
# Repeat for KEY2, KEY3

# Decode the encoded token with the OTP

Service Can't Authenticate

# Test AppRole login manually
ROLE_ID=$(op item get "OpenBao AppRole - Norns" --vault ravenmask --fields role_id --reveal)
SECRET_ID=$(op item get "OpenBao AppRole - Norns" --vault ravenmask --fields secret_id --reveal)

curl -X POST https://vault.ravenhelm.dev/v1/auth/approle/login \
  -d "{\"role_id\":\"$ROLE_ID\",\"secret_id\":\"$SECRET_ID\"}"

Permission Denied

# Check token's policies
curl -s -H "X-Vault-Token: $TOKEN" \
  https://vault.ravenhelm.dev/v1/auth/token/lookup-self | jq '.data.policies'

# Check if path is covered by policy
ssh ravenhelm@100.115.101.81 "docker exec -e BAO_TOKEN=$ROOT_TOKEN openbao bao policy read <policy-name>"

Maintenance Tasks

Weekly

Verify backup exists and is recent
Check logs for errors
Verify services are authenticating successfully

Monthly

Review access policies
Rotate AppRole secret IDs
Update any expiring credentials
Test backup restore procedure (in staging)

Quarterly

Audit who has root token access
Review and clean up unused secrets
Update OpenBao version if available

Emergency Contacts

Role	Contact
Primary Admin	Nate Walker
1Password Vault	`ravenmask`

OpenBao - Main documentation
OpenBao Auto-Unseal - Auto-unseal setup
OpenBao AppRole - Service authentication

Quick Reference​

Essential Commands​

Important URLs​

1Password Items​

Daily Operations​

Check OpenBao Health​

View Recent Activity​

Secret Management​

List All Secrets​

Read a Secret​

Create/Update a Secret​

Delete a Secret​

View Secret Versions​

Policy Management​

List Policies​

Read a Policy​

Create/Update a Policy​

Delete a Policy​

AppRole Management​

List AppRoles​

View AppRole Details​

Create New AppRole​

Rotate Secret ID​

Backup & Restore​

Create Backup​

Restore from Backup​

Troubleshooting​

OpenBao is Sealed​

Container Not Starting​

Lost Root Token​

Service Can't Authenticate​

Permission Denied​

Maintenance Tasks​

Weekly​

Monthly​

Quarterly​

Emergency Contacts​

Related Documentation​

Quick Reference

Essential Commands

Important URLs

1Password Items

Daily Operations

Check OpenBao Health

View Recent Activity

Secret Management

List All Secrets

Read a Secret

Create/Update a Secret

Delete a Secret

View Secret Versions

Policy Management

List Policies

Read a Policy

Create/Update a Policy

Delete a Policy

AppRole Management

List AppRoles

View AppRole Details

Create New AppRole

Rotate Secret ID

Backup & Restore

Create Backup

Restore from Backup

Troubleshooting

OpenBao is Sealed

Container Not Starting

Lost Root Token

Service Can't Authenticate

Permission Denied

Maintenance Tasks

Weekly

Monthly

Quarterly

Emergency Contacts

Related Documentation