Skip to content

Troubleshooting

Common issues and their solutions when working with Carbon Connect.

PostgreSQL Port Conflict

Problem

psycopg2.OperationalError: could not connect to server: Connection refused
Is the server running on host "localhost" and accepting TCP/IP connections on port 5432?

Cause

Windows may have a local PostgreSQL installation running on the default port 5432. Carbon Connect uses port 5433 to avoid this conflict.

Solution

Verify your .env file has:

POSTGRES_PORT=5433

Check the Docker container is running on the correct port:

docker compose ps
# Should show: 0.0.0.0:5433->5432/tcp

If a local PostgreSQL is blocking the port:

# Check what's using port 5432
netstat -an | findstr "5432"

# Stop the local PostgreSQL service (Windows)
net stop postgresql-x64-16

pgvector Migration Error

Problem

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedFunction)
function vector(...) does not exist

Cause

The pgvector extension is not installed in the database, or the migration file references pgvector.sqlalchemy.Vector instead of Vector.

Solution

  1. Ensure the pgvector extension is installed:
CREATE EXTENSION IF NOT EXISTS vector;
  1. In migration files, use the correct import:
# CORRECT
from pgvector.sqlalchemy import Vector
# Column definition:
sa.Column("embedding", Vector(768))

# WRONG
sa.Column("embedding", pgvector.sqlalchemy.Vector(dim=768))

SQLAlchemy "metadata is a reserved attribute" Error

Problem

sqlalchemy.exc.InvalidRequestError: Attribute name 'metadata' is reserved
when using the Declarative mapping API.

Cause

Using metadata as a column name in SQLAlchemy models conflicts with the built-in metadata attribute of the declarative base class.

Solution

Use extra_data instead of metadata for JSONB columns:

# CORRECT
extra_data = Column(JSONB, nullable=True)

# WRONG
metadata = Column(JSONB, nullable=True)

Docker Containers Not Starting

Problem

ERROR: for postgres  Cannot start service postgres: driver failed programming
external connectivity on endpoint granted_carbon_postgres

Solution

  1. Ensure Docker Desktop is running
  2. Check for port conflicts:
netstat -an | findstr "5433 6379 7700"
  1. Remove old volumes and restart:
docker compose down -v
docker compose up -d
  1. Check container logs:
docker compose logs -f postgres
docker compose logs -f valkey
docker compose logs -f meilisearch

AsyncMock vs MagicMock Errors

Problem

TypeError: object MagicMock can't be used in 'await' expression

Cause

Using MagicMock() for methods that are awaited (like aclose(), execute(), etc.).

Solution

Use AsyncMock() for any method that will be awaited:

# WRONG
mock_client = MagicMock()
mock_client.request = MagicMock(return_value=mock_response)

# CORRECT
mock_client = MagicMock()
mock_client.request = AsyncMock(return_value=mock_response)
mock_client.aclose = AsyncMock()  # Required for async context managers

Rule of thumb: If the source code has await method(), the mock must be AsyncMock().

bcrypt/passlib Compatibility

Problem

AttributeError: module 'bcrypt' has no attribute '__about__'

Cause

bcrypt version 5.x is incompatible with passlib version 1.7.4.

Solution

The project pins bcrypt to version 4.x in pyproject.toml:

[tool.poetry.dependencies]
bcrypt = "^4.3.0"

If you encounter this error, verify the correct version is installed:

poetry show bcrypt
# Should show version 4.x.x

MagicMock Truthy Trap in Tests

Problem

Tests pass when they should fail, or tests fail with unexpected behavior around None checks.

Cause

MagicMock attributes are always truthy, which can mask bugs:

mock = MagicMock()
if mock.prompt_feedback:  # Always True!
    raise ContentFilterError()  # Unexpected error

Solution

Always set explicit values for attributes that will be evaluated:

mock = MagicMock()
mock.prompt_feedback = None       # Explicitly None
mock.total_tokens = 500           # Explicit integer
mock.block_reason = None          # Explicitly None

See Testing Philosophy for detailed guidance.

Alembic Migration Conflicts

Problem

alembic.util.exc.CommandError: Can't locate revision identified by 'abc123'

Cause

Migration history has diverged, typically from a branch merge.

Solution

  1. Check current migration state:
poetry run alembic current
poetry run alembic history
  1. If heads have diverged, merge them:
poetry run alembic merge heads -m "merge migration heads"
  1. Apply the merged migration:
poetry run alembic upgrade head

Celery Worker Not Processing Tasks

Problem

Tasks are queued but never executed.

Solution

  1. Verify Valkey is running:
docker exec granted_carbon_valkey valkey-cli ping
# Should return: PONG
  1. Check the worker is listening on the correct queue:
# Start worker for all queues
poetry run celery -A backend.app.worker.celery_app worker -l info

# Or specific queue
poetry run celery -A backend.app.worker.celery_app worker -Q sync -l info
  1. Inspect queued tasks:
poetry run celery -A backend.app.worker.celery_app inspect active
poetry run celery -A backend.app.worker.celery_app inspect reserved

Meilisearch Index Not Found

Problem

meilisearch.errors.MeilisearchApiError: Index `grants` not found

Solution

  1. Verify Meilisearch is running:
curl http://localhost:7700/health
# Should return: {"status":"available"}
  1. Create the index:
from backend.app.services.meilisearch_service import MeilisearchService
service = MeilisearchService()
await service.setup_grants_index()
  1. The API uses search_grants_with_fallback() which automatically falls back to PostgreSQL when Meilisearch is unavailable.

Verification Commands

Quick commands to verify all services are operational:

# Check all services at once
poetry run python scripts/verify_services.py

# Check PostgreSQL
docker exec granted_carbon_postgres psql -U grant_user -d grant_engine -c "\dt"

# Check PostgreSQL connection
docker exec granted_carbon_postgres psql -U grant_user -d grant_engine -c "SELECT 1;"

# Check Valkey
docker exec granted_carbon_valkey valkey-cli ping

# Check Meilisearch
curl http://localhost:7700/health

# Check migration status
poetry run alembic current
poetry run alembic history

# Check running Docker containers
docker compose ps

# Check port usage
netstat -an | findstr "5433 6379 7700 8000 3000"