Search Service¶

Carbon Connect uses Meilisearch as its primary full-text search engine with an automatic PostgreSQL fallback for resilience.

Source: backend/app/services/meilisearch_service.py

Architecture¶

flowchart LR
    A[API Request] --> B{Meilisearch<br/>Available?}
    B -->|Yes| C[Meilisearch<br/>sub-100ms]
    B -->|No| D[PostgreSQL ILIKE<br/>Fallback]
    C --> E[Search Results]
    D --> E

The search_grants_with_fallback() function transparently tries Meilisearch first and falls back to PostgreSQL if Meilisearch is unavailable or returns an error.

Meilisearch Integration¶

Index Configuration¶

The service manages a grants index with the following configuration:

Searchable Attributes¶

Fields that Meilisearch indexes for full-text search, listed in priority order:

1. title
2. description
3. summary
4. thematic_areas
5. carbon_categories

Filterable Attributes¶

Fields that support faceted filtering:

• countries
• nace_codes
• company_sizes
• status
• source
• is_carbon_focused
• carbon_categories
• is_active
• grant_type
• eu_taxonomy_aligned

Sortable Attributes¶

Fields that support sorting:

• deadline
• funding_amount_min
• funding_amount_max
• created_at
• updated_at

Index Management¶

from backend.app.services.meilisearch_service import MeilisearchService

service = MeilisearchService()

# Create or update the grants index
await service.setup_grants_index()

# Index a batch of grants
await service.index_grants(grants)

# Delete a grant from the index
await service.delete_grant(grant_id)

# Full reindex
await service.reindex_all_grants(db_session)

Search API¶

The search function supports filtering, pagination, and sorting:

results = await service.search_grants(
    query="renewable energy",
    filters={
        "countries": ["DE", "FR"],
        "is_carbon_focused": True,
        "status": "active",
    },
    sort=["deadline:asc"],
    page=1,
    per_page=20,
)

Fallback Strategy¶

search_grants_with_fallback()¶

This is the primary search function used by the API. It provides automatic degradation:

async def search_grants_with_fallback(
    query: str,
    db: AsyncSession,
    filters: dict | None = None,
    sort_by: str = "deadline",
    page: int = 1,
    per_page: int = 20,
) -> SearchResult:
    """
    Search grants with Meilisearch, falling back to PostgreSQL.

    1. Try Meilisearch for sub-100ms full-text search
    2. If Meilisearch fails, fall back to PostgreSQL ILIKE
    3. Return normalized results from either source
    """

When Fallback Activates¶

The PostgreSQL fallback is used when:

• Meilisearch service is not configured (no MEILI_HOST set)
• Meilisearch is unreachable (connection error)
• Meilisearch returns an error (index not found, invalid filter)
• Request timeout (configurable, default 5 seconds)

PostgreSQL Fallback Implementation¶

The fallback uses ILIKE queries on title and description columns:

SELECT * FROM grants
WHERE (title ILIKE '%query%' OR description ILIKE '%query%')
  AND is_active = true
  AND country = ANY(:countries)
ORDER BY deadline ASC
LIMIT :per_page OFFSET :offset

API Endpoints¶

Index Maintenance¶

The Meilisearch index is maintained through scheduled tasks:

Manual Reindex¶

To manually trigger a full reindex:

# Via Celery task
poetry run celery -A backend.app.worker.celery_app call \
    backend.app.worker.tasks.maintenance_tasks.reindex_search

# Via Python
from backend.app.services.meilisearch_service import MeilisearchService
service = MeilisearchService()
await service.reindex_all_grants(db_session)

Configuration¶

Local Development¶

Meilisearch runs as a Docker container:

docker compose up -d meilisearch

# Verify
curl http://localhost:7700/health