Skip to main content

IMAP Indexers

EmailEngine uses an indexing strategy to track email changes in IMAP accounts. The indexer determines how EmailEngine detects new, updated, and deleted messages, affecting both webhook notifications and Redis storage usage.

IMAP Only

Indexers only apply to IMAP accounts. Gmail API and MS Graph accounts use their own change detection mechanisms provided by the respective APIs.

Indexing Strategies​

Full Indexer (Default)​

The full indexer maintains a complete reference of all emails in each mailbox, stored in Redis.

How it works:

  • Stores a reference (UID and flags) for every message in each folder
  • Compares current mailbox state against stored references
  • Detects new messages, deleted messages, and flag changes

Capabilities:

EventDetected
New messagesYes
Deleted messagesYes
Flag changes (read/unread, starred, etc.)Yes
Message moves between foldersYes (as delete + new)

Trade-offs:

  • Higher Redis memory usage (stores reference for every message)
  • Slower initial sync (must index all messages)
  • Complete change detection

Best for:

  • Applications that need to track all email changes
  • CRM integrations requiring deletion and flag sync
  • Full mailbox synchronization use cases

Fast Indexer​

The fast indexer only tracks the highest UID (unique identifier) seen in each mailbox.

How it works:

  • Stores only the last known UID for each folder
  • Detects new messages by checking for UIDs higher than stored value
  • Does not track individual message states

Capabilities:

EventDetected
New messagesYes
Deleted messagesNo
Flag changes (read/unread, starred, etc.)No
Message moves between foldersPartial (new in destination only)

Trade-offs:

  • Minimal Redis memory usage
  • Faster initial sync
  • Limited change detection

Best for:

  • Processing pipelines that only need new emails
  • High-volume accounts where storage is a concern
  • Feed-forward processing (AI analysis, archival)

Comparison​

FeatureFull IndexerFast Indexer
messageNew webhooksYesYes
messageDeleted webhooksYesNo
messageUpdated webhooksYesNo
Redis storageHigh (all messages)Low (one UID per folder)
Initial sync speedSlowerFaster
Ongoing sync speedSimilarSimilar

Configuration​

Global Default​

Set the default indexer for all new IMAP accounts via API:

curl -X POST https://your-ee.com/v1/settings \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "imapIndexer": "full"
  }'

Or via the web interface: Configuration > General Settings > IMAP Processing > Indexing Method

Per-Account Setting​

Set the indexer when creating an account:

curl -X POST https://your-ee.com/v1/account \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "account": "user123",
    "imap": {
      "host": "imap.example.com",
      "port": 993,
      "secure": true,
      "auth": {
        "user": "user@example.com",
        "pass": "password"
      }
    },
    "imapIndexer": "fast"
  }'

Changing Indexer for Existing Account​

Use the flush API to change the indexer for an existing account:

curl -X PUT https://your-ee.com/v1/account/user123/flush \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "flush": true,
    "imapIndexer": "full"
  }'

This resets the account index and rebuilds it using the new indexer strategy.

Index Reset

Changing the indexer flushes the existing index. During re-indexing:

  • The account temporarily shows as syncing
  • messageNew webhooks may be sent for existing messages (depending on notifyFrom)
  • Normal operations resume after sync completes

Use Cases​

Email Processing Pipeline​

For pipelines that only process incoming emails (AI analysis, vector embeddings, archival):

# Use fast indexer - only need new messages
curl -X POST https://your-ee.com/v1/account \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "account": "pipeline-account",
    "imapIndexer": "fast",
    "notifyFrom": "2024-01-01T00:00:00.000Z",
    "imap": { ... }
  }'

CRM Integration​

For CRM systems that need complete email sync including deletions:

# Use full indexer - need all changes
curl -X POST https://your-ee.com/v1/account \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "account": "crm-account",
    "imapIndexer": "full",
    "imap": { ... }
  }'

Processing Existing Emails​

To trigger webhooks for existing emails (e.g., initial data import):

# Flush with notifyFrom in the past
curl -X PUT https://your-ee.com/v1/account/user123/flush \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "flush": true,
    "notifyFrom": "1970-01-01T00:00:00.000Z"
  }'

This triggers messageNew webhooks for all existing emails in the account.

Monitoring​

Check the current indexer setting for an account:

curl https://your-ee.com/v1/account/user123 \
  -H "Authorization: Bearer YOUR_TOKEN"

The response includes the imapIndexer field showing the active strategy.

See Also​