Skip to main content

Outbox Queue

EmailEngine uses queues to process background tasks including email sending. Understanding the queue system helps you monitor delivery, troubleshoot issues, and optimize performance.

Why Queues Matter​

When you submit an email, EmailEngine doesn't send it immediately. Instead, it:

  1. Validates your request
  2. Adds the message to a queue
  3. Returns immediately with a queue ID
  4. Processes the queue asynchronously
  5. Handles retries automatically
  6. Notifies you via webhooks

This approach provides:

  • Reliability: Automatic retries on failures
  • Scalability: Handle high-volume sending
  • Monitoring: Track delivery status
  • Resilience: Survive crashes and restarts

Queue Technology​

EmailEngine uses BullMQ for queue management, backed by Redis. BullMQ provides:

  • Persistent job storage
  • Automatic retry logic
  • Priority queues
  • Delayed jobs (scheduled sending)
  • Job progress tracking

Queue Types​

EmailEngine maintains three queue types:

1. Submit Queue​

Handles all email sending jobs.

  • Purpose: Process outbound email
  • Jobs: Individual send requests
  • Lifecycle: Waiting → Active → Completed/Failed/Delayed

2. Notify Queue​

Handles all webhook delivery jobs.

  • Purpose: Send webhook notifications
  • Jobs: Webhook HTTP requests
  • Retries: Automatic retry on webhook failures

3. Documents Queue​

Handles document indexing jobs.

  • Purpose: Index emails for search functionality
  • Jobs: Document indexing tasks
  • Used when: Document Store/Elasticsearch integration is enabled

Job Lifecycle​

Jobs in the submit queue move through different states:

1. Waiting​

Description: Jobs ready to be processed immediately.

How jobs get here:

  • New submissions without sendAt property
  • Delayed jobs whose sendAt time has been reached
  • Jobs moved from Paused when queue is unpaused

What happens: Jobs are picked up one by one and moved to Active.

# View waiting jobs
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

2. Active​

Description: Jobs currently being processed.

What happens:

  • EmailEngine connects to SMTP server
  • Transmits the message
  • Waits for SMTP response

Outcomes:

  • Success → Moved to Completed
  • Temporary failure → Moved to Delayed (will retry)
  • Permanent failure (retries exhausted) → Moved to Failed
# View active jobs (these are currently being processed)
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

3. Completed​

Description: Successfully delivered jobs.

What happens:

  • SMTP server accepted the message (250 OK)
  • messageSent webhook is emitted
  • Job stored for informational purposes

Note: By default, completed jobs are not stored. To enable storage, configure "How many completed/failed queue entries to keep" in Configuration → Service.

# View completed jobs (if enabled)
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

4. Failed​

Description: Jobs that failed too many times and won't be retried.

How jobs get here:

  • Retried deliveryAttempts times (default: 10)
  • All attempts failed

What happens:

  • messageFailed webhook is emitted
  • Job stored for debugging

Common failure reasons:

  • Invalid credentials
  • Network errors (persistent)
  • Recipient address rejected
  • Message rejected by spam filter
# View failed jobs
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

5. Delayed​

Description: Jobs waiting for future processing.

How jobs get here:

  • New submissions with sendAt property (scheduled sending)
  • Failed jobs that will be retried (retry delay calculated)

What happens:

  • Job waits until the delay time
  • Then moved to Waiting
  • If failure: messageDeliveryError webhook is emitted

Retry schedule (exponential backoff with 5 second base delay):

  • Attempt 1: Immediate
  • Attempt 2: +10 seconds (2^1 x 5s)
  • Attempt 3: +20 seconds (2^2 x 5s)
  • Attempt 4: +40 seconds (2^3 x 5s)
  • Attempt 5: +80 seconds (2^4 x 5s)
  • Attempt 6: +160 seconds (2^5 x 5s)
  • And so on, doubling each time
# View delayed jobs
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

6. Paused​

Description: Jobs held when queue is paused.

How to pause: Use the Bull Board UI or API to pause the queue.

What happens:

  • New jobs go to Paused instead of Waiting
  • Active jobs finish processing
  • When unpaused, jobs move to Waiting

Use cases:

  • Maintenance windows
  • Debugging issues
  • Rate limit management
# Pause queue
curl -XPUT "https://ee.example.com/v1/settings/queue/submit" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"paused": true}'

# Resume queue
curl -XPUT "https://ee.example.com/v1/settings/queue/submit" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"paused": false}'

Monitoring the Queue​

Bull Board UI​

EmailEngine includes Bull Board, a web UI for BullMQ queues.

Access: Navigate to Tools → Bull Board in EmailEngine UI, or go directly to /admin/bull-board.

Features:

  • View job counts by state
  • Inspect individual jobs
  • Retry failed jobs
  • Delete jobs
  • Pause/resume queues
  • View job logs

API Access​

Query queue status via the outbox API:

# Get queue summary
curl "https://ee.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

Response:

{
  "account": "example",
  "queued": 5,
  "states": {
    "waiting": 3,
    "active": 1,
    "delayed": 1,
    "completed": 0,
    "failed": 0
  }
}

List Jobs by State​

# List waiting jobs
curl "https://ee.example.com/v1/outbox?pageSize=10" \
  -H "Authorization: Bearer <token>"

Response:

{
  "jobs": [
    {
      "id": "abc123",
      "queueId": "4646ac53857fd2b2",
      "messageId": "<message-id@example.com>",
      "state": "waiting",
      "to": ["recipient@example.com"],
      "subject": "Test message",
      "created": "2025-05-14T10:00:00.000Z"
    }
  ],
  "total": 3,
  "page": 0,
  "pages": 1
}

Get Job Details​

# Get specific job
curl "https://ee.example.com/v1/outbox/abc123" \
  -H "Authorization: Bearer <token>"

Response:

{
  "id": "abc123",
  "queueId": "4646ac53857fd2b2",
  "messageId": "<message-id@example.com>",
  "state": "delayed",
  "attemptsMade": 2,
  "attempts": 10,
  "nextAttempt": "2025-05-14T10:15:00.000Z",
  "lastError": "Connection timeout",
  "envelope": {
    "from": "sender@example.com",
    "to": ["recipient@example.com"]
  },
  "created": "2025-05-14T10:00:00.000Z",
  "updated": "2025-05-14T10:05:00.000Z"
}

Managing Queue Jobs​

Delete a Job​

Use the delete outbox entry API:

# Delete job from queue
curl -XDELETE "https://ee.example.com/v1/outbox/abc123" \
  -H "Authorization: Bearer <token>"

Useful for:

  • Removing stuck jobs
  • Canceling scheduled sends
  • Clearing failed jobs
Retrying Failed Jobs

To retry a failed job, you need to delete it from the queue and resubmit the message using the submit API. The outbox API does not support automatic retry of individual jobs.

Configuration​

Delivery Attempts​

Configure maximum retry attempts via the web UI or API:

Via Web UI:

  1. Navigate to Configuration → Service
  2. Set "Delivery Attempts" (default: 10)

Via API when submitting a message:

{
  "to": [{"address": "recipient@example.com"}],
  "subject": "Test",
  "text": "Hello",
  "deliveryAttempts": 5
}

Default: 10 attempts

Keep Completed/Failed Jobs​

By default, completed and failed jobs are removed to save Redis memory.

Enable storage:

  1. Navigate to Configuration → Service
  2. Set "Job History Limit" to the number of completed/failed jobs to keep
  3. Example: Set to 100 to keep last 100 completed and 100 failed jobs

Note: This only applies to new jobs, not existing ones.

SMTP Timeout​

The SMTP socket timeout is set to 2 minutes (120 seconds). This is the maximum time allowed for SMTP operations before timing out.

Webhook Events​

The queue system triggers webhooks at key points:

messageSent​

Emitted when job moves to Completed:

{
  "event": "messageSent",
  "account": "example",
  "date": "2025-05-14T10:32:39.499Z",
  "data": {
    "messageId": "<message-id@example.com>",
    "queueId": "4646ac53857fd2b2",
    "response": "250 2.0.0 Ok: queued as 5755482356",
    "envelope": {
      "from": "sender@example.com",
      "to": ["recipient@example.com"]
    }
  }
}

messageDeliveryError​

Emitted when job moves to Delayed after failure:

{
  "event": "messageDeliveryError",
  "account": "example",
  "date": "2025-05-14T10:05:35.832Z",
  "data": {
    "queueId": "4646ac53857fd2b2",
    "messageId": "<message-id@example.com>",
    "error": "Connection timeout",
    "errorCode": "ETIMEDOUT",
    "smtpResponseCode": null,
    "job": {
      "attemptsMade": 2,
      "attempts": 10,
      "nextAttempt": "2025-05-14T10:10:35.465Z"
    },
    "envelope": {
      "from": "sender@example.com",
      "to": ["recipient@example.com"]
    }
  }
}

messageFailed​

Emitted when job moves to Failed:

{
  "event": "messageFailed",
  "account": "example",
  "date": "2025-05-14T11:58:50.181Z",
  "data": {
    "messageId": "<message-id@example.com>",
    "queueId": "4646ac53857fd2b2",
    "error": "Error: Invalid login: 535 5.7.8 Error: authentication failed",
    "envelope": {
      "from": "sender@example.com",
      "to": ["recipient@example.com"]
    }
  }
}