Outbox Queue

A submitted message is not sent inside the API call. EmailEngine stores it, answers with a queue ID, and a separate worker delivers it. This page covers what happens between "queued" and "sent": the states a message passes through, how retries are timed, what the outbox API shows, and which settings control it.

Why Queues Matter

When you submit an email, EmailEngine:

1. Validates your request
2. Stores the message and adds a job to the submit queue
3. Returns immediately with a queue ID
4. Delivers the message from a worker, independently of the API request
5. Retries a failed delivery on a schedule
6. Reports the outcome through webhooks

Because the job lives in Redis rather than in the API process, a scheduled or retrying message survives an EmailEngine restart.

Queue Technology

EmailEngine uses BullMQ for its queues, backed by Redis. Outbound mail is the submit queue; webhooks and exports have queues of their own. The queue management page describes the queues as a set, along with Bull Board and worker concurrency. This page stays with the submit queue.

Job Lifecycle

A job in the submit queue moves through these states:

1. Waiting

Description: Jobs ready to be processed.

How jobs get here:

• New submissions without a sendAt in the future
• Delayed jobs whose time has come

What happens: A submit worker picks the job up and moves it to Active.

2. Active

Description: Jobs currently being processed.

What happens:

• EmailEngine connects to the account's SMTP server (or the gateway named in the submission)
• Transmits the message
• Waits for the SMTP response

Outcomes:

• Success - Moved to Completed
• Temporary failure - Moved to Delayed and retried
• Permanent failure - Moved to Failed, even if attempts remain
• Retries exhausted - Moved to Failed after the last attempt

A worker holds a lock on an active job for 3 minutes and renews it while the delivery runs. A job whose worker died (a crash, or a shutdown that could not wait for a long transaction) is detected as stalled within a minute and run again, up to 3 times, after which it fails. Since v2.79.8 a re-run reads the progress the previous run stored and does not send again once that says the server accepted the message, so a stall after the 250 costs the bookkeeping that followed it (the Sent copy, the flags on a replied message), not a duplicate delivery.

3. Completed

Description: The SMTP server accepted the message.

What happens:

• messageSent webhook is emitted
• The stored message content is removed from Redis

By default a completed job is removed from the queue as soon as it finishes. To keep completed jobs for debugging, set the Job History Limit described under Configuration. Even then only the job metadata remains; the message content is gone.

4. Failed

Description: Jobs that will not be retried.

How jobs get here:

• Every attempt allowed by deliveryAttempts (default 10) failed with a retriable error
• A permanent error occurred, on any attempt

What happens:

• messageFailed webhook is emitted
• The stored message content is removed from Redis

Failed jobs are kept, unlike completed ones: a failure is the only record that a delivery was given up on. By default the last 500 failed entries are retained for 7 days; EENGINE_QUEUE_KEEP_FAILED and EENGINE_QUEUE_KEEP_FAILED_AGE (seconds) change those bounds. The retained entry holds the job metadata and the last error, not the message itself.

How EmailEngine decides that an error is permanent:

1. When the SMTP server replied with a status code, that code decides: a 5xx reply is permanent, except 503, which is retried. A 4xx reply is retried
2. When there was no SMTP reply (the connection or the handshake failed), the error code decides. These codes are permanent:

Error Code	Meaning
EAUTH	SMTP authentication failed
ENOAUTH	No credentials provided
EOAUTH2	OAuth2 token failure
ETLS	TLS handshake failed
EENVELOPE	Invalid sender or recipients
EMESSAGE	Message content error
EPROTOCOL	SMTP protocol mismatch
GatewayNotFound	The gateway the message was queued for has been deleted since

Everything else (network timeouts, connection resets, a server that closed the connection) is retried.

5. Delayed

Description: Jobs waiting for a future time.

How jobs get here:

• New submissions with a sendAt in the future
• Failed delivery attempts that will be retried

What happens:

• The job waits until its time
• Then it moves to Waiting
• A messageDeliveryError webhook is emitted for every failed delivery attempt, including the final one

Retry schedule: BullMQ's exponential backoff with a 5 second base. The delay before retry n is 2^(n-1) x 5 s, reduced by a random amount of up to 20% (the jitter), so each retry lands between 80% and 100% of the nominal delay:

Attempt	Nominal delay after the previous failure
1	Immediate
2	5 s
3	10 s
4	20 s
5	40 s
6	80 s
10	about 21 minutes

With the default of 10 attempts, a message that keeps failing with a retriable error is given up on roughly 43 minutes after the first attempt. The jitter keeps a batch of messages that failed together from retrying at the same instant.

nextAttempt is the nominal time

The nextAttempt value in the outbox API and in the messageDeliveryError payload is computed from the nominal delay without the jitter, so it is the latest moment the retry can happen; the actual retry is up to 20% earlier. Before v2.70.0 the outbox API doubled the delay once too often and reported a time later than the nominal one.

Pausing the queue

BullMQ 6 (EmailEngine v2.79.0 and later) has no separate "paused" job state. Pausing the queue stops workers from taking new jobs; the jobs that are already waiting stay in Waiting, an active job finishes, and new submissions join Waiting too. Resuming lets the workers continue.

Pause and resume from the queue browser, or through the queue settings API:

# Pause queue
curl -XPUT "https://emailengine.example.com/v1/settings/queue/submit" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"paused": true}'

# Resume queue
curl -XPUT "https://emailengine.example.com/v1/settings/queue/submit" \
  -H "Authorization: Bearer <token>" \
  -H "Content-Type: application/json" \
  -d '{"paused": false}'

Use it for maintenance windows, for debugging, or to hold outgoing mail while a provider rate limit clears.

Monitoring the Queue

Queue browser

The admin interface has a browser for the queues under System > Queues (/admin/queues). Releases before v2.84.0 bundled the third-party Bull Board page at /admin/bull-board; that address now redirects to the queue browser.

Outgoing messages lists the submit queue one state at a time, each entry shown as a message: subject, sender and recipients, account, when it is due or when it failed, the attempt count, and the outcome. Opening an entry shows the envelope, the timing, what went wrong in plain terms with the server's reason under it, and the raw job record for support cases. The search box looks through the newest entries of the selected state by subject, address, message ID, queue ID or account.

From there you can:

• Send now a message that is waiting for its scheduled time or for the backoff before its next attempt
• Retry a failed message whose content is still kept (see Retrying failed messages)
• Remove a message that is not being delivered right now - its content goes with it
• Pause and resume the queue, and remove every completed or failed entry

Outbox API

The outbox API lists the jobs in the delayed, waiting, active, and failed states, in that order, and each entry carries its state. Completed jobs never appear here, even when the Job History Limit retains them; they are visible in the queue browser only.

List queued messages

curl "https://emailengine.example.com/v1/outbox" \
  -H "Authorization: Bearer <token>"

Response:

{
  "total": 120,
  "page": 0,
  "pages": 6,
  "messages": [
    {
      "queueId": "4646ac53857fd2b2",
      "account": "example",
      "source": "api",
      "messageId": "<test123@example.com>",
      "envelope": {
        "from": "sender@example.com",
        "to": ["recipient@example.com"]
      },
      "subject": "Test message",
      "gateway": null,
      "proxy": null,
      "localAddress": null,
      "idempotencyKey": null,
      "created": "2025-05-14T10:00:00.000Z",
      "scheduled": "2025-05-14T10:00:00.000Z",
      "nextAttempt": "2025-05-14T10:00:15.465Z",
      "attemptsMade": 2,
      "attempts": 10,
      "progress": {
        "status": "error",
        "error": {
          "message": "Connection timeout",
          "code": "ETIMEDOUT",
          "statusCode": null
        }
      }
    }
  ]
}

Use the page and pageSize query parameters for pagination (pageSize defaults to 20 and goes up to 1000):

curl "https://emailengine.example.com/v1/outbox?page=0&pageSize=10" \
  -H "Authorization: Bearer <token>"

source says how the message entered the queue: api, smtp (the SMTP server), ui (a test message from the admin interface), or test (a delivery test). idempotencyKey is the key the submission carried, or null. An API submission also carries useStructuredFormat, the value of the query parameter of the same name, which only matters for MS Graph deliveries.

The progress field tracks the delivery status of each message:

Status	Meaning
queued	Accepted and waiting for a delivery attempt
processing	A submit worker has picked the message up
smtp-starting	Opening the SMTP connection
smtp-completed	The SMTP transaction finished and the message was handed over (includes response and messageId, plus originalMessageId when the server assigned a new Message-ID)
submitted	Delivered to the receiving server (includes the SMTP response)
error	The last attempt failed (includes error with message, code, and the SMTP statusCode)

nextAttempt is when the next delivery attempt is due, or false when no attempts remain. attemptsMade against attempts shows how much of the retry budget is left.

Get a specific message

Retrieve a single queued message by its queue ID with the get outbox entry API:

curl "https://emailengine.example.com/v1/outbox/4646ac53857fd2b2" \
  -H "Authorization: Bearer <token>"

The response is one entry in the same shape as the list. The endpoint works while the message content is still stored, which means the waiting, active, and delayed states. Once a message has completed or failed for good, its content is removed and the endpoint returns 404, even when the Job History Limit keeps the job's metadata in Bull Board.

Managing Queue Jobs

Delete a Job

Use the delete outbox entry API:

curl -XDELETE "https://emailengine.example.com/v1/outbox/4646ac53857fd2b2" \
  -H "Authorization: Bearer <token>"

Response:

{
  "deleted": true
}

This removes both the stored message and the job. It works for waiting, delayed, and failed jobs. A job that a worker holds at that moment (active) cannot be removed: the response carries "deleted": false together with "locked": true (v2.79.8 and later), and the call can be repeated once the attempt is over. An unknown queue ID answers "deleted": false alone.

Useful for:

• Cancelling a scheduled send
• Clearing a retained failed job

Retrying Failed Jobs

A failed message can be retried while its content is still kept: POST /v1/outbox/{queueId}/retry over the API, or Retry on the message's page in the queue browser. See Retrying failed messages below.

Retry a failed message

curl -X POST "https://emailengine.example.com/v1/outbox/1812477e7ed5330eda4/retry" \
  -H "Authorization: Bearer <token>"

{
  "retried": true,
  "queueId": "1812477e7ed5330eda4",
  "messageId": "<uuid@example.com>"
}

The message goes back to Waiting with a fresh set of delivery attempts. A 409 answer names why it cannot be retried in its code: NotFailed when the message is not in the failed state, ContentGone when its content is no longer kept - resubmit it through the submit API in that case.

Send a scheduled message now

curl -X POST "https://emailengine.example.com/v1/outbox/1812477e7ed5330eda4/promote" \
  -H "Authorization: Bearer <token>"

{
  "promoted": true,
  "queueId": "1812477e7ed5330eda4",
  "messageId": "<uuid@example.com>"
}

Moves a message out of Delayed so it is delivered at the next opportunity, whether it was waiting for its sendAt or for the backoff before its next attempt. Any other state answers 409 with code NotDelayed.

Retrying failed messages

When the last delivery attempt fails, the message record stays in the failed set and, since v2.84.0, so does the message content, so the message can be retried as it was. The content is bounded separately from the job record, because a message can be as large as EENGINE_MAX_SMTP_MESSAGE_SIZE:

• EENGINE_QUEUE_KEEP_FAILED_CONTENT - how many of the most recent failed messages keep their content (default 100)
• EENGINE_QUEUE_KEEP_FAILED_CONTENT_SIZE - how much that content may add up to in total (default 100MB)

Past either bound the oldest content is dropped first, and content is dropped with its job record after EENGINE_QUEUE_KEEP_FAILED_AGE, when the failed set is cleaned, and when the account is deleted. Setting either variable to 0 keeps no content, which is how releases before v2.84.0 behaved. A failed message whose content was dropped is still listed, marked Content discarded, and can only be sent again by resubmitting it. The overview of the queue browser shows how many messages currently keep their content and how much space they take.

Configuration

Delivery Attempts

The maximum number of delivery attempts is set per instance, and can be overridden per message.

Instance default:

1. Open Configuration > Email Processing
2. Set Delivery attempts in the Email Delivery card (default: 10)

The same value is the deliveryAttempts key in the Settings API. Both the setting and the per-message override take 0 to 100; 0 and 1 both mean a single attempt.

Per message, in the submit payload:

{
  "to": [{"address": "recipient@example.com"}],
  "subject": "Test",
  "text": "Hello",
  "deliveryAttempts": 5
}

Permanent errors (see Failed) end the job immediately regardless of this setting.

Keep Completed and Failed Jobs

Completed jobs are removed as soon as they finish, to save Redis memory. To keep them:

1. Open Configuration > General
2. Set Completed job history limit (queueKeep) to the number of completed jobs to keep. 0 disables the history

Retained completed jobs are also dropped after 24 hours, whichever limit is reached first. Failed jobs have their own floor, described under Failed: at least 500 entries for 7 days, or the Job History Limit when that is higher.

warning

The retention policy is attached to a job when it is created, so a change applies to jobs queued after the change. Retained jobs are visible in the queue browser, and retained failed jobs also in GET /v1/outbox. GET /v1/outbox/{queueId} returns 404 for a completed job, because the message content is removed on completion; a failed job answers while its content is kept (see Retrying failed messages).

SMTP Timeout

The SMTP socket timeout during delivery is 2 minutes. A server that stays silent longer than that fails the attempt with a retriable error.

Throttling

EENGINE_SUBMIT_DELAY makes each submit worker wait that long between two deliveries, so the queue drains at a fixed rate rather than as fast as the servers accept mail. It is a startup variable, described under environment variables.

Webhook Events

The queue emits three events. Each has a page of its own with the full payload; the examples here show the fields that relate to the queue.

messageSent

Emitted when a job completes:

{
  "event": "messageSent",
  "account": "example",
  "date": "2025-05-14T10:32:39.499Z",
  "data": {
    "messageId": "<message-id@example.com>",
    "queueId": "4646ac53857fd2b2",
    "response": "250 2.0.0 Ok: queued as 5755482356",
    "envelope": {
      "from": "sender@example.com",
      "to": ["recipient@example.com"]
    }
  }
}

messageDeliveryError

Emitted on every failed delivery attempt, including the final one, where messageFailed follows. job.nextAttempt is the nominal time of the next retry, or false after the last attempt:

{
  "event": "messageDeliveryError",
  "account": "example",
  "date": "2025-05-14T10:05:35.832Z",
  "data": {
    "queueId": "4646ac53857fd2b2",
    "messageId": "<message-id@example.com>",
    "error": "Connection timeout",
    "errorCode": "ETIMEDOUT",
    "smtpResponseCode": null,
    "job": {
      "attemptsMade": 2,
      "attempts": 10,
      "nextAttempt": "2025-05-14T10:05:55.465Z"
    },
    "envelope": {
      "from": "sender@example.com",
      "to": ["recipient@example.com"]
    }
  }
}

messageFailed

Emitted when a job is finished for good, after the last attempt or on a permanent error:

{
  "event": "messageFailed",
  "account": "example",
  "date": "2025-05-14T11:58:50.181Z",
  "data": {
    "messageId": "<message-id@example.com>",
    "queueId": "4646ac53857fd2b2",
    "error": "Error: Invalid login: 535 5.7.8 Error: authentication failed"
  }
}

See Also

• Queue Management - Every queue, the queue browser, worker concurrency, and Redis tuning
• Basic Sending - How a message enters the queue
• messageDeliveryError and messageFailed - The full payloads of the failure events
• Sending API - The outbox endpoints in the API reference
• Bounces - A failure that arrives by mail after the queue reported success