Pre-Processing Functions

Pre-processing functions allow you to run custom JavaScript code to filter or transform data before EmailEngine processes it. Use these functions to implement custom business logic, filter unwanted events, or modify webhook payloads.

Overview

EmailEngine supports pre-processing for:

• Webhooks - Filter or modify webhook payloads before delivery
• Custom Routes - Transform data before processing

Pre-processing functions are small JavaScript snippets that run in a secure sandbox environment.

Function Types

Code Format

Pre-processing code is function body content, not a full function definition. The code runs in the main scope and must return a value directly. The webhook payload is available as payload.

if (payload.path === "INBOX") {
  return true;
}
return false;

1. Filter Functions

Filter functions determine whether an event should be processed or discarded.

Return value:

• true - Process the event (must be exactly true, not just truthy)
• false or any other value - Discard the event
• Exception thrown - Discard the event

Use cases:

• Skip webhooks for automated messages
• Filter out spam or promotional emails

Example - Skip auto-reply emails:

// Return true to send webhook, false to skip

// Skip auto-replies
if (payload.data.headers && payload.data.headers["auto-submitted"]) {
  return false;
}

// Skip out-of-office messages
if (payload.data.subject && /out of office/i.test(payload.data.subject)) {
  return false;
}

// Process all other webhooks
return true;

2. Mapping Functions

Mapping functions modify the structure or content of data before processing.

Return value:

• Modified data object
• Original data unchanged if no return value

Use cases:

• Add custom fields to webhooks
• Redact sensitive information
• Normalize data formats
• Enrich with additional context

Example - Add custom fields:

// Add custom tracking ID
payload.customId = `${payload.account}-${Date.now()}`;

// Add priority based on subject
if (payload.data.subject && /urgent/i.test(payload.data.subject)) {
  payload.priority = "high";
} else {
  payload.priority = "normal";
}

// Redact sensitive content
if (payload.data.text && payload.data.text.plain) {
  payload.data.text.plain = payload.data.text.plain.replace(/ssn:\s*\d{3}-\d{2}-\d{4}/gi, "ssn: [REDACTED]");
}

return payload;

Configuration

Webhook Pre-Processing

Configure pre-processing for webhook routes in the EmailEngine UI.

Step 1: Navigate to Webhook Settings

1. Go to Settings → Webhooks
2. Click on the webhook route to configure
3. Scroll to Pre-Processing Function section

Step 2: Add Filter Function

// Filter: return true to send webhook, false to skip

// Only send webhooks for inbox messages
if (payload.path !== "INBOX") {
  return false;
}

// Skip notifications (usually automated)
if (payload.data.from && /noreply|no-reply/i.test(payload.data.from.address)) {
  return false;
}

// Skip old messages (older than 1 hour)
if (payload.data.date) {
  const messageAge = Date.now() - new Date(payload.data.date).getTime();
  if (messageAge > 3600000) {
    // 1 hour in milliseconds
    return false;
  }
}

return true;

Step 3: Add Mapping Function

// Mapping: modify payload and return it

// Add custom fields
payload.metadata = {
  receivedAt: new Date().toISOString(),
  environment: "production",
  version: "1.0",
};

// Categorize by subject
if (payload.data.subject) {
  const subject = payload.data.subject.toLowerCase();
  if (subject.includes("invoice") || subject.includes("payment")) {
    payload.category = "billing";
  } else if (subject.includes("support") || subject.includes("help")) {
    payload.category = "support";
  } else {
    payload.category = "general";
  }
}

// Extract ticket ID from subject
const ticketMatch = payload.data.subject && payload.data.subject.match(/#(\d+)/);
if (ticketMatch) {
  payload.ticketId = ticketMatch[1];
}

return payload;

Step 4: Test Function

Use the Set test payload button to provide sample data for testing your function. The editor runs your filter and mapping functions in the browser and shows:

• Evaluation result - Whether the filter function returns true (matches) or not
• Mapping preview - The transformed payload after the mapping function runs

You can select from predefined example payloads or enter custom JSON data to test different scenarios.

Common Use Cases

1. Skip Automated Emails

// Check Auto-Submitted header
if (payload.data.headers && payload.data.headers["auto-submitted"] && payload.data.headers["auto-submitted"][0] !== "no") {
  return false;
}

// Check for common automated addresses
const automatedPatterns = [/noreply/i, /no-reply/i, /donotreply/i, /notifications?/i, /mailer-daemon/i, /postmaster/i];

if (payload.data.from && payload.data.from.address) {
  for (const pattern of automatedPatterns) {
    if (pattern.test(payload.data.from.address)) {
      return false;
    }
  }
}

// Check subject for automated patterns
const automatedSubjects = [/out of office/i, /automatic reply/i, /auto-reply/i, /mail delivery fail/i];

if (payload.data.subject) {
  for (const pattern of automatedSubjects) {
    if (pattern.test(payload.data.subject)) {
      return false;
    }
  }
}

return true;

2. Extract and Normalize Data

// Extract email addresses from CC and BCC
const allRecipients = [...(payload.data.to || []), ...(payload.data.cc || []), ...(payload.data.bcc || [])].map((r) => r.address);

payload.allRecipients = [...new Set(allRecipients)]; // Remove duplicates

// Parse subject line for ticket/order numbers
if (payload.data.subject) {
  // Extract ticket ID (e.g., "#12345", "TICKET-12345")
  const ticketMatch = payload.data.subject.match(/#(\d+)|TICKET-(\d+)/i);
  if (ticketMatch) {
    payload.ticketId = ticketMatch[1] || ticketMatch[2];
  }

  // Extract order ID (e.g., "Order #12345", "Order ID: 12345")
  const orderMatch = payload.data.subject.match(/order\s*#?:?\s*(\d+)/i);
  if (orderMatch) {
    payload.orderId = orderMatch[1];
  }
}

// Normalize sender domain
if (payload.data.from && payload.data.from.address) {
  const domain = payload.data.from.address.split("@")[1];
  payload.senderDomain = domain.toLowerCase();

  // Flag internal emails
  payload.isInternal = ["example.com", "company.com"].includes(domain);
}

return payload;

3. Priority and Categorization

// Determine priority
payload.priority = "normal";

// High priority indicators
const urgentKeywords = ["urgent", "asap", "important", "critical"];
const subject = (payload.data.subject || "").toLowerCase();

for (const keyword of urgentKeywords) {
  if (subject.includes(keyword)) {
    payload.priority = "high";
    break;
  }
}

// VIP senders
const vipDomains = ["important-client.com", "executive.com"];
if (payload.data.from && payload.data.from.address) {
  const domain = payload.data.from.address.split("@")[1];
  if (vipDomains.includes(domain)) {
    payload.priority = "high";
    payload.isVip = true;
  }
}

// Categorize by content
const categories = {
  billing: ["invoice", "payment", "receipt", "billing"],
  support: ["support", "help", "question", "issue"],
  sales: ["quote", "proposal", "pricing", "demo"],
  hr: ["benefits", "payroll", "pto", "vacation"],
};

for (const [category, keywords] of Object.entries(categories)) {
  for (const keyword of keywords) {
    if (subject.includes(keyword)) {
      payload.category = category;
      break;
    }
  }
  if (payload.category) break;
}

payload.category = payload.category || "general";

return payload;

4. Redact Sensitive Information

// Patterns for sensitive data
const patterns = {
  ssn: /\b\d{3}-\d{2}-\d{4}\b/g,
  creditCard: /\b\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}\b/g,
  phone: /\b\d{3}[-.]?\d{3}[-.]?\d{4}\b/g,
  email: /\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b/g,
};

// Redact from text (note: text content may be in payload.data.text.plain)
if (payload.data.text && payload.data.text.plain) {
  payload.data.text.plain = payload.data.text.plain.replace(patterns.ssn, "SSN:[REDACTED]");
  payload.data.text.plain = payload.data.text.plain.replace(patterns.creditCard, "CARD:[REDACTED]");
}

// Flag as containing sensitive data
const originalText = (payload.data.text && payload.data.text.plain) || "";
if (patterns.ssn.test(originalText) || patterns.creditCard.test(originalText)) {
  payload.containsSensitiveData = true;
}

return payload;

5. Add Metadata and Context

// Add processing metadata
payload.processing = {
  receivedAt: new Date().toISOString(),
  version: "2.0",
};

// Count attachments by type
if (payload.data.attachments) {
  payload.attachmentStats = {
    total: payload.data.attachments.length,
    images: payload.data.attachments.filter((a) => a.contentType?.startsWith("image/")).length,
    documents: payload.data.attachments.filter((a) => a.contentType?.includes("pdf") || a.contentType?.includes("word")).length,
    totalSize: payload.data.attachments.reduce((sum, a) => sum + (a.encodedSize || 0), 0),
  };
}

return payload;

Available Data

Webhook Payload

Pre-processing functions receive the complete webhook payload. The payload has a nested structure with top-level webhook metadata and message details inside the data property:

{
  // Top-level webhook metadata
  serviceUrl: "https://emailengine.example.com",
  account: "testaccount",
  date: "2024-10-13T14:23:45.678Z",
  path: "INBOX",
  specialUse: "\\Inbox",
  event: "messageNew",

  // Message data (nested inside `data`)
  data: {
    id: "AAAABgAAAdk",
    uid: 123,
    path: "INBOX",
    date: "2024-10-13T14:20:00.000Z",
    flags: [],
    unseen: true,
    subject: "Important Message",
    from: {
      name: "John Doe",
      address: "john@example.com"
    },
    to: [
      { name: "Jane Smith", address: "jane@example.com" }
    ],
    replyTo: [],
    messageId: "<abc123@example.com>",
    headers: {
      "return-path": ["<john@example.com>"],
      "message-id": ["<abc123@example.com>"],
      "from": ["John Doe <john@example.com>"],
      "to": ["Jane Smith <jane@example.com>"],
      "subject": ["Important Message"],
      "content-type": ["text/plain; charset=utf-8"],
      "auto-submitted": ["no"]  // Present for automated messages
    },
    text: {
      id: "...",
      encodedSize: { plain: 1234, html: 5678 }
    },
    attachments: [
      {
        id: "abc123",
        contentType: "application/pdf",
        encodedSize: 12345,
        filename: "document.pdf",
        embedded: false,
        inline: false
      }
    ]
  }
}

Accessing data in filter/mapping functions:

// Top-level properties (webhook metadata)
payload.event; // "messageNew"
payload.account; // "testaccount"
payload.path; // "INBOX"

// Message properties (inside payload.data)
payload.data.subject; // "Important Message"
payload.data.from; // { name: "...", address: "..." }
payload.data.headers; // { "auto-submitted": [...], ... }
payload.data.attachments; // [...]

Sandbox Environment

Pre-processing functions run in a secure sandbox with limited access:

Available:

• Standard JavaScript (ES6+)
• Top-level await is supported
• Date, Math, JSON, RegExp
• fetch - Make HTTP requests to external services
• URL - URL parsing and manipulation
• logger - Pino.js logger instance (logs to EmailEngine's stdout)
• env - The scriptEnv settings object (configure via Settings API)

Not Available:

• require() - Cannot import modules
• fs - No filesystem access
• process, child_process - No system access

Using env for Configuration:

Store sensitive values like API keys in scriptEnv settings rather than hardcoding them:

// Access values from scriptEnv settings
const apiKey = env.MY_API_KEY;
const webhookSecret = env.WEBHOOK_SECRET;

if (apiKey) {
  // Use the API key
  const response = await fetch("https://api.example.com/validate", {
    headers: { Authorization: `Bearer ${apiKey}` },
  });
}

return true;

Configure scriptEnv via the Settings API (note: the value must be a JSON string):

curl -X POST "https://emailengine.example.com/v1/settings" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "scriptEnv": "{\"MY_API_KEY\":\"your-api-key-here\",\"WEBHOOK_SECRET\":\"your-secret\"}"
  }'

Using logger for Structured Logging:

logger.info({ account: payload.account, path: payload.path, msg: "Processing webhook" });
logger.warn({ subject: payload.data.subject, msg: "Suspicious email detected" });

return true;

Performance Considerations

1. Keep Functions Fast

Pre-processing runs for every event. Keep functions lightweight:

// Fast - simple checks
return payload.path === "INBOX" && !payload.data.headers["auto-submitted"];

// Slow - avoid external requests in pre-processing
const response = await fetch("https://api.openai.com/v1/chat/completions", {
  method: "POST",
  headers: {
    Authorization: `Bearer ${env.OPENAI_API_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    model: "gpt-4",
    messages: [{ role: "user", content: `Is this spam? ${payload.data.subject}` }],
  }),
});
const result = await response.json();
return result.choices[0].message.content.includes("not spam");

Use Webhooks for Slow Operations

Instead of calling external APIs in pre-processing, let the webhook pass through and handle AI classification in your webhook handler. This keeps EmailEngine responsive while your application handles the slow operations asynchronously.

2. Cache Computed Values

If checking multiple conditions, store intermediate results:

const subject = (payload.data.subject || "").toLowerCase(); // Compute once

// Use cached value
payload.isUrgent = subject.includes("urgent") || subject.includes("important");

payload.isBilling = subject.includes("invoice") || subject.includes("payment");

return payload;

3. Exit Early

Return as soon as you know the result:

// Exit early if conditions not met
if (payload.path !== "INBOX") return false;
if (!payload.data.from || !payload.data.from.address) return false;
if (payload.data.headers && payload.data.headers["auto-submitted"]) return false;

// Only process if all checks passed
return true;

Debugging

Using the Logger

Use logger (Pino.js) for debugging:

logger.info({ account: payload.account, path: payload.path, msg: "Processing webhook" });
logger.debug({ autoSubmitted: !!payload.data.headers?.["auto-submitted"], msg: "Header check" });

const result = payload.path === "INBOX";
logger.info({ result, msg: "Filter decision" });

return result;

Log entries appear in EmailEngine's stdout alongside other application logs.

Check EmailEngine Logs

EmailEngine logs to stdout. Use your process manager or Docker logs to view output:

# If running directly
node server.js 2>&1 | grep "subscript"

# If using Docker
docker logs -f emailengine 2>&1 | grep "subscript"

# If using systemd
journalctl -u emailengine -f | grep "subscript"

Monitor Execution Time

Log timing information:

const start = Date.now();

// Your transformations
payload.customField = "processed";

const duration = Date.now() - start;
logger.info({ duration, msg: "Processing completed" });

return payload;

HTML Email Pre-Processing for Web Display

When displaying email HTML content on a webpage (such as in a webmail client), you face several challenges:

Problems with Raw HTML:

• Broken tags can break your page layout
• CSS styles can override your page styles
• JavaScript could execute malicious code
• Images with cid: URLs won't load in browsers
• Unclosed tags can corrupt surrounding content

Traditional Solution: iFrames

Many webmail clients use <iframe> containers with sandbox attributes to isolate email HTML. However, this approach has drawbacks:

• Difficult to size correctly (scrollbars or blank space)
• Responsive design challenges
• Additional complexity for mobile views

EmailEngine's HTML Pre-Processing

EmailEngine provides built-in HTML sanitization and transformation to make email HTML safe for inline display.

Enable Pre-Processing

Use query parameters when fetching message data:

API Request:

curl "https://ee.example.com/v1/account/example/message/AAAAGQAACeE?embedAttachedImages=true&preProcessHtml=true&textType=*" \
  -H "Authorization: Bearer TOKEN"

Query Parameters:

• embedAttachedImages=true: Convert cid: image links to base64 Data URLs
• preProcessHtml=true: Sanitize and fix HTML structure
• textType=*: Include all content types for processing

What Pre-Processing Does

1. HTML Sanitization

Uses DOMPurify to:

• Remove dangerous tags (<script>, <object>, <embed>)
• Strip JavaScript event handlers (onclick, onerror, etc.)
• Clean malicious attributes
• Remove suspicious content

2. Structure Fixes

• Closes unclosed tags
• Fixes broken HTML structure
• Normalizes malformed markup
• Ensures valid HTML5

3. CSS Scoping

• Removes global style overrides
• Scopes styles to prevent interference
• Strips !important declarations that affect page layout
• Preserves email-specific styles

4. Image Handling

Converts embedded image references to inline data:

Before (in email):

<img src="cid:image-123" />

After (pre-processed):

<img src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..." />

Usage Example

Fetch and Display Email HTML:

// Pseudo code - implement in your preferred language

function display_email(accountId, messageId, token):
  // Build URL with query parameters
  url = CONCAT(
    "https://ee.example.com/v1/account/", accountId, "/message/", messageId,
    "?embedAttachedImages=true&preProcessHtml=true&textType=*"
  )

  // Make HTTP GET request
  response = HTTP_GET(url, headers={
    "Authorization": CONCAT("Bearer ", token)
  })

  // Parse JSON response
  message = PARSE_JSON(response.body)

  // Inject HTML into page (safe due to pre-processing)
  SET_ELEMENT_HTML("email-content", message.text.html)
end function

HTML Structure:

<div class="webmail-container">
  <div class="email-header">
    <strong>From:</strong> <span id="from"></span><br />
    <strong>Subject:</strong> <span id="subject"></span>
  </div>

  <!-- Pre-processed HTML injected here safely -->
  <div id="email-content" class="email-body"></div>
</div>

CSS for Email Container:

.email-body {
  /* Isolate email styles */
  all: initial;

  /* Apply safe defaults */
  font-family: Arial, sans-serif;
  font-size: 14px;
  line-height: 1.5;
  color: #333;

  /* Prevent layout breaking */
  max-width: 100%;
  overflow: hidden;
  word-wrap: break-word;
}

.email-body img {
  max-width: 100%;
  height: auto;
}

Attachment Handling

EmailEngine distinguishes between two attachment types:

1. Regular Attachments

• Meant to be downloaded
• Listed in attachments array
• Include download URLs

2. Embedded Attachments (Inline Images)

• Displayed within HTML
• Referenced via cid: protocol
• Converted to data URLs when embedAttachedImages=true

Example Response:

When embedAttachedImages=true, inline images referenced by cid: in the HTML are converted to base64 data URIs directly in the HTML content. The attachments array still contains metadata about all attachments:

{
  "id": "AAAAGQAACeE",
  "subject": "Newsletter",
  "text": {
    "html": "<p>Check out our new product:</p><img src=\"data:image/png;base64,iVBORw0...\" />"
  },
  "attachments": [
    {
      "id": "ATT123",
      "filename": "product-catalog.pdf",
      "contentType": "application/pdf",
      "encodedSize": 524288
    },
    {
      "id": "IMG456",
      "filename": "product-image.png",
      "contentType": "image/png",
      "contentId": "<image-123@example.com>",
      "encodedSize": 12345
    }
  ]
}

Note: Inline images remain in the attachments array but their cid: references in HTML are replaced with data URIs.

Implementation Example

// Pseudo code - implement in your preferred language

function email_viewer(accountId, messageId, token):
  // Fetch email data
  url = CONCAT(
    "https://ee.example.com/v1/account/", accountId, "/message/", messageId,
    "?embedAttachedImages=true&preProcessHtml=true&textType=*"
  )

  response = HTTP_GET(url, headers={
    "Authorization": CONCAT("Bearer ", token)
  })

  email = PARSE_JSON(response.body)

  if email is null:
    DISPLAY("Loading...")
    return
  end if

  // Display email header
  DISPLAY("From: " + email.from.name + " <" + email.from.address + ">")
  DISPLAY("Subject: " + email.subject)
  DISPLAY("Date: " + FORMAT_DATE(email.date))

  // Display email body (pre-processed HTML)
  SET_HTML("email-body", email.text.html)

  // Display attachments if present
  if email.attachments exists AND LENGTH(email.attachments) > 0:
    DISPLAY("Attachments:")
    for each attachment in email.attachments:
      DISPLAY_LINK(attachment.downloadUrl,
                   attachment.filename + " (" + attachment.encodedSize + " bytes)")
    end for
  end if
end function