Gmail Pub/Sub Integration

This guide covers the Google Cloud Pub/Sub integration used by EmailEngine to receive real-time notifications from Gmail accounts using the Gmail REST API.

Prerequisites

This guide assumes you have already configured Gmail API access following the Gmail API Setup Guide. Pub/Sub is configured when you create a Gmail API OAuth2 application with a linked service account.

Overview

When using Gmail REST API (instead of IMAP), EmailEngine relies on Google Cloud Pub/Sub for push notifications:

1. Gmail detects changes in an account (new message, flag change, etc.)
2. Gmail publishes a notification to a Pub/Sub topic
3. EmailEngine pulls the Pub/Sub subscription for messages
4. EmailEngine processes the notification and fires webhooks

This is different from IMAP, where EmailEngine maintains persistent connections and uses IMAP IDLE for notifications.

Pub/Sub is optional. A Gmail API application without a linked service account polls each account for changes about every 10 minutes instead, so webhooks still fire, with up to that much delay.

Configuration Settings

Service Account Settings

When creating a Gmail Service Account application in EmailEngine for Pub/Sub, these fields matter:

Setting	Description
provider	gmailService
baseScopes	pubsub for a service account that manages Pub/Sub
googleProjectId	Google Cloud project ID containing the Pub/Sub resources (project_id in the key file)
serviceClient	Service account unique ID (client_id in the key file)
serviceClientEmail	Service account email address (client_email in the key file)
serviceKey	Service account private key in PEM format (the private_key field of the JSON key file)
authMethod	serviceKey (default), externalAccount, or metadataServer for the attached service account
googleTopicName, googleSubscriptionName	Optional overrides for the generated resource names, see below

A service account can also authenticate without a stored key through Workload Identity Federation (authMethod: "externalAccount" with externalAccount); see Google Service Accounts.

Attached Service Account (Google Cloud)

When EmailEngine runs on Google Cloud, a Pub/Sub application can authenticate as the service account attached to the machine it runs on, with nothing stored in EmailEngine. EmailEngine asks the Google Cloud metadata server for an access token, the same way Google's own client libraries do. This works on:

• Compute Engine: the service account attached to the VM.
• GKE: the Google service account bound to the pod's Kubernetes service account through Workload Identity.
• Cloud Run: the service's service account. See the note on Cloud Run below.

Create the application with authMethod: "metadataServer":

curl -X POST https://emailengine.example.com/v1/oauth2 \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "provider": "gmailService",
    "name": "Gmail Pub/Sub",
    "baseScopes": "pubsub",
    "authMethod": "metadataServer",
    "googleProjectId": "my-project-123456"
  }'

serviceClient, serviceClientEmail and serviceKey are not used. In the admin interface, select the Cloud Pub/Sub tab in the Base scope card, then the Attached service account authentication method in the Credentials card. Outside Cloud Pub/Sub that tab carries a Pub/Sub only badge, and choosing it switches the base scope to Cloud Pub/Sub. Detect from this host reads the service account and project from the metadata server and fills in the project ID.

Requirements:

• Grant the attached service account the Pub/Sub permissions in the project.
• On Compute Engine, the VM's access scopes cap every token the metadata server issues, whatever roles the service account holds. Give the VM the cloud-platform access scope (Allow full access to all Cloud APIs).
• EmailEngine has to reach metadata.google.internal directly. The request never goes through the configured proxy. EENGINE_GCP_METADATA_HOST points it elsewhere, for example at an emulator.

Limitations:

• Pub/Sub applications only. The metadata server's token belongs to the service account itself and cannot act as a Workspace user, so it cannot be used for domain-wide delegation to IMAP or the Gmail API. Mailbox access still needs a service account key or Workload Identity Federation.
• Not available to a token with restricted permissions. Such an application acts as the deployment's own cloud identity, so only the admin interface or an unrestricted API token can create one.
• Fixed after creation, like the other authentication methods.

Verify setup on the application page reports the metadata server it asked, the service account the token was issued for, and whether that token can read the application's Pub/Sub topic. Each failure comes with a hint: EmailEngine not running on Google Cloud, no service account attached, a missing access scope, or a missing role.

Cloud Run

EmailEngine is a long-running process that keeps IMAP connections open and pulls Pub/Sub in the background, and it must run as a single instance: two instances on the same Redis both sync every account. Cloud Run runs a new revision alongside the old one during a deploy and can replace an instance at any time, so a second copy is possible however max-instances is set. If you run it there anyway, use instance-based billing (CPU always allocated) with min-instances and max-instances both set to 1. A Compute Engine VM, or GKE with a single replica and the Recreate deployment strategy, avoids these problems.

OAuth2 Application Settings

When creating a Gmail OAuth2 application, link it to a service account:

Setting	Description
pubSubApp	ID of the service account application to use for Pub/Sub management
baseScopes	api for Gmail REST API access

The service account must belong to the same Google Cloud project as the OAuth2 client.

Subscription Expiration (TTL)

You can configure how long a Pub/Sub subscription persists without activity before Google automatically deletes it.

Setting	Description
gmailSubscriptionTtl	Subscription TTL in days. Empty for Google's default (31 days), 0 for indefinite (never expires), 1-365 for a custom TTL

Configure via the API:

curl -X POST "https://emailengine.example.com/v1/settings" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "gmailSubscriptionTtl": 0
  }'

Or through the admin UI at Integrations > OAuth2 Apps > Gmail Subscriptions.

When do expiration changes take effect?

Changing this setting does not immediately update existing subscriptions. The new TTL is applied only when EmailEngine next calls Google's Pub/Sub API for that subscription, which happens when:

• A Pub/Sub service account application is created (the new subscription gets the current TTL)
• A Pub/Sub service account application is updated (the existing subscription is patched to match the current TTL, unless EmailEngine adopted it rather than creating it; an adopted subscription is never modified)
• A subscription is re-created after expiring (Google deleted it due to inactivity, and EmailEngine's recovery process creates a new one with the current TTL)

If you need the change to apply immediately, edit and save each Pub/Sub service account application to trigger a subscription update.

Generated Pub/Sub Resources

EmailEngine creates and manages these Pub/Sub resources for each service account application with the pubsub base scope:

Resource	Naming Pattern	Description
Topic	projects/{projectId}/topics/ee-pub-{appId}	Receives notifications from Gmail
Subscription	projects/{projectId}/subscriptions/ee-sub-{appId}	EmailEngine pulls this for messages

The default names can be overridden with the googleTopicName and googleSubscriptionName fields when creating or updating the service account application. If a topic or subscription with the configured name already exists, EmailEngine adopts it and records that it did not create it; such adopted resources are left in place when the application is deleted.

API Configuration

Creating a Service Account Application

curl -X POST "https://emailengine.example.com/v1/oauth2" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Gmail Pub/Sub Manager",
    "provider": "gmailService",
    "enabled": true,
    "baseScopes": "pubsub",
    "googleProjectId": "your-project-id",
    "serviceClient": "113457912345678901234",
    "serviceClientEmail": "emailengine@your-project-id.iam.gserviceaccount.com",
    "serviceKey": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC7\n-----END PRIVATE KEY-----\n"
  }'

The values come from the service account JSON key file downloaded from Google Cloud: serviceClient is the numeric client_id, serviceClientEmail is the client_email, googleProjectId is the project_id, and serviceKey is the PEM-formatted private_key string - not the base64-encoded JSON file.

The response carries the application ID:

{
  "id": "AAABkQ3c5eQ",
  "created": true
}

Registering the application also creates the topic and subscription and grants Gmail's push service publish access to the topic. A failure in that setup does not fail the request; it is recorded on the application as lastError and shown in the Gmail Subscriptions list.

Creating a Gmail OAuth2 Application with Pub/Sub

curl -X POST "https://emailengine.example.com/v1/oauth2" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Gmail API OAuth2",
    "provider": "gmail",
    "enabled": true,
    "baseScopes": "api",
    "clientId": "YOUR_CLIENT_ID",
    "clientSecret": "YOUR_CLIENT_SECRET",
    "redirectUrl": "https://emailengine.example.com/oauth",
    "pubSubApp": "AAABkQ3c5eQ"
  }'

The pubSubApp field links this OAuth2 application to the service account for Pub/Sub management.

Checking Pub/Sub Status

Retrieve application details to check Pub/Sub status:

curl "https://emailengine.example.com/v1/oauth2/AAABkQ3c5eQ" \
  -H "Authorization: Bearer YOUR_TOKEN"

The response includes the Pub/Sub resource names alongside the other application fields:

{
  "id": "AAABkQ3c5eQ",
  "name": "Gmail Pub/Sub Manager",
  "provider": "gmailService",
  "enabled": true,
  "baseScopes": "pubsub",
  "googleProjectId": "my-project",
  "pubSubTopic": "projects/my-project/topics/ee-pub-AAABkQ3c5eQ",
  "pubSubSubscription": "projects/my-project/subscriptions/ee-sub-AAABkQ3c5eQ",
  "pubSubIamPolicy": true
}

pubSubIamPolicy is true once Gmail's push service has been granted publish access to the topic. A lastError or pubSubError object appears in this response only while there is an error to report; the status listing below returns null for them instead.

Listing All Pub/Sub Applications

To list all Pub/Sub service account applications and their errors, use GET /v1/pubsub/status:

curl "https://emailengine.example.com/v1/pubsub/status" \
  -H "Authorization: Bearer YOUR_TOKEN"

{
  "total": 2,
  "page": 0,
  "pages": 1,
  "apps": [
    {
      "id": "AAABkQ3c5eQ",
      "name": "Gmail Pub/Sub Manager",
      "lastError": null,
      "pubSubError": null
    },
    {
      "id": "AAABkR7d2fR",
      "name": "Another App",
      "lastError": null,
      "pubSubError": {
        "message": "Failed to process subscription loop",
        "description": "Subscription not found"
      }
    }
  ]
}

• lastError is the last setup error for the application (creating the topic or subscription, setting the IAM policy) or a failed token renewal for the service account; its response field carries the message, for example Enable the Cloud Pub/Sub API
• pubSubError is the latest error from the subscription pull loop, with message and description

null for both means the subscription is healthy.

Subscription expiration is not shown in the status list

The /v1/pubsub/status endpoint does not include subscription expiration dates. Expiration is managed by Google Cloud based on the TTL configured in EmailEngine. To check the actual expiration policy on a subscription, use the Google Cloud Console or the gcloud CLI:

gcloud pubsub subscriptions describe ee-sub-AAABkQ3c5eQ \
  --project=your-project-id \
  --format="value(expirationPolicy.ttl)"

Required Google Cloud Permissions

EmailEngine uses the service account to get, create and delete the topic, read and set the topic's IAM policy, get, create, patch and delete the subscription, and pull and acknowledge messages.

Pub/Sub Admin Role

The Pub/Sub Admin role (roles/pubsub.admin) covers all of these.

Alternative: Custom Role

For least-privilege access, create a custom role with only:

title: "EmailEngine Pub/Sub Manager"
description: "Manages Pub/Sub for EmailEngine Gmail integration"
includedPermissions:
  - pubsub.topics.create
  - pubsub.topics.delete
  - pubsub.topics.get
  - pubsub.topics.setIamPolicy
  - pubsub.topics.getIamPolicy
  - pubsub.topics.attachSubscription
  - pubsub.subscriptions.create
  - pubsub.subscriptions.update
  - pubsub.subscriptions.delete
  - pubsub.subscriptions.get
  - pubsub.subscriptions.consume

Gmail Watch Renewal

A Gmail watch (users.watch) tells Gmail to publish an account's changes to the topic, and it expires after about 7 days. EmailEngine renews it:

• On account initialization and on every reconnect, if the last successful watch is more than a day old
• On a timer that fires hourly, or an hour before the expiration Gmail reported, whichever is later

A renewal that fails is retried on the next timer run. Before v2.80.0 a failed renewal counted as a renewal, so a broken watch was retried once a day and lapsed after a week of failures.

Watch State

GET /v1/account/{account} reports the watch as gmailWatch (since v2.80.0):

Field	Description
state	active while the watch is armed and has not lapsed, expired once it has lapsed, error when the last renewal failed
lastCheck	When the watch was last renewed, or when a renewal was last attempted
expires	When the current watch lapses

The field is absent for an account that has never armed a watch, and for an account whose OAuth2 application has no linked Pub/Sub app, whatever an earlier configuration left behind; an account re-authorized onto an application without push clears its old record (since v2.80.0). The reason a renewal failed is Google's own response text, so it is shown on the account page in the admin interface rather than returned by the API. A watch that is not active does not stop the account syncing: the fallback poll keeps running, so new mail is delayed rather than lost.

Topic region policies

Renewal only succeeds if Google's Gmail push service can publish to the topic. A message storage policy that restricts the topic to specific regions with in-transit enforcement blocks renewals - see Watch renewal fails with a message storage policy error.

Force Watch Renewal

To force a watch renewal for an account, request a reconnect:

curl -X PUT "https://emailengine.example.com/v1/account/user123/reconnect" \
  -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reconnect": true}'

A reconnect re-initializes the account and renews the watch regardless of its age. The response is {"reconnect": true} when the reconnect was requested. Since v2.79.4 it is {"reconnect": false} for an account that EmailEngine switched off after repeated authentication failures; re-authorize the account instead.

Troubleshooting

Pub/Sub Setup Failed When Registering the App

Symptoms:

• The application appears in Integrations > OAuth2 Apps > Gmail Subscriptions with a setup error
• lastError.response in GET /v1/pubsub/status carries a message such as Enable the Cloud Pub/Sub API

Causes:

• Cloud Pub/Sub API not enabled in the Google Cloud project
• Service account lacks the permissions listed above
• Project quota exceeded

Solution:

1. Enable the Cloud Pub/Sub API in Google Cloud Console
2. Verify the service account has the Pub/Sub Admin role
3. Edit and save the application in EmailEngine to retry the setup

Subscription Messages Not Received

Symptoms:

• Webhooks not firing for Gmail accounts
• Account shows as connected but no events

Causes:

• Gmail watch expired and not renewed
• Subscription not receiving messages
• EmailEngine webhook worker not processing

Solution:

1. Read gmailWatch.state from GET /v1/account/{account}; error or expired means the watch is the problem, and the account page shows Google's response
2. Check EmailEngine logs for Pub/Sub errors:

   journalctl -u emailengine | grep -i "pub/sub\|pubsub"

3. Force account reconnection to renew the watch
4. Verify the webhook worker is running (threads{type="webhooks"} metric)

Watch Renewal Fails with a Message Storage Policy Error

Symptoms:

• An account's Change subscription status shows Failed in the admin UI, with the error OAuth2 request failed
• The Show details view contains a FAILED_PRECONDITION region error, for example: "The topic's message storage policy requires enforcement in transit, but the Publish request was received by a Pub/Sub server in a non-allowed region."

Cause:

The topic has a message storage policy with in-transit enforcement (enforce_in_transit) that limits it to specific regions - often applied through an organization-level "Enforce in-transit regions for Pub/Sub messages" policy. Notifications are published to the topic by Google's Gmail push service (gmail-api-push@system.gserviceaccount.com), not by EmailEngine. That publisher routes through Google's global infrastructure and cannot be pinned to a region, so the publish is rejected. This is a general limitation of Gmail push - no client controls which region Gmail publishes from - and it cannot be avoided by pointing EmailEngine at a regional Pub/Sub endpoint.

Solution:

Choose one of the following:

1. Keep real-time push - relax the topic policy. Remove in-transit enforcement on the EmailEngine topic, or ask whoever manages your Google Cloud organization policies to exclude this topic or project so the Gmail publisher is allowed. In-transit enforcement also rejects EmailEngine's own subscription pulls on the global endpoint, so lifting it is required for Pub/Sub notifications to work end to end.
2. Drop Pub/Sub - use polling instead. Create the Gmail OAuth2 application without a linked service account (leave pubSubApp unset). EmailEngine then skips the watch entirely and falls back to polling each account about every 10 minutes for changes. Real-time push is lost, but the accounts sync normally and the policy is never triggered.

Debugging Pub/Sub in Logs

EmailEngine logs Pub/Sub activity at various levels:

# View Pub/Sub processing logs
journalctl -u emailengine | grep -i pubsub

# View Google API errors
journalctl -u emailengine | grep "googleapis"

# View watch renewal logs
journalctl -u emailengine | grep -i "watch"

Monitoring

Prometheus Metrics

Monitor Pub/Sub-related activity through these metrics:

Metric	Description
oauth2_api_request{provider="gmail"}	Gmail API requests, including watch calls; labeled by status and statusCode
events{event="messageNew"}	New message events (triggered by Pub/Sub notifications or fallback polling)
threads{type="webhooks"}	The webhook worker threads, which run the subscription pull loop

Health Indicators

A healthy Pub/Sub integration shows:

• Regular messageNew events for active Gmail accounts
• Successful oauth2_api_request metrics
• pubSubError: null for every application in GET /v1/pubsub/status

How Notifications Reach EmailEngine

EmailEngine creates one Pub/Sub topic and subscription per service account application when the application is registered, registers a Gmail watch for each account that points at the topic, and pulls the subscription in a loop, acknowledging each batch of notifications once it is processed. A notification only says that the account's history changed; the messages themselves are fetched through the Gmail API, and a 10-minute fallback poll covers notifications that never arrive. For how this compares with IMAP IDLE, and when to prefer either, see Why use the Gmail API instead of IMAP.

See Also

• Gmail API Setup - Complete setup guide for Gmail API access
• Gmail IMAP OAuth2 - Alternative setup using IMAP/SMTP
• Google Service Accounts - The service account that manages the Pub/Sub resources
• Monitoring - Prometheus metrics for monitoring
• Webhooks Overview - Understanding EmailEngine webhooks