Etrans C4

tungntMarch 11, 202628 min read
Export HTMLExport PDF

Email Transactional - System Overview

Architecture documentation following the C4 model — zoom from high-level context down to component details.

C4 Level 1: System Context Diagram

Who uses the system and what external systems does it depend on?

C4 Level 2: Container Diagram

What are the running applications, data stores, and how do they communicate?

C4 Level 3: Component Diagrams

API Server Components

What are the internal modules of the Python API?

Sendmail Worker Components

Fetch Log Worker Components

Data Flow: Send Email (Sequence)

Three Services

ServiceLanguagePortRole
API ServerPython 3.10 / FastAPI8000HTTP endpoints, business logic, publishes to Kafka
Sendmail WorkerGo 1.24-Consumes email requests from Kafka, delivers via SMTP
Fetch Log WorkerGo 1.24-Consumes SMTP delivery logs from Kafka, stores in MongoDB

End-to-End Flow: Sending an Email

StepComponentActions
1ClientSends POST /emails with email payload
2APIValidates auth (Bearer/API key)
Checks quotas (daily/hourly from mkt_domain in Billing DB)
Caches SMTP credentials to Redis (AES-256-CBC encrypted)
Saves EmailMessage record to USEND DB (status: queued)
Publishes email payload to Kafka sendmail topic
3Sendmail WorkerConsumes message from Kafka
Fetches SMTP credentials from Redis (hash auth, key=user_id)
Decrypts credentials with AES-256-CBC
Sends email via SMTP (go-mail library, 5 retries, exponential backoff)
4SMTP ServerDelivers the email, writes delivery logs to disk
5FilebeatParses SMTP logs, publishes structured events to Kafka email-logs topic
6Fetch Log WorkerConsumes log event from Kafka
Filters by event type (Sender/Queue/Deferred/Reject)
Skips internal messages (bizfly.vn pattern)
Inserts log document into MongoDB
7ClientQueries GET /logs to see delivery status
APIReads from MongoDB, maps raw statuses to display labels

Shared Infrastructure

InfrastructureWho WritesWho ReadsShared Config
Kafka (sendmail topic)API (Python)Sendmail Worker (Go)KAFKA_BOOTSTRAP_SERVERS, KAFKA_TOPIC
Kafka (email-logs topic)Filebeat (external)Fetch Log Worker (Go)KAFKA_BOOTSTRAP_SERVERS, LOG_TOPIC
RedisAPI (caches SMTP creds)Sendmail Worker (decrypts creds)REDIS_URI, TOKEN_SECRET, TOKEN_IV
MongoDBFetch Log WorkerAPI (reads logs)MONGODB_URI, MONGODB_DBNAME
Billing DB (PostgreSQL)External / Billing SyncAPI (read-only queries)DB_URI
USEND DB (PostgreSQL)APIAPIUSEND_DB_URI

API Service (Python)

Tech Stack

  • Python 3.10, FastAPI, SQLAlchemy 2.0, Pydantic
  • Clean Architecture: Domain -> App -> Adapters -> API
  • Command Bus pattern for all use cases

Dual Database Architecture

DatabaseConfigAccessTables
Billing DBDB_URIREAD-ONLYdomain, dkim, mkt_domain, users, temporary_password
USEND DBUSEND_DB_URIRead/Writeapi_keys, email_messages, email_transaction_pack, domain_subscription_status, user_pack, domain_tracking_settings, transport

CRITICAL: Never run migrations or modify schema on the Billing DB. It is managed externally.

Authentication

MethodHeaderPermission
Bearer TokenAuthorization: Bearer <JWT>Always full_access
API KeyX-API-KEY: <key>Depends on key's configured level
CookieSession cookiePassthrough

Permission Hierarchy: read_only < sending_access < full_access (higher includes lower)

API Endpoints Quick Reference

GroupMethodPathAuthDescription
EmailsPOST/emailssending_accessSend email
POST/emails/batchsending_accessSend batch emails
GET/emailsread_onlyList emails (paginated)
GET/emails/{message_id}read_onlyGet email details
LogsGET/logsread_onlyGet transaction logs
DomainsPOST/domainsfull_accessCreate domain
GET/domainsread_onlyList domains
GET/domains/{domain_id}read_onlyGet domain details
GET/domains/{domain_id}/quotaread_onlyGet domain quota
POST/domains/{domain_id}/verifyfull_accessVerify domain DNS
PATCH/domains/{domain_id}full_accessUpdate tracking
DELETE/domains/{domain_id}full_accessDelete domain
API KeysPOST/api-keysfull_accessCreate API key
GET/api-keysread_onlyList API keys
GET/api-keys/{key_id}read_onlyGet API key
DELETE/api-keys/{key_id}full_accessDelete API key
BillingGET/billing/subscription/packsread_onlyList packages
GET/billing/subscription/current-packread_onlyGet current pack
POST/billing/subscription/subsfull_accessCreate subscription
GET/billing/payg/packagesread_onlyGet PAYG packages
POST/billing/payg/subscriptionsfull_accessCreate PAYG sub
GET/billing/payg/subscriptions/{domain_id}read_onlyGet current PAYG sub
DELETE/billing/payg/subscriptions/{domain_id}full_accessCancel PAYG sub
SendersPOST/sendersfull_accessCreate SMTP sender
TestingPOST/testing/set-packfull_accessSet pack for testing
HealthGET/healthznoneHealth check

Detailed API Reference

Emails

POST /emails -- Send Email

Permission: sending_access

Request Body:

FieldTypeRequiredDescription
fromstringYes"email@domain.com" or "Name <email@domain.com>"
tostring[]YesRecipient email addresses
subjectstringYesEmail subject
htmlstringNo*HTML body (*at least one of html or text required)
textstringNo*Plain text body
ccstring[]NoCC recipients
bccstring[]NoBCC recipients
reply_tostringNoReply-to address
attachmentsobject[]NoAttachments (see below)
headersobjectNoCustom email headers
tagsobject[]NoTags (name/value pairs)
template_idstringNoTemplate identifier
template_dataobjectNoTemplate rendering data

Attachment object:

FieldTypeRequired
filenamestringYes
contentstringYes (base64-encoded)
content_typestringYes

Response (200):

{
  "id": "<message-id>",
  "from": "sender@example.com",
  "to": ["recipient@example.com"],
  "status": "queued",
  "created_at": "2026-03-10T04:16:54Z"
}

Errors: 400 (no body, invalid attachment, size > 25MB, quota exceeded, domain not verified) | 401 | 403

POST /emails/batch -- Send Batch Emails

Permission: sending_access

Request Body:

FieldTypeRequired
emailsSendEmailRequest[]Yes (same schema as single send)

Response (200):

{
  "data": [{"id": "<message-id>"}],
  "errors": [{"index": 2, "error": "Domain not verified"}]
}

Per-email errors are returned in the errors array, not as HTTP errors.

GET /emails -- List Emails

Permission: read_only

Query Parameters:

ParamTypeDefaultDescription
pageint1Page number (min: 1)
per_pageint20Items per page (1-100)
statusstring-Filter: queued,sent,delivered,bounced,deferred,delayed (comma-separated)
created_at_fromstring-ISO-8601 UTC lower bound
created_at_tostring-ISO-8601 UTC upper bound

Response (200):

{
  "data": [
    {
      "id": "msg_abc123",
      "from": "sender@example.com",
      "to": ["recipient@example.com"],
      "subject": "Hello",
      "status": "Delivered",
      "created_at": "2026-03-10T04:16:54Z"
    }
  ],
  "total": 42,
  "page": 1,
  "per_page": 20,
  "stats": [
    {"label": "Delivered", "count": 30, "percentage": 71.4},
    {"label": "Bounced", "count": 12, "percentage": 28.6}
  ]
}

Notes: Status values in response are display labels (Delivered, Delayed, Bounced, Sent, Queued). Filter accepts both raw (deferred) and display (delayed) names.

Errors: 400 (invalid status, invalid timestamp, from > to, per_page out of range)

GET /emails/{message_id} -- Get Email

Permission: read_only

Response (200):

{
  "id": "msg_abc123",
  "from": "sender@example.com",
  "to": ["recipient@example.com"],
  "subject": "Hello",
  "html": "<html>...</html>",
  "status": "Delivered",
  "created_at": "2026-03-10T04:16:54Z",
  "delivered_at": "2026-03-10T04:16:58Z"
}

Errors: 404 (not found)

Logs

GET /logs -- Get Transaction Logs

Permission: read_only

Query Parameters:

ParamTypeDefaultDescription
pageint1Page number
per_pageint20Items per page (1-100)
log_beforestring-Filter before timestamp (ISO-8601)
log_afterstring-Filter after timestamp (ISO-8601)
actionstring-Filter by raw action (delivered, deferred, rejected)
senderstring-Filter by sender email
recipientstring-Filter by recipient email
message_idstring-Filter by message ID
sourcestring-Filter: api, smtp, or comma-separated
sortsstring-timestampSort field (- prefix = descending)

Response (200):

{
  "logs": [
    {
      "event": "Sender",
      "timestamp": "2026-03-10T04:16:54Z",
      "status": "Delivered",
      "action": "250 2.0.0 OK",
      "action_detail": "250 2.0.0 OK 1772679259 xyz",
      "sender": "hello@example.com",
      "recipient": "user@example.com",
      "from_domain": "example.com",
      "to_domain": "example.com",
      "message_id": "msg_abc123",
      "source": "API"
    }
  ],
  "total": 150,
  "page": 1,
  "per_page": 20,
  "stats": [
    {"label": "Delivered", "count": 100, "percentage": 66.7},
    {"label": "Bounced", "count": 30, "percentage": 20.0},
    {"label": "API", "count": 80, "percentage": 53.3},
    {"label": "SMTP", "count": 70, "percentage": 46.7}
  ]
}

Notes: status is the mapped display label. action is the short SMTP code. action_detail is the full SMTP response. source is "API" or "SMTP" (computed by cross-referencing message_id with USEND DB).

Errors: 400 (invalid source, per_page out of range)

Domains

POST /domains -- Create Domain

Permission: full_access

Request Body:

FieldTypeRequired
domainstringYes

Response (200):

{
  "id": "dom_123",
  "domain": "example.com",
  "status": "pending",
  "dns_records": [
    {"type": "TXT", "name": "@", "value": "v=spf1 include:...", "purpose": "spf"},
    {"type": "MX", "name": "@", "value": "capmx-re30...", "priority": 5, "purpose": "mx"},
    {"type": "TXT", "name": "dkim._domainkey", "value": "v=DKIM1;...", "purpose": "dkim"}
  ],
  "created_at": "2026-03-10T04:16:54Z"
}

Errors: 400 (duplicate domain, domain limit reached)

GET /domains -- List Domains

Permission: read_only

Query Parameters:

ParamTypeDefaultDescription
pageint1Page number
per_pageint20Items per page (1-100)
searchstring""Search domain names

Response (200):

{
  "data": [
    {"id": "dom_123", "domain": "example.com", "status": "verified", "created_at": "..."}
  ],
  "total": 3,
  "total_verified": 2,
  "page": 1,
  "per_page": 20
}
GET /domains/{domain_id} -- Get Domain

Permission: read_only | Path: domain_id in dom_123 format

Response (200):

{
  "id": "dom_123",
  "domain": "example.com",
  "status": "verified",
  "dns_records": [...],
  "verified_at": "2026-03-10T04:16:54Z"
}

Errors: 400 (invalid format) | 404 (not found)

GET /domains/{domain_id}/quota -- Get Domain Quota

Permission: read_only

Response (200):

{
  "domain": "example.com",
  "daily_limit": 500,
  "daily_used": 120,
  "daily_remaining": 380,
  "hourly_limit": 50,
  "hourly_used": 10,
  "hourly_remaining": 40,
  "monthly_limit": 15500,
  "monthly_used": 3600,
  "monthly_remaining": 11900,
  "pack_code": "DEDICATED_100K",
  "pack_name": "Dedicated 100K"
}
POST /domains/{domain_id}/verify -- Verify Domain

Permission: full_access | Request Body: None

Response (200):

{
  "id": "dom_123",
  "domain": "example.com",
  "status": "verified",
  "dns_records": [
    {"type": "TXT", "name": "@", "value": "...", "purpose": "spf", "verified": true},
    {"type": "MX", "name": "@", "value": "...", "purpose": "mx", "verified": true},
    {"type": "TXT", "name": "dkim._domainkey", "value": "...", "purpose": "dkim", "verified": false}
  ],
  "verified_at": "2026-03-10T04:16:54Z"
}

Errors: 400 (DNS verification failed) | 404

PATCH /domains/{domain_id} -- Update Domain Tracking

Permission: full_access

Request Body:

FieldTypeRequired
click_trackingboolNo
open_trackingboolNo

Response (200):

{
  "id": "dom_123",
  "domain": "example.com",
  "click_tracking": true,
  "open_tracking": false
}
DELETE /domains/{domain_id} -- Delete Domain

Permission: full_access | Request Body: None

Response (200):

{"id": "dom_123", "deleted": true}

API Keys

POST /api-keys -- Create API Key

Permission: full_access

Request Body:

FieldTypeRequiredDefaultDescription
namestringYes-Key name
permissionstringNo"full_access"full_access, sending_access, or read_only
expires_in_daysintNonull (never)7, 30, 60, 90, 180, or 365

Response (200):

{
  "id": "key_abc123",
  "name": "My API Key",
  "key": "etrans_live_abc123def456...",
  "permission": "full_access",
  "expires_at": "2026-06-10T04:16:54Z",
  "created_at": "2026-03-10T04:16:54Z"
}

Important: The key field is shown only once at creation. It cannot be retrieved later.

GET /api-keys -- List API Keys

Permission: read_only

Query Parameters: page (default: 1), per_page (default: 20, max: 100)

Response (200):

{
  "data": [
    {
      "id": "key_abc123",
      "name": "My API Key",
      "permission": "full_access",
      "expires_at": "2026-06-10T04:16:54Z",
      "is_expired": false,
      "created_at": "2026-03-10T04:16:54Z"
    }
  ],
  "total": 5,
  "page": 1,
  "per_page": 20
}
GET /api-keys/{key_id} -- Get API Key

Permission: read_only

Response (200):

{
  "id": "key_abc123",
  "name": "My API Key",
  "permission": "full_access",
  "expires_at": null,
  "is_expired": false,
  "active": true,
  "created_at": "2026-03-10T04:16:54Z",
  "updated_at": "2026-03-10T04:16:54Z"
}
DELETE /api-keys/{key_id} -- Delete API Key

Permission: full_access | Request Body: None

Response (200):

{"id": "key_abc123", "deleted": true}

Billing - Subscription

GET /billing/subscription/packs -- List Packages

Permission: read_only

Response (200):

{
  "success": true,
  "packages": [
    {
      "pack_code": "DEDICATED_100K",
      "pack_type": "dedicated_ip",
      "daily_email_limit": 100000,
      "price": 500000.0,
      "is_active": true
    }
  ]
}
GET /billing/subscription/current-pack -- Get Current Pack

Permission: read_only

Response (200):

{
  "pack_code": "DEDICATED_100K",
  "pack_name": "Dedicated 100K",
  "pack_type": "dedicated_ip",
  "daily_email_limit": 100000,
  "domain_limit": 5,
  "quota": {
    "daily_limit": 100000,
    "daily_used": 5000,
    "daily_remaining": 95000,
    "hourly_limit": 4166,
    "hourly_used": 200,
    "hourly_remaining": 3966,
    "monthly_limit": 3100000,
    "monthly_used": 150000,
    "monthly_remaining": 2950000
  },
  "domains": [
    {"domain_id": "dom_123", "domain_name": "example.com", "status": "verified"}
  ]
}
POST /billing/subscription/subs -- Create Subscription

Permission: full_access

Request Body:

FieldTypeRequired
pack_codestringYes
domain_idintYes

Response (200):

{
  "success": true,
  "message": "Subscription created successfully",
  "domain_id": 123,
  "pack_code": "DEDICATED_100K"
}

Errors: 400 | 403 (not Owner, insufficient balance) | 404

Billing - Pay As You Go

GET /billing/payg/packages -- Get PAYG Package

Permission: read_only

Response (200):

{
  "pack_code": "PAYG_SHARED",
  "pack_name": "Pay As You Go",
  "pack_type": "shared_ip",
  "pricing": {}
}
POST /billing/payg/subscriptions -- Create PAYG Subscription

Permission: full_access

Request Body:

FieldTypeRequired
domain_idintYes

Response (200):

{
  "success": true,
  "message": "PAYG subscription created",
  "subscription": {},
  "previous_pack": "DEDICATED_100K"
}
GET /billing/payg/subscriptions/{domain_id} -- Get Current PAYG Subscription

Permission: read_only

Response (200):

{"subscription": {}}

Errors: 404 (no PAYG subscription)

DELETE /billing/payg/subscriptions/{domain_id} -- Cancel PAYG Subscription

Permission: full_access

Response (200):

{"success": true, "message": "PAYG subscription cancelled"}

Senders

POST /senders -- Create Sender

Permission: full_access

Request Body:

FieldTypeRequiredConstraints
accountstringYesEmail address (e.g., noreply@example.com)
domainstringYesDomain name
passwordstringYesmin 8 characters

Response (200):

{
  "id": "sender_123",
  "account": "noreply@example.com",
  "domain": "example.com",
  "active": true,
  "daily_limit": 100,
  "hourly_limit": 4,
  "created_at": "2026-03-10T04:16:54Z"
}

Errors: 400 (sender already exists)

Testing

POST /testing/set-pack -- Set Pack (Testing Only)

Permission: full_access

Request Body:

FieldTypeRequired
pack_codestringYes

Response (200):

{
  "pack_code": "DEDICATED_100K",
  "pack_name": "Dedicated 100K",
  "domain_limit": 5,
  "daily_email_limit": 100000,
  "domains_updated": 3
}

Sets the pack for ALL user domains. Skips billing API and Kafka events. Works even with 0 domains (stores user-level pack assignment).

Errors: 400 (invalid pack_code)

Health Check

GET /healthz

Auth: None

Response (200):

{"status": "ok", "service": "email-transactional"}

Command Bus (19 Use Cases)

DomainCommandsExternal Services
Email SendingSend, Get, ListKafka, Redis, MongoDB
DomainsCreate, Verify, Get, List, Delete, UpdateTracking, GetQuotaBillingSync
BillingGetPacks, GetCurrentPack, CreateSub, PAYG (4 cmds), SetPackExternal Billing API
API KeysCreate, List, Get, DeleteRedis (cache eviction)
SendersCreateRedis, Auth, BillingSync

Business Rules

RuleWhereHow
Domain limitCreateDomainUseCaseChecks pack.domain_limit via domain subscriptions or user_pack fallback
Email quotaSendMailUseCaseChecks mkt_domain.daily/hourly limits
Rate limit trackingmkt_domain tabledaily_tracking/hourly_tracking counters
API key authauth.pySHA-256 hashed keys, plaintext verified on request
SMTP credential cachesend.py -> RedisAES-256-CBC encrypted, stored in Redis hash auth

API Environment Variables

CategoryVariablePurpose
DatabasesDB_URIBilling PostgreSQL (read-only)
USEND_DB_URIMain app PostgreSQL
PROD_BILLING_DB_URIProd billing DB (for staging sync)
KafkaKAFKA_BOOTSTRAP_SERVERSKafka brokers
KAFKA_TOPICSendmail topic (default: sendmail)
KAFKA_BILLING_TOPICBilling events topic
KAFKA_SASL_*SASL authentication
RedisREDIS_URIRedis connection
MongoDBMONGODB_URI / MONGODB_DBNAME / MONGODB_COLLECTIONLog storage
AuthMANAGE_AUTH_URIBizFly auth API
KEYSTONE_*Keystone admin auth
EncryptionTOKEN_SECRET / TOKEN_IVAES-256-CBC for SMTP credentials
BillingBILLING_V4_URIExternal billing API
DNSDEFAULT_SPF_VALUE, DEFAULT_MX_*DNS record templates
MonitoringSENTRY_DSNError tracking
APM_*Elastic APM (currently disabled)

Sendmail Worker (Go)

Purpose

Consumes email send requests from Kafka and delivers them via SMTP.

How It Works

See C4 Level 3: Sendmail Worker Components for the full component diagram.

Processing pipeline: Kafka Consumer → Worker Pool (10 goroutines) → Email Service → Redis (fetch creds) → AES Decrypt → SMTP Send

Key behaviors:

  • Manual Kafka offset commit with back-pressure (blocks until task.Done)
  • 5-minute timeout per email
  • 5 retries with exponential backoff (1s, 2s, 4s, 8s, 16s)
  • TLS opportunistic mode

Kafka Message Format (Input)

{
  "payload": {
    "sender": { "name": "Sender Name", "email": "sender@example.com" },
    "recipients": ["to@example.com"],
    "cc": ["cc@example.com"],
    "bcc": ["bcc@example.com"],
    "subject": "Subject line",
    "body": "<html>Email body</html>",
    "message_id": "<unique-message-id>",
    "attachments": [
      {
        "filename": "file.pdf",
        "content": "<base64-encoded>",
        "content_type": "application/pdf"
      }
    ],
    "user_id": "12345"
  }
}

Environment Variables

VariableDefaultPurpose
KAFKA_BOOTSTRAP_SERVERSrequiredKafka brokers
KAFKA_TOPICrequiredTopic to consume (sendmail)
KAFKA_GROUP_IDrequiredConsumer group ID
KAFKA_SASL_*-SASL authentication
REDIS_HOST / REDIS_PORT / REDIS_DBrequiredRedis for SMTP creds
SMTP_HOST / SMTP_PORTrequiredSMTP server
SMTP_SECUREfalseTLS mode
TOKEN_SECRETrequiredAES-256 key (32 bytes)
TOKEN_IVrequiredAES-256 IV (16 bytes)
WORKER_POOL_SIZE10Concurrent goroutines

Key Files

FilePurpose
sendmail_worker/main.goEntry point, wiring
sendmail_worker/config/config.goEnv var loading + validation
sendmail_worker/config/crypto.goAES-256-CBC decryption
sendmail_worker/kafka/consumer.goSarama consumer with back-pressure
sendmail_worker/kafka/message.goMessage structs
sendmail_worker/worker/processor.goGoroutine pool
sendmail_worker/email/service.goSMTP sending + retry logic
sendmail_worker/redis/client.goCredential fetch from Redis

Fetch Log Worker (Go)

Purpose

Consumes SMTP delivery log events from Kafka (published by Filebeat) and stores them in MongoDB.

How It Works

See C4 Level 3: Fetch Log Worker Components for the full component diagram.

Processing pipeline: Filebeat → Kafka → Consumer → Worker Pool (10 goroutines) → Event Filter → MongoDB Insert

Key behaviors:

  • Manual Kafka offset commit with back-pressure (blocks until task.Done)
  • Default offset: earliest (processes historical events)
  • Only stores: Sender, Queue, Deferred, Reject events
  • Skips messages matching bizfly.vn pattern (internal emails)
  • Auto-creates MongoDB indexes on startup
  • Errors reported to Sentry

Log Event Format (Input from Kafka)

{
  "Event": "Sender",
  "@timestamp": "2026-02-01T10:30:15.000Z",
  "SMTP_Action": "delivered",
  "from": "hello@yourdomain.com",
  "from_name": "Hello",
  "recipient": "user@example.com",
  "to_name": "User",
  "response": "250 2.0.0 OK",
  "from_domain": "yourdomain.com",
  "Event_ID": "evt_123",
  "message_id": "msg_1234567890",
  "to_domain": "example.com",
  "agent": {
    "id": "agent_1",
    "type": "filebeat",
    "hostname": "host1",
    "name": "agent-name",
    "ephemeral_id": "eph_1"
  }
}

Event Types

Event TypeMeaningStored?
SenderEmail accepted by SMTP serverYes
QueueEmail queued for deliveryYes
DeferredDelivery temporarily failed, will retryYes
RejectEmail rejectedYes
OthersVarious SMTP eventsNo (filtered out)

MongoDB Document (Output)

MongoDB FieldSource
eventEvent
timestamp@timestamp
actionSMTP_Action
senderfrom
recipientrecipient
responseresponse (full SMTP response)
from_domainfrom_domain
message_idmessage_id
to_domainto_domain
event_idEvent_ID
agent_*Agent metadata (type, name, hostname)

MongoDB Indexes

Index NameFieldsPurpose
idx_from_domain_timestampfrom_domain ASC, timestamp DESCLog queries filtered by domain
idx_message_idmessage_idSingle email log lookup
idx_event_timestampevent ASC, timestamp DESCEvent type filtering

Environment Variables

VariableDefaultPurpose
KAFKA_BOOTSTRAP_SERVERSrequiredKafka brokers
LOG_TOPICrequiredTopic to consume (email-logs)
LOG_GROUP_IDrequiredConsumer group ID
KAFKA_SASL_*-SASL authentication
MONGODB_URIrequiredMongoDB connection
MONGODB_DBNAMErequiredDatabase name
MONGODB_COLLECTIONrequiredCollection name
LOG_WORKER_POOL_SIZE10Concurrent goroutines
LOG_PROCESS_TIMEOUT_SECONDS10Processing timeout
LOG_MONGO_INSERT_TIMEOUT_SECONDS10Insert timeout
STORE_EVENT_TYPESSender,Queue,Deferred,RejectWhich events to store
SKIP_MESSAGE_ID_PATTERNbizfly.vnSkip internal messages
SENTRY_DSN-Error tracking

Key Files

FilePurpose
main.goEntry point, wiring
config/config.goEnv var loading + filter config
kafka/consumer.goSarama consumer with back-pressure
worker/pool.goGoroutine pool + event filtering
events/log_sender.goLogEvent struct + MongoDB document conversion
mongo/client.goMongoDB insert + index management
observability/sentry.goCustom Sentry error reporting

Deployment

Docker Services

ServiceDockerfileExposed Port
email_transaction_api./Dockerfile8000
sendmail_workerfetch_log_worker_go/sendmail_worker/Dockerfile-
fetch_log_workerfetch_log_worker_go/Dockerfile-

Git Repositories

RepoPathRemote
API (root)/email-transactional/git@git.paas.vn:devteam/vce/email-transactional.git
Combined Workers/email-transactional/fetch_log_worker_go/git@git.paas.vn:devteam/vce/etrans-fetch-log-worker.git

The standalone sendmail_worker/ repo is deprecated. Source of truth is fetch_log_worker_go/sendmail_worker/.

Status Mapping (API Display Labels)

The API maps raw SMTP statuses to user-friendly labels:

Raw Status (DB/Logs)Display LabelSource
queuedQueuedAPI sets on send
sender / sentSentSMTP accepted
delivered / deliveryDeliveredSMTP delivered
deferredDelayedSMTP temporary failure
bounced / reject / rejectedBouncedSMTP permanent failure

Filters on GET /emails accept both raw and display names (delayed -> deferred).

State Machine Diagrams

Email Lifecycle

Domain Lifecycle

API Key Lifecycle

Subscription Lifecycle

Sender Lifecycle

Email Sending Flow (Detailed State Machine)

Domain Creation with Limit Check