1. Context: High-Volume Verification

The primary objective was to process 5 million unique emails across concurrent jobs within a 12-hour window, using limited server resources, whilst maintaining an 80% success rate. The vision was to transform a resource-intensive, expensive process into a lean, automated commodity without sacrificing accuracy.

2. Technical Constraints: Scaling SMTP

The primary friction point was "adversarial engineering". Sending 5 million SMTP handshakes in a short burst is functionally indistinguishable from a DDoS attack. Standard approaches lead to immediate IP blacklisting by major providers (Gmail, Outlook) who obfuscate their blocking logic.

We identified three core technical barriers:

1. Anti-Spam Triggers: Using proxies or VPNs is ineffective as port 25 is blocked by default, and IP reputation is generally poor. Furthermore, mail servers flag mismatches between IPs and PTR records instantly.
2. Resource Asymmetry: A naive queue system storing 1 job per email creates massive memory spikes. Conversely, processing a single 4-million-email payload risks timeouts and crashes.
3. The SQLite Concurrency Bottleneck: We chose SQLite for its operational simplicity and backup capabilities, but it is typically single-threaded for writes. Handling high-concurrency updates without locking the database (the SQLITE_BUSY error) at this volume required specific architectural adjustments.

3. Hardware Selection

To fulfil the "lean automated commodity" vision, we architected a tiered infrastructure designed to maximise output while maintaining a minimal monthly burn.

I. Main Server

Role: Central API, SQLite Database Host, Redis Queue Manager, Heuristic Filtration Engine.
Selection Logic: The bottleneck here is Disk I/O (for SQLite writes) and RAM (for Redis queues and Node.js heap), not CPU. We selected a mid-range instance with high-performance storage to handle the Write-Ahead Logging (WAL) without locking.

Component	Specification	Purpose
Instance Type	CCX23 Hetzner VPS	Cost-effective balance of Compute/RAM.
CPU	4 vCPUs	Sufficient for regex filtration and managing concurrent Axios requests.
RAM	8 GB	Dedicated 2GB for Redis (Queue), 4GB for Node Heap, remainder for OS/Cache.
Storage	160 GB NVMe SSD	Critical: High IOPS required for SQLite WAL mode to handle concurrent writes without SQLITE_BUSY errors.
Cost	~$28 - $30 / month	Significantly cheaper than managed DB solutions.

II. MTA Cluster Nodes

Role: SMTP Handshakes, IP Rotation, "Warm-up" execution.
Selection Logic: These nodes are treated as infrastructure-as-code and are disposable. The value is not the hardware, but the unique IPv4 address. We use the smallest possible instance size that can run a Node.js runtime, allowing us to maximise the number of unique IPs within the budget. We selected DigitalOcean as the provider for this deployment because Hetzner VPS IPs suffered from reputation issues and no smaller alternative to the 2GB RAM instance was available.

Component	Specification	Purpose
Instance Type	DigitalOcean Basic Droplet	Minimal footprint
CPU	1 vCPU	Only requires simple socket handling (SMTP handshakes).
RAM	1 GB	Sufficient to buffer 1,000 email chunks in memory.
Cost	~$5 - $6 / node / month	Allows linear scaling. 10 more nodes = only $50/mo increase.

4. Implementation: Distributed Infrastructure

To address these constraints, we designed a system that relies on distributed orchestration and intelligent preprocessing rather than brute force.

The Distributed MTA Cluster

We determined that a standard proxy gateway was insufficient. Instead, we built a bespoke cluster of small VPS nodes acting as Mail Transfer Agents (MTAs). By controlling the VPS, we could set correct PTR records (mapping IPs to domains) to satisfy strict sender verification protocols. We implemented a Node.js dispatcher on these nodes to manage "warm-up" algorithms, gradually ramping sending limits from 1,000 to 10,000 emails per day to emulate organic traffic patterns.

Architecture of the Distributed MTA Cluster.

// Main Server Orchestration
for (let i = 0; i < emails.length; i += BATCH_SIZE) {
    const batch = emails.slice(i, i + BATCH_SIZE);
    // Round-robin selection ensuring even load distribution
    const workerUrl = WORKER_NODES[activeWorkerIndex];
    activeWorkerIndex = (activeWorkerIndex + 1) % WORKER_NODES.length;

    const p = axios.post(`${workerUrl}/process-batch`, {
            batchId,
            emails: batch
        }, { headers: { 'x-api-key': 'secret-agent-key' } })
        .then(response => handleSuccess(response))
        .catch(err => handleFailure(err));

    pendingJobs.push(p);
}

The Validation Perimeter

To protect the SMTP layer, we engineered a multi-stage validation pipeline. Before an SMTP handshake is even attempted, the email passes through regex verification, direct username blacklists, and a heuristic pattern matcher designed to identify disposable accounts (e.g., inconsistencies like multiple periods or symbol abuse). We also implemented a Domain Intelligence layer to categorise TLDs.

export const usernameBlacklist = [
	'0000',
	'1111',
	'123',
	'1234',
	'12345',
	'123456',
	'12345678',
    ...
]
const blacklistedPatterns = [
    /^test[0-9]*$/,
    /^user[0-9]*$/,
    /^(asdf|qwer|zxcv|1234)[a-z0-9]*$/,
]
export function analyzeUsername(username: string): HeuristicResult {
	...
	if (COMMON_NAMES.includes(lower)) { ... }
	if (checkRepeatedChars(username)) { ... }
	if (checkKeyboardWalk(username)) { ... }
	if (checkNumericHeavy(username)) { ... }
	if (checkVowelConsonantRatio(username)) { ... }
	const entropy = calculateShannonEntropy(username)
	if (entropy > 4.2) { ... }
	...
}

Optimising the Monolith (SQLite WAL Mode)

While MySQL or PostgreSQL are more traditional choices for high-concurrency tasks, we selected SQLite for its zero-latency local socket connections and reduced network overhead. To solve the concurrency issue without abandoning SQLite, we enabled Write-Ahead Logging (WAL) mode. This allows simultaneous readers and writers by appending changes to a log file before merging. We coupled this with atomic updates (UPDATE jobs SET count = count + ?) and a dynamic micro-batching strategy. The batch size is calculated as a fraction of the total volume, ensuring the system remains responsive without blocking the UI, even during 5-million-record stress tests.

const sqlite = new Database(process.env.DB_FILE_NAME);
sqlite.pragma('busy_timeout = 5000'); // Queue workers rather than failing
if (process.env.NODE_ENV === 'production') sqlite.pragma('journal_mode = WAL');

5. Results: Performance & Efficiency

The transition from abstraction to reality was validated through rigorous stress testing.

The final verification dashboard showing processed lists and success rates.

• Validation Efficiency: The heuristic and domain intelligence layers successfully resolved 80% of incoming lists without ever touching the SMTP layer. This dramatically reduced the infrastructure load and cost per verification.
• Performance Metrics: The system successfully processed a 5-million email payload, distributed across 5 concurrent jobs, in under 12 hours.
• Resilience: During the stress test, the system experienced 0% IP blacklisting and zero crashes. The warm-up algorithms successfully evaded provider automated blocks. Inevitable occurrences, such as greylisting and soft bouncing, are managed by an intelligent retry system with a maximum attempt threshold.
• Caching Strategy: We implemented a 24-hour caching window for results and a tracker for high-frequency emails, ensuring instant verification for recurring checks and further reducing external network requests.

6. Unit Economics and Infrastructure ROI

The primary value of this architecture is reflected in the unit economics. By resolving 80% of the 5-million-email list via local heuristics and caching, we reduced the expensive SMTP workload to just 1 million records.

Managed services typically charge per credit regardless of validation complexity. For a 5-million-record batch, industry benchmarks range from $4,500 on budget-focused platforms (e.g. DeBounce at $0.0009/credit) to over $20,000 on enterprise-grade providers such as ZeroBounce or NeverBounce (~$0.004/credit).

In contrast, this custom infrastructure operates at a fixed cost of ~$102/month. The system operates on a capacity-based model rather than a transaction-based one. The volume of queries is immaterial, provided it remains within the processing capacity. Based on the infrastructure above, the pre-SMTP layer can handle approximately 2,000–3,000 emails per second. Conversely, for emails requiring deep checks (including SMTP), the server processes 15–20 emails per second (aggregated across the cluster).

By shifting from a variable "per-unit" expense to a flat overhead, the system achieved an ROI of over 7,000% per batch while allowing for unlimited re-runs.

7. Future Roadmap: BEAM Migration

The current architecture delivers exceptional ROI for the stakeholders' current growth phase. However, as the platform scales to tens of millions of users, we are preparing for the next evolution.

Phase 2: The BEAM Migration

The current system is capable of supporting our strategic partners over the long term and handling millions of records. As the platform scales towards 10 million users, the roadmap includes a pivot to Elixir and the BEAM VM. This technology stack is uniquely suited for massive concurrency and fault tolerance. By leveraging Elixir, we will be able to scale horizontally, utilising every ounce of the VPS cluster's CPU power for real-time processing.

8. Stakeholder Feedback

The project concluded with high stakeholder satisfaction, specifically highlighting the proactive design improvements and technical execution.

I am very impressed with this project. You always work hard and very well but this job is, in my opinion, your best one so far. It has been executed perfectly. You took your own initiative to add elements, you added design that made it even better in my opinion. You even added some details that make this system better. This is by far the best project you did... You are improving your skills very fast. — Project Lead