Designing Reliable Payment Webhook Ingestion & Processing

Processing payment webhooks inline within an API controller endpoint creates fragile, brittle architectures. If your application database or downstream fulfillment service experiences a minor delay during an incoming webhook request, the gateway’s HTTP request will time out, causing delivery retries, duplicate events, and out-of-order state transitions.

To build payment infrastructure that remains correct under failure, you must decouple Webhook Ingestion (receiving and authenticating) from Webhook Processing (domain state transitions and order fulfillment).

---

1. The Ingestion vs Processing Pipeline Architecture

graph TD
    A[Payment Gateway HTTP POST] --> B[1. Authenticate / Verify HMAC Signature]
    B -->|Invalid| C[Return HTTP 401 Unauthorized]
    B -->|Valid| D[2. Persist Raw Event to Immutable Store]
    D --> E[3. Acknowledge HTTP 200 OK Immediately]
    E --> F[Background Event Processor]
    F --> G[4. Deduplicate Check Event ID / Reference]
    G --> H[5. Execute State Transition & Order Fulfillment]
    H -->|Success| I[Mark Event Processed]
    H -->|Transient Error| J[Retry with Exponential Backoff]
    J -->|Max Retries Exceeded| K[Move to Dead-Letter Queue & Alert Operator]

---

2. Ingestion Step: Raw Event Persistence

Store every incoming webhook payload in an immutable WebhookEvent table before processing it. This ensures you never lose a signed payload even if downstream services crash.

namespace Tofali.Payments.Webhooks;

public enum WebhookProcessingStatus
{
    Received = 1,
    Processing = 2,
    Processed = 3,
    FailedRetryable = 4,
    DeadLetter = 5
}

public sealed class RawWebhookPayload
{
    public Guid Id { get; set; } = Guid.NewGuid();
    public string ProviderId { get; set; } = string.Empty; // "paystack", "flutterwave", "monnify"
    public string EventType { get; set; } = string.Empty;
    public string Reference { get; set; } = string.Empty;
    public string RawJsonBody { get; set; } = string.Empty;
    public string SignatureHeader { get; set; } = string.Empty;
    public WebhookProcessingStatus Status { get; set; } = WebhookProcessingStatus.Received;
    public int RetryCount { get; set; } = 0;
    public string? LastError { get; set; }
    public DateTimeOffset ReceivedAt { get; set; } = DateTimeOffset.UtcNow;
    public DateTimeOffset? ProcessedAt { get; set; }
}

---

3. Dealing with Eventual Consistency & Out-of-Order Delivery

Payment gateways do not guarantee strictly ordered delivery. You may receive a successful payment webhook event BEFORE the user returns to your checkout callback page, or you may receive a transaction reversal event before the original success confirmation.

Architectural Controls for Eventual Consistency:

1. State Machine Guards: Reject invalid state transitions. For example, if a transaction is already Completed, ignore late-arriving Pending or duplicate successful payment webhooks without failing the HTTP response.
2. Transaction Reference Scope: Match events using the cryptographically unique reference or tx_ref key created during payment initialization.
3. Timestamp Validation: Compare event payload timestamps (paid_at or event.createdAt) against your existing transaction records to ensure older events never overwrite newer state.

---

4. Dead-Letter Processing & Poison Messages

When a webhook handler throws an unhandled exception (e.g. database schema mismatch or missing customer record), automatic retries will eventually exhaust their retry quota.

The Dead-Letter Lifecycle:

1. Retry Limit: Attempt processing up to 5 times using exponential backoff (e.g., 1 min, 5 mins, 15 mins, 1 hour, 4 hours).
2. Quarantine (Dead-Lettering): Once retries are exhausted, set Status = WebhookProcessingStatus.DeadLetter and emit a high-priority alert.
3. Operator Inspection & Replay: Build an internal developer tool or administrative CLI that allows engineers to inspect dead-letter payloads, fix underlying configuration or code issues, and trigger Idempotent Event Replay.

namespace Tofali.Payments.Webhooks;

public interface IDeadLetterProcessor
{
    Task MoveToDeadLetterAsync(Guid eventId, Exception exception, CancellationToken cancellationToken = default);
    Task ReplayDeadLetterEventAsync(Guid eventId, CancellationToken cancellationToken = default);
}

---

5. Summary Checklist for Production Webhook Handlers

• Fast HTTP 200 Response: Return HTTP 200 OK within 2 seconds of signature verification.
• Raw Body Verification: Always verify HMAC signatures against raw request stream bytes before JSON parsing.
• Durable Event Storage: Save raw payloads to a database or message queue before attempting business logic.
• Deduplication Key: Use ProviderId + EventType + Reference as a unique processing key.
• Replay Safety: Design all event processing logic to be idempotent so replay commands do not duplicate actions.

---

6. Official Gateway References

• Paystack Webhook Security: https://paystack.com/docs/payments/webhooks/
• Flutterwave Webhook Best Practices: https://developer.flutterwave.com/docs/integration-guides/webhooks/
• Monnify Webhook Documentation: https://docs.monnify.com
• Last Tested Date: August 22, 2026 UTC