Transport, Storage, & Event-Driven Processing
Context
The existing implementation streams each extracted artifact into an intermediate document collaboration platform via vendor APIs. A manifest file summarizing extracted records is uploaded last as a completion signal.
Downstream data processing does not run against the document repository directly. It requires retrieving data from that intermediate location into object storage before transformation begins.
Decision
```typescript filename="presigned-upload-flow.ts"
interface JobDescriptor {
jobId: string;
collectionName: string;
/** Short-lived presigned PUT URL issued specifically for this object key */
uploadUrl: string;
/** Expected SHA-256 checksum after streaming */
checksumAlgorithm: "sha256";
}
```
Presigned URL Trust Model
Presigned URLs are issued per job by the server using temporary, least-privilege credentials tied to the exact destination key.
• Bounded Expiry: Presign TTL is constrained by underlying role duration.
• Renewal: The agent can request fresh presigned URLs if transfer rates indicate risk of TTL expiry during large extractions.
Server-Side Completion Verification
The server does not advance a company's stored watermark based on client self-reporting alone. It independently verifies the uploaded object via a HEAD request confirming existence, size, and SHA-256 checksum.
Consequences
• Direct-to-object-storage streaming eliminates intermediate hops.
• Atomic manifest-last pattern guarantees completeness without complex 2PC protocols.
• Server verification prevents corrupted watermark progression.
Alternatives Considered
Continue intermediate repository upload with scheduled retrieval. Rejected: Retains extra hops and credential exposure.
Scheduled daily bucket polling for manifests. Rejected: Adds 24-hour latency floor and scales poorly.
Direct object storage streaming with server verification. Accepted: Fast, reliable, least-privilege architecture.