An artist finishes a marathon ComfyUI session. They have five hundred new images in their output folder. They drag the folder into their asset manager and expect to start browsing within seconds. Behind that expectation sits a processing pipeline that must handle content hashing, metadata extraction, thumbnail generation, and asynchronous enrichment — for every single file — without blocking the artist's workflow or degrading the experience for other operations happening simultaneously.
Part of our AI-Native DAM Architecture
Batch processing patterns address the gap between what users expect (instant availability) and what the system must do (substantial per-file computation). The key insight is that not all processing needs to happen at the same time or at the same priority. By decomposing the ingest pipeline into stages with different urgency levels and managing those stages through priority queues with backpressure controls, the system can serve the artist immediately while completing deeper analysis in the background.
The Forces at Work
- Imports arrive in bursts, not streams: Generative AI work is session-based. An artist generates nothing for hours, then imports five hundred images at once. The processing system must handle extreme spikes without requiring infrastructure that sits idle between sessions.
- Different stages have different urgency: Content hashing and thumbnail generation must complete in seconds so the artist can browse. Metadata extraction should complete in minutes so search works. Embedding generation and quality scoring can take hours without anyone noticing the delay.
- Interactive operations must not wait behind batch operations: While five hundred images are being processed, an artist might upload a single urgent file, run a search, or browse their existing library. These interactive operations must take priority over background batch work, or the system feels broken during imports.
- Failures must not cascade: If one image in a batch of five hundred has corrupted metadata that crashes the parser, the remaining four hundred and ninety-nine images must still process successfully. A single failure should never take down the entire batch.
The Problem
The naive approach to processing a batch of files is a simple loop: for each file, run every processing stage sequentially, then move to the next file. This approach has two catastrophic failure modes at scale. First, the last file in a batch of five hundred waits behind all processing of all four hundred and ninety-nine preceding files — meaning the artist cannot browse their complete import until every single file has been fully processed. Second, any concurrent operation — a search query, a single file upload, a collection edit — must wait behind the entire batch, making the system unresponsive during imports.
Batch Processing Strategies
| Strategy | Throughput | Responsiveness |
|---|---|---|
| Sequential (one at a time) | Low — processes one file fully before starting next | None — all operations blocked during batch |
| Parallel (all at once) | High burst — but overwhelms system resources | Poor — resource contention degrades everything |
| Fixed thread pool | Medium — bounded parallelism | Medium — pool may be exhausted by batch |
| Priority queue with backpressure | High — parallel with rate limiting | Excellent — interactive operations always prioritized |
The goal of batch processing is not to process files as fast as possible. It is to process files as fast as necessary. Thumbnails in seconds, metadata in minutes, embeddings in hours. Matching processing speed to user need is what makes a system feel fast even when doing enormous amounts of work in the background.
The Solution: Tiered Priority Queues
Batch processing decomposes the ingest pipeline into independent stages, each managed by its own priority queue. Work items flow through these queues based on urgency, with interactive operations always receiving higher priority than background batch work.
Stage Decomposition
The processing pipeline splits into three tiers based on how quickly the user needs results. Tier 1 (synchronous) handles content hashing and thumbnail generation — the minimum needed for the asset to appear in the library. Tier 2 (near-real-time) handles metadata extraction and basic indexing — the minimum needed for search to work. Tier 3 (background) handles embedding generation, quality scoring, and session clustering — deep analysis that improves the library progressively. Cost-aware processing determines which assets proceed beyond Tier 2.
Priority Scheduling
Each queue supports multiple priority levels. Interactive operations (a single file upload, a manual re-process request) enter at the highest priority and execute immediately. Batch imports enter at standard priority. Background enrichment runs at the lowest priority, only when higher-priority work is not waiting. This ensures that an artist uploading one urgent file during a five-hundred-file batch import sees their file appear instantly, not after the batch completes.
Backpressure Management
When batch imports arrive faster than the system can process them, backpressure prevents resource exhaustion. The queue tracks its depth and processing rate. When depth exceeds a threshold, the system signals upstream to slow intake — pausing file reading from disk while allowing already-queued items to drain. This prevents memory exhaustion during massive imports while maintaining predictable processing times for items already in the queue.
Failure Isolation
Each work item in the queue is independent. If processing fails for one file — a corrupted image, an unrecognized metadata format, a temporary resource shortage — the failure is logged, the item is moved to a retry queue, and processing continues for all remaining items. After a configurable number of retries, permanently failed items are flagged for manual review. The batch as a whole never fails because of individual item failures.
Progress Reporting
For large batches, the system reports progress in real time: how many files have completed each tier, how many are in progress, how many have failed. This gives the artist confidence that the import is working and an estimate of when full functionality (search, similarity, quality scores) will be available for the imported batch.
Consequences
- Responsive during bulk operations: The priority queue ensures that interactive operations — browsing, searching, single uploads — are never blocked by batch processing. The system feels responsive even during a thousand-file import because interactive work always jumps the queue.
- Progressive availability: Assets become browsable within seconds (Tier 1), searchable within minutes (Tier 2), and fully enriched within hours (Tier 3). The artist does not wait for the slowest stage to complete before they can work with their imported content.
- Queue management complexity: Priority queues with backpressure and retry logic add significant operational complexity compared to a simple processing loop. Queue depth, processing latency, failure rates, and retry counts all need monitoring. Dead letter queues need periodic review.
- Ordering guarantees are relaxed: Because items process in parallel and at different speeds, there is no guarantee that file number one hundred completes before file number two hundred. For most operations this is fine, but features that depend on processing order (like session boundary detection) must handle out-of-order completion gracefully.
Related Patterns
- Ingest Architecture defines the processing stages that batch processing distributes across priority queues.
- Cost-Aware Processing determines which assets proceed to deeper processing tiers and which stop at basic analysis.
- Creative Session Clustering runs as a background batch operation after individual assets are processed.
- When Your Library Hits 10,000 Assets describes the scale at which batch processing becomes essential rather than optional.