What is batch processing in AI? Explained with a print shop's drop-off tray
Batch processing means sending an AI service many requests at once and collecting the answers later, for a lower price. Explained with a print shop's drop-off tray.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
This week's model launches from OpenAI, Google and Alibaba all list whether "batch" is supported, and that one word can halve the bill for the right kind of AI job.
In one sentence: Batch processing means sending an AI service a large set of requests in one go and collecting all the answers later, usually within 24 hours, in return for a much lower price than asking one question at a time and waiting for each reply.
The everyday picture
Picture a busy print shop. At the front counter you can wait while your job is printed. It is quick, but you pay the full price, and the shop must keep machines free for people who walk in.
The shop also has a drop-off tray. You leave a whole box of jobs, each with its own label, and come back for them by tomorrow. The shop prints them when its machines would otherwise sit idle, so it charges you less.
That tray is batch processing. The counter is the normal, real-time way of using an AI model. The box is your batch of requests, and each label is a request ID that tells you which answer belongs to which question.
Your software drops a labelled batch of requests in the tray (the Batch API). The model works through it when it has room, the answers go into a results file, and each one is matched back to its request by its label.
How it works, step by step
- You prepare the batch (pack the box). Your software lists every request, such as thousands of support emails to sort, and gives each one its own ID (a label).
- You send the whole batch to the Batch API (leave the box in the tray). Depending on the provider, this is one file or one list in a single call. You get a batch number straight away, not answers.
- The provider works through it when it has room (the shop prints in quiet hours). Each request is handled on its own by the same models used for normal requests. The promise is to finish within a set window, often 24 hours, and many batches finish much sooner.
- The answers are collected in a results file (the finished box). Any request that failed or ran out of time is marked as such, so you can send it again.
- Your software matches answers to questions by ID (reads the labels). Results may come back in a different order from the one you sent, so the ID is what links each answer to its request.
Try it · What would you do?
Which job belongs in the drop-off tray?
A small online shop uses an AI model for several jobs. It wants to move one of them to batch processing to cut costs. Which is the best fit?
Pick what you would do. You can change your mind.
Where the picture stops
- A print shop prints the same thing every time. An AI model may word an answer differently on each run, batch or not.
- You cannot stand at the tray and watch. You check the batch's status, or ask to be told when it is done, then download the results.
- The shop's discount is a favour. With AI providers, the batch price and time window are published terms, and they differ by provider and model.
Why it matters for your business
- Lower cost for big jobs. At the time of writing, OpenAI, Anthropic and Google's Gemini API each charge batch requests at half their standard price.
- Room for large volumes. Batch jobs run under their own limits, separate from normal real-time use, so a one-off clean-up of an old archive does not compete with your live chatbot for capacity.
- Not for anything a customer is waiting on. A live chat reply or a check during checkout needs the normal, real-time route.
- Check before you choose a model. Not every model supports batch. Our Qwen4-Max story notes that batch processing is not offered for that model, so a cheaper list price may not be cheaper for bulk work.
An example
Imagine a 30-person insurance broker with five years of customer emails. They want each email tagged by topic, such as claims, renewals or complaints, to see where time goes. Nobody needs the tags this minute. Their developer sends all the emails as one batch on Friday afternoon, each labelled with the email's ID. By Saturday the results file is ready. The software matches each tag to its email, re-sends the few that failed, and the broker pays the batch rate instead of the standard one.
Frequently asked questions
Are batch answers worse than normal answers?
No. The providers' guides describe batch as a different way of sending requests, not a weaker model. Each request goes to the same model, with the same settings, as a normal call would. You trade waiting time for price.
How long does a batch take?
It depends on the provider and how busy it is. OpenAI and Google describe a 24-hour target, often quicker. Anthropic says most batches finish within an hour, and anything not done in 24 hours expires.
What happens if some requests fail?
The results show which requests succeeded and which failed or expired, by ID. You resend only those. Each provider's guide explains how failed or expired requests are billed.
Can I combine batch with prompt caching?
With some providers, yes. Anthropic and Google document using cached prompts inside batches, which can cut costs further when many requests share the same long instructions.
Related explainers
The DevDrift builds AI automation that picks the right way to run each job, from live replies to overnight batches: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.