Batch Requests in Python for an OpenAI-Compatible LLM API
For llm api batch python work, you do not need a special batch endpoint: you need a worker pool that respects the rate limits, a retry rule for HTTP 429, and a results file you can resume from. This guide builds all three with the OpenAI SDK pointed at your base URL.

Key points
- The API has two methods, chat completions and models, so batching is done on your side with a pool of concurrent requests.
- A full key allows 8 concurrent requests and 300 per minute; a trial key allows 2 and 60. Size your pool to match.
- Retry on 429 using the Retry-After header, and tell rate_limited apart from concurrency.
- Write each result to a JSONL file as it finishes so a crash costs minutes, not the whole run.
How batching works here
The endpoint is OpenAI-compatible and exposes POST /v1/chat/completions and GET /v1/models. There is no separate job-submission endpoint that takes a file and returns later. A batch is therefore a loop over normal requests, run concurrently, with the limits built into your code. That is simpler than it sounds, and it gives you results as they finish instead of at the end.
The numbers that shape the design are the limits. A full key allows 8 concurrent requests and 300 requests per minute. A trial key allows 2 concurrent and 60 per minute. The maximum request body is 8 MB. A streamed request holds its concurrency slot until the stream ends, so for batch work, leave streaming off.
Why not just fire everything at once? Because a batch is a throughput problem with a ceiling, and the ceiling is yours to respect. A loop that sends 5,000 requests as fast as Python can create them gets a wall of 429 responses, wastes time on retries, and buries the useful errors in noise. A pool that holds at most 8 requests in flight keeps the line moving at the speed the service allows, and it is the same code whether you have 50 rows or 500,000. Plan for three pieces: a bounded pool, one retry rule, and an append-only results file.
The model ID is uncensored, authentication is Authorization: Bearer <key>, and the base URL is https://api.unfilteredvideo.com/v1. If you have not got a key yet, the API key page has a one-click $0.50 trial.
Install and set up
You need Python 3.9 or newer and the official openai package. Put the key in an environment variable rather than in the script.
pip install openai
export API_KEY="paste-your-key-here"
# quick check that the key and base URL work
curl https://api.unfilteredvideo.com/v1/models -H "Authorization: Bearer $API_KEY"
The model list should show uncensored. Now prepare your input as a JSONL file, one object per line with an id and the text to process. The id is what lets you resume later.
{"id": "r001", "text": "Summarise this product review in one sentence: ..."}
{"id": "r002", "text": "Summarise this product review in one sentence: ..."}
A thread pool that stays inside the limits
For most scripts a ThreadPoolExecutor is the shortest path. The OpenAI client is safe to share between threads, and the work is network-bound, so threads fit. Set the pool size to your concurrency limit, not higher: extra workers just collect 429 errors.
import json, os
from concurrent.futures import ThreadPoolExecutor, as_completed
from openai import OpenAI
client = OpenAI(api_key=os.environ["API_KEY"], base_url="https://api.unfilteredvideo.com/v1", max_retries=0)
MODEL = "uncensored"
WORKERS = 8 # use 2 on a trial key
def run_one(row):
resp = client.chat.completions.create(
model=MODEL,
temperature=0.2,
max_tokens=200,
messages=[{"role": "user", "content": row["text"]}],
)
return {
"id": row["id"],
"output": resp.choices[0].message.content,
"in": resp.usage.prompt_tokens,
"out": resp.usage.completion_tokens,
}
rows = [json.loads(line) for line in open("input.jsonl")]
with ThreadPoolExecutor(max_workers=WORKERS) as pool, open("output.jsonl", "a") as out:
futures = [pool.submit(run_one, r) for r in rows]
for f in as_completed(futures):
out.write(json.dumps(f.result()) + "\n")
out.flush()
This version has no retry and no resume, so it is a starting point. The client is created with max_retries=0 on the client because the SDK retries on its own by default, and you want one place that owns the retry rule. The next two sections add that rule and the resume logic.
Handle 429 the right way
There are three kinds of 429 on this API, and they call for slightly different reactions:
rate_limited: you sent too many requests in a minute. Slow down. A pool with fewer workers, or a short sleep, fixes it.concurrency: too many requests were in flight at once. Your pool is bigger than your limit, or you have open streams you forgot about.- Temporary overload: a 429 with a
Retry-After: 5header. Nothing is wrong with your code. Wait the stated time and try again.
One retry helper covers all three. It waits for Retry-After when present and otherwise backs off exponentially with a little jitter. Failed and refused requests are not charged, so retrying is free.
import random, time
import openai
from openai import OpenAI
client = OpenAI(api_key=__import__("os").environ["API_KEY"], base_url="https://api.unfilteredvideo.com/v1", max_retries=0)
MODEL = "uncensored"
def call_with_retry(messages, attempts=6):
for n in range(attempts):
try:
return client.chat.completions.create(model=MODEL, messages=messages, max_tokens=200)
except openai.RateLimitError as e:
wait = e.response.headers.get("Retry-After")
delay = float(wait) if wait else min(60, 2 ** n) + random.random()
print("429", getattr(e, "code", None), "waiting", round(delay, 1), "s")
time.sleep(delay)
except (openai.APIConnectionError, openai.APITimeoutError):
time.sleep(min(30, 2 ** n))
raise RuntimeError("gave up after %d attempts" % attempts)
Do not retry blindly on every error. A 400 means your request is wrong, for example a body over 8 MB or a malformed message, and sending it again will fail the same way. Log it with the row id and move on.
An async version with a semaphore
If your script already runs in an event loop, or you want to feed a web service, use AsyncOpenAI and an asyncio.Semaphore that caps in-flight requests. The semaphore plays the role of the pool size.
import asyncio, json, os
from openai import AsyncOpenAI
client = AsyncOpenAI(api_key=os.environ["API_KEY"], base_url="https://api.unfilteredvideo.com/v1", max_retries=0)
MODEL = "uncensored"
sem = asyncio.Semaphore(8)
async def run_one(row, out):
async with sem:
resp = await client.chat.completions.create(
model=MODEL,
max_tokens=200,
messages=[{"role": "user", "content": row["text"]}],
)
out.write(json.dumps({"id": row["id"], "output": resp.choices[0].message.content}) + "\n")
out.flush()
async def main():
rows = [json.loads(l) for l in open("input.jsonl")]
with open("output.jsonl", "a") as out:
await asyncio.gather(*(run_one(r, out) for r in rows))
asyncio.run(main())
Wrap the call in the same retry logic as above, using await asyncio.sleep instead of time.sleep. One caution: gather with tens of thousands of coroutines creates them all at once. For very large inputs, process the file in slices of a few thousand rows.
Make the run resumable
Long jobs fail in boring ways: a laptop sleeps, a connection drops, you press Ctrl+C. Because each finished row is appended to output.jsonl with its id, resuming means skipping ids you already have.
import json, os
from openai import OpenAI
client = OpenAI(api_key=os.environ["API_KEY"], base_url="https://api.unfilteredvideo.com/v1")
MODEL = "uncensored"
def done_ids(path="output.jsonl"):
if not os.path.exists(path):
return set()
return {json.loads(line)["id"] for line in open(path) if line.strip()}
finished = done_ids()
todo = [r for r in map(json.loads, open("input.jsonl")) if r["id"] not in finished]
print(len(finished), "done,", len(todo), "to go")
Feed todo to either pool above. Open the output file in append mode, never write mode, or a restart will erase your progress. Also write errors to a second file, errors.jsonl, so you can rerun only the failures after reading why they failed.
For repeatable output across reruns, pass a fixed seed along with a low temperature. The seed is passed through to the model along with top_p, stop and the penalties.
Test on a small slice first
Run 20 rows before you run 20,000. Pick rows that include your worst cases: the longest input, one with odd characters, one that is empty. Read the outputs by eye, check that usage matches your cost estimate, and confirm the retry helper behaves by temporarily raising the worker count above your limit. You should see a 429 with the code concurrency in the log, a wait, and a successful retry. It is better to see that on 20 rows than discover it at row 9,000.
If the job needs structured output, add JSON mode with response_format={"type": "json_object"}, describe the keys in the prompt, and parse each reply inside a try block. Count parse failures during the test slice. A rate above a percent or two usually means the prompt needs a clearer schema, and that is cheaper to fix now than to re-run later. Rows that fail to parse go to errors.jsonl with their id, and the resume logic treats them as not done.
Finally, set a client timeout that fits your longest expected output. The SDK accepts timeout=60 on the client or per request; a stuck request that never returns will otherwise hold one of your 8 slots for as long as it sits there.
Sizing the pool and estimating time
Throughput is bounded by two things: concurrency divided by the time one request takes, and the requests-per-minute cap. On a full key the cap is 300 per minute, which is 5 per second. If a typical request takes 2 seconds, 8 workers produce 8 / 2 = 4 requests per second, or 240 per minute, which is under the cap. If requests take 1 second, 8 workers would try for 8 per second and hit the cap, so add a small delay or accept occasional 429s that your retry helper absorbs.
At 4 per second, 10,000 rows take 2,500 seconds, about 42 minutes. On a trial key, with 2 concurrent requests and 60 per minute, the ceiling is 1 per second, so the same 10,000 rows would take 10,000 seconds, nearly 2 hours 47 minutes. In practice the $0.50 trial credit will run out before that, which makes it a good place to test the script on a few hundred rows and not a place to run production.
What a large batch costs
Pricing is $0.15 per 1M input tokens and $0.50 per 1M output tokens. Take a classification-style job: 50,000 rows, each about 200 input tokens and 100 output tokens.
- Input: 50,000 × 200 = 10,000,000 tokens, which is 10 × $0.15 = $1.50.
- Output: 50,000 × 100 = 5,000,000 tokens, which is 5 × $0.50 = $2.50.
- Total: $4.00, or $0.00008 per row.
The script above records in and out for each row, so you can compute the same figure from your real run. The X-Balance-USD response header also shows remaining credit, and X-Request-Id is what to quote if you ask for help on the docs or support pages.
Credit is prepaid in whole-dollar top-ups from $20 to $500, with +5% from $50, +10% from $100 and +15% from $200, paid in crypto (USDT on TRC20 or USDC on Base). It never expires. The pricing page has the full detail. One note for sensitive datasets: prompts are not used for training.
More guides
Questions and answers
Does the API have a batch endpoint?
No. It exposes chat completions and a model list. You batch by running many normal requests concurrently from your own code, within 8 concurrent and 300 per minute on a full key.
How many threads should I use?
Use 8 workers on a full key and 2 on a trial key, matching the concurrency limits. More workers only produce 429 errors with the code concurrency.
What should I do when I get HTTP 429?
Wait for the Retry-After value if the response has one, otherwise back off for a few seconds, then retry. Failed requests are not charged, so retrying costs nothing.
Can I stream responses in a batch job?
You can, but you should not. A streamed request holds its concurrency slot until the stream ends, which lowers throughput. Leave streaming off for batch work.
Your key is one form away
Create an account, copy the key, change the base URL. That is the whole setup.