Batch jobs

Submit large collections of Serverless inference requests as named batch jobs, then monitor asynchronous processing and retrieve results.

Batch jobs let you run large volumes of inference requests against a queue-based Serverless endpoint without waiting for each result in real time. Batches run asynchronously on dedicated workers that are separate from your endpoint’s standard /run traffic. A large batch never takes capacity from your interactive requests.

Requirements

• A queue-based Serverless endpoint. To create one, see the Serverless quickstart. Batch jobs aren’t available on load balancing endpoints.
• A Runpod API key, if you use the API.

When to use batch jobs instead of /run

The following table compares batch jobs with standard /run requests:

	Batch	/run
Use case	Bulk, offline workloads	Interactive, real-time inference
Latency	Multi-hour	Seconds to minutes
Traffic isolation	Dedicated batch workers	Standard Serverless workers
Result delivery	Poll or subscribe to notifications	Synchronous or async poll

Use batch when your workload can tolerate multi-hour latency. Common examples include:

• Nightly dataset processing.
• Pre-computing embeddings.
• Running evaluations.

Keep using /run or /runsync for any request a user is waiting on.

Batch requests are billed at a discounted rate compared to on-demand Serverless pricing. There are separate discount tiers for consumer and enterprise GPUs. For discount rates, contact sales. To estimate cost, run a small test batch (for example, 100 requests) and extrapolate.

Batch lifecycle

A batch moves through the following API states:

DRAFT → FINALIZED → COMPLETED
                  → FAILED
                  → CANCELLED

• DRAFT: The batch is a draft, and you can still change its requests. Batch workers haven’t started any work.
• FINALIZED: The batch is locked, and you can’t add or remove requests. Batch workers process the requests while the batch stays in this state. Track progress with the requestTotal, requestInProgress, requestCompleted, and requestFailed counts. When requestCompleted + requestFailed equals requestTotal, the batch moves to COMPLETED.
• COMPLETED: All requests in the batch have finished. Some individual requests might have failed, so check requestFailed.
• FAILED: The batch itself failed before or during execution. This is different from individual request failures in a batch whose other requests finished.
• CANCELLED: You cancelled the batch. For details, see Cancellation.

A batch doesn’t start processing until you finalize it. A DRAFT batch never runs.

The console shows different status labels. For the mapping, see Console statuses.

API walkthrough

The following steps create a batch, finalize it, check its progress, and read the results. Replace ENDPOINT_ID, API_KEY, and BATCH_ID with your own values.

Step 1: Create a batch

Create a batch with an initial list of requests:

curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '[
    { "input": { "text": "The quick brown fox" } },
    { "input": { "text": "Jumped over the lazy dog" } }
  ]'

The request body is a top-level JSON array. Each element uses the same shape as a standard /run request: a JSON object with an input field. To create an empty batch and add requests later, send an empty array ([]).

The response includes the batch ID, which you use in the remaining steps:

{
  "id": "5af6bdb3-551f-4195-8872-bad522efbad7",
  "status": "DRAFT"
}

Step 2: Add more requests (optional)

While the batch is a draft, you can append more requests:

curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/requests \
  -H "Authorization: Bearer API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "requests": [
      { "input": { "text": "More text to embed" } },
      { "input": { "text": "Another piece of text" } }
    ]
  }'

Each call accepts up to 10 MiB. To build a large batch, call this endpoint as many times as you need.

Step 3: Finalize the batch

Finalize the batch to lock its requests and make it eligible to run:

curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/finalize \
  -H "Authorization: Bearer API_KEY"

The batch status changes to FINALIZED. You can no longer add or remove requests.

Step 4: Check batch progress

Fetch the batch summary to check progress:

curl https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID \
  -H "Authorization: Bearer API_KEY"

The response shows the batch status and request counts:

{
  "id": "5af6bdb3-551f-4195-8872-bad522efbad7",
  "endpointId": "abc123xyz",
  "status": "FINALIZED",
  "requestTotal": 1000,
  "requestInProgress": 8,
  "requestCompleted": 244,
  "requestFailed": 6,
  "createdAt": 1783584000000
}

This batch is still processing: 244 of 1,000 requests are complete and 6 have failed. A batch reports FINALIZED while its requests are processing. When all requests have finished (requestCompleted + requestFailed equals requestTotal), the status changes to COMPLETED. To check whether a batch is done, use the status field. If the batch itself fails, the status is FAILED, and if you cancel it, the status is CANCELLED. The createdAt field is a Unix timestamp in milliseconds.

You can poll the batch summary at whatever interval suits your workflow, or use notifications instead.

Step 5: Retrieve results

Fetch the results for every request in the batch:

curl "https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/requests?offset=0&limit=50" \
  -H "Authorization: Bearer API_KEY"

The response lists each request with its status and its output or error:

{
  "requests": [
    {
      "id": "req_abc001",
      "status": "COMPLETED",
      "output": { "embedding": [0.12, 0.34, ...] },
      "delayTime": 1200,
      "executionTime": 2000
    },
    {
      "id": "req_abc002",
      "status": "FAILED",
      "error": "Handler raised an exception: timeout exceeded",
      "delayTime": 1500,
      "executionTime": 9000
    }
  ],
  "total": 1000,
  "offset": 0,
  "limit": 50,
  "hasMore": true
}

Each request includes delayTime (how long the request waited in the queue) and executionTime (how long it took to run), both in milliseconds.

Results are paginated. Use the offset and limit query parameters to page through them. When hasMore is true, more pages remain.

API reference

Use the following endpoints to manage batches:

Method	Path	Description
POST	/v2/ENDPOINT_ID/batch	Create a batch, optionally with initial requests.
POST	/v2/ENDPOINT_ID/batch/BATCH_ID/requests	Append requests to a draft batch.
POST	/v2/ENDPOINT_ID/batch/BATCH_ID/finalize	Lock the batch and make it eligible to run.
PUT	/v2/ENDPOINT_ID/batch/BATCH_ID	Update batch attributes, such as the display name.
DELETE	/v2/ENDPOINT_ID/batch/BATCH_ID/requests/REQUEST_ID	Remove a single request from a draft batch.
GET	/v2/ENDPOINT_ID/batch	List all batches for an endpoint, newest first.
GET	/v2/ENDPOINT_ID/batch/BATCH_ID	Get the batch summary with request counts.
POST	/v2/ENDPOINT_ID/batch/BATCH_ID/cancel	Cancel a batch.
GET	/v2/ENDPOINT_ID/batch/BATCH_ID/requests	List the batch’s requests, paginated.

Monitor batches in the console

To see all batches for an endpoint, open the endpoint in the Runpod console and select the Bulk Requests tab. Each row shows the batch name, status, and progress counts.

To create a batch in the console:

1. Open the Bulk Requests tab

   Open your endpoint and select the Bulk Requests tab.

2. Add your requests

   Click Create batch, then enter a JSON request object or an array of request objects. You can edit the requests until you run the batch.

3. Run the batch

   Click Run this batch to finalize the batch.

Click a batch to open its detail view, which shows:

• The batch’s status and progress.
• A row for each request, with its status, timestamps, and an error message if it failed.
• Links to the full detail view for each request.

The request list shows failed requests first, then in-progress, queued, and completed requests.

Console statuses

The console uses different status labels from the API:

Console status	API status
Draft	DRAFT
Running	FINALIZED
Completed	COMPLETED
Failed	FAILED
Cancelled	CANCELLED

Notifications

When a batch reaches a terminal state, Runpod sends the following notifications. The terminal states are COMPLETED, FAILED, and CANCELLED.

• Console Inbox notification: Includes the batch ID, endpoint name, terminal status, and request counts (completed, failed, and total).
• Webhook event: Sent if your account has a webhook subscription for batch events.

Each notification is sent once per terminal state. Runpod doesn’t send notifications for intermediate progress.

Cancellation

To cancel a batch:

curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/cancel \
  -H "Authorization: Bearer API_KEY"

When you cancel a batch:

• Queued requests are cancelled immediately and aren’t billed.
• In-progress requests finish and are billed normally.

The batch status changes to CANCELLED once all in-progress requests have finished.

If you cancel a batch before any requests start, it moves to CANCELLED immediately and nothing is billed.

Limits

The following limits apply to batch jobs:

Limit	Value
Active batches per endpoint	10
Requests per batch	5,000
Requests per day per endpoint	50,000

Each endpoint can process up to 50,000 batch requests a day. For example, that’s 10 active batches with 5,000 requests each.

Need more? Contact support or sales to raise your limit, up to 3,000,000 requests a day.

Billing

Batch requests are billed at a discounted rate compared to on-demand Serverless pricing, with separate discount tiers for consumer and enterprise GPUs. For discount rates, contact sales.

Billing is based on the compute time each request uses. If you cancel a batch, requests that were already in progress are still billed. For general Serverless pricing, see Pricing.

Error handling

A failed request doesn’t fail the batch unless failures reach the request failure limit. Until then, the batch stays FINALIZED and keeps processing the remaining requests. To inspect failed requests, use the console or the GET /v2/ENDPOINT_ID/batch/BATCH_ID/requests endpoint. Each failed request includes the error message from your handler.

The request failure limit is 10 failed requests or 2% of the batch’s requests (rounded up), whichever is greater. For example:

Requests in batch	Failed requests that fail the batch
100	10
1,000	20
5,000	100

If your handler returns an error in its output instead of raising an exception, Runpod marks the request COMPLETED, not FAILED. These requests don’t count toward the failure limit. To catch them, check each request’s output for errors.

If the batch itself has the FAILED status, it stopped before finishing its requests. Check the failureCode and the request-level errors, and contact support if they don’t explain the failure.

A failed batch includes a failureCode that explains why it failed, and a failedAt timestamp. For example, a batch that reached the request failure limit returns:

{
  "id": "5210fe3f-ee61-4a81-bc4e-7078c955c24d",
  "endpointId": "bo3oj8kx6jz411",
  "status": "FAILED",
  "requestTotal": 12,
  "requestInProgress": 0,
  "requestCompleted": 0,
  "requestFailed": 10,
  "failureCode": "REQUEST_FAILURE_LIMIT",
  "failedAt": 1793132360580,
  "createdAt": 1793132294147,
  "name": "intentional-failed-batch-example"
}

In this example, 10 of 12 requests failed, which reached the limit, so the batch stopped and the remaining 2 requests didn’t run.

failureCode	Meaning
REQUEST_FAILURE_LIMIT	Too many requests failed. Check the request-level errors, fix your handler or inputs, and submit a new batch.
ALL_BATCH_WORKERS_UNHEALTHY	No batch worker could start, so no requests ran. Check your endpoint configuration and worker logs. The response also includes failedWorkerIds.

The failedAt field is a Unix timestamp in milliseconds.

Batch requests use the same queue as standard Serverless requests. If the queue fails, you might lose queued batch requests. This also applies to /run traffic.

Known limitations

• Batches use the GPU type configured on your endpoint. You can’t choose a different GPU per batch or per request.
• Requests within a batch run in no particular order, and you can’t schedule individual requests.
• Runpod doesn’t show a cost estimate before you finalize a batch. To estimate cost, run a small test batch and extrapolate.
• Runpod schedules batch workers based on global queue urgency and off-peak capacity, so start times aren’t guaranteed. Runpod attempts to run each batch within 24 hours on a best-effort basis.

Next steps

Send requests to an endpoint

Submit standard /run and /runsync requests to a Serverless endpoint.

Job states

Learn about the states a Serverless job moves through.

Serverless pricing

Review per-second pricing for Serverless workers.