Your personal operating system for career transition.

Async provider client with semaphore rate limiting

Running LLM calls serially on a batch is 10–50x slower than running them concurrently. But unlimited concurrency hammers rate limits. A semaphore is the standard tool for bounded concurrency.

What you are building

Implement an AsyncBatchProcessor class that:

1. Accepts a process_fn: Callable[[str], Awaitable[str]] — the async function to call for each item (simulates an LLM API call)
2. Accepts max_concurrent: int = 5 — maximum simultaneous calls
3. async def run(self, items: list[str]) -> tuple[list[str], list[dict]] — processes all items concurrently within the semaphore limit. Returns (results, failures) where failures is a list of {"index": i, "error": str} dicts.

Requirements

• Use asyncio.Semaphore(max_concurrent) to bound concurrency
• Use asyncio.gather(..., return_exceptions=True) to collect results
• A failed item should not abort other items
• Results list must have the same length as items, with None for failed items
• Failures list must contain the index and error string for each failed item

Example

import asyncio

async def fake_llm(text: str) -> str:
    await asyncio.sleep(0.01)
    if text == "fail":
        raise ValueError("Simulated failure")
    return f"processed: {text}"

processor = AsyncBatchProcessor(process_fn=fake_llm, max_concurrent=3)
results, failures = asyncio.run(processor.run(["a", "b", "fail", "d"]))
# results == ["processed: a", "processed: b", None, "processed: d"]
# failures == [{"index": 2, "error": "Simulated failure"}]