Snapshot
Continue to About ↓- What it does
- idler is a B2B data and evaluation vendor that builds reinforcement-learning environments and graded task suites from real production work — real company operations, law-firm matters and codebases — for labs and enterprises training, post-training or benchmarking frontier models.
More detail
- It also makes bespoke evaluations for a customer's own model and domain.
- Best for
- Post-training and RL teams at frontier labs, and enterprise AI teams building domain agents in finance/ops, legal, security or software engineering, that need realistic, unsaturated, graded task suites for training or evaluation.
- Pricing
- Pricing is not disclosed.
More detail
- No subscription, per-seat or per-task rates are published, and no minimums, fees, spreads, lockups or withdrawal terms appear in the sources.
- Delivery appears to be contractual and arranged case by case, so cost must be quoted directly by idler; do not assume it is free or
- Main limitation
- Buyers who want self-serve access, published pricing, SLAs or a low-cost entry tier — there is no public sign-up, trial or API documentation, and several collections are gated behind access upon request.
More detail
- Consumers and general app builders are not the audience.
- Record
- Static profile
- Reviewed
- Evidence
- Public web research
- Machine access
- Structured profile ↗
About
- idler is a B2B data and evaluation vendor that builds reinforcement-learning environments and graded task suites from real production work — real company operations, law-firm matters and codebases — for labs and enterprises training, post-training or benchmarking frontier models.
- It also makes bespoke evaluations for a customer's own model and domain.
What it helps with
- Benchmark saturation and synthetic-data leakage: idler argues statistical business simulators can be reverse-engineered
- Weak long-horizon agentic training signal: idler's premise is that long-horizon RL drives agentic progress and that a re
- No domain-specific evals for economically valuable work such as legal, finance, security, software engineering and tool
- No way to test a model against deliberately hard, unsaturated tasks: idler reports frontier models score far below passi
Capabilities
A practical view of what is available and where limitations apply.
| Offering | Available | Detail |
|---|---|---|
| ShelfLife environments | Yes | Digital twin of a live multi-brand e-commerce retailer (9 stores, 3 commerce platforms), 200 environments from about a decade of operational data; measured Aug 13, 2026. |
| ShelfLife leaderboard and skill breakdown | Yes | Pass@1 scores with rollout cost per model, plus skill-tagged sub-scores (planning, numerical finance precision, hallucination avoidance and others); top score Claude Opus 5 at 69.0%. |
| ShelfLife E-Sim | Yes | Stateful retail-operations simulator, 12 tasks, 120-day fixed turn budget, with late suppliers, returns and postponable purchase orders; measured Aug 26, 2026. |
| CorpLaw | Limited | 50 tasks from a real law firm's anonymized data covering legal writing, client counsel, negotiation and judgment; access upon request. |
| Long-Horizon SWE | Limited | Tasks with 100+ steps and pass rates below 50% for frontier models; access upon request. |
| Cybersecurity, Recursive Self-Improvement, Terminal Agents collections | Limited | Listed collections, each marked access upon request; individual detail is not fully published. |
| Bespoke evaluations | Yes | idler states it makes bespoke evaluations for a customer's model. |
| Transfer evidence from post-training | Yes | Post-training Nemotron 3 Nano on 41 ShelfLife tasks raised its Finance Agent Benchmark score from 30.0% to 41.8% (p ≤ 0.001), per idler's launch post. |
| Self-serve sign-up, trial or public API docs | No | None described in the sources; access is by request and arrangement. |
| Published pricing or SLAs | Unknown | Not disclosed anywhere in the evidence reviewed. |
Bot trust
How confidently AI systems can verify this profile from public evidence.
—of 100
ChatGPT—/100
Claude—/100
This profile has not completed a Bot Score assessment yet.
Evidence
Where this record came from, when it was reviewed, and how confidently it can be used.
- Verification
- Scribble researched these facts from the linked public sources.
- The owner has not manually confirmed every field.
- Last reviewed
- Primary source
- idler.ai
Technical profile details
Record ID
64d598ac-dc68-4832-9396-bcf19c27b741Canonical URLhttps://profiles.scribble.network/listing/idler-64d598ac