Substrate Labs
Viveka Hoshi
- Generation
- Fourth
- Answers
- Choose one, rate on a scale, yes or no
- Access
- Private beta, by waitlist
Decisions from your app’s data. Send a state and a question. Get a choice, a score or a yes/no answer, with a probability for every option and a flag when it isn’t sure.
Three kinds of question
- Choose one
- Route a ticket, a request or an action to one of the options you define, with a probability for each.
- Rate on a scale
- Score urgency, sentiment or relevance across levels you set, with a probability for each level.
- Yes or no
- The probability of yes, for a rule you provide: approve, block or escalate.
Every answer is structured, so there is no free text to parse. Set a threshold for acting, and check it on examples from your own workflow before you automate anything.
Accuracy
- Open-Jev test3,000 decisions94.9%
- Open-Jev, unfamiliar tasks3,000 decisions83.7%
- Mind2Web, website actions*1,008 decisions87.8%
- When2Call, tool use*1,067 decisions81.8%
- Procedural5,701 decisions86.3%
- Procedural, long inputs708 decisions74.3%
- tasksource7,673 decisions71.6%
- BoolQ, reading3,270 decisions78.6%
Decisions with one clear right answer. *Hoshi trained on separate training splits of Mind2Web and When2Call, so these two are not unseen data.
Calibration
A probability is only useful if it can be trusted: when Hoshi says it is 90% sure, it should be right about 90% of the time.
- 2,512
- of 3,000 Open-Jev test decisions were 90% sure or more
- 99.7%
- of those were right
- Open-Jev test3,000 decisions0.014
- Open-Jev, unfamiliar tasks3,000 decisions0.129
- Mind2Web, website actions*1,008 decisions0.079
- When2Call, tool use*1,067 decisions0.118
- Procedural5,701 decisions0.009
- Procedural, long inputs708 decisions0.025
- tasksource7,673 decisions0.089
- BoolQ, reading3,270 decisions0.099
- LocalLLaMA, soft labels2,000 decisions0.041
Expected calibration error: the gap between stated confidence and actual accuracy, over 10 confidence bins and every row of each set.
Hoshi is best calibrated on familiar decisions. Between 30% and 60% confidence it is overconfident: those answers are right less often than it says.
The unsure flag
Each answer returns the set of options that could be right, and is marked unsure when that set holds more than one. Hoshi aims for the right answer to be inside the set 95% of the time.
- Open-Jev test2.9% marked unsure96.5%
- Open-Jev validation4.4% marked unsure95.5%
- Open-Jev, unfamiliar tasks2.4% marked unsure85.7%
The threshold was fit on Open-Jev validation. The 95% holds on familiar decisions; on unfamiliar task variants it held for 85.7%.
Four training rounds
Hoshi was trained in four rounds, each starting from the one before. Every round improved both accuracy and calibration on Open-Jev validation (3,272 decisions).
- 91.8%
- Round 1 · NLL 0.243
- 92.3%
- Round 2 · NLL 0.234
- 93.4%
- Round 3 · NLL 0.222
- 93.7%
- Round 4, Hoshi · NLL 0.211
Validation accuracy, and NLL: how much probability the model put on the right answer. Lower NLL is better.
Speed
- 463 ms
- median time per decision
- 1,274 ms
- 95th percentile, the slowest 5%
1,000 Open-Jev test decisions sent from India, 12 at a time. Results depend on the workload.
Limits
- BoolQ, yes/no questions about a passage of text, is the weakest result at 78.6%. It is the main target of the next version.
- On unfamiliar task variants Hoshi is right more often than not, but too confident when it is wrong.
- The answer must be in the input you send. Viveka does not supply missing world knowledge, and a confident answer can still be wrong.
All measurements by Substrate Labs on 22 and 23 October 2026.
Join the waitlist
Viveka Hoshi is in private beta. Leave your email and we’ll write when there’s a place.