Rendered at 20:10:42 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
overgard 9 minutes ago [-]
> Just as a painter knows exactly what brush they need to use, and the craftsman carefully chooses their tools, engineers should understand trade-offs and subtleties of the different models.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
mmargenot 28 minutes ago [-]
I find this routing problem to be opaque and I’m generally skeptical that the label people are trying to predict is meaningful.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
I think fail-over for a production critical service is an equally important responsibility of the router.
ramraj07 3 minutes ago [-]
You cant fail over to a different model though. Even failing over to a different provider isn't always as simple as set it and forget it. For example bedrock doesn't support the tool search tool feature from anthropic.
velcrovan 1 hours ago [-]
Ironically, my confidence that a human had at least an active part in writing/editing this article went up because of this train wreck of a sentence:
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."
hhh 48 minutes ago [-]
what’s wrong with the sentence? reads fine to me
1bpp 40 minutes ago [-]
'keeps' is the wrong tense and could be 'will keep'
thehamkercat 45 minutes ago [-]
i think it should be "by adding stickiness to the initially chosen provider of that model"
coffinbirth 17 minutes ago [-]
Some routing services not just route to different LLMs, they also handle all the legal issues (GDPR compliance, ISO certification, guaranteed Zero-Data-Retention, domestic data processing/European based clouds, etc.). In regulated industries, these things matter a lot, especially when processing of sensitive data is involved.
maxrev17 36 minutes ago [-]
Yeah routers suck, been doing this myself for the best part of a year now and it’s really difficult to make it behave. A good model for your task is the best bet.
I'm really skeptical of this idea. Pragmatically: who has time to understand the nuances of these models when there's like a new one every week? Also without any view into the training, figuring out what each model is potentially good at is more or less just throwing spaghetti against the wall, except the spaghetti is potentially very expensive and might insert subtle issues into your code base.
If you really need more discrimination of the complexity of an input to get an efficient response, sft or rl tuning something for your harness would be more effective.
> "A cache-aware model router will take that into account by adding stickiness to the initially chosen model and keeps querying it."