OpenRouter support for Langsmith

[kvyb]

Feature request

OpenRouter is very popular as a provider for LLM inference, and should be available for use in playground with all of its models, next to OpenAI etc.

Motivation

Its a pity that it is not available in LangSmith Playground. Meaning I can't really do experiments with traces since all of our production inference is done on OpenRouter LLMs, and some of these LLMs are not even available with other providers.

[hinthornw]

Will add to backlog, and if this ticket gets a lot of demand we can re-prioritize

[kvyb]

Thanks

[hahaCrazy]

Some providers of openrouter are very unstable, and problems often occur in the model. If you do not specify a provider, when calling the api, the providers will be sorted by openrouter's default order. This order changes dynamically. Therefore, it is often easy for the model to fail to work properly.

The order in which langchain adds providers to openrouter is really important!

[winglian]

Is it not sufficient to simply change the api base to point to openrouter for langsmith integration to work?

[hinthornw]

If it is openai compatible than it would be

[kvyb]

It seems to be in fact openai compatible

[hahaCrazy]

Some providers of openrouter are very unstable, and problems often occur in the model. If you do not specify a provider, when calling the api, the providers will be sorted by openrouter's default order. This order changes dynamically. Therefore, it is often easy for the model to fail to work properly.

The order in which langchain adds providers to openrouter is really important!