Add a "lax" or "disabled" option for validating schemas for tool calls and JSON mode

[njpearman]

I'd really like to have a way to disable the strictness around schema validation for tool calls and JSON modes in genkit. I keep getting stung by models not quite getting the defined JSON structure right and these fail the very strict validation that genkit enforces. This is frustrating as I want to see what the model is struggling to get right at least but I don't get the opportunity because genkit fails hard without any trace of the underlying structure that didn't validate.

I'm using some pretty complex JSON API schemas (with discriminated unions and things), and having mostly a lot of joy, but little model "misunderstandings" are causing hard fails in genkit that are disrupting my ability to make quick trial-and-error adjustments. This problem definitely has a bit of overlap with the underlying model integrations, as they lack adequate logging for a quick glance from that side, so I think there are different approaches to take to help here. It's super frustrating with things the way they are at the moment though.

I'd be interested in hearing others thoughts on this!

[mbleigh]

We've heard this request several times so I'm pretty tempted to go ahead and implement it, but every time I think about it I run into this question in my mind:

What use is non-conforming JSON coming back from the generate call?

If I specify a specific schema for my data, and the returned data does not match that schema, is it still useful to me? Unless I write manual "repair" logic or have an additional "fix my bad JSON" generate call, I can't just pass through the result to my UI or downstream processing logic.

That's not a reason not to offer the ability to disable schema validation, it's just something I keep thinking about.

Would it be more useful if in addition to an outputSchema you could provide a validationSchema? So let's say you have a complex schema but there's only 2-3 fields you really care about - you can pass a validation schema that only looks for those fields, and then everything else is a "bonus". And if you really don't care, you could pass z.any() to the validationSchema.

[njpearman]

Thanks both for the thoughtful replies.

In principle, I'd agree with @mbleigh that if you're providing an output schema, you are setting a strong expectation about the output structure. That is my aim with the things I'm working on - give a defined Schema and expect the model to understand it and return something that conforms to it.

What I find interesting is the world we're operating in - I can write a schema that strongly conforms to the tool or integration that I'm using, but I have to tease the model to obey the schema with a bit of trial and error in a prompt. It's quite different from coding out an API endpoint and client that conforms to a schema or contract.

I think that sums up my position on what I'd like to see in genkit. My pain point is when I'm developing a prompt-based feature, I specifically want to have full visibility of how the model has gone haywire. This allows me to make quick, informed adjustments to a prompt (or schema) and iterate towards a solid-enough prompt for my use case.

I possibly also want to turn off validation to just let things break down stream while I'm experimenting, as this is likely to help inform what the model has screwed up.

At the moment, genkit doesn't give me this debugging exit hatch - when the output doesn't conform, I get a terse error message with no context. As I mentioned in my opening post, I think part of the issue is in the model plugins, as they don't necessarily record the direct response from the LLM service. How I'm working around this is to copy the plugin code (openai) and put my own logging in.

I'm not actually interested in complicating the production situation, e.g. validationSchema + outputSchema. If I'm putting a tool call or JSON mode prompt into production, I will be at a point where I have enough confidence that it is following the schema I'm providing often enough. So I'm mostly (but not entirely) fine with strict validation there. Zod + JSON API give me a lot of power to craft a schema for anything the prompt struggles with at dev time. However, I definitely want to be confident that if something fails in prod, I have all of the information captured to tell me why.

To speak to @ariel-pettyjohn's comments, my experience is that validation errors are generally due to missing properties and/or hallucinating extra properties, which is a pretty straightforward failure case, rather than any semantic naming issues. I've never seen that sort of semantic drift in structured JSON - I'd be interested to know more about the prompt set up causing this sort of problem !

[ariel-pettyjohn]

Sure thing, here's a quick example of what I'm describing!

I have a very simple test app called Johnny Five, whose job is to learn the names of people during conversation. I've simplified the code in the example below, but, because of the no candidate errors, I've been wrapping the generate call in a try-catch. I try using the output schema, but then omit the output schema in the catch statement while still calling result.output(). Like I said, I don't know how JSON mode or validation work under the hood, but my guess is that I'm still using JSON mode here just without validation?

In this example, I've given the app the text "My name is Walt Disney and this is my friend Mickey Mouse" and commented the output schema to simulate the catch case. In the first screenshot, without the output schema, you can see it still returns JSON containing the specified "people" array as prompted, but it isn't guaranteed to format the names separately like it does in the second screenshot. That makes sense, and I've learned from experimentation that I can inject the fields from the schema into the prompt and it will format the names as "firstName" and "lastName" even in the catch case. At that point though, it feels like I'm both reinventing the wheel and missing something along the way.

My main use case for something like a "lax" mode would be as a fallback in these scenarios, but they might also be avoidable in this case with debugging/logging and better prompt engineering. If I really wanted to restrict validation only to certain fields, then it seems like I could just generate those select fields using an output schema, and then generate the rest using an approach like the fallback in this example.

[mbleigh]

Thanks for the discussion. What I'm hearing after this conversation is that you're not necessarily looking to turn off schema validation, but that the iterative process of improving a prompt to accurately return the desired schema is currently opaque and frustrating, and you'd also probably like better ways to recover from these types of errors (e.g. automatic retrying or self-healing output).

Would it help if the error message was extremely verbose and printed out the parsed output object as well as the specific schema validation errors that were encountered? What % of the overall problem do you think better error messages (and potentially documentation around handling errors) would solve?

[njpearman]

I'd say comprehensive logging addresses 90%+ of my current difficulties - I solved my problems by dropping in my own custom logging. In terms of what's useful, I think being able to get the full underlying LLM response is what I want ideally. I'd also be very keen to have this sort of thing always-traced as well as logged, as I think it's super important to be able to collect detailed, structured information. I feel this is much more important with LLM-driven features than classic dev (for which tracing is important but often neglected).

[njpearman]

And just to add, anything above and beyond what I've put above would be great!

[ariel-pettyjohn]

💯 I think some combination of automatic retrying and that level of verbosity would do the trick. Judging by how close I can get with fallback methods, I'd say more insight into the specific schema validation errors would probably address about 70% of the issues I've encountered.

I'd say 15% of my issues are probably documentation related just because Genkit's still in beta, and maybe another 15% could be real edge cases where some kind of retry or repair logic would be helpful. I've been hesitant to implement any retry logic on my own just because some prompts seem to fail about 50% of the time, whereas similar prompts might fail 80% of the time.

Thanks again for everyone's hard work on the Firebase team!

[njpearman]

@mbleigh something that occurred to me today with validation errors is that I might want to feed those back into a "fixer" prompt as a follow up. It would therefore be useful to have the validation details returned so they're available for use downstream. I imagine this would be something of a pain to implement but I really do think it would be a useful aspect to the schema validation part of genkit.

[ariel-pettyjohn]

@njpearman and @mbleigh Hey there! I hacked some simple console.error statements into my local copy of the schema module in order to enumerate the errors found in the validateSchema function, and I was able to solve the errors I'd been experiencing immediately! Specifically, I'd 1) assumed that fields were optional by default; and 2) didn't realize that Zod distinguishes between optional and nullable. So I quickly realized all I had to do was add .nullish() to my fields and now I'm error-free! For my purposes so far, I think I just need a more verbose logging option for those errors, because I actually found them super helpful and easy to understand as-is 👍

[mbleigh]

For those of you who requested more helpful error messages...good news! #536 is pending 😄