Source dvc.yaml model from official pydantic.BaseModel

[blakeNaccarato]

Please check for similar discussions first

• I searched and couldn't find an appropriate discussion to participate in instead.

Description

May need to convert to v2 with the migration tool, then use it to generate the dvc.yaml from the CLI structure, with params.yaml having stages and params per stage, then generate a dvc.yaml with params marching per stage AND cmd with templated params substitute via CLI to allow injecting different values to a stage.

https://github.com/iterative/dvcyaml-schema/blob/master/gen.py

[blakeNaccarato]

This one's almost done, the derived DVC models inform the dvc.yaml generator now, though it's still ad-hoc. Would be nice to encode behavior as field annotations and in model config in this fashion:

• Context toggle that turns on the dvc.yaml dumping mode
• Field-level wrap serializers to mark a single field as a dep/out/plot/param/metric
• Model-level wrap serializer (activated by config option?) such that all fields within that model are one of the above
• Field-level annotations override model-level config
• Annotation to exclude a given field from e.g. params/deps even if it's in a model-level mark
• Dumping while in DVC mode will append to an object in shared context, incrementally building out stages and their parts
• Assembled context variable recoverable on the dumped object context, easily dumped to YAML
• Helper function to encapsulate that last mile

[blakeNaccarato]

This is all done and mostly isolated in boilercv_pipeline.sync_dvc.