v2 -> v3 API translation #8082

htuch · 2019-08-29T18:02:08Z

Implement a v2 --> v3 translation code generator. This will take v2 inputs and re-express them as v3. As part of this, an internal only (not API visible) deprecated_v2 field will be synthesized that can smuggle v2 fields into the mostly v3 Envoy internals.
Update Envoy to accept v3 internally, using the above translator output for v2 inputs.

The text was updated successfully, but these errors were encountered:

Split out the generic plugin and FileDescriptorProto traversal bits from protodoc. This is in aid of the work in envoyproxy#8082 ad envoyproxy#8083, where additional protoc plugins will be responsible for v2 -> v3alpha API migrations and translation code generation. This is only the start really of the api_proto_plugin framework. I anticipate additional bits of protodoc will move here later, including field type analysis and oneof handling. In some respects, this is a re-implementation of some of https://github.com/lyft/protoc-gen-star in Python. The advantage is that this is super lightweight, has few dependencies and can be easily hacked. We also embed various bits of API business logic, e.g. annotations, in the framework (for now). Risk level: Low Testing: diff -ru against previous protodoc.py RST output, identical modulo some trivial whitespace that doesn't appear in generated HTML. There are no real tests yet, I anticipate adding some golden proto style tests. Signed-off-by: Harvey Tuch <htuch@google.com>

Split out the generic plugin and FileDescriptorProto traversal bits from protodoc. This is in aid of the work in #8082 ad #8083, where additional protoc plugins will be responsible for v2 -> v3alpha API migrations and translation code generation. This is only the start really of the api_proto_plugin framework. I anticipate additional bits of protodoc will move here later, including field type analysis and oneof handling. In some respects, this is a re-implementation of some of https://github.com/lyft/protoc-gen-star in Python. The advantage is that this is super lightweight, has few dependencies and can be easily hacked. We also embed various bits of API business logic, e.g. annotations, in the framework (for now). Risk level: Low Testing: diff -ru against previous protodoc.py RST output, identical modulo some trivial whitespace that doesn't appear in generated HTML. There are no real tests yet, I anticipate adding some golden proto style tests. Signed-off-by: Harvey Tuch <htuch@google.com>

This patch introduces a new tool, protoxform, that will be the basis of the v2 -> v3 migration tooling. It operates as a Python protoc plugin, within the same framework as protodoc, and provides the ability to operate on protoc AST input and generate proto output. As a first step, the tool is applied reflexively on v2, and functions as a formatting tool. In later patches, this will be added to check_format/fix_format scripts and CI. Part of #8082. Risk level: medium (it's possible that some inadvertent wire changes occur, if they do, this patch should be rolled back). Testing: manual inspection of diff, bazel test //test/..., some grep/diff scripts to ensure we haven't lost any comments. Signed-off-by: Harvey Tuch <htuch@google.com>

Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

* ci: add protoxform as a check/fix_format step. Part of #8082. Signed-off-by: Harvey Tuch <htuch@google.com>

Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

…8335) Part of #8082. Signed-off-by: Harvey Tuch <htuch@google.com>

…yproxy#8157) Split out the generic plugin and FileDescriptorProto traversal bits from protodoc. This is in aid of the work in envoyproxy#8082 ad envoyproxy#8083, where additional protoc plugins will be responsible for v2 -> v3alpha API migrations and translation code generation. This is only the start really of the api_proto_plugin framework. I anticipate additional bits of protodoc will move here later, including field type analysis and oneof handling. In some respects, this is a re-implementation of some of https://github.com/lyft/protoc-gen-star in Python. The advantage is that this is super lightweight, has few dependencies and can be easily hacked. We also embed various bits of API business logic, e.g. annotations, in the framework (for now). Risk level: Low Testing: diff -ru against previous protodoc.py RST output, identical modulo some trivial whitespace that doesn't appear in generated HTML. There are no real tests yet, I anticipate adding some golden proto style tests. Signed-off-by: Harvey Tuch <htuch@google.com>

This patch introduces a new tool, protoxform, that will be the basis of the v2 -> v3 migration tooling. It operates as a Python protoc plugin, within the same framework as protodoc, and provides the ability to operate on protoc AST input and generate proto output. As a first step, the tool is applied reflexively on v2, and functions as a formatting tool. In later patches, this will be added to check_format/fix_format scripts and CI. Part of envoyproxy#8082. Risk level: medium (it's possible that some inadvertent wire changes occur, if they do, this patch should be rolled back). Testing: manual inspection of diff, bazel test //test/..., some grep/diff scripts to ensure we haven't lost any comments. Signed-off-by: Harvey Tuch <htuch@google.com>

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

…yproxy#8157) Split out the generic plugin and FileDescriptorProto traversal bits from protodoc. This is in aid of the work in envoyproxy#8082 ad envoyproxy#8083, where additional protoc plugins will be responsible for v2 -> v3alpha API migrations and translation code generation. This is only the start really of the api_proto_plugin framework. I anticipate additional bits of protodoc will move here later, including field type analysis and oneof handling. In some respects, this is a re-implementation of some of https://github.com/lyft/protoc-gen-star in Python. The advantage is that this is super lightweight, has few dependencies and can be easily hacked. We also embed various bits of API business logic, e.g. annotations, in the framework (for now). Risk level: Low Testing: diff -ru against previous protodoc.py RST output, identical modulo some trivial whitespace that doesn't appear in generated HTML. There are no real tests yet, I anticipate adding some golden proto style tests. Signed-off-by: Harvey Tuch <htuch@google.com>

This patch introduces a new tool, protoxform, that will be the basis of the v2 -> v3 migration tooling. It operates as a Python protoc plugin, within the same framework as protodoc, and provides the ability to operate on protoc AST input and generate proto output. As a first step, the tool is applied reflexively on v2, and functions as a formatting tool. In later patches, this will be added to check_format/fix_format scripts and CI. Part of envoyproxy#8082. Risk level: medium (it's possible that some inadvertent wire changes occur, if they do, this patch should be rolled back). Testing: manual inspection of diff, bazel test //test/..., some grep/diff scripts to ensure we haven't lost any comments. Signed-off-by: Harvey Tuch <htuch@google.com>

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

…nvoyproxy#8335) Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

This patch introduces a new tool, protoxform, that will be the basis of the v2 -> v3 migration tooling. It operates as a Python protoc plugin, within the same framework as protodoc, and provides the ability to operate on protoc AST input and generate proto output. As a first step, the tool is applied reflexively on v2, and functions as a formatting tool. In later patches, this will be added to check_format/fix_format scripts and CI. Part of envoyproxy#8082. Risk level: medium (it's possible that some inadvertent wire changes occur, if they do, this patch should be rolled back). Testing: manual inspection of diff, bazel test //test/..., some grep/diff scripts to ensure we haven't lost any comments. Signed-off-by: Harvey Tuch <htuch@google.com>

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

As part of envoyproxy#8082, we want to be able to (1) automatically generate BUILD files and (2) treat packages as atomic from a "upgrade / do not upgrade" decision perspective. This is simplified by having our BUILD targets at package granularity, since this is what the protoxform plugin operates on. This PR broadens the package-level treatment that was already introduced for Go in envoyproxy#8003 to Python and C++. This simplifies BUILD files significantly and opens the way to automated generation. There is some technical debt introduced, since all visibility controls have been removed. This is slated for reintroduction in envoyproxy#8491. As a bonus (useful for BUILD file generation), also removed the inconsistency in BUILD package target naming for packages in envoy.api.* and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg. Risk level: Low (but this will break internal builds and require BUILD fixups to consuming projects). Testing: bazel test //test/... @envoy_api//... Signed-off-by: Harvey Tuch <htuch@google.com>

As part of #8082, we want to be able to (1) automatically generate BUILD files and (2) treat packages as atomic from a "upgrade / do not upgrade" decision perspective. This is simplified by having our BUILD targets at package granularity, since this is what the protoxform plugin operates on. This PR broadens the package-level treatment that was already introduced for Go in #8003 to Python and C++. This simplifies BUILD files significantly and opens the way to automated generation. There is some technical debt introduced, since all visibility controls have been removed. This is slated for reintroduction in #8491. As a bonus (useful for BUILD file generation), also removed the inconsistency in BUILD package target naming for packages in envoy.api.* and envoy.type.*. E.g. //envoy/api/v2:v2 is now //envoy/api/v2:pkg. Risk level: Low (but this will break internal builds and require BUILD fixups to consuming projects). Testing: bazel test //test/... @envoy_api//... Signed-off-by: Harvey Tuch <htuch@google.com>

These allow specific types, expressions and files to be excluded from API boosting via annotation. API_NO_BOOST_FILE (anywhere in the file) will skip boosting of the file, API_NO_BOOST() wrapped around an expression or type will bypass boosting of that text. The idea is that there are some types that we do not ever want to upgrade as they are relevant to the cross-version wire impedance matching, or testing v2 compat when the tree is v3alpha. The sites identified in this PR were taken from my WiP in which I have the entire tree boosted and tests passing. It's possible we might need to tune some more later, but this should go a reasonable way towards reducing the amount of manual fixups required after boosting to get tests passing again. Risk level: Low (macros are nops) Testing: bazel test //test/..., new api_booster golden tests. Part of #8082 Signed-off-by: Harvey Tuch <htuch@google.com>

This PR replaces the previous unknown field smuggling style of upgrading with shadow-based. Ultimately, we want to move away from shadows, but for 1.13.0 this simplifies by removing smuggling. In addition, we make two major changes that are backported from the WiP v3alpha branch: 1. Rather than make translateOpaqueConfig() smart and version/type aware, we switch to an iterative loadFromJson() that first attempts to load as v2 and then falls back to v3. This relies on the property that any v3 configuration that is ingestible as v2 has the same semantics in v2/v3, which holds due to the highly structured vN/v(N+1) mechanical transforms. 2. Support for downgrading to earlier versions is introduced to make it easier for tests that are within the v(N-1) subset of vN to continue to validate v(N-1), while still mostly using vN messages. This is particularly useful for ensuring that e2e integration tests are using v2 resources. As new v3 tests are introduced, tests will need to eschew downgrading, but this should be a viable approach for 1.13.0. Risk level: Low Testing: There's a bunch of new tests that need to be added for version converter, loadFroJson/unpackTo() and translateOpaqueConfig(). Putting this PR out for comment to get feedback. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com>

This is intended to simplify the internal handling of deprecations during API boosting. See https://docs.google.com/document/d/1mGO9LtVo7t4Lph7WlmyGCxXye3h6j29z3JZvIBbs_D0/edit Ultimately the plan is to hide this all as a build artifact in Bazel cache, but due to the technical complexity of the pure Bazel solution (involving changes spanning PGV, protoxform, API build rules), we will use checked-in artifacts for 1.13.0. Risk level: Low Testing: Additional API test and protoxform golden test. Part of #8082 This should be unwound in the future with #9479 Signed-off-by: Harvey Tuch <htuch@google.com>

…9502) * config: shadow-based upgrading/downgrading for versioned messages. This PR replaces the previous unknown field smuggling style of upgrading with shadow-based. Ultimately, we want to move away from shadows, but for 1.13.0 this simplifies by removing smuggling. In addition, we make two major changes that are backported from the WiP v3alpha branch: 1. Rather than make translateOpaqueConfig() smart and version/type aware, we switch to an iterative loadFromJson() that first attempts to load as v2 and then falls back to v3. This relies on the property that any v3 configuration that is ingestible as v2 has the same semantics in v2/v3, which holds due to the highly structured vN/v(N+1) mechanical transforms. 2. Support for downgrading to earlier versions is introduced to make it easier for tests that are within the v(N-1) subset of vN to continue to validate v(N-1), while still mostly using vN messages. This is particularly useful for ensuring that e2e integration tests are using v2 resources. As new v3 tests are introduced, tests will need to eschew downgrading, but this should be a viable approach for 1.13.0. Risk level: Low Testing: additional unit tests added to cover opaque config conversion, Any unpacking, JSON loading and version conversion. Part of #8082 Signed-off-by: Harvey Tuch <htuch@google.com>

…oxy#9329) This is a beachhead PR for a Clang Libtooling based workflow that automagically updates Envoy's source tree to the latest API version for every referenced package. So far, ths tool is only capable of inferring types and performing header fixups, later PRs will expand this. Risk level: Low Testing: Manual cleanup of all headers in source/ test/ and include/, all tests pass. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

…xy#9375) This PR introduces API boosting support for ElaboratedTypeLoc, one of a number of Clang AST nodes that we need to analyze and rewrite to move to the latest type according to the type database. The approach taken is to generate a replacements YAML file suitable for consumption by clang-apply-replacements; this is how clang-tidy works internally. Replacements are a series of patch like file transformation operations, Clang has nice support for building replacement sets and serializing to YAML. This PR also starts to split out api_booster into more modules (just some utils to start with), introduces some more unit tests and a golden C++ source file test framework. Risk level: Low Testing: New unit and e2e golden tests added. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

…yproxy#9418) Via AST node matchers on UsingDecl and DeclRefExpr. This commit also refactors the clang::tooling::SourceFileCallbacks run() routine, by moving to a per AST node dispatch model. This will support scaling to additional AST node types in the future without having run() grow to an unreadable length. Risk level: Low Testing: Additional golden tests. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

…9450) Mangled names exist only to help with forward declarations, proto style explicitly discourages this, see https://developers.google.com/protocol-buffers/docs/reference/cpp-generated#enum. By avoiding the mangled names, we simplify API boosting tooling. Risk level: Low Testing: New check_format test. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

The type database now has rename information. Also some misc. fixes for enum and generated static method handling support. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

These allow specific types, expressions and files to be excluded from API boosting via annotation. API_NO_BOOST_FILE (anywhere in the file) will skip boosting of the file, API_NO_BOOST() wrapped around an expression or type will bypass boosting of that text. The idea is that there are some types that we do not ever want to upgrade as they are relevant to the cross-version wire impedance matching, or testing v2 compat when the tree is v3alpha. The sites identified in this PR were taken from my WiP in which I have the entire tree boosted and tests passing. It's possible we might need to tune some more later, but this should go a reasonable way towards reducing the amount of manual fixups required after boosting to get tests passing again. Risk level: Low (macros are nops) Testing: bazel test //test/..., new api_booster golden tests. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

This is intended to simplify the internal handling of deprecations during API boosting. See https://docs.google.com/document/d/1mGO9LtVo7t4Lph7WlmyGCxXye3h6j29z3JZvIBbs_D0/edit Ultimately the plan is to hide this all as a build artifact in Bazel cache, but due to the technical complexity of the pure Bazel solution (involving changes spanning PGV, protoxform, API build rules), we will use checked-in artifacts for 1.13.0. Risk level: Low Testing: Additional API test and protoxform golden test. Part of envoyproxy#8082 This should be unwound in the future with envoyproxy#9479 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

…nvoyproxy#9502) * config: shadow-based upgrading/downgrading for versioned messages. This PR replaces the previous unknown field smuggling style of upgrading with shadow-based. Ultimately, we want to move away from shadows, but for 1.13.0 this simplifies by removing smuggling. In addition, we make two major changes that are backported from the WiP v3alpha branch: 1. Rather than make translateOpaqueConfig() smart and version/type aware, we switch to an iterative loadFromJson() that first attempts to load as v2 and then falls back to v3. This relies on the property that any v3 configuration that is ingestible as v2 has the same semantics in v2/v3, which holds due to the highly structured vN/v(N+1) mechanical transforms. 2. Support for downgrading to earlier versions is introduced to make it easier for tests that are within the v(N-1) subset of vN to continue to validate v(N-1), while still mostly using vN messages. This is particularly useful for ensuring that e2e integration tests are using v2 resources. As new v3 tests are introduced, tests will need to eschew downgrading, but this should be a viable approach for 1.13.0. Risk level: Low Testing: additional unit tests added to cover opaque config conversion, Any unpacking, JSON loading and version conversion. Part of envoyproxy#8082 Signed-off-by: Harvey Tuch <htuch@google.com> Signed-off-by: Prakhar <prakhar_au@yahoo.com>

htuch added no stalebot Disables stalebot from closing an issue api/v3 Major version release @ end of Q3 2019 labels Aug 29, 2019

htuch added this to the 1.12.0 milestone Aug 29, 2019

htuch self-assigned this Aug 29, 2019

htuch mentioned this issue Aug 29, 2019

v3 API release plan #8085

Closed

5 tasks

htuch mentioned this issue Sep 5, 2019

protodoc/api_proto_plugin: generic API protoc plugin framework. #8157

Merged

htuch mentioned this issue Sep 20, 2019

api: protoxform tool and API reformat. #8309

Merged

htuch added a commit to htuch/envoy that referenced this issue Sep 23, 2019

ci: add protoxform as a check/fix_format step.

e38aa1c

Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

htuch mentioned this issue Sep 23, 2019

ci: add protoxform as a check/fix_format step. #8331

Merged

lizan pushed a commit that referenced this issue Sep 23, 2019

ci: add protoxform as a check/fix_format step. (#8331)

74ced53

* ci: add protoxform as a check/fix_format step. Part of #8082. Signed-off-by: Harvey Tuch <htuch@google.com>

htuch added a commit to htuch/envoy that referenced this issue Sep 23, 2019

protoxform: remove blank line spacing in any enum with no comments.

d4c4b4a

Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

htuch mentioned this issue Sep 23, 2019

protoxform: remove blank line spacing in any enum with no comments. #8335

Merged

htuch added a commit that referenced this issue Sep 23, 2019

protoxform: remove blank line spacing in any enum with no comments. (#…

5518034

…8335) Part of #8082. Signed-off-by: Harvey Tuch <htuch@google.com>

danzh2010 pushed a commit to danzh2010/envoy that referenced this issue Sep 24, 2019

ci: add protoxform as a check/fix_format step. (envoyproxy#8331)

926eacb

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

htuch mentioned this issue Sep 27, 2019

re-deprecate opaque Struct config #8403

Closed

danzh2010 pushed a commit to danzh2010/envoy that referenced this issue Oct 4, 2019

ci: add protoxform as a check/fix_format step. (envoyproxy#8331)

002f445

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

danzh2010 pushed a commit to danzh2010/envoy that referenced this issue Oct 4, 2019

protoxform: remove blank line spacing in any enum with no comments. (e…

dda3851

…nvoyproxy#8335) Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

danzh2010 pushed a commit to danzh2010/envoy that referenced this issue Oct 4, 2019

ci: add protoxform as a check/fix_format step. (envoyproxy#8331)

a7db44b

* ci: add protoxform as a check/fix_format step. Part of envoyproxy#8082. Signed-off-by: Harvey Tuch <htuch@google.com>

htuch mentioned this issue Oct 4, 2019

api/build: switch to only package level targets in API. #8495

Merged

htuch mentioned this issue Dec 27, 2019

config: shadow-based upgrading/downgrading for versioned messages. #9502

Merged

sthagen mentioned this issue Dec 29, 2019

api: shadowing for API protos. (#9429) sthagen/envoyproxy-envoy#127

Merged

htuch mentioned this issue Jan 3, 2020

all: upgrade Envoy source to v3alpha API. #9546

Merged

mattklein123 closed this as completed in #9546 Jan 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

v2 -> v3 API translation #8082

v2 -> v3 API translation #8082

htuch commented Aug 29, 2019 •

edited

Loading

v2 -> v3 API translation #8082

v2 -> v3 API translation #8082

Comments

htuch commented Aug 29, 2019 • edited Loading

htuch commented Aug 29, 2019 •

edited

Loading