-
Notifications
You must be signed in to change notification settings - Fork 4.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
protobuf: refactor proto visitor pattern. #9807
Closed
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -13,6 +13,7 @@ | |
#include "common/config/version_converter.h" | ||
#include "common/protobuf/message_validator_impl.h" | ||
#include "common/protobuf/protobuf.h" | ||
#include "common/protobuf/visitor.h" | ||
#include "common/protobuf/well_known.h" | ||
|
||
#include "absl/strings/match.h" | ||
|
@@ -375,42 +376,25 @@ void checkForDeprecatedNonRepeatedEnumValue(const Protobuf::Message& message, | |
} | ||
} | ||
|
||
void checkForUnexpectedFields(const Protobuf::Message& message, | ||
ProtobufMessage::ValidationVisitor& validation_visitor, | ||
Runtime::Loader* runtime) { | ||
// Reject unknown fields. | ||
const auto& unknown_fields = message.GetReflection()->GetUnknownFields(message); | ||
if (!unknown_fields.empty()) { | ||
std::string error_msg; | ||
for (int n = 0; n < unknown_fields.field_count(); ++n) { | ||
if (unknown_fields.field(n).number() == ProtobufWellKnown::OriginalTypeFieldNumber) { | ||
continue; | ||
} | ||
error_msg += absl::StrCat(n > 0 ? ", " : "", unknown_fields.field(n).number()); | ||
} | ||
// We use the validation visitor but have hard coded behavior below for deprecated fields. | ||
// TODO(htuch): Unify the deprecated and unknown visitor handling behind the validation | ||
// visitor pattern. https://github.com/envoyproxy/envoy/issues/8092. | ||
if (!error_msg.empty()) { | ||
validation_visitor.onUnknownField("type " + message.GetTypeName() + | ||
" with unknown field set {" + error_msg + "}"); | ||
} | ||
} | ||
class UnexpectedFieldProtoVisitor : public ProtobufMessage::ConstProtoVisitor { | ||
public: | ||
UnexpectedFieldProtoVisitor(ProtobufMessage::ValidationVisitor& validation_visitor, | ||
Runtime::Loader* runtime) | ||
: validation_visitor_(validation_visitor), runtime_(runtime) {} | ||
|
||
const Protobuf::Descriptor* descriptor = message.GetDescriptor(); | ||
const Protobuf::Reflection* reflection = message.GetReflection(); | ||
for (int i = 0; i < descriptor->field_count(); ++i) { | ||
const Protobuf::FieldDescriptor* field = descriptor->field(i); | ||
absl::string_view filename = filenameFromPath(field->file()->name()); | ||
const void* onField(const Protobuf::Message& message, const Protobuf::FieldDescriptor& field, | ||
const void*) override { | ||
const Protobuf::Reflection* reflection = message.GetReflection(); | ||
absl::string_view filename = filenameFromPath(field.file()->name()); | ||
|
||
// Before we check to see if the field is in use, see if there's a | ||
// deprecated default enum value. | ||
checkForDeprecatedNonRepeatedEnumValue(message, filename, field, reflection, runtime); | ||
checkForDeprecatedNonRepeatedEnumValue(message, filename, &field, reflection, runtime_); | ||
|
||
// If this field is not in use, continue. | ||
if ((field->is_repeated() && reflection->FieldSize(message, field) == 0) || | ||
(!field->is_repeated() && !reflection->HasField(message, field))) { | ||
continue; | ||
if ((field.is_repeated() && reflection->FieldSize(message, &field) == 0) || | ||
(!field.is_repeated() && !reflection->HasField(message, &field))) { | ||
return nullptr; | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we ever return something which is used, or plan to? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, in |
||
} | ||
|
||
#ifdef ENVOY_DISABLE_DEPRECATED_FEATURES | ||
|
@@ -421,22 +405,22 @@ void checkForUnexpectedFields(const Protobuf::Message& message, | |
// Allow runtime to be null both to not crash if this is called before server initialization, | ||
// and so proto validation works in context where runtime singleton is not set up (e.g. | ||
// standalone config validation utilities) | ||
if (runtime && field->options().deprecated()) { | ||
if (runtime_ && field.options().deprecated()) { | ||
// This is set here, rather than above, so that in the absence of a | ||
// registry (i.e. test) the default for if a feature is allowed or not is | ||
// based on ENVOY_DISABLE_DEPRECATED_FEATURES. | ||
warn_only &= !field->options().GetExtension(envoy::annotations::disallowed_by_default); | ||
warn_only = runtime->snapshot().deprecatedFeatureEnabled( | ||
absl::StrCat("envoy.deprecated_features:", field->full_name()), warn_only); | ||
warn_only &= !field.options().GetExtension(envoy::annotations::disallowed_by_default); | ||
warn_only = runtime_->snapshot().deprecatedFeatureEnabled( | ||
absl::StrCat("envoy.deprecated_features:", field.full_name()), warn_only); | ||
} | ||
|
||
// If this field is deprecated, warn or throw an error. | ||
if (field->options().deprecated()) { | ||
if (field.options().deprecated()) { | ||
std::string err = fmt::format( | ||
"Using deprecated option '{}' from file {}. This configuration will be removed from " | ||
"Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated " | ||
"for details.", | ||
field->full_name(), filename); | ||
field.full_name(), filename); | ||
if (warn_only) { | ||
ENVOY_LOG_MISC(warn, "{}", err); | ||
} else { | ||
|
@@ -448,29 +432,43 @@ void checkForUnexpectedFields(const Protobuf::Message& message, | |
throw ProtoValidationException(err + fatal_error, message); | ||
} | ||
} | ||
return nullptr; | ||
} | ||
|
||
// If this is a message, recurse to check for deprecated fields in the sub-message. | ||
if (field->cpp_type() == Protobuf::FieldDescriptor::CPPTYPE_MESSAGE) { | ||
if (field->is_repeated()) { | ||
const int size = reflection->FieldSize(message, field); | ||
for (int j = 0; j < size; ++j) { | ||
checkForUnexpectedFields(reflection->GetRepeatedMessage(message, field, j), | ||
validation_visitor, runtime); | ||
void onMessage(const Protobuf::Message& message, const void*) override { | ||
// Reject unknown fields. | ||
const auto& unknown_fields = message.GetReflection()->GetUnknownFields(message); | ||
if (!unknown_fields.empty()) { | ||
std::string error_msg; | ||
for (int n = 0; n < unknown_fields.field_count(); ++n) { | ||
if (unknown_fields.field(n).number() == ProtobufWellKnown::OriginalTypeFieldNumber) { | ||
continue; | ||
} | ||
} else { | ||
checkForUnexpectedFields(reflection->GetMessage(message, field), validation_visitor, | ||
runtime); | ||
error_msg += absl::StrCat(n > 0 ? ", " : "", unknown_fields.field(n).number()); | ||
} | ||
// We use the validation visitor but have hard coded behavior below for deprecated fields. | ||
// TODO(htuch): Unify the deprecated and unknown visitor handling behind the validation | ||
// visitor pattern. https://github.com/envoyproxy/envoy/issues/8092. | ||
if (!error_msg.empty()) { | ||
validation_visitor_.onUnknownField("type " + message.GetTypeName() + | ||
" with unknown field set {" + error_msg + "}"); | ||
} | ||
} | ||
} | ||
} | ||
|
||
private: | ||
ProtobufMessage::ValidationVisitor& validation_visitor_; | ||
Runtime::Loader* runtime_; | ||
}; | ||
|
||
} // namespace | ||
|
||
void MessageUtil::checkForUnexpectedFields(const Protobuf::Message& message, | ||
ProtobufMessage::ValidationVisitor& validation_visitor, | ||
Runtime::Loader* runtime) { | ||
::Envoy::checkForUnexpectedFields(API_RECOVER_ORIGINAL(message), validation_visitor, runtime); | ||
UnexpectedFieldProtoVisitor unexpected_field_visitor(validation_visitor, runtime); | ||
ProtobufMessage::traverseMessage(unexpected_field_visitor, API_RECOVER_ORIGINAL(message), | ||
nullptr); | ||
} | ||
|
||
std::string MessageUtil::getYamlStringFromMessage(const Protobuf::Message& message, | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
#include "common/protobuf/visitor.h" | ||
|
||
namespace Envoy { | ||
namespace ProtobufMessage { | ||
|
||
void traverseMutableMessage(ProtoVisitor& visitor, Protobuf::Message& message, const void* ctxt) { | ||
visitor.onMessage(message, ctxt); | ||
const Protobuf::Descriptor* descriptor = message.GetDescriptor(); | ||
const Protobuf::Reflection* reflection = message.GetReflection(); | ||
for (int i = 0; i < descriptor->field_count(); ++i) { | ||
const Protobuf::FieldDescriptor* field = descriptor->field(i); | ||
const void* field_ctxt = visitor.onField(message, *field, ctxt); | ||
// If this is a message, recurse to scrub deprecated fields in the sub-message. | ||
if (field->cpp_type() == Protobuf::FieldDescriptor::CPPTYPE_MESSAGE) { | ||
if (field->is_repeated()) { | ||
const int size = reflection->FieldSize(message, field); | ||
for (int j = 0; j < size; ++j) { | ||
traverseMutableMessage(visitor, *reflection->MutableRepeatedMessage(&message, field, j), | ||
field_ctxt); | ||
} | ||
} else if (reflection->HasField(message, field)) { | ||
traverseMutableMessage(visitor, *reflection->MutableMessage(&message, field), field_ctxt); | ||
} | ||
} | ||
} | ||
} | ||
void traverseMessage(ConstProtoVisitor& visitor, const Protobuf::Message& message, | ||
const void* ctxt) { | ||
visitor.onMessage(message, ctxt); | ||
const Protobuf::Descriptor* descriptor = message.GetDescriptor(); | ||
const Protobuf::Reflection* reflection = message.GetReflection(); | ||
for (int i = 0; i < descriptor->field_count(); ++i) { | ||
const Protobuf::FieldDescriptor* field = descriptor->field(i); | ||
const void* field_ctxt = visitor.onField(message, *field, ctxt); | ||
// If this is a message, recurse to scrub deprecated fields in the sub-message. | ||
htuch marked this conversation as resolved.
Show resolved
Hide resolved
|
||
if (field->cpp_type() == Protobuf::FieldDescriptor::CPPTYPE_MESSAGE) { | ||
if (field->is_repeated()) { | ||
const int size = reflection->FieldSize(message, field); | ||
for (int j = 0; j < size; ++j) { | ||
traverseMessage(visitor, reflection->GetRepeatedMessage(message, field, j), field_ctxt); | ||
} | ||
} else if (reflection->HasField(message, field)) { | ||
traverseMessage(visitor, reflection->GetMessage(message, field), field_ctxt); | ||
} | ||
} | ||
} | ||
} | ||
|
||
} // namespace ProtobufMessage | ||
} // namespace Envoy |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,43 @@ | ||
#pragma once | ||
|
||
#include "common/protobuf/protobuf.h" | ||
|
||
namespace Envoy { | ||
namespace ProtobufMessage { | ||
|
||
class ProtoVisitor { | ||
public: | ||
virtual ~ProtoVisitor() = default; | ||
|
||
// Invoked when a field is visited, with the message, field descriptor and context. Returns a new | ||
// context for use when traversing the sub-message in a field. | ||
virtual const void* onField(Protobuf::Message&, const Protobuf::FieldDescriptor&, | ||
const void* ctxt) { | ||
return ctxt; | ||
} | ||
|
||
// Invoked when a message is visited, with the message and a context. | ||
virtual void onMessage(Protobuf::Message&, const void*){}; | ||
}; | ||
|
||
class ConstProtoVisitor { | ||
public: | ||
virtual ~ConstProtoVisitor() = default; | ||
|
||
// Invoked when a field is visited, with the message, field descriptor and context. Returns a new | ||
// context for use when traversing the sub-message in a field. | ||
virtual const void* onField(const Protobuf::Message&, const Protobuf::FieldDescriptor&, | ||
const void* ctxt) { | ||
return ctxt; | ||
} | ||
|
||
// Invoked when a message is visited, with the message and a context. | ||
virtual void onMessage(const Protobuf::Message&, const void*){}; | ||
}; | ||
|
||
void traverseMutableMessage(ProtoVisitor& visitor, Protobuf::Message& message, const void* ctxt); | ||
void traverseMessage(ConstProtoVisitor& visitor, const Protobuf::Message& message, | ||
const void* ctxt); | ||
|
||
} // namespace ProtobufMessage | ||
} // namespace Envoy |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
optional, but it seems like this might be useful enough to make part of the onField call. I'd think folks would either want to be traversing structure (not care) or looking at valid fields (care) and if they care it's easy enough to get this wrong.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Surprisingly tricky, since what is counted as used/unused is subtle for scalar fields. This can be seen in this code, since it first performs the scalar enum check before conditioning on whether the field is set in some sense.