Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

28857 openapi include exclude #29769

Draft
wants to merge 3 commits into
base: integration
Choose a base branch
from

Conversation

benjamin-confino
Copy link
Contributor

@benjamin-confino benjamin-confino commented Oct 2, 2024

resolves #28857

This PR

  • Creates a centralized service for openAPI to read the relevent parts of the server.xml
  • Uses that service to read what applications/modules are included or excluded, overriding mpConfig if both are in use.

@benjamin-confino benjamin-confino marked this pull request as draft October 2, 2024 15:22
@LibbyBot
Copy link

LibbyBot commented Oct 2, 2024

Code analysis and actions

DO NOT DELETE THIS COMMENT.
  • 11 product code files were changed.

  • Please describe in a separate comment how you tested your changes.

  • 1 test infrastructure code files were changed.

  • Test failures/errors in the build could be due to these changes.

  • 12 FAT files were changed, added, or removed.

  • Check that the build did not break the affected FAT suite(s).

  • 1 NLS files were changed and need an ID review.

  • @OpenLiberty/message-reviewer Please review.

  • dev/io.openliberty.microprofile.openapi.internal.common/resources/OSGI-INF/l10n/metatype.properties

Copy link

@jantley-ibm jantley-ibm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ID review comments attached; there appears to be a placeholder reminder in a couple of spots, so when that is completed I will review that text.
If you have questions or concerns about any of the suggested changes, please let me know.
I will add the "ID reviewed" label when we agree on changes. Thanks!


include=MicroProfile OpenAPI included modules.
include=A comma separated list of modules that will be included in the OpenAPI document. Modules are in the format of {ApplicationName}/{ModuleName} or {ApplicationName} which refer to to <TODO, sort out the confusion of names>. An ApplicationName which does not specify a module will include all modules in the application. The special value "all" will include all available modules.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following appears to be a note that further work needs to be done:
to to <TODO, sort out the confusion of names>

Same appears in line 32

Change
comma separated
to
comma-separated

Change
that will be included
to
that are to be included

Change
An ApplicationName which
to
An ApplicationName that

Change
will include all modules
to
includes all modules

Change
The special value "all" will include
to
The "all" special value includes


exclude=MicroProfile OpenAPI excluded modules.
exclude=A comma separated list of that will be excluded from the OpenAPI document. Modules are in the format of {ApplicationName}/{ModuleName} or {ApplicationName} which refer to to <TODO, sort out the confusion of names>. An ApplicationName which does not specify a module will exclude all modules in the application. An exclude takes precedent over an include.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following appears to be a note that further work needs to be done:
to to <TODO, sort out the confusion of names>

Change
A comma separated list of that will be excluded
to
A comma-separated list of modules that are excluded

Change
An exclude takes precedent over an include.
to
Exclusion supersedes inclusion.

@benjamin-confino benjamin-confino force-pushed the 28857-openapi-include-exclude branch 5 times, most recently from 4595c11 to 6af9c5b Compare October 9, 2024 15:59
@benjamin-confino
Copy link
Contributor Author

#build

@LibbyBot
Copy link

LibbyBot commented Oct 9, 2024

Your personal pipeline request is at https://libh-proxy1.fyre.ibm.com/cognitive/pipelineAnalysis.html?uuid=8008f055-9474-4534-83f0-158272300a50

Target locations of links might be accessible only to IBM employees.

@LibbyBot
Copy link

@benjamin-confino benjamin-confino force-pushed the 28857-openapi-include-exclude branch 2 times, most recently from 269270e to 5ce8c23 Compare October 10, 2024 08:55
@benjamin-confino benjamin-confino force-pushed the 28857-openapi-include-exclude branch 2 times, most recently from 404b48c to e5127e8 Compare October 10, 2024 10:31
@benjamin-confino
Copy link
Contributor Author

#build

@LibbyBot
Copy link

Your personal pipeline request is at https://libh-proxy1.fyre.ibm.com/cognitive/pipelineAnalysis.html?uuid=b9f42711-27e0-4f75-ba46-0a8d69789982

Target locations of links might be accessible only to IBM employees.

@LibbyBot
Copy link

Your personal build request is at https://wasrtc.hursley.ibm.com:9443/jazz/resource/itemOid/com.ibm.team.build.BuildResult/_IoGZUIbrEe-vDLerZarrlA

Target locations of links might be accessible only to IBM employees.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would suggest moving this change into a separate PR as it's the only change in fattest.simplicity.

Changing any of the core test framework projects will mean your personal builds need to run every bucket, rather than only the ones which use your feature and therefore take a while longer to complete.

I wouldn't shy away from making changes to the test framework if they're needed for your PR, but if it's just doc changes like this I'd move it to its own PR to avoid the larger PB.

Comment on lines +39 to +43
public static interface OpenAPIAppConfigListener extends Comparable<OpenAPIAppConfigListener> {
public void processConfigUpdate();

public int getConfigListenerPriority();
}
Copy link
Member

@Azquelt Azquelt Oct 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This interface needs documented.

If it implements Comparable, it needs to also document how two objects are compared and whether that comparison is consistent with equals. However, in this case I think it would make more sense not to make this implement Comparable, and instead use a Comparator which compares based on the value returned by getConfigListenerPriority().

In addition, please document whether listeners are called with the largest priority first or the smallest priority first.

@LibbyBot
Copy link

Your personal build request is at https://wasrtc.hursley.ibm.com:9443/jazz/resource/itemOid/com.ibm.team.build.BuildResult/_NFwGMIc0Ee-vDLerZarrlA

Target locations of links might be accessible only to IBM employees.

@LibbyBot
Copy link

@LibbyBot
Copy link

The build benjamin-confino-29769-20241010-0344
https://wasrtc.hursley.ibm.com:9443/jazz/resource/itemOid/com.ibm.team.build.BuildResult/_IoGZUIbrEe-vDLerZarrlA
completed and has errors or failures.

For help analyzing your personal build, go to https://libh-proxy1.fyre.ibm.com/cognitive/buildAnalysis.html?uuid=_IoGZUIbrEe-vDLerZarrlA

# Copyright (c) 2020, 2024 IBM Corporation and others.
# Copyright (c) 2020, 2022 IBM Corporation and others.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Incorrect copyright date change?

Comment on lines +70 to +71
@SuppressWarnings("unused")
private OpenAPIAppConfigProvider openAPIAppConfigProvider;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove SuppressWarnings

@SuppressWarnings("unused")
private OpenAPIAppConfigProvider openAPIAppConfigProvider;

@Reference(cardinality = ReferenceCardinality.MANDATORY, policy = ReferencePolicy.STATIC, unbind = "unbindAppConfigListener")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unbind is not necessary here, it gets inferred based on the method names.

Comment on lines +74 to +88
public void bindAppConfigListener(OpenAPIAppConfigProvider openAPIAppConfigProvider) {
this.openAPIAppConfigProvider = openAPIAppConfigProvider;
openAPIAppConfigProvider.registerAppConfigListener(this);
}

public void unbindAppConfigListener(OpenAPIAppConfigProvider openAPIAppConfigProvider) {
openAPIAppConfigProvider.unregisterAppConfigListener(this);
this.openAPIAppConfigProvider = null;
}

@Reference
public void bindOpenAPIAppConfigProvider(OpenAPIAppConfigProvider openAPIAppConfigProvider) {
this.openAPIAppConfigProvider = null;
openAPIAppConfigProvider.registerAppConfigListener(this);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like you have two references to OpenAPIAppConfigProvider, but the first one is named appConfigListener.

Comment on lines +62 to +71
@Reference(cardinality = ReferenceCardinality.MANDATORY, policy = ReferencePolicy.STATIC, unbind = "unbindAppConfigListener")
public void bindAppConfigListener(OpenAPIAppConfigProvider openAPIAppConfigProvider) {
this.configFromServerXML = openAPIAppConfigProvider;
openAPIAppConfigProvider.registerAppConfigListener(this);
}

public void unbindAppConfigListener(OpenAPIAppConfigProvider openAPIAppConfigProvider) {
openAPIAppConfigProvider.unregisterAppConfigListener(this);
this.configFromServerXML = null;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is a static reference, it might be easier to annotate the field and register as a listener in activate/deactivate.

Technically here we're registering as a listener before we're activated.

Comment on lines +330 to +339
@Override
public synchronized void processConfigUpdate() {
//Reset everything for the next lazyInit
hasBeenInit = false;
isAll = false;
isFirst = false;
serverxmlmode = false;
included = null;
excluded = null;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have an issue with thread safety because we have this method to reset everything and re-initialize on next use.

If lazyInit was only called once it would be sufficient for only the lazyInit method to be synchronized as long as every method which needs to access any of these values calls lazyInit before doing so.

However, since we have the ability to reset, every use of any of these variables must be synchronized, otherwise they could be reset in the middle of a method which uses them.

To avoid this, I would suggest:

  • add a new static inner class to hold these values - let's call it ConfigValues
  • rather than having a lazyInit method which sets all these fields, have a method which creates, caches and returns a ConfigValues object
    • all other code which needs the ConfigValues object must retrieve it via this method
  • processConfigUpdate then only needs to delete the cached ConfigValues object
  • this method and the method to get ConfigValues may need synchronization or correct use of volatile, but everything else should be correct without synchronization

Comment on lines +213 to +215

System.out.println("GREPGREP " + a + " " + b + " " + comparator.getClass().toGenericString());

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove this (or make it proper entry trace)

Comment on lines 153 to 157
@Override
public String toString() {
lazyInit();
StringBuilder sb = new StringBuilder("Module Selection Config[");
if (isFirst) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a bit concerned about toString() doing initialization here. That's the sort of thing that can cause enabling trace to change behaviour...

Could we make it so that toString() reports "Uninitialized Config` or something similar if initialization hasn't happened yet?

Comment on lines +247 to +262
if (serverxmlmode && ProductInfo.getBetaEdition()) {
for (Iterator<ModuleName> iterator = includedNotYetSeen.iterator(); iterator.hasNext();) {
ModuleName moduleName = iterator.next();
for (ModuleInfo moduleInfo : moduleInfos) {
if (matches(moduleName, moduleInfo, false)) {
includedNotYetSeenButSeenUnderOldNaming.add(moduleName);
iterator.remove();
break;
}
}
}

for (String unmatchedInclude : includedNotYetSeenButSeenUnderOldNaming.stream().map(ModuleName::toString).collect(toList())) {
Tr.warning(tc, MessageConstants.OPENAPI_USING_WRONG_NAME_SOURCE_CWWKO1680W, Constants.MERGE_INCLUDE_CONFIG, unmatchedInclude);
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we do this check that the may have used the wrong name for excluded apps as well as included apps please?

(The other check for a name that's missing should remain as only checking the included apps).

Co-authored-by: Andrew Rouse <anrouse@uk.ibm.com>
Comment on lines +293 to +300
Map<String, ApplicationRecord> oldApps = applications;
applications = new LinkedHashMap<>();
for (ApplicationRecord record : oldApps.values()) {
//Add application uses config to decide if it creates and registers any providers in ApplicationInfo
//Rather than map from the old state to the new state when the config changes, KISS and start again.
addApplication(record.info);
}
cachedProvider = null;
Copy link
Member

@Azquelt Azquelt Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, we use synchronized (this) in the method body rather than synchronized in the signature so that the generated trace is outside the synchronized block, making it easier to debug lock contention.

Since addApplication can take a while to complete, I think we should do that here.

Comment on lines +322 to +324
public int getConfigListenerPriority() {
return 2;
}
Copy link
Member

@Azquelt Azquelt Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment that this is to ensure we're called after the ModuleSelectionConfig

Comment on lines +63 to +74
@Modified
protected void modified(Map<String, Object> properties) {
if (ProductInfo.getBetaEdition()) {
if (TraceComponent.isAnyTracingEnabled() && tc.isEventEnabled()) {
Tr.event(tc, "Processing update to server.xml");
}
processModuleConfig(properties);
for (OpenAPIAppConfigListener listener : openAPIAppConfigListeners) {
listener.processConfigUpdate();
}
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This class only cares about a subset of properties from the <mpOpenAPI> config element. If someone changes a config value that isn't related to module inclusion/exclusion then we shouldn't call the listeners since re-generating all the OpenAPI documents is a relatively expensive operation.

Comment on lines +1 to +9
<?xml version="1.0" encoding="UTF-8"?>
<!-- Used for tests that need a non-default deployment name for a war -->
<web-app version="3.0"
xmlns="http://java.sun.com/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/javaee web-app_3_0.xsd"
id="testID">
<module-name>nameFromWar</module-name>
</web-app>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use WebXmlAsset instead of having a web.xml file included manually.

Comment on lines +148 to +149
server.swapInServerXMLFromPublish(SERVER_XML_NAME);
server.waitForStringInLogUsingMark("CWWKG0017I"); //The server configuration was successfully updated
Copy link
Member

@Azquelt Azquelt Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server.swapInServerXMLFromPublish(SERVER_XML_NAME);
server.waitForStringInLogUsingMark("CWWKG0017I"); //The server configuration was successfully updated
server.setMarkToEndOfLog();
server.swapInServerXMLFromPublish(SERVER_XML_NAME);
server.waitForConfigUpdateInLogUsingMark(null);

I think we need to manually set the mark ourselves before starting the update if we want to use it afterwards.

Also there's a specific method for waiting for config updates.

But also, I would prefer it if we had a method to update the config that made it more obvious in the test what the value of the config is so you don't have to jump between lots of files to understand the test.

E.g. something that let you make calls like this:

updateConfig(server, includeApp("all"));
updateConfig(server, includeApp("app1"));
updateConfig(server,
    includeApp("app1"),
    includeApp("app2"),
    excludeModule("app2/war1"));

or like this:

openapiConfig(server).includeApp("all").apply();
openapiConfig(server).includeApp("app1").apply();
openapiConfig(server).includeApp("app1")
    .includeApp("app2")
    .excludeModule("app2/war1")
    .apply();

Edit: I've since commented on MergeServerXmlTest that we should probably cover the same cases that are covered in MergeConfigTest. Those tests could be conducted using dynamic updates, though I'm unsure whether it would cover the same cases you have in this class where there are apps deployed and you update the config.

I wanted to update this message though in case you were working through and implemented this suggestion before seeing the other comment.

Comment on lines +121 to +122
server.swapInServerXMLFromPublish(APP1_XML_NAME);
server.waitForStringInLogUsingMark("CWWKG0017I"); //The server configuration was successfully updated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
server.swapInServerXMLFromPublish(APP1_XML_NAME);
server.waitForStringInLogUsingMark("CWWKG0017I"); //The server configuration was successfully updated
server.setMarkToEndOfLog();
server.swapInServerXMLFromPublish(APP1_XML_NAME);
server.waitForConfigUpdateInLogUsingMark(null);

Comment on lines +38 to +66
@Test
public void tesWarningWhenMatchesOldNamingScheme() throws Exception {

EnterpriseArchive ear = null;
WebArchive war1 = ShrinkWrap.create(WebArchive.class, "test1.war")
.addClasses(DeploymentTestApp.class, DeploymentTestResource.class);

WebArchive war2 = ShrinkWrap.create(WebArchive.class, "test2.war")
.addClasses(DeploymentTestApp.class, DeploymentTestResource.class);

WebArchive war3 = ShrinkWrap.create(WebArchive.class, "test3.war")
.addClasses(DeploymentTestApp.class, DeploymentTestResource.class)
.addAsWebInfResource(DeploymentTestApp.class.getPackage(), "web.xml", "web.xml");

ear = ShrinkWrap.create(EnterpriseArchive.class, "testEar.ear")
.addAsModules(war1, war2, war3);
ShrinkHelper.exportAppToServer(server, ear, SERVER_ONLY, DISABLE_VALIDATION);

server.setAdditionalSystemProperties(
Collections.singletonMap("mp_openapi_extensions_liberty_merged_include", "testEar"));
server.startServer();

//Test that a warning message saying we cant find the app but could under the pre OpenAPI 4.0 rules
assertThat(
server.findStringsInLogs(
"CWWKO1680W"),
hasSize(1));

}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test method contains lots of things are aren't relevant and hides or doesn't explain everything important about the test.

I don't think it's relevant that the app is an ear, or that it contains three modules, or that one of the modules contains a web.xml. Please simplify this.

I'm unsure if the mp_openapi_extensions_liberty_merged_include property is important - this looks to be identical to the config in the server.xml? Should it be removed?

The server.xml gives a custom name to the application and adds configuration which includes the app using the wrong name. These details should either be explained in a comment or be moved into the test method.

The error message appears because the server.xml name is "nameFromServerXml" while the deployment descriptor name is "testEar" because there's no deployment descriptor so it defaults to the archive name without the extension. This detail should be explained in a comment.

There's no description of what the CWWKO1680W message is or why we expect it.

Comment on lines +60 to +64
//Test that a warning message saying we cant find the app but could under the pre OpenAPI 4.0 rules
assertThat(
server.findStringsInLogs(
"CWWKO1680W"),
hasSize(1));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The warning message is wrong, the rules differ depending on whether the config came from server.xml or MP Config, not whether we're using mpOpenAPI-4.0 or an earlier feature.

We need to ensure the message has been substituted correctly as the name of the message also includes CWWKO1680W. Since this message has placeholders, we should include the expected values for those placeholders in the regex.

I'm not sure if this message is guaranteed to have been written by the time that server.startServer() returns. We should use server.waitForStringInLogs here unless we're certain that waiting is never necessary.

Comment on lines +52 to +54
ear = ShrinkWrap.create(EnterpriseArchive.class, "testEar.ear")
.addAsModules(war1, war2, war3);
ShrinkHelper.exportAppToServer(server, ear, SERVER_ONLY, DISABLE_VALIDATION);
Copy link
Member

@Azquelt Azquelt Oct 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We're disabling validation here because the name of the app doesn't match the archive name. That's probably worth a comment.

Since we do expect the app to start, we should do the validation manually by calling server.addInstalledAppForValidation("nameFromServerXml"); ourselves after exporting the app.

Comment on lines +66 to +67
@RunWith(FATRunner.class)
public class MergeServerXMLTest {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test class appears to have been adapted from MergeTest?

MergeTest is meant to test that we correctly merge documents, all tests run with a simple config which merges all apps.

I don't think we need to repeat the same tests with with the same basic config in server.xml.

MergeConfigTest is our main test for the merging config - we should include all the include and exclude cases from there in our server.xml config testing. Since the server.xml config is dynamic, we won't have to restart the server between each test like we do in MergeConfigTest.

The new cases that you've covered in this class where you're updating the config are obviously still valuable.

@@ -48,6 +48,7 @@
import io.openliberty.microprofile.openapi20.internal.cache.ConfigSerializer;
import io.openliberty.microprofile.openapi20.internal.services.ConfigFieldProvider;
import io.openliberty.microprofile.openapi20.internal.services.ModelGenerator;
import io.openliberty.microprofile.openapi20.internal.services.ModuleSelectionConfig;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spurious import

Comment on lines +248 to +261
for (Iterator<ModuleName> iterator = includedNotYetSeen.iterator(); iterator.hasNext();) {
ModuleName moduleName = iterator.next();
for (ModuleInfo moduleInfo : moduleInfos) {
if (matches(moduleName, moduleInfo, false)) {
includedNotYetSeenButSeenUnderOldNaming.add(moduleName);
iterator.remove();
break;
}
}
}

for (String unmatchedInclude : includedNotYetSeenButSeenUnderOldNaming.stream().map(ModuleName::toString).collect(toList())) {
Tr.warning(tc, MessageConstants.OPENAPI_USING_WRONG_NAME_SOURCE_CWWKO1680W, Constants.MERGE_INCLUDE_CONFIG, unmatchedInclude);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This code needs updating because the placeholders in the message have been updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add MP OpenAPI server.xml config to include or exclude apps (2.0+)
5 participants