[Observability] Enabling distributed tracing for LMS workers

[robrap]

A.C.

• Enable distributed tracing for the New Relic app (e.g. LMS workers).
• It would be nice to see where else this should be enabled.

Two options for configuring:

1. Enable server-side config, which means turn on configuration in the New Relic UI, or
2. Add to a New Relic configuration file, but there is only one template in edx-configuration.

Downsides of option 1:
a. defaults that I think are different from what we have now, so we’d need to look that up and set things appropriately to not start getting different behavior, and
b. it’s not config as code, and
c. there are other configurations I’d like to add later, but they are only available via option 2.

Downsides of option 2:
a. We don’t have a way to provide different config for different apps.

We are actively deprecating edx-configuration. What will be the ultimate solution for files like this New Relic configuration? Can we fast track this type of work so we have an option here?

[robrap]

FYI: @jmbowman @adzuci

[adzuci]

If we do 1 now can we do 2 later? Also, could you link me to the place in configuration where this is set currently?

[adzuci]

I tried to add the escalate-to-psre label to push a request to SRE which could be edited to add ACs to ask us to provide guidance on where to add code for option 2. If that GHA workflow still doesn't work could you add a ticket directly to PSRE and tag @nadeemshahzad and I on it so we can slate it for our next sprint?

[robrap]

@adzuci:

• Here's a link to the DEPR ticket and to the New Relic config, all in one: [DEPR]: Phase out edx-configuration openedx/public-engineering#51 (comment)
• Yes, choosing option 1 is reversable. We just need to be careful, and should probably document it if we do so. It's strange that they couldn't have kept the same defaults either way.

[jristau1984]

I believe this can now be closed.