Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A/B test header on the /Cody page #6931

Closed
iskyOS opened this issue May 9, 2024 · 1 comment · Fixed by #7005 · May be fixed by #6946
Closed

A/B test header on the /Cody page #6931

iskyOS opened this issue May 9, 2024 · 1 comment · Fixed by #7005 · May be fixed by #6946
Assignees

Comments

@iskyOS
Copy link
Contributor

iskyOS commented May 9, 2024

Context
We would like to A/B test to headline section of the sourcegraph.com/cody page to test some new copy and a new format.

We have a feature flag set up in PostHog to automatically group users into "A" and "B" groups.

  • Feature flag: cody-value-headline-test
  • Control key: control
  • Test key: test-value

Product requirements:
This Figma file shows both the current headline section and the new section we'd like to try:
https://www.figma.com/file/oQgh5R0iSBG3UII6C8CPJ8/Cody-page?type=design&node-id=4582-14344&mode=dev

(For the purpose of this task, please ignore that the Figma mockups use a light/white theme instead of the current dark theme. We'll put the light theme request in a separate request)

  • These changes should not impact any other page, including sourcegraph.com/demo/cody.

  • Please make sure the feature flag from PostHog is loaded before the full pageload, so that the headline does not appear as the control group and then change after loading in the user's view.

  • For "control" group members, they should see the current headline section that's on the page. It has this copy:

"Cody is a coding AI assistant that uses AI and a deep understanding of your codebase to help you write and understand code faster."

  • For "test" group members, they should see the new copy that appears in list format:

"Cody is the AI coding assistant that understands your code.

  • Ingest Code from any code host
  • Choose from the latest AI models
  • Get relevant AI suggestions through enhanced context"
  • For "undefined" group members (in the case PostHog fails), they should see the same headline as "control" group members.

Telemetry requirements:

  • When a user lands on the Cody page, please log an enrollment event to eventLogger. The event name can be ValueHeadlineTestEnrolled
  • In the event metadata, please also log the user's variant. control or test
  • If you have any questions about telemetry, please let us know and we can clarify!
Copy link
Contributor

gitstart-app bot commented May 9, 2024

Here is the GitStart Ticket for this issue: https://clients.gitstart.com/sourcegraph/3436/tickets/SGM-6931

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants