Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize tags dashboards #3101

Merged
merged 9 commits into from
Feb 22, 2024
Merged

Conversation

mathjazz
Copy link
Collaborator

@mathjazz mathjazz commented Feb 20, 2024

Fix #2264.

I rewrote the code for retrieving data used on all three Tags dashboards:

I also:

  • dropped the unused code (which represents the vast majority of changes in this patch)
  • added an extra DB index, which significantly impacts Tags tab performance

The functionality should remain identical, but the views should be significantly faster. In the case of Firefox Tag dashboards, the load times were in the 20-30 second range (or timing out), now these pages take max. 2-3 seconds to load.

Deployed to stage.

@mathjazz mathjazz force-pushed the 2264-optimize-tags-dashboards branch 5 times, most recently from bc3f4f4 to 884c6c0 Compare February 20, 2024 21:21
Copy link
Member

@eemeli eemeli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only one minor stylistic quibble inline about the code, but I have two larger concerns:

  1. The tests for the old implementation are removed, but I don't see any corresponding tests being added for the new class Tags. Is that really covered by the remaining prior tests?
  2. The file paths and naming is messed up. TagsTool is now only used for the admin form, and it's in tags/utils/tags.py. Tags is the presumably more common non-admin part, and it's in tags/utils/utils.py. Could/should the former class and file be renamed as TagsAdmin or something similar, and Tags defined in tags.py?

Comment on lines 90 to 101
if tr["approved_date"] is not None and tr["approved_date"] > tr["date"]:
date = "approved_date"
else:
date = "date"

dates[tr[date]] = tr[group_by]
prefix = "entity__" if group_by == "resource__tag" else ""

# Find translations with matching date and tag/locale
translations |= Translation.objects.filter(
Q(**{date: tr[date], f"{prefix}{group_by}": tr[group_by]})
).prefetch_related("user", "approved_user")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if tr["approved_date"] is not None and tr["approved_date"] > tr["date"]:
date = "approved_date"
else:
date = "date"
dates[tr[date]] = tr[group_by]
prefix = "entity__" if group_by == "resource__tag" else ""
# Find translations with matching date and tag/locale
translations |= Translation.objects.filter(
Q(**{date: tr[date], f"{prefix}{group_by}": tr[group_by]})
).prefetch_related("user", "approved_user")
date = max(tr["date"], tr["approved_date"] or 0)
dates[date] = tr[group_by]
prefix = "entity__" if group_by == "resource__tag" else ""
# Find translations with matching date and tag/locale
translations |= Translation.objects.filter(
Q(**{date: date, f"{prefix}{group_by}": tr[group_by]})
).prefetch_related("user", "approved_user")

@mathjazz
Copy link
Collaborator Author

mathjazz commented Feb 21, 2024

  1. The tests for the old implementation are removed, but I don't see any corresponding tests being added for the new class Tags. Is that really covered by the remaining prior tests?

Codecov says the coverage of the utils.py is above 80% (although I can't currenty access the results page). Still, I should add a test or two for that.

  1. The file paths and naming is messed up. TagsTool is now only used for the admin form, and it's in tags/utils/tags.py. Tags is the presumably more common non-admin part, and it's in tags/utils/utils.py. Could/should the former class and file be renamed as TagsAdmin or something similar, and Tags defined in tags.py?

OK, I'll move things around a little. Maybe rename tags/utils/utils.py to tags/utils.py and move the rest from tags/utils to tags/admin.

I didn't want to touch the old code before wiping it out the way I did with the rest of the code. I should file a bug for that.

I could also add a README file to /pontoon/tags/utils explaining that everything but the utils.py is legacy code that should not be touched.

@mathjazz
Copy link
Collaborator Author

Applied the suggested change, move Admin code to /admin and filed #3108.

@mathjazz mathjazz requested a review from eemeli February 21, 2024 21:38
Comment on lines 3 to 5
from pontoon.tags.admin.exceptions import InvalidProjectError

from .base import TagsDataTool
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
from pontoon.tags.admin.exceptions import InvalidProjectError
from .base import TagsDataTool
from .base import TagsDataTool
from .exceptions import InvalidProjectError

@mathjazz mathjazz merged commit 1dcd738 into mozilla:main Feb 22, 2024
4 checks passed
@mathjazz mathjazz deleted the 2264-optimize-tags-dashboards branch February 22, 2024 20:41
ayanaar pushed a commit to ayanaar/pontoon that referenced this pull request Feb 23, 2024
The functionality should remain identical, but the views should be significantly faster. In the case of Firefox Tag dashboards, the load times were in the 20-30 second range (or timing out), now these pages take max. 2-3 seconds to load.

The patch also:
* adds an extra DB index, which significantly impacts Tags tab performance
* drops the unused code (which represents the vast majority of changes in this patch)
* moves remaining files from tags/utils to tags/admin
mathjazz added a commit to mathjazz/pontoon that referenced this pull request Feb 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimize generation of data for tags
2 participants