Ignore diacritics by default when searching

[longrunningprocess]

Description

This feature will change the default search to include "diacritic-agnostic" results. It will also add a new advanced option allowing users to narrow results based on diacritics.

Fixes #1114

Type of Change

• New feature (non-breaking change which adds functionality)
• UI change

Screenshots

Provided with test cases below

How Has This Been Tested?

Project used for testing (5491 entries): Alex's project.zip

Functional testing (should work the same in both list and entry view)

• Searching for chaca or CHACA should result in 124 matches with no advanced options selected

• Searching for chaca or CHACA should result in 90 matches when "matching diacritics"

• Searching for chacä or CHACÄ should result in 124 matches with no advanced options selected

• Searching for chacä or CHACÄ should result in 9 matches when "matching diacritics"

• Searching for chacå or CHACÅ should result in 124 matches with no advanced options selected

• Searching for chacå or CHACÅ should result in 0 matches when "matching diacritics"

• Checking the "Match diacritics" option should cause the "Options" changed indicator to appear

• see Add "whole word" search option #1229 for more regression tests

Checklist:

• I have performed a self-review of my own code
• I have reviewed the title/description of this PR which will be used as the squashed PR commit message
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works

[longrunningprocess]

Reviewer can check this for browser support: https://caniuse.com/?search=unicode%20property%20escapes. Please ensure these are ok.

I saw something about another property called InCombiningDiacriticalMarks and I'm not sure yet whether that is more appropriate or not, if reviewer has some thoughts I'd appreciate them.

[megahirt]

Approved with a ⭐ !
I'm impressed with your independent work to take advantage of research resources (C Hubbard) and to design and implement the solution. You also took care to make sure that the UX is consistent across products and consulted Alex as power user to make sure we are getting it right.

This is great work and will be immediately helpful to users when it is shipped.

Your PR description and test scenarios are first class - well done.

[megahirt]

💥 This is it! Well done, and elegant too. I didn't know about Regex Unicode Property Escapes so my own solution probably would have been multiple lines.

[longrunningprocess]

great, I'm glad you're pleased with it, thanks for reviewing. Just to confirm, you're good with the browser support and you think \p{Diacritic} is good, no need to further investigate InCombiningDiacriticalMarks?

[megahirt]

I would like to add an additional test case that demonstrates diacritics keyed in both NFC and NFD form. The test should demonstrate that diacritics are ignored regardless of NFD/NFC input. I snagged some images from the Unicode docs on normalization forms to show some sample characters to search for with their codepoints, where these diacritics are safely ignored in search. How you key in these characters depends on your OS and keyboard software.

As I write this, I realize that "exact match" might not work in certain edge cases where the input query is not in the same normalized form (NFC/NFD) as the data being searched. The fix is of course to normalize the query and data before comparison. This would be a separate issue to look into, not part of this PR.

[longrunningprocess]

I would like to add an additional test case that demonstrates diacritics keyed in both NFC and NFD form. The test should demonstrate that diacritics are ignored regardless of NFD/NFC input. I snagged some images from the Unicode docs on normalization forms to show some sample characters to search for with their codepoints, where these diacritics are safely ignored in search. How you key in these characters depends on your OS and keyboard software.

As I write this, I realize that "exact match" might not work in certain edge cases where the input query is not in the same normalized form (NFC/NFD) as the data being searched. The fix is of course to normalize the query and data before comparison. This would be a separate issue to look into, not part of this PR.

ok, thanks for thinking of some additional use cases, I've felt like I was on shaking ground with my limited testing. I'll see if I can figure out if I can doing something in the browser console to test this.

for my ref:

String.fromCharCode(0x1e69)
'ṩ'
String.fromCharCode(0x1e0b, 0x0323)
'ḍ̇'
String.fromCharCode(0x0071, 0x0307, 0x0323)
'q̣̇'

[longrunningprocess]

FYI @megahirt a brief NFD/NFC test:

good (except for highlighting) in the default search mode:

no good when attempting to match diacritics:

[longrunningprocess]

I'm going to go ahead and merge this under the assumption you did check the browser support and were ok as well as anticipating InCombiningDiacriticalMarks is probably irrelevant.

[megahirt]

@longrunningprocess in my testing this, I realized that we didn't address the highlighting that matches a string. I'm pretty sure you are already aware of it, and that's fine. I'm mostly just commenting about it explicitly for future reference.

The highlightMatches() method would need to be rewritten or modified to take into a account the part about ignoring diacritics.

When I search for "chu" in the default mode, only the exact matches for "chu" are selected, even though the results show those that include diacritics, see below.

(whole word is selected as an option to reduce hits)

What I would expect to see is that all headwords that contain a form of "chu" are highlighted, including:

• ćhü
• ćhu
  etc.

[longrunningprocess]

@longrunningprocess in my testing this, I realized that we didn't address the highlighting that matches a string. I'm pretty sure you are already aware of it, and that's fine. I'm mostly just commenting about it explicitly for future reference.

I wasn't even looking at it, I'm glad you put fresh eyes on it. I'll convert your finding to a new issue.