Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fulltext refactor, including dupes checking #196

Open
wants to merge 78 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
78 commits
Select commit Hold shift + click to select a range
eb4d977
feat(test): add fixtures and docs to facilitate adding `Fulltext`
spool Aug 18, 2023
c298557
feat(test): add fixtures and docs to facilitate adding `Fulltext`
spool Aug 18, 2023
07b97ef
feat(fulltext): update `Fulltext` model for `alto2txt2fixtures` changes
spool Sep 23, 2023
05662dd
[pre-commit.ci] pre-commit autoupdate
pre-commit-ci[bot] Sep 26, 2023
00d8141
fix(test): correct `Fulltext.plaintext_path` -> `Fulltext.path`
spool Sep 28, 2023
890851e
merge: main with `fulltext-hmd`
spool Sep 28, 2023
887b066
[pre-commit.ci] pre-commit autoupdate
pre-commit-ci[bot] Sep 26, 2023
d8d9aa6
Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int…
spool Sep 29, 2023
dd42c7f
merge: `poetry.lock` fix
spool Oct 1, 2023
c3a7986
fix: update deps for `linux` server build
griff-rees Oct 2, 2023
ff7d89b
fix: `bootstrap.js` in `vendors.js` 5.3.1 -> 5.3.2
griff-rees Oct 2, 2023
e621e7e
chore: squash `gazetteer`, `mitchells` and `newspaper` database migra…
griff-rees Oct 2, 2023
3f4d3d6
merge: fetch from GitHub
spool Oct 4, 2023
257c618
Merge remote-tracking branch 'origin/fulltext-hmd' into fulltext-hmd-…
griff-rees Oct 4, 2023
2d0bcfc
feat: add `lwmdb.utils::DupeRemoveConfig` and helper functions
griff-rees Oct 6, 2023
9adc487
fix(ci): apply `pre-commit` to last commit
griff-rees Oct 6, 2023
55abc50
chore: squash `newspaper` database migrations and fix docstrings
spool Oct 7, 2023
ee44fc2
chore: remove backup `newspaper` and `mitchells` migrations
spool Oct 7, 2023
3a9ed15
chore: remove residual `mitchells` migration squash
spool Oct 7, 2023
ee8e81f
chore: clean up comments in `lwmdb/utils.py`
spool Oct 7, 2023
a660141
fix(DedupeConfig): add some tests and fixes around `DedupeConfig`
spool Oct 7, 2023
83d2b06
fix: clean up `Mitchells` migration
spool Oct 7, 2023
defdade
feat(utils): improve info `dupes_to_rm` returns and logs
spool Oct 7, 2023
637b4d4
fix(convert_similar_qs_to_records): aggregate multiple dupe cases
spool Oct 7, 2023
6cae85d
feat(test): parameterize `test_dupes_to_rm` for dupes > 1
spool Oct 7, 2023
c9a70be
feat(test): ensure full `Newspaper` aggregation tested in `test_dupes…
spool Oct 7, 2023
fe34277
fix(fulltext): mv previous 0001_initial.py -> 0001_initial.py.bak
spool Oct 7, 2023
7430c3d
fix(fulltext): fix migrations squash to 0001_initial.py
spool Oct 7, 2023
eb76ee9
fix(DedupeConfig): attempt to ensure `records_to_keep` are filterable…
spool Oct 7, 2023
96ac85e
fix(filter_by_not_all_null_fk): qs.Model -> qs.model
spool Oct 7, 2023
ebcb0c3
fix(filter_by_not_all_null_fk): `qs.model.filter` -> `qs.model.object…
spool Oct 7, 2023
8a4d5bf
fix(doc): replace mount config in `local.yml` to fix live reload
spool Oct 8, 2023
b1b0efa
fix(utils): fix and rename `filter_by_not_all_null_fk` -> `qs_to_full…
griff-rees Oct 8, 2023
6e5562e
merge: resolve conflict
griff-rees Oct 8, 2023
937450a
(doc): add `manage_duplication.md` to document dedupe options
griff-rees Oct 8, 2023
c348b66
feat(doc): expand examples demonstrating managing duplicate records
griff-rees Oct 8, 2023
07af889
feat(ci): remove `fulltext-hmd-dedupe` and add `fulltext-hmd` branche…
griff-rees Oct 8, 2023
4cbfb80
doc: update `lwmdb-doc.svg` database diagram schema
spool Oct 8, 2023
2c28b88
feat(doc): update `lwmd-doc.svg` -> `lwmdb-doc-fulltext.svg`
spool Oct 8, 2023
97d9a02
feat(doc): rename `lwmdb-doc-fulltext.svg` -> `docs/assets/lwmdb-sche…
spool Oct 8, 2023
ae6959f
feat: add required `errors` field to `Fulltext` and fix `mkdocs` `loc…
spool Oct 8, 2023
dbfdd8c
fix(fulltext): rm `fulltext` `fixture_path` field
spool Oct 14, 2023
f57156e
fix(fulltext): squash `fulltext` merges.
spool Oct 14, 2023
afe8421
fix(fulltext): `rm` squashed migrations
spool Oct 14, 2023
d7962d2
feat(fulltext): finish `fulltext/migrations` squash
spool Oct 15, 2023
518d6b1
fix(ci): fix static `whitenoise` `production` deployment
griff-rees Oct 16, 2023
f629fd6
fix(chore): apply `pre-commit` lenting etc.
griff-rees Oct 16, 2023
b03af4b
feat(refactor): move `Fulltext` model to `newspapers` and add `produc…
spool Oct 24, 2023
3040f5c
feat(FullText): rename Fulltext to FullText and tidy references/tests
spool Oct 29, 2023
58eda6e
feat(FullText): replace `fulltext` references with `full_text`
spool Oct 29, 2023
ba52244
feat(FullText): add `item_code` and tests for matching `Item` records
spool Oct 29, 2023
0a835fa
feat(test): add `test_plaintext.json.zip` and `fixture` loading test
griff-rees Oct 30, 2023
26651fa
fix(DataProvider): set 'code' fiel to use '-' like 'bl_hmd' -> 'bl-hmd'
spool Nov 4, 2023
9f22e87
fix(test): add `newspapers/tests/__init__.py`
spool Nov 4, 2023
dc6cd41
fix(doc): typo in `lwmdb/utils.py` `docstring`
spool Nov 7, 2023
b59b4aa
feat(FullText): add means of logging edge cases for Item -> FullText
spool Nov 9, 2023
4210faf
[pre-commit.ci] pre-commit autoupdate
pre-commit-ci[bot] Nov 27, 2023
a97f4f8
fix(chore): merge `main` to `fulltext-refactor`
spool Nov 30, 2023
4b79b40
feat(ci): update `fulltext-refactor` from 'origin/pre-commit-ci-updat…
spool Nov 30, 2023
a93292f
chore(deps): bump jupyter-server from 2.9.1 to 2.11.2
dependabot[bot] Dec 5, 2023
51c6817
[pre-commit.ci] pre-commit autoupdate
pre-commit-ci[bot] Dec 11, 2023
41ccffe
refactor: rename `DupeRemov` -> `DupeFix`
spool Dec 15, 2023
b0f49e2
Merge remote-tracking branch 'origin/dependabot/pip/jupyter-server-2.…
spool Dec 15, 2023
e9cf7be
Merge remote-tracking branch 'origin/pre-commit-ci-update-config' int…
spool Dec 15, 2023
8fd2fb1
fix: rename `DupeFixConfig` -> `DupeConfig` and fix `mitchells_data_p…
spool Dec 15, 2023
4ddfee7
chore(deps-dev): bump jinja2 from 3.1.2 to 3.1.3
dependabot[bot] Jan 11, 2024
b2fc6e8
fix(doc): docstring typos, improve func names, mailhog -> mailpit
spool Jan 17, 2024
382d419
fix(chore): update jinja2 -> 3.1.3
spool Jan 17, 2024
4bcc27a
fix(chore): merge `pre-commit` updates
spool Jan 17, 2024
607672c
fix(ci): add `pre-fulltext-backups` to `.dockerignore`
spool Jan 17, 2024
3cbe379
feat(ui): front end layout and branding
spool Jan 18, 2024
1dc1c02
fix(ci): add admin_cursor_navigator to mypy pre-commit config
spool Jan 18, 2024
0fa1aa2
fix: rm file
spool Jan 18, 2024
7199134
fix(ci): add migrations used in server config
griff-rees Jan 18, 2024
44bc2c6
Merge remote-tracking branch 'origin/fulltext-refactor' into fulltext…
griff-rees Jan 18, 2024
d63b056
fix: fulltext.__str__, mkdocs docker, svg formatting
griff-rees Jan 20, 2024
1a3d189
fix: rm `mitchells` and `newspaper` migrations
griff-rees Feb 18, 2024
204f3c9
fix(test): add of VITURAL_HOST to ProductionENVGenConfig write test
spool Feb 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix(fulltext): squash fulltext merges.
  • Loading branch information
spool committed Oct 14, 2023
commit f57156e206b2efed480d93b1ba7a93d151bb8e8e
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Generated by Django 4.2.4 on 2023-10-14 11:04

from django.db import migrations, models


class Migration(migrations.Migration):
replaces = [
("fulltext", "0001_initial"),
("fulltext", "0002_fulltext_errors"),
("fulltext", "0003_remove_fulltext_fixture_path"),
]

initial = True

dependencies = []

operations = [
migrations.CreateModel(
name="Fulltext",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
("created_at", models.DateTimeField(auto_now_add=True)),
("updated_at", models.DateTimeField(auto_now=True)),
("text", models.TextField()),
(
"compressed_path",
models.CharField(blank=True, max_length=200, null=True),
),
("path", models.CharField(blank=True, max_length=200, null=True)),
("errors", models.TextField(blank=True, null=True)),
],
),
]
Loading