Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Core/Google Drive datasource #135

Merged
merged 43 commits into from
Nov 21, 2023
Merged

Core/Google Drive datasource #135

merged 43 commits into from
Nov 21, 2023

Conversation

osala-eng
Copy link
Contributor

Description

Please include a summary of the change, e.g. what the new feature # is and/or what bug # it fixes. Please also include relevant motivation and context. List any dependencies that are required for this change.

Fixes/Implements #(issue/feature)

implements #134

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Refactor and code improvement (non-breaking change which improves code quality and/or performance)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation
  • Tests
  • Other chores such as maintenance

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration by modifying the list below.

  • Test A
  • Test B
  • Test C

Test Configuration:

Please describe the test setup. List them below as bullet points.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings
  • The commit message follows the convention of this project

@osala-eng osala-eng linked an issue Oct 28, 2023 that may be closed by this pull request
CORE: G Drive as data source #134
Closed
osala-eng
osala-eng commented Oct 28, 2023
View reviewed changes
Copy link
Contributor Author

@osala-eng osala-eng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@janaka This is ready for a look, left a comment on opendal reader

source/docq/data_source/googledrive.py Outdated Show resolved Hide resolved
@osala-eng osala-eng marked this pull request as ready for review November 4, 2023 23:12
@osala-eng osala-eng requested a review from a team as a code owner November 4, 2023 23:12
janaka
janaka requested changes Nov 6, 2023
View reviewed changes
Copy link
Contributor

@janaka janaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments essentially around how the datasource settings are implemented in a way that couples the backend to the UI too much.

If it makes it simpler drop the fancy folder selection UI for not (even though a nicer UX). But it feels like the creds setting still requires a new funcitonality unless we chang that to "paste your key here" which I don't think we should do.

source/docq/data_source/googledrive.py Outdated Show resolved Hide resolved
source/docq/data_source/googledrive.py Outdated Show resolved Hide resolved
source/docq/data_source/support/opendal_reader/base.py Show resolved Hide resolved
source/docq/domain.py Outdated Show resolved Hide resolved
web/utils/layout.py Outdated Show resolved Hide resolved
source/docq/manage_documents.py Show resolved Hide resolved
web/utils/layout.py Outdated Show resolved Hide resolved
web/utils/layout.py Outdated Show resolved Hide resolved
@osala-eng
chore: Add Google Drive env configs to template files.
9b29644
@osala-eng
chore: Add Google Drive env configs to template files.
e94e67d
@osala-eng
chore: Add Google Drive API modules to .toml files.
1beff9b
@osala-eng
feat: Add UI for requesting google Authorization
5cfe7aa
@osala-eng
feat: Create Google Drive API helper service
c187a18
@osala-eng
feat: Create Google Drive data source
598d907
@osala-eng
chore: Create a local credentials store.
870c265
@osala-eng
chore: Setup first draft of Google Drive data source.
6a54215
@osala-eng
fix: Update data_source base reader.
c73935a
@osala-eng
chore: Try using opendal to load gdrive and switch to local loader if…
c0fe0dc 
… it fails.
@osala-eng
chore: Update services init method
1c57ab0 
Update poe env
@osala-eng
chore: Update env templates.
032a010
@osala-eng
chore: Update env templates.
f387855
@osala-eng
chore: Update GDrive data source.
4cb974a
@osala-eng
chore: Fix minor bugs.
b8fb4c5
@osala-eng
chore: Added support for arguments in ConfigKeyHandlers.
fa374fc
@osala-eng
chore: Added support for arguments in ConfigKeyHandlers.
403ff67
@osala-eng
chore: Allow indexer to use filenames when downloading files from Goo…
8066178 
…gle Drive.
@osala-eng
chore: Remove test in google drive module __doc__.
4ca779a
@osala-eng
chore: Make google drive downloader async.
6fb2289
@osala-eng
chore: Enable downloading google doc files as pdf.
b900d5e
@osala-eng
chore: Update base reader.
51ff552
@osala-eng osala-eng mentioned this pull request Nov 9, 2023
Closed
osala-eng and others added 3 commits November 9, 2023 23:23
@osala-eng
fix: shared space selector (#146)
714161d 
* chore: Fix shared ask space selector not syncing with rest of the page.

* chore: Make space selector expanded by default.
@osala-eng
Merge branch 'main' of github.com:docqai/docq into core/g-drive-datas…
cb134ef 
…ource
@osala-eng @janaka
Update web/utils/layout.py
a92c108 
Co-authored-by: Janaka Abeywardhana <contact@janaka.co.uk>
janaka
janaka requested changes Nov 14, 2023
View reviewed changes
source/docq/data_source/main.py Outdated Show resolved Hide resolved
web/utils/layout.py Outdated Show resolved Hide resolved
source/docq/data_source/main.py Outdated Show resolved Hide resolved
web/utils/layout.py Outdated Show resolved Hide resolved
@janaka
Copy link
Contributor

janaka commented Nov 20, 2023

Scanning through on my phone seems all good. Will look on my laptop in a bit.

janaka
janaka approved these changes Nov 20, 2023
View reviewed changes
Copy link
Contributor

@janaka janaka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - Nice work cleaning it up.

Can you please make sure to do a full QA. That personal ask issue is the only known issue iirc.

Also, the integration tests broke for some reason a few PRs back, I couldn't figure it out so left it. Worth checking that as well.

@osala-eng
Copy link
Contributor Author

LGTM - Nice work cleaning it up.

Can you please make sure to do a full QA. That personal ask issue is the only known issue iirc.

Also, the integration tests broke for some reason a few PRs back, I couldn't figure it out so left it. Worth checking that as well.

Okay, Doing a full QA

@osala-eng osala-eng merged commit c9cd9f0 into main Nov 21, 2023
1 check passed
@osala-eng osala-eng deleted the core/g-drive-datasource branch November 21, 2023 08:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janaka janaka janaka approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

CORE: G Drive as data source
2 participants
@osala-eng @janaka