Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Lightning: Support Custom File Iteration Algorithm when Constructing MDLoader #38598

Closed
dsdashun opened this issue Oct 21, 2022 · 1 comment · Fixed by #38599
Closed

Lightning: Support Custom File Iteration Algorithm when Constructing MDLoader #38598

dsdashun opened this issue Oct 21, 2022 · 1 comment · Fixed by #38599
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.

Comments

@dsdashun
Copy link
Contributor

Enhancement

The MDLoader is the abstraction that stores the result of scanning the data source with file route rules and filters. However, in some situations, we don't need to iterate ALL the data source files to construct a complete MDLoader object. Instead, we just need to pick a few of key files in the data source to construct the information we want in MDLoader, so that the performance for generating an MDLoader can be boosted.

Currently, the file iteration logic cannot be customized, all we have is using the ExternalStorage interface, which provides a WalkDir method to iterate ALL the files in the data source. We want to make the iteration logic customizable.

@dsdashun dsdashun added the type/enhancement The issue or PR belongs to an enhancement. label Oct 21, 2022
@dsdashun
Copy link
Contributor Author

/component lightning

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component/lightning This issue is related to Lightning of TiDB. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants