-
Notifications
You must be signed in to change notification settings - Fork 3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crates are more than tables #274
Comments
How much do we want to change (if at all) the Crate API? Could we do something like this for feature service sources? new Crate(source_name='https://url-to-feature-service', source_workspace=None...) And then either to a URL regex on Or maybe What could this look like for custom API requests? Do we need to pass some sort of optional def get_data():
#: request data
#: build a table and then return the path to it
new Crate(source_name=None, source_workspace=None, get_data=get_data...) |
What about creating some basic data readers, storing the common ones in forklift, and then passing the data reader function to the crate at creation time. |
Something like? from forklift.readers import feature_service
new Crate(source_name='https://url-to-feature-service', source_workspace=None, get_data=feature_service...) |
what about from forklift.readers import feature_service, json_api, csv_api
data_reader = lambda: feature_service(url, and, other, options, like, credentials)
json_data_reader = lambda: json_api(url, and, maybe another function for filtering and transforming?)
new Crate(source_name='My data set', source_workspace=None, get_data=data_reader)
new Crate(source_name='Sure sites', source_workspace=None, get_data=json_data_reader) The reader may also need another function for filtering fields etc. |
I like it! I wasn't thinking about the issue with new Crate(destination_name='MyFeatureClass', destination_workspace='database.gdb', get_data=data_reader) |
Do you think a breaking change to the crate constructor/name lookup would make this nicer? Do you think it would create a lot of work for our pallets? Do you think it would be worth the improvement to the API surface to simplify the crate constructor? |
I'm not sure. It would definitely be a lot of work. But the best that I could come up with as for required init params is:
Maybe that will become a mess in the future? Forklift is already a big, hairy beast so I'm open to simplification even if it requires some up-front work. |
In the future crates will have data sources that do not resemble the tables that we expect. They will be web api requests to map servers or custom apis.
We need to come up with a solution to take structured data and make it a table for the hashing to work or use a different solution.
We should brainstorm some solutions and maybe prototype a few things.
The text was updated successfully, but these errors were encountered: