fix: respect storage options on create_index

[joshua-auchincloss]

Currently, if create_index is used with non-standard storage backends, or configuration options passed at runtime, lance does not respect the storage option specified. This can be resolved by checking and passing storage_options from the create_index methods to allow for the LanceDataset to be recreated.

[wjones127]

Hi, thanks for the PR. This method is indeed flawed, but I think there is a more sensible solution.

[wjones127]

self._ds.create_index() just updates self here:

lance/python/src/dataset.rs

Lines 928 to 934 in 8746737

	let mut new_self = self.ds.as_ref().clone();
	RT.block_on(
	None,
	new_self.create_index(&columns, idx_type, name, params.as_ref(), replace),
	)?
	.map_err(|err| PyIOError::new_err(err.to_string()))?;
	self.ds = Arc::new(new_self);

So, we might as well do return self instead. The current code re-opens the dataset, but there's no good reason we can't re-use the instance given we are already mutating it. This will make the index_cache_size parameter un-used, but I think we can just deprecate it in that case.

[joshua-auchincloss]

agree - I find it pretty weird there is a recreation in the first place

from an end-user perspective the current implementation is a blocker for usage of indices in meaningful fashions (not just local fs) so the "dirty" fix was implemented. since the set_index method is featured in the readme, should at least function with all object storages though

[wjones127]

the current implementation is a blocker for usage of indices

Why is this? Why can't you just use the original dataset?

ds = lance.write_dataset(...)
ds.create_index(...) # ignore return value
# use ds now?

[joshua-auchincloss]

Hi, thanks for the PR. This method is indeed flawed, but I think there is a more sensible solution.

no prob, happy to contribute to such an interesting project

agree completely, i view this as a pretty "dirty" fix created with the intent to patch the functionality until further review of process is implemented. would be ideal to just reuse the current instances and retain the current object store, since object storage should be static anyways relative to object instantiation at runtime