Multiple storage accounts

[alexwilcoxson-rel]

Hi I am evaluating databend for our use case. We have multiple Azure storage accounts for different clients' data. Can a single databend instance have separate databases that are managed in separate storage accounts?

[sundy-li]

We support external location in fuse engine, see https://databend.rs/doc/sql-commands/ddl/table/ddl-create-table#create-table--external_location

[alexwilcoxson-rel]

Thanks! That looks like it only supports AWS at this time. Also are there disadvantages with external tables compared to within FUSE?

[sundy-li]

Yes, you can have your custom UriLocation.

databend/src/query/sql/src/planner/binder/location.rs

Lines 268 to 332 in 993421e

	pub fn parse_uri_location(l: &mut UriLocation) -> Result<(StorageParams, String)> {
	// Path endswith `/` means it's a directory, otherwise it's a file.
	// If the path is a directory, we will use this path as root.
	// If the path is a file, we will use `/` as root (which is the default value)
	let (root, path) = if l.path.ends_with('/') {
	(l.path.clone(), "/".to_string())
	} else {
	("/".to_string(), l.path.clone())
	};
	let protocol = l.protocol.parse::<Scheme>()?;
	let sp = match protocol {
	Scheme::Azblob => parse_azure_params(l, root)?,
	// Wait for https://github.com/datafuselabs/opendal/pull/1101
	//
	// Scheme::Ftp => StorageParams::Ftp(StorageFtpConfig {
	// endpoint: if !l.protocol.is_empty() {
	// format!("{}://{}", l.protocol, l.name)
	// } else {
	// // no protocol prefix will be seen as using FTPS connection
	// format!("ftps://{}", l.name)
	// },
	// root: root.to_string(),
	// username: l.connection.get("username").cloned().unwrap_or_default(),
	// password: l.connection.get("password").cloned().unwrap_or_default(),
	// }),
	Scheme::Gcs => parse_gcs_params(l)?,
	#[cfg(feature = "storage-hdfs")]
	Scheme::Hdfs => parse_hdfs_params(l)?,
	Scheme::Ipfs => parse_ipfs_params(l)?,
	Scheme::S3 => parse_s3_params(l, root)?,
	Scheme::Oss => parse_oss_params(l, root)?,
	Scheme::Http => {
	// Make sure path has been percent decoded before parse pattern.
	let path = percent_decode_str(&l.path).decode_utf8_lossy();
	let cfg = StorageHttpConfig {
	endpoint_url: format!("{}://{}", l.protocol, l.name),
	paths: globiter::Pattern::parse(&path)
	.map_err(|err| {
	Error::new(
	ErrorKind::InvalidInput,
	anyhow!("input path is not a valid glob: {err:?}"),
	)
	})?
	.iter()
	.collect(),
	};
	// HTTP is special that we don't support dir, always return / instead.
	return Ok((StorageParams::Http(cfg), "/".to_string()));
	}
	Scheme::Fs => {
	let cfg = StorageFsConfig { root };
	StorageParams::Fs(cfg)
	}
	v => {
	return Err(Error::new(
	ErrorKind::InvalidInput,
	anyhow!("{v} is not allowed to be used as uri location"),
	));
	}
	};
	Ok((sp, path))

@ariesdevil need to improve the documents.