[FEA] need to support cudf.DataFrame([[‘a’,‘b’]]).corr() when set ‘method=spearman’, we only support ‘method=pearson’ so far

[dominicshanshan]

Is your feature request related to a problem? Please describe.
from time series analysis, need to support spearman correlation matrix calculation in cuDF

Describe the solution you'd like
similar like pandas.DataFrame().corr(method='spearman')

Additional context
is it possible to let me know this feature adding roadmap? big thanks !

[beckernick]

This would be a welcome addition. This could be implemented (in cuDF Python) by combining series.rank() with the existing pearson correlation. A libcudf implementation would probably want to start with Pearson correlation.

Would you be interested in contributing this?

[dominicshanshan]

love to .... but need your guidance ;) @beckernick

[beckernick]

Pearson correlation is currently the implementation of correlation in cuDF Python currently. Spearman correlation is defined mathematically as the pearson correlation of the ranks of the columns. Rank is implemented currently. This means we have everything needed in cuDF Python.

There are several ways to do this.

One way would be to use a branching statement inside corr to check whether the method is "spearman". If it is, call rank on both columns before continuing on to call cov. Another way might be to implement two "private" functions, _corr_pearson and _corr_spearman. _corr_pearson could be the existing implementation, and _corr_spearman would use rank and then call _corr_pearson. The corr function could then delegate appropriately depending on the method argument. These could even be sub-functions inside corr, since it's unlikely they will be used elsewhere. Either way, you could update the numericalcolumn and series level corr function to allow for spearman.

The dataframe implementation would need to be tweaked less, as we use CuPy's pearson correlation for efficiency. So you'd need to pass self.rank().values and support the method argument.

cudf/python/cudf/cudf/core/column/numerical.py

Lines 330 to 339 in de5577c

	def corr(self, other):
	if len(self) == 0 or len(other) == 0:
	return cudf.utils.dtypes._get_nan_for_dtype(self.dtype)
	cov = self.cov(other)
	lhs_std, rhs_std = self.std(), other.std()
	if not cov or lhs_std == 0 or rhs_std == 0:
	return cudf.utils.dtypes._get_nan_for_dtype(self.dtype)
	return cov / lhs_std / rhs_std

cudf/python/cudf/cudf/core/dataframe.py

Lines 6789 to 6795 in de5577c

	def corr(self):
	"""Compute the correlation matrix of a DataFrame.
	"""
	corr = cupy.corrcoef(self.values, rowvar=False)
	df = DataFrame(cupy.asfortranarray(corr)).set_index(self.columns)
	df.columns = self.columns
	return df

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[dominicshanshan]

need more time to finish, holiday season in China ;)