Concat only unique fields in DatasetInfo.from_merge

[mariosasko]

I thought someone from the community with less experience would be interested in fixing this issue, but that wasn't the case.

Fixes #2103

[bhavitvyamalik]

Hi @mariosasko,
Just came across this PR and I was wondering if we can use
description = "\n\n".join(OrderedDict.fromkeys([info.description for info in dataset_infos]))

This will obviate the need for unique and is almost as fast as set. We could have used dict inplace of OrderedDict but it's available 3.7+ onwards

[mariosasko]

Hi,

let's see what @lhoestq thinks. Although my approach adds more code, it's more readable IMO.

[bhavitvyamalik]

Yeah, that's true. Your approach is more readable.

[lhoestq]

Thanks !

Small preference on this definition on unique since it's way more readable that using OrderedDict.