Add support for other languages for rouge

[alexyalunin]

I calculate rouge with

from datasets import load_metric
rouge = load_metric("rouge")
rouge_output = rouge.compute(predictions=['тест тест привет'], references=['тест тест пока'], rouge_types=[
    "rouge2"])["rouge2"].mid
print(rouge_output)

the result is
Score(precision=0.0, recall=0.0, fmeasure=0.0)
It seems like the rouge_score library that this metric uses filters all non-alphanueric latin characters
in rouge_scorer/tokenize.py with text = re.sub(r"[^a-z0-9]+", " ", six.ensure_str(text)).
Please add support for other languages.

[m3hrdadfi]

@alexyalunin

I did something similar for others languages.

Repo: rouge-metric

[lvwerra]

We could update ROUGE similiar to BLEU (#19) where optionally a tokenizer can be passed. The RougeScorer accepts a tokenizer keyword argument.

cc @sashavor