Add Accept-Language header to URL previews requests

[vurpo]

Because some webservers decide upon what language to serve their content in by what country the HTTP client is located in (if no language header is present in the request), this can lead to URL previews being in foreign languages for users if their homeserver is located in a foreign country.

For example, vurpo.fi is hosted on OVH and physically located in France, and I get some URL previews (for example Facebook) in French. Back when I hosted my server in Finland, I had the same problem but in Finnish.

The solution would be to add an Accept-Language header to the URL previews request. Webservers use this as a hint for what language it should serve its content in. The simplest way to configure this would just be to have a key in homeserver.yaml decide what language this header should contain (although some people might want to configure this per-user, it's probably not worth the complexity to implement that at this time).

[rkfg]

For the record, sometimes it's even worse. My server is in Netherlands but the previews from google.com are in Ukrainian (wat?), I really have no idea why.

[rkfg]

A simple patch with a hardcoded language:

diff --git a/synapse/rest/media/v1/preview_url_resource.py b/synapse/rest/media/v1/preview_url_resource.py
index adca49064..00b63a57c 100644
--- a/synapse/rest/media/v1/preview_url_resource.py
+++ b/synapse/rest/media/v1/preview_url_resource.py
@@ -295,7 +295,7 @@ class PreviewUrlResource(Resource):
             try:
                 logger.debug("Trying to get url '%s'" % url)
                 length, headers, uri, code = yield self.client.get_file(
-                    url, output_stream=f, max_size=self.max_spider_size,
+                    url, output_stream=f, max_size=self.max_spider_size, headers={b"Accept-language": [b"en"]},
                 )
             except Exception as e:
                 # FIXME: pass through 404s and other error messages nicely

I haven't tested this against the master branch, only patched the current 0.32.1 from the repo. I set the language to ru-RU and the previews are in Russian now instead of Dutch.

[ara4n]

currently our youtube previews are coming in Indonesian(!?)... and apparently YT is blocked there currently, causing element-hq/element-web#11812

[ptman]

What should the language be, then? A configurable default per home server? A state key per room?

[rkfg]

I think a good start would be a server-level default that can be set in the config. Then the per room overrides could be added if required but it might be a part of a bigger idea like an attribute specifying the room language for search/grouping purposes.

I suppose most of the Matrix servers are not public and all users there speak the same language, often not the language of the country the server physically located in. Hence a server-wide language for all rooms there would be a very welcome feature.

[anoadragon453]

The ability to specify the Accept-Language header on a server-wide scale in the config file has been added in #7265.

I imagine we want to be able to do this at a more granular level in the future. I've created #7277 to track progress on that.