Unexpected exception ('ascii' codec can't decode byte 0xae in position 621: ordinal not in range(128)) occurred fetching URL

[bcoles]

2020-09-09 14:42:57,715 [ERROR] Unexpected exception ('ascii' codec can't decode byte 0xae in position 621: ordinal not in range(128)) occurred fetching URL: http://<redacted>
2020-09-09 14:42:57,716 [ERROR] Traceback (most recent call last):
  File "/spiderfoot/sflib.py", line 2647, in fetchUrl
    result['content'] = res.content.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xae in position 621: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/spiderfoot/sflib.py", line 2649, in fetchUrl
    result['content'] = res.content.decode("ascii")
UnicodeDecodeError: 'ascii' codec can't decode byte 0xae in position 621: ordinal not in range(128)

[leotrubach]

Maybe just

 except UnicodeDecodeError:
    pass

or

for encoding in ("utf-8", "ascii"):
    try:
        result["content"] = res.content.decode(encoding)
    except UnicodeDecodeError:
        pass
    else:
        break
else:
    result["content"] = res.content

[bcoles]

The latter would probably be best.

This issue is left over from when SpiderFoot was written in Python2. Consumers of fetchUrl() expect res['content'] to be str(); however, in some instances it must be bytes(). It would be better to have the code fail in the module than fail in the library.

[smicallef]

I've pushed the fix provided but let's leave this bug open a little while longer until more testing has been done.