feat: support post requests

[SethFalco]

Adds support for Anilist by allowing POST requests. Note, I don't do Python much, so just let me know if the code is crap. ^-^'

This is just a first iteration, it should already work without breaking any existing functionality. You're welcome to give feedback though if you feel I'd done something wrong.

Main Changes

• Adds Anilist back in to data.json. (For testing the implementation of this PR.)
• Adds an interpolate_string function, because I thought it would be nicer if we could define the request_payload as JSON rather than as a string literal. Opinion welcome!
• Allow data.json to specify the request method, defaulting to HEAD/GET based on the errorType if not specified.
• Replaces the now redundant "request_head_only": false with "request_method": "GET"

Queries

This replaces the use of .format with .replace which might be unfavorable, but I wasn't sure of what the best way to go about this was.

1. .format throws an exception if there is no {} in the string. This can be annoying since we may want to insert the username into the url, probeUrl, or request_payload. But different websites do things differently, some will want the username in the url, others might want it in the request_payload, so one of them will throw an exception. This is avoided with .replace.
2. I think it'd be a better developer/user experience if the request_payload was an object rather than a string, so the interpolate_string function would want to use .replace in this case anyway since many of the keys may not require templating.

Related

• Closes Add request method #861

[SethFalco]

If possible, it'd be wonderful if this could be reviewed/merged soon. Then I'd be happy to look into bringing back some of the removes sites.

(Already have Coil on standby, assuming this were to be merged as is.)

[sdushantha]

I tried your code with Fiverr's data but it seems like Sherlock says the username blueman does not exist while the username noonewouldeverusethis11 exists.

  "Fiverr": {
    "errorType": "message",
    "errorMsg": "user_taken",
    "url": "https://www.fiverr.com/{}",
    "headers": {
        "User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0",
        "Content-Type": "application/json;charset=utf-8"
    },
    "urlMain": "https://www.fiverr.com/",
    "urlProbe": "https://www.fiverr.com/validate_username",
    "request_method": "POST",
    "request_payload": {
        "username": "{}"
    },
    "username_claimed": "blue",
    "username_unclaimed": "noonewouldeverusethis7"
  }

I made this data by referring to the Anilist data but I have a feeling I might have done something wrong 🤔

[SethFalco]

I think it's inverted.

"errorType": "message", will say the profile doesn't exist if the response contains errorMsg, right?
But if the response contains user_taken (as in, the username was taken) then the profile does exist?

I think it should be "errorMsg": "\"status\":\"success\""? (The response when the username is available, therefore not taken by another user.)

If the response contains user_taken, then the profile exists.
If the response contains "status":"success", then the profile doesn't exist.

"Fiverr": {
  "errorType": "message",
  "errorMsg": "\"status\":\"success\"",
  "regexCheck": "^[A-Za-z][A-Za-z\\d_]{5,14}$",
  "url": "https://www.fiverr.com/{}",
  "urlMain": "https://www.fiverr.com/",
  "urlProbe": "https://www.fiverr.com/validate_username",
  "request_method": "POST",
  "request_payload": {
      "username": "{}"
  },
  "username_claimed": "blueman",
  "username_unclaimed": "noonewouldeveru"
},

This should work, but we'll need to add a regex for invalid usernames. (This has false positives for invalid usernames otherwise.)
I think something like ^[A-Za-z][A-Za-z\d_]{5,14}$ should be good from trial and error.

• Must start with a letter.
• Minimum of 6 characters.
• Maximum of 15 characters.
• Can only contain letters, numbers, and underscores.

---

Note: The check in my comment has false positives for deleted profiles. That's because this endpoint is actually checking if the username is available or not. Profiles that have been deleted still do not have their username available for others to use. With that in mind using response_url might be better unless that has its own problems.

which was found when running the tests

It looks like Fiverr was removed because of the tests, but actually both tests for Fiverr are invalid anyway based on the username regex. ^-^'

[sdushantha]

As for the actual check, I think it's inverted.
"errorType": "message", will say the profile doesn't exist if the response contains errorMsg, right?
But if the response contains user_taken (as in, the username was taken) then the profile does exist?

Ah yes! Sorry my bad, I for some reason pasted the message for the wrong request. I really appreciate the detailed response :)

[sdushantha]

Thank you for putting a lot of time and effort into this PR, its greatly appreciated! 😊