Add $removeheader modifier support

[ameshkov]

$removeheader

Rules with $removeheader modifier are intended to remove headers from HTTP requests and responses. The initial motivation for this rule type is to be able to get rid of the Refresh header which is often used to redirect users to an undesirable location. However, this is not the only case where this modifier can be useful.

Just like $csp, $redirect, $removeparam, and $cookie, this modifier exists independently, rules with it do not depend on the regular basic rules, i.e. regular exception or blocking rules will not affect it. By default, it only affects response headers. However, you can also change it to remove headers from HTTP requests as well.

Syntax

Basic syntax

• ||example.org^$removeheader=header-name -- removes a response header called header-name
• ||example.org^$removeheader=request:header-name -- removes a request header called header-name

Please note, that $removeheader is case-insensitive, but we suggest always use lower case.

Negating $removeheader

This type of rules works pretty much the same way it works with $csp and $redirect modifiers.

Use @@ to negate $removeheader:

• @@||example.org^$removeheader -- negates all $removeheader rules for URLs that match ||example.org^.
• @@||example.org^$removeheader=header -- negates the rule with $removeheader=header for any request matching ||example.org^.
• $removeheader rules can also be disabled by $document and $urlblock exception rules. But basic exception rules without modifiers don't do that. For example, @@||example.com^ will not disable $removeheader=p for requests to example.com, but @@||example.com^$urlblock will.

Multiple rules matching a single request
In case if multiple $removeheader rules match a single request, we will apply each of them one by one.

Restrictions

1. Please note that this type of rules can be used only in trusted filters. This category includes your own User filter and all the filters created by AdGuard Team.
2. We don't want to lower the security level so $removeheader cannot remove headers from the list below:

• access-control-allow-origin
• access-control-allow-credentials
• access-control-allow-headers
• access-control-allow-methods
• access-control-expose-headers
• access-control-max-age
• access-control-request-headers
• access-control-request-method
• origin
• timing-allow-origin
• allow
• cross-origin-embedder-policy
• cross-origin-opener-policy
• cross-origin-resource-policy
• content-security-policy
• content-security-policy-report-only
• expect-ct
• feature-policy
• origin-isolation
• strict-transport-security
• upgrade-insecure-requests
• x-content-type-options
• x-download-options
• x-frame-options
• x-permitted-cross-domain-policies
• x-powered-by
• x-xss-protection
• public-key-pins
• public-key-pins-report-only
• sec-websocket-key
• sec-websocket-extensions
• sec-websocket-accept
• sec-websocket-protocol
• sec-websocket-version
• p3p
• sec-fetch-mode
• sec-fetch-dest
• sec-fetch-site
• sec-fetch-user
• referrer-policy
• content-type
• content-length
• accept
• accept-encoding
• host
• connection
• transfer-encoding
• upgrade

3. $removeheader rules are not compatible with any other modifiers except $domain, $third-party, $app, $important, $match-case and content type modifiers (e.g. $script, $stylesheet, etc). The rules which have any other modifiers are considered invalid and will be discarded.

Examples

• ||example.org^$removeheader=refresh - removes Refresh header from all HTTP responses returned by example.org and it's subdomains.
• ||example.org^$removeheader=request:x-client-data - removes X-Client-Data header from all HTTP requests
• Removes Refresh and Location headers from all HTTP responses returned by example.org save for requests to example.org/path/* for which no headers will be removed.

  ||example.org^$removeheader=refresh
  ||example.org^$removeheader=location
  @@||example.org/path/$removeheader
  

[ameshkov]

Once this task is completed, we should also:

1. Add a new task to tsurlfilter repo
2. Add its description to the Knowledge base
3. Implement automatic conversion: Automatically convert uBO-type responseheader rules to AdGuard-specific #1429
4. Add a test for it to testcases.adguard.com

[sxgunchenko]

any other headers you'd like to keep protected?

Maybe the following:

1. related to security and privacy:

• Sec-Fetch-Mode
• Sec-Fetch-Dest
• Sec-Fetch-Site
• Sec-Fetch-User
• Referrer-Policy

2. related to filtering, removing which would decrease the filtering capabilities:

• Content-Type
• Accept
• Accept-Encoding

3. related to HTTP data consistency:

• Host
• Connection
• Transfer-Encoding
• Upgrade

[ameshkov]

Yep, sounds good. Please edit the spec accordingly.

[sxgunchenko]

It's not mentioned in the spec, but should we provide a way to negate such rules only for requests or responses?
Like @@||example.org^$removeheader=request which would negate all $removeheader rules for requests to the URLs that match ||example.org^.

But with the current syntax, we cannot provide the same possibility for responses.

[ameshkov]

I suppose negating ALL removeheader rules should be enough, i.e. @@||example.org^$removeheader.

[dnmTX]

We don't want to lower the security level....

@ameshkov not able to remove:
sec-websocket-key
sec-websocket-extensions
sec-websocket-accept
sec-websocket-protocol
sec-websocket-version
....is exactly the opposite. WebSockets are uncontrolablle,even iptables is not able to sniff the packets. It's pure hacking.
So....let me spend few more nights,figuring out how to block them with my makeshift slash amateurish hour scripts.
Very nice !!!!!

[ameshkov]

@dnmTX well, removing headers won't help you control WebSockets. Blocking them is easy already (something like this: |wss://*), and if you want to look "inside" a WebSocket, you probably need to use Javascript for that.

[Bluscream]

I would also suggest allowing blocking ANY header in the user filter atleast. If a user manually lowers their security, it's their responsibilitiy, not yours. I specifically got Adguard because it did not take away my decisions of what is "safe" and what isn't

For anyone else who want's full control over their traffic: https://alternativeto.net/software/requestly/about/

https://app.requestly.io/rules/#sharedList/1673471223673-Strip-Restricting-Headers