Skip to content

Commit

Permalink
Reorganize AGTree readmes
Browse files Browse the repository at this point in the history 
Merge in ADGUARD-FILTERS/tsurlfilter from feature/reorganize-agtree-readmes to master

Squashed commit of the following:

commit 77e00ec
Author: scripthunter7 <d.tota@adguard.com>
Date:   Thu Aug 10 13:14:40 2023 +0200

    Remove empty line

commit 811beea
Author: scripthunter7 <d.tota@adguard.com>
Date:   Thu Aug 10 13:08:39 2023 +0200

    Contributing guide

commit 5bd1c8c
Author: scripthunter7 <d.tota@adguard.com>
Date:   Thu Aug 10 12:53:17 2023 +0200

    Update main README

commit b9fd4b1
Author: scripthunter7 <d.tota@adguard.com>
Date:   Thu Aug 10 12:50:21 2023 +0200

    Parser TODO list

commit 37ac787
Author: scripthunter7 <d.tota@adguard.com>
Date:   Thu Aug 10 12:49:59 2023 +0200

    Separate parser README
  • Loading branch information
@scripthunter7
scripthunter7 committed Aug 10, 2023
1 parent 8419fd3 commit 45069db
Show file tree
Hide file tree
Showing 4 changed files with 400 additions and 221 deletions.
90 changes: 90 additions & 0 deletions packages/agtree/CONTRIBUTING.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Contributing

You can contribute to the project by opening a pull request. People who
contribute to AdGuard projects can receive various rewards, see
[this page][contribute] for details.

[contribute]: https://adguard.com/contribute.html

## Development & Contribution

Here is a guide on how to set up the development environment and how to
submit your changes:

- Pre-requisites: [Node.js][nodejs] (v14 or higher), [Yarn][yarn] (v2 or
higher), [Git][git]. It is important to use Yarn and not NPM, because the
project is optimized for Yarn.
- Fork the repository on GitHub. You will need to have a GitHub account for
this. If you already have a fork, make sure to update it with the latest
changes from the main repository.
- Clone *your forked repository* to your local machine with
`git clone <repository-url>`. It is important to clone your forked repository
and not the main repository, because you will not be able to push your
changes to the main repository, since you do not have the permissions to do
so.
- Install dependencies by following [this guide][main-dev-guide].
- Create a new branch with `git checkout -b <branch-name>`. Example:
`git checkout -b feature/add-some-feature`. Please add `feature/` or `fix/`
prefix to your branch name, and refer to the issue number if there is one.
Example: `fix/42`.
- Open the **project root** folder in your editor.
- Make your changes and test them.
- Check code by running `yarn check-types`, `yarn lint` and `yarn test`
commands (Husky will run these commands automatically before each commit).
- If everything is OK, commit your changes and push them to your forked
repository. Example:
- Add files to commit with `git add .`
- Commit files with `git commit -m "Add some feature"`
- Push changes to your forked repository with
`git push origin feature/add-some-feature`.
- When you are ready to submit your changes, go to your forked repository on
GitHub and create a pull request. Make sure to select the correct branch.
Example: `feature/add-some-feature` branch in your forked repository to
`master` branch in the main repository.
- After you open a pull request, GitHub Actions will run the tests on your
changes. If the tests fail, you can see the error details in the "Checks"
tab. If the tests pass, a green checkmark will appear in the "Checks" tab.
- Finally, wait for the maintainers to review your changes. If there are any
issues, you can fix them by pushing new commits to your branch. If everything
is OK, the maintainers will merge your pull request.

We would be happy to review your pull request and merge it if it is suitable for
the project.

[git]: https://git-scm.com/
[main-dev-guide]: https://github.com/AdguardTeam/tsurlfilter#development
[nodejs]: https://nodejs.org/en/
[yarn]: https://yarnpkg.com/

### Available commands

During development, you can use the following commands (listed in
`package.json`):

- `yarn build` - builds the library with [rollup][rollup] to the `dist` folder
- `yarn build-txt` - creates a `dist/build.txt` file which contains the version
of the library.
- `yarn build-types` - build type definitions with [TypeScript][typescript] to
the `dist/types` folder.
- `yarn check-compatibility-tables` - checks if the
[compatibility tables][compatibility-tables] are valid.
- `yarn check-types` - check type definitions with
[TypeScript][typescript].
- `yarn clean` - remove the `dist` folder.
- `yarn clean-types` - remove the `dist/types` folder.
- `yarn coverage` - run tests with [Jest][jest] and generate a code coverage
report.
- `yarn increment` - increment the version of the library in `package.json`
(patch version by default).
- `yarn lint` - run all linters.
- `yarn lint:md` - lint the markdown files with [markdownlint][markdownlint].
- `yarn lint:ts` - lint the code with [ESLint][eslint].
- `yarn precommit` - run all checks before committing.
- `yarn test` - run tests with [Jest][jest].

[compatibility-tables]: https://github.com/AdguardTeam/tsurlfilter/tree/master/packages/agtree/src/compatibility-tables
[eslint]: https://eslint.org/
[jest]: https://jestjs.io/
[markdownlint]: markdownlint
[rollup]: https://rollupjs.org/
[typescript]: https://www.typescriptlang.org/
265 changes: 44 additions & 221 deletions packages/agtree/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,258 +67,81 @@
Table of Contents:

- [Introduction](#introduction)
- [Usage](#usage)
- [Parsing rules](#parsing-rules)
- [Parsing filter lists](#parsing-filter-lists)
- [Development \& Contribution](#development--contribution)
- [Ideas \& Questions](#ideas--questions)
- [License](#license)
- [References](#references)

## Introduction

AGTree is a universal adblock filter list parser. It supports all syntaxes
currently in use: AdGuard, uBlock Origin and AdBlock / Adblock Plus.
AGTree is a universal tool for working with adblock filter lists. It contains
the following modules:

## Usage
- [Adblock rule converter][converter-url]
- [Adblock rule parser][parser-url]
- [Compatibility tables][compatibility-tables-url]

AGTree provides a powerful, error-tolerant parser for all kinds of ad blocking
rules. It fully supports AdGuard, uBlock Origin and Adblock Plus syntaxes, and
provides a high-detail AST for all rules.
AGTree supports all syntaxes currently in use:

Basically, the parser API has two main parts:
- <img src="https://cdn.adguard.com/website/github.com/AGLint/adg_logo.svg" width="14px"> [AdGuard][adg-url]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/ubo_logo.svg" width="14px"> [uBlock Origin][ubo-url]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/abp_logo.svg" width="14px"> [Adblock Plus][abp-url]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/ab_logo.svg" width="14px"> [AdBlock][ab-url]

- Parser: parsing rules (string &#8594; AST)
- Generator: serialization of ASTs (AST &#8594; string)

### Parsing rules

You can parse individual rules using the `RuleParser` class. It has a `parse`
method that takes a rule string and returns an AST. For example, this code:

```typescript
import { RuleParser } from "@adguard/agtree";

// RuleParser automatically determines the rule type
const ast = RuleParser.parse("/ads.js^$script");
```

will gives you this AST:

```json
{
"type": "NetworkRule",
"category": "Network",
"syntax": "Common",
"exception": false,
"pattern": {
"type": "Value",
"value": "/ads.js^"
},
"modifiers": {
"type": "ModifierList",
"children": [
{
"type": "Modifier",
"modifier": {
"type": "Value",
"value": "script"
},
"exception": false
}
]
}
}
```

<details>
<summary>Show full AST (with locations)</summary>

```json
{
"type": "NetworkRule",
"loc": {
"start": {
"offset": 0,
"line": 1,
"column": 1
},
"end": {
"offset": 15,
"line": 1,
"column": 16
}
},
"raws": {
"text": "/ads.js^$script"
},
"category": "Network",
"syntax": "Common",
"exception": false,
"pattern": {
"type": "Value",
"loc": {
"start": {
"offset": 0,
"line": 1,
"column": 1
},
"end": {
"offset": 8,
"line": 1,
"column": 9
}
},
"value": "/ads.js^"
},
"modifiers": {
"type": "ModifierList",
"loc": {
"start": {
"offset": 9,
"line": 1,
"column": 10
},
"end": {
"offset": 15,
"line": 1,
"column": 16
}
},
"children": [
{
"type": "Modifier",
"loc": {
"start": {
"offset": 9,
"line": 1,
"column": 10
},
"end": {
"offset": 15,
"line": 1,
"column": 16
}
},
"modifier": {
"type": "Value",
"loc": {
"start": {
"offset": 9,
"line": 1,
"column": 10
},
"end": {
"offset": 15,
"line": 1,
"column": 16
}
},
"value": "script"
},
"exception": false
}
]
}
}
```

</details>

As you can see, this AST is very detailed and contains all the information about
the rule: syntax, category, exception, modifiers, node locations, and so on.
Locations are especially useful for linters, since they allow you to point to
the exact place in the rule where the error occurred.

And this code:

```typescript
RuleParser.generate(ast);
```

will generate the adblock rule from the AST (serialization):

```adblock
/ads.js^$script
```

Please keep in mind that the parser omits unnecessary whitespaces, so the
generated rule may not match with the original rule character by character.
Only the formatting can change, the rule itself remains the same. You can
pass any rule to the parser, it automatically determines the type and category
of the rule. If the rule is syntactically incorrect, the parser will throw an
error.

### Parsing filter lists

You can also parse complete filter lists using the `FilterListParser` class.
It works the same way as the `RuleParser` class. Here is an example of parsing
[EasyList](https://easylist.to/easylist/easylist.txt) and generating it back:

```typescript
import { FilterListParser } from "@adguard/agtree";
import { writeFile } from "fs/promises";
// Requires installing "node-fetch" package
import fetch from "node-fetch";

// Download EasyList
const easyList = await (
await fetch("https://easylist.to/easylist/easylist.txt")
).text();

// Or read it from file
// const easyList = await readFile('easylist.txt', 'utf-8');

// Parse EasyList to AST. By default, parser is very tolerant,
// if it can't parse some rules, it will just mark them as "raw".
// If you want to disable this behavior, you can pass the second
// argument as "false" to the "parse" method, like this:
// const ast = FilterListParser.parse(easyList, false);
const ast = FilterListParser.parse(easyList);

// Generate filter list from filter list AST
const easyListGenerated = FilterListParser.generate(ast);

// Write generated filter list to file
await writeFile("easylist-generated.txt", easyListGenerated);
```
[ab-url]: https://getadblock.com
[abp-url]: https://adblockplus.org
[adg-url]: https://adguard.com
[compatibility-tables-url]: https://github.com/AdguardTeam/tsurlfilter/tree/master/packages/agtree/src/compatibility-tables
[converter-url]: https://github.com/AdguardTeam/tsurlfilter/tree/master/packages/agtree/src/converter
[parser-url]: https://github.com/AdguardTeam/tsurlfilter/tree/master/packages/agtree/src/converter
[ubo-url]: https://github.com/gorhill/uBlock

## Development & Contribution

Please read the [CONTRIBUTING.md](CONTRIBUTING.md) file for details on how to
Please read the [CONTRIBUTING.md][contributing-url] file for details on how to
contribute to this project.

[contributing-url]: https://github.com/AdguardTeam/tsurlfilter/tree/master/packages/agtree/CONTRIBUTING.md

## Ideas & Questions

If you have any questions or ideas for new features, please open an issue or a
discussion. We will be happy to discuss it with you.
If you have any questions or ideas for new features, please
[open an issue][new-issue-url] or a [discussion][discussions-url]. We will be
happy to discuss it with you.

[discussions-url]: https://github.com/AdguardTeam/tsurlfilter/discussions
[new-issue-url]: https://github.com/AdguardTeam/tsurlfilter/issues/new

## License

AGTree is licensed under the MIT License. See the [LICENSE](LICENSE) file for
details.
AGTree is licensed under the MIT License. See the [LICENSE][license-url] file
for details.

[license-url]: https://github.com/AdguardTeam/tsurlfilter/blob/master/packages/agtree/LICENSE

## References

Here are some useful links to help you write adblock rules. This list is not
exhaustive, so if you know any other useful resources, please let us know.

<!--markdownlint-disable MD013-->
- Syntax documentation:
- [AdGuard: _How to create your own ad filters_][adg-filters]
- [uBlock Origin: _Static filter syntax_][ubo-filters]
- [Adblock Plus: _How to write filters_][abp-filters]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/adg_logo.svg" width="14px"> [AdGuard: *How to create your own ad filters*][adg-filters]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/ubo_logo.svg" width="14px"> [uBlock Origin: *Static filter syntax*][ubo-filters]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/abp_logo.svg" width="14px"> [Adblock Plus: *How to write filters*][abp-filters]
- Extended CSS documentation:
- [MDN: _CSS selectors_][mdn-css-selectors]
- [AdGuard: _Extended CSS capabilities_][adg-ext-css]
- [uBlock Origin: _Procedural cosmetic filters_][ubo-procedural]
- [Adblock Plus: _Extended CSS selectors_][abp-ext-css]
- [MDN: *CSS selectors*][mdn-css-selectors]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/adg_logo.svg" width="14px"> [AdGuard: *Extended CSS capabilities*][adg-ext-css]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/ubo_logo.svg" width="14px"> [uBlock Origin: *Procedural cosmetic filters*][ubo-procedural]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/abp_logo.svg" width="14px"> [Adblock Plus: *Extended CSS selectors*][abp-ext-css]
- Scriptlets:
- [AdGuard scriptlets][adg-scriptlets]
- [uBlock Origin scriptlets][ubo-scriptlets]
- [Adblock Plus snippets][abp-snippets]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/adg_logo.svg" width="14px"> [AdGuard scriptlets][adg-scriptlets]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/ubo_logo.svg" width="14px"> [uBlock Origin scriptlets][ubo-scriptlets]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/abp_logo.svg" width="14px"> [Adblock Plus snippets][abp-snippets]
- Third party libraries:
- [CSSTree docs][css-tree-docs]
- [AdGuard's compatibility table][adg-compatibility-table]
- <img src="https://raw.githubusercontent.com/csstree/csstree/master/assets/csstree-logo-rounded.svg" width="14px"> [CSSTree docs][css-tree-docs]
- <img src="https://cdn.adguard.com/website/github.com/AGLint/adg_logo.svg" width="14px"> [AdGuard's compatibility table][adg-compatibility-table]
<!--markdownlint-enable MD013-->

[abp-ext-css]: https://help.eyeo.com/adblockplus/how-to-write-filters#elemhide-emulation
[abp-filters]: https://help.eyeo.com/adblockplus/how-to-write-filters
Expand Down
Loading

0 comments on commit 45069db

Please sign in to comment.