Experimental pagination feature

[szmarczak]

Fixes #722

Checklist

• I have read the documentation.
• I have included a pull request description of my changes.
• I have included some tests.
• If it's a new feature, I have included documentation updates.

---

IssueHunt Summary

Referenced issues

This pull request has been submitted to:

• #722: Pagination support

---

IssueHunt has been backed by the following sponsors. Become a sponsor

[sindresorhus]

We need an option for itemCount too.

[szmarczak]

We need an option for itemCount too.

Instead of an itemCount option I think it's better to have this:

const results = await got.paginate.all({
	pagination: {
		shouldContinue: allItems => allItems.length !== 3
	}
});

because people may want have custom conditions.

[szmarczak]

@rarkins Could you look at the tests and tell if the API looks ok?

[rarkins]

@rarkins Could you look at the tests and tell if the API looks ok?

I think it looks good so far, thanks

[szmarczak]

I need to make more tests here

[rarkins]

@szmarczak one day I would like to see pagination with a limit, e.g. you might want the first 1000 Pull Requests from GitHub's API. Would you anticipate that being an option to got.paginate.all or would it be a new function like got.paginate.some? Also we'd ideally like to combine it with a filter too eventually, e.g. you can get the first X paginated items that match logic Y.

[szmarczak]

A side note for me: There's a missing test for options.pagination.shouldContinue.

@rarkins You can do something like this:

const results = await got.paginate.all({
    pagination: {
        ...,
        shouldContinue: items => items.length < 1000
    }
});

That will give you max 1000 items. But if it's a common use case, then I can make options.pagination.countLimit.

[rarkins]

But if it's a common use case

I'm pretty sure it would be commonly used

[szmarczak]

I'm pretty sure it would be commonly used

So you have a new countLimit option now :) I just need to update the docs and the PR is good to merge.

[szmarczak]

I'm gonna write the docs tomorrow.

[sindresorhus]

How about we prefix the option with an underscore and document that the option is experimental and that users should lock the Got dependency as it might change in minor releases? This will give users a chance to try it out, but also gives us a chance to improve it without having to do lots of major releases?

[szmarczak]

Sure, will finish this today.

[sindresorhus]

I think we should explain the Link header logic in English prose, not just link to the source.

[sindresorhus]

#1053

[sindresorhus]

This is looking great! Nice work, @szmarczak 👌