Pagination in MongoDB for 100k+ documents: keyset cursor instead of skip
Why does the feed slow down on distant pages?
Classic pagination looks harmless:
db.posts.find({ status: "published" })
.sort({ createdAt: -1 })
.skip((page - 1) * 20)
.limit(20);On the first page it is fast. On page 3000, the server needs to traverse the index of 60,000 records, discard them, and only then return 20. The cost increases linearly with the page number. In the aggregation pipeline, $skip works the same way: replacing find with aggregate does not solve the problem, and comparing “aggregation versus cursor” actually comes down to comparing offset pagination with keyset pagination.
There is a second defect: if new posts appear between page requests, the posts are shifted and the user sees duplicates or skips elements.
Keyset pagination
Idea: instead of the page number, the client passes the value of the sorting key of the last element, and the request continues from this point by index.
The sort key must be unique. createdAt itself is not unique - two posts can have the same time within a millisecond. Therefore, we sort by the pair (createdAt, _id).
db.posts.createIndex({ status: 1, createdAt: -1, _id: -1 });The order of the fields in the index follows the equality, sort, range (ESR) rule as described in the MongoDB documentation: exact match fields first (status), then sort fields, then range conditions.
Implementation in Node.js
import { ObjectId, type Collection } from "mongodb";
type Cursor = { c: string; id: string }; // createdAt ISO + _id hex
const encode = (doc: { createdAt: Date; _id: ObjectId }) =>
Buffer.from(JSON.stringify({ c: doc.createdAt.toISOString(), id: doc._id.toHexString() })).toString("base64url");
const decode = (s: string): Cursor => JSON.parse(Buffer.from(s, "base64url").toString());
export async function listPosts(posts: Collection, after?: string, limit = 20) {
const filter: Record<string, unknown> = { status: "published" };
if (after) {
const { c, id } = decode(after);
const createdAt = new Date(c);
const _id = new ObjectId(id);
filter.$or = [
{ createdAt: { $lt: createdAt } },
{ createdAt, _id: { $lt: _id } },
];
}
const docs = await posts
.find(filter, { projection: { title: 1, slug: 1, createdAt: 1 } })
.sort({ createdAt: -1, _id: -1 })
.limit(limit + 1) // на один больше, чтобы узнать, есть ли следующая страница
.toArray();
const hasMore = docs.length > limit;
const page = hasMore ? docs.slice(0, limit) : docs;
return {
items: page,
nextCursor: hasMore ? encode(page[page.length - 1] as any) : null,
};
}The cursor is opaque to the client: it passes the string back as is. Be sure to validate the decoded value (valid date, 24-character hex for ObjectId), otherwise a garbage cursor will result in an exception and a 500 response.
Check via explain
db.posts.find({
status: "published",
$or: [
{ createdAt: { $lt: ISODate("2026-10-01T00:00:00Z") } },
{ createdAt: ISODate("2026-10-01T00:00:00Z"), _id: { $lt: ObjectId("6710f0c2a1b2c3d4e5f60718") } }
]
}).sort({ createdAt: -1, _id: -1 }).limit(21).explain("executionStats")Look at the three values in executionStats:
totalKeysExaminedandtotalDocsExaminedshould be close tonReturned, not tens of thousands;- the plan should not have the
SORTstage - sorting must be done by index; winningPlanuses your composite index, not just{ createdAt: -1 }.
If the scheduler selects a bad plan with $or, you can split the condition into two queries or suggest an index using hint.
Total quantity
Interfaces love "page 5 of 4312". An exact countDocuments filter on a large collection goes through all matching index keys on every query. Options:
- do not show the total number in the endless feed at all;
estimatedDocumentCount()- fast, but only for the entire collection without a filter;- cache the counter and update it according to a schedule or when published.
Combining $facet with data and counter in one aggregation looks elegant, but performs both heavy passes on each request.
Common errors
- Sorting only by
createdAt: with the same values, elements are lost at page boundaries. - Index
{ createdAt: -1 }withoutstatusat the beginning: filter by status is done after reading documents. - Cursor in the form of a page number, which still turns into
skipon the server. - The comparison direction does not coincide with the sorting: with
-1you need$lt, with1-$gt.
When keyset is not suitable
If the user needs a jump to an arbitrary page (a table in the admin panel with page numbers), keyset does not provide it. For small collections where pagination depth is limited to tens of pages, skip is simpler and works well. The compromise is to limit the maximum page number and offer filters and search instead of deep scrolling.