[ADP-312] Pagination

Overview

Pagination in RESTful APIs is a technique for dividing large datasets into smaller, more manageable chunks or "pages." This helps improve performance, reduce server load, and provide a better user experience.

There are mainly two methods of pagination: offset-based pagination and cursor-based pagination. See Design Considerations below for when to use each.

Offset-based Pagination

Offset-based pagination uses an offset value to specify the starting point of the dataset and a limit to specify the number of items to retrieve.

Example

GET /resources?offset=20&limit=10 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
Link: </resources?offset=20&limit=10>; rel="self", </resources?offset=30&limit=10>; rel="next", </resources?offset=10&limit=10>; rel="prev", </resources?offset=0&limit=10>; rel="first", </resources?offset=90&limit=10>; rel="last"
X-Total-Count: 100

{
  "resources": [ /* Resources from offset 20 to 29 */ ],
  "meta": {
    "offset": 20,
    "limit": 10,
    "totalItems": 100
  }
}

Cursor-based Pagination

Cursor-based pagination uses a cursor (usually a unique identifier) to fetch the next set of results.

Example

GET /resources?cursor=abc123&limit=10 HTTP/1.1

HTTP/1.1 200 OK
Content-Type: application/json
Link: </resources?cursor=abc123&limit=10>; rel="self", </resources?cursor=def456&limit=10>; rel="next"
X-Total-Count: 256

{
  "resources": [ /* Resources starting after cursor abc123 */ ],
}

TIP

Vendors use different names for the cursor value: page_token / next_page_token (Google AIP-158), pagination_token / next_token (Twitter API v2), nextPageToken (Atlassian Jira), after / before (GitHub), search_after (Elasticsearch). Some don't have a separate parameter at all — Stripe uses the last object's ID via starting_after / ending_before. They all refer to the same concept: an opaque string telling the server where to resume. This document uses cursor consistently.

Guidance

• SHOULD pick pagination strategy by trading off requirements against cost. See Design Considerations below.

• SHOULD cap the maximum reachable offset when using offset-based pagination on a non-trivial dataset, to prevent unbounded deep-scan queries (where the server has to scan through many rows to skip past them).

  TIP

  GitHub's Search API caps total results at 1,000 for this reason.

• SHOULD use common query parameters: cursor, offset, limit.

• SHOULD provide pagination links using Link headers per RFC 8288. The presence of rel="next" means more results follow; its absence means end of results. Do NOT embed pagination links in the response body.

  HTTP/1.1 200 OK
  Content-Type: application/json
  Link: </resources?cursor=abc123&limit=10>; rel="self",
        </resources?cursor=def456&limit=10>; rel="next",
        </resources?cursor=xyz789&limit=10>; rel="prev"
  X-Total-Count: 256
  
  {
    "resources": [ /* Resource items */ ]
  }

• SHOULD include X-Total-Count only when the count is exact and cheap to compute. Omit it otherwise.

  TIP

  Computing COUNT(*) on every paginated response is O(N) on most backends and becomes a scalability bottleneck on large or unbounded datasets. See Design Considerations.

• SHOULD use registered custom headers for pagination metadata. These headers MUST be registered in the organization's internal registry:

  • X-Total-Count: total number of items (use only when exact and cheap)
  • X-Page-Count: total number of pages (offset-based pagination only)
  • X-Current-Page: current page number (offset-based pagination only)

Design Considerations

Offset vs cursor

Offset suits small datasets, random page access (like a numbered pager UI), and stateless URLs that clients can bookmark. The cost of offset=N scales with N, so this only holds while the dataset stays small. Once it grows, offset queries get expensive, and unbounded ones can take down the database. GitHub's Search API caps total results at 1,000 for exactly this reason.

Cursor suits large or unbounded datasets, jobs that need to read through the entire dataset (e.g., data exports, generating reports), infinite scroll, and cases where rows are inserted or deleted while clients paginate. A cursor only points to the end of the previous page, so clients can only fetch the next page — not jump to an arbitrary one.

Atlassian's 2025 migration of Jira search from offset to cursor is a useful reference. Their RFC-61 explains the reasoning: offset pagination requires counting all preceding rows on every page, which they call "slow for the user, costly for us, and not scalable". Their migration writeup reports 99th percentile latency (P99) dropping from 1502 ms to 358 ms after the change.

Why cursor can't jump to page N

A cursor encodes a position within the result set of a query (whether filtered or an unfiltered list) — typically the last returned item's sort key. It does not encode a page number. Producing the cursor for page 47 would require traversing pages 1 through 46 first, putting back the deep-scan cost cursor pagination was meant to avoid.

APIs that need numbered-page UI should use offset (with a cap). APIs that don't should drop the page-number concept and use infinite scroll, "load more" affordances, or filter-based navigation (jump by date, status, or sort key).

Total count: include or omit

Counting all rows matching a query is O(N) on most backends. PostgreSQL COUNT(*) does a sequential scan because MVCC (multi-version concurrency control) requires checking each row's visibility individually; published benchmarks put it at roughly 85 ms per million rows. Maintaining an exact counter via INSERT/DELETE triggers eliminates the read cost but slows inserts by roughly 50× at typical workloads.

Both Stripe and Atlassian removed per-page total counts from their APIs in 2024–2025 citing this cost. Stripe deprecated the total_count expansion in API version 2025-03-31.basil and requires clients to rely on has_more. Atlassian removed total from Jira's new search endpoint and exposes a separate approximate-count endpoint instead.

When a service has clients that genuinely need count despite the cost, vendors have taken different approaches:

• Omit count entirely. Stripe (post-2025-03-31) returns only has_more and requires clients to construct the next page using the last object's ID via starting_after. Twitter API v2 returns meta.next_token, which clients pass back as pagination_token. Both push count concerns out of the API contract.
• Opt-in via request header. PostgREST's Prefer: count=exact|planned|estimated lets clients pick an accuracy level (exact = full scan, planned = PostgreSQL statistics, estimated = exact under a row threshold then planned). Zalando uses Prefer: return=total-count (boolean opt-in). Both rely on RFC 7240 — server may silently ignore.
• Separate count resource. Atlassian's POST /rest/api/3/search/approximate-count exposes count as its own endpoint, allowing independent caching and rate limits.

Compared to the opt-in approaches above, Microsoft Azure has an explicit SHOULD NOT rule: "YOU SHOULD NOT return a count of all objects in the collection as this may be expensive to compute."

Choosing whether and how to provide count is a three-way trade-off: requirements (do clients need it?), implementation (how to compute it?), and cost (what each implementation pays). There's no universal right answer. Start with whether clients actually need count. Most don't, and not building it is the simplest call. When they do, the choice usually comes down to how often count is requested versus what it costs to compute. Pick the simplest mechanism that handles both, and record the decision in a service ADR.

The right way to compute count depends on how accurate it needs to be, how often the data is written, and how big the dataset is.

TIP

Common implementation techniques for large datasets:

• Trigger or counter table — exact count. A sync trigger updates the counter on every write, which slows writes ~50× (Citus benchmark). An async background sync mechanism (CDC, listening to the DB write log) updates the counter instead, so writes stay fast but the counter lags by a few seconds.
• DB statistics estimate (Postgres pg_class.reltuples, MySQL INFORMATION_SCHEMA.TABLES) — near-instant, accuracy depends on when the statistics were last refreshed (PostgreSQL ANALYZE command); good for "about N items" UI.
• HyperLogLog (HLL) — probabilistic, ~0.5–2% error, scales to billions in kilobytes of memory; built into BigQuery, Snowflake, Presto, ClickHouse.
• Search engine capped count (Elasticsearch track_total_hits: N) — exact up to N, then "N+"; short-circuits past the cap.

Example

OpenAPI Example

openapi: 3.1.0
info:
  title: Example API
  version: 1.0.0
paths:
  /resources:
    get:
      summary: Retrieve resource list
      parameters:
        - name: cursor
          in: query
          description: Cursor for pagination
          required: false
          schema:
            type: string
        - name: limit
          in: query
          description: Number of items per page
          required: false
          schema:
            type: integer
            default: 20
            maximum: 100
        - name: filter
          in: query
          description: Filtering criteria
          required: false
          schema:
            type: string
        - name: sort
          in: query
          description: Sorting order
          required: false
          schema:
            type: string
      responses:
        '200':
          description: Paginated resource list
          headers:
            Link:
              description: |
                Pagination links per RFC 8288. Presence of `rel="next"` indicates
                more results are available; absence indicates end of results.
              schema:
                type: string
              example: </resources?cursor=def456&limit=10>; rel="next", </resources?cursor=xyz789&limit=10>; rel="prev"
            X-Total-Count:
              description: |
                Total number of items. Include only when the count is exact and
                cheap to compute; omit otherwise.
              schema:
                type: integer
              example: 256
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: array
                    items:
                      type: object

References

Specification References

• RFC 8288: Web Linking
• RFC 9110: HTTP Semantics §14.4 Content-Range
• RFC 7240: Prefer Header for HTTP

Design References

• GitHub REST API: Pagination
• GitHub REST API: Search — offset pagination capped at 1,000 results
• Stripe API: Pagination — cursor pagination with has_more
• Stripe Changelog: deprecate total_count expansion (2025-03-31)
• PostgREST: Pagination and Count — Prefer: count=exact|planned|estimated negotiation
• Elasticsearch: Search API — track_total_hits, search_after
• Zalando RESTful API Guidelines: Pagination — Prefer: return=total-count for opt-in count
• Microsoft Azure REST API Guidelines: Collections — explicit "SHOULD NOT return a count" by default
• Google AIP-158: Pagination — total_size field convention (may be estimate)
• Atlassian RFC-61: Evolving Search Capabilities (2024-08-29) — rationale for Jira search migration from offset to cursor
• Avoiding Pitfalls: Migration to Enhanced JQL APIs — Atlassian's published P90/P99 latency comparisons
• Citus: Faster PostgreSQL Counting — PostgreSQL count benchmarks across techniques

Changelog

• 2026-05-05: Major rewrite — added Design Considerations (offset/cursor selection, total count trade-offs), reframed Guidance around the requirement/cost trade-off, standardized cursor terminology.
• 2025-04-14: Updated to recommend standard Link headers and custom headers for pagination metadata instead of embedding links in response body.