At PeopleTray, the size and scope of the data we expose via our APIs has changed dramatically since the product first launched. Endpoints that were designed around the expectation that they would, in the most extreme cases, return several hundred records, are now returning hundreds of thousands of records. To handle this rapid growth, we’ve had to rethink how we paginate data — from no pagination to a new cursor-based pagination scheme.
Cursor-based Pagination Scheme
PeopleTray uses Cursor-based pagination that works by returning a pointer to a specific item in the dataset. On subsequent requests, the server returns results after the given pointer.
This will scale well for large datasets, by using a WHERE clause to fetch rows with id values that is indexed in the database.
Pagination Metadata in the Response Object
| Property Name | Description |
|---|---|
| TotalRecords | Total count of records of the requested query |
| MaxPageSize | Maximum number of records per page. This is set based on the response size and varies by endpoint. |
| CurrentPageSize | Number of records returned in the current page (or call) |
| NextPageLink | This is a pre-constructed link that contains the pagination parameters built in and ready for the client to call to get the next set of paged records. |
| NextCursor | Pagination parameter that is used by the API to return results after this pointer. In standard scenarios this value is for reference only as the NextPageLink value already contains the NextCursor in the url query parameters. |
| data | An array of objects containing the requested data |
Example Response:
Pagination parameters in the Request
PaginationParameter Object:
| Name | Description |
|---|---|
| Cursor | This is built into the NextPageLink url |
| PageSize | Optional. Can be used to request custom page size and should be less than the MaxPageSize of the endpoint. The MaxPageSize value takes precedence if the requested PageSize value is greater. |
Usage
-
First call by the client will be the regular endpoint of the Api.
-
The response from the call contains pagination metadata for the subsequent calls.
-
The client will use the "Next Page Link" from the response to make subsequent calls. The "Next Page Link" would be null on the last page, where the client would end the call.
Example Request:
First Call:
http://localhost:50195/api/reporting/48F6EFC2-20D7-4B42-85C5-2F5BD187E1DB/checklists?fromDate=1-Jan-2020
Subsequent Calls (from the NextPageLink response object):
http://localhost:50195/api/reporting/48F6EFC2-20D7-4B42-85C5-2F5BD187E1DB/checklists?fromDate=1-Jan-2020&cursor=12935