In the newer version of our Integration API, support for Pagination was added to several open-ended GET endpoints. These endpoints were both highly used and tended to return the largest data sets. While a single open-ended GET call is efficient from the client’s perspective, this is not the case at scale. When many clients make open-ended GET calls, job queues can get backed up, leading to poor response time for all API users. Like time-sliced multi-threading, breaking large jobs into small pieces allows many simultaneous requests to be serviced while maintaining responsiveness.
Best Practices
When paginating through a dataset that is being constantly updated, there is a risk that returned records will differ from the database. It is possible that records are altered during Pagination shift position in the record set, causing them to be returned on more than one page, or in extreme cases not be returned at all.
Two new attributes: dateAdded and dateEdited have been added to resources returned from paginated calls. Not only do they enable highly targeted filtering, but they also can be used to reduce the risk of data accuracy issues described above.
Below we describe two commonly used methods:
Frozen Snapshot
To closely emulate the behavior of an open-ended GET call, use filtering to return only records added or updated before the first pagination call. Like a non-paginated open-ended call, anything added after Pagination starts will not be included.
Sorting Oldest to Newest
To reduce the risk of excluding added records, use sorting to return the oldest records first (sorting on dateAdded), and any records added after Pagination started should show up on the last page(s). It’s a best practice to never sort results from newest to oldest.
If you have any questions or feedback about this article, please contact your Customer Success Manager or Account Manager for further assistance.
Comments
Article is closed for comments.