MongoDB Pagination with Java and SpringBoot

1. Overview

In this post, we will walk you through the MongoDB Pagination with Java and Springboot. Pagination is a process that is used to divide large data into smaller discrete pages.

Let’s see the ways available in MongoDB to implement pagination and compare the better approach.

2. MongoDB Pagination with skip and limit in Java and Springboot

Let’s explore the first basic approach that we have in MongoDB. We have the skip and limit cursor methods available in MongoDB. However, this native approach has a performance impact on a very large dataset.

The root cause is the skip() command which is not efficient and can not take big benefit from the index. The skip() method requires the server to scan from the beginning of the input results set before beginning to return results. As the offset increases, skip() will become slower.

You can directly go to these alternate approaches in case you want to work on a large dataset.

Before that, let’s understand this basic concept. Unless you specify the sort() method or $near operator, MongoDB does not guarantee the order of query results. Most of the time, it is an insertion order but not guaranteed.

For example, the following query, which returns all documents from the orders collection, does not specify a sort order and the query returns the documents in indeterminate order.

db.orders.find()

So it is recommended to use the sort method along with skip/limit pagination.

2.1. Sort before pagination

MongoDB does not store documents in a collection in a particular order. When sorting on a field which contains duplicate values, documents containing those values may be returned in any order.

To achieve a consistent sort, add a field which contains exclusively unique values to the sort. The following command uses the sort() method to sort both the borough field and the _id field:

db.restaurants.find().sort( { "borough": 1, "_id": 1 } )

Since the _id field is always guaranteed to contain exclusively unique values, the returned sort order will always be the same across multiple executions of the same sort.

MongoDB can obtain the results of a sort operation from an index which includes the sort fields. MongoDB may use multiple indexes to support a sort operation if the sort uses the same indexes as the query predicate.

If MongoDB cannot use an index or indexes to obtain the sort order, MongoDB must perform a blocking sort operation on the data. A blocking sort indicates that MongoDB must consume and process all input documents to the sort before returning results.

2.2. Skip

You can use the skip() method on a cursor to control where exactly the MongoDB must start returning the results. This would be useful in implementing paginated results.

The skip method takes the offset as the parameter that shows the number of documents to skip in the result set.

2.3. limit

You can leverage the limit() method to specify the maximum number of documents the MongoDB should return. 

You can use the limit() method to maximize performance and prevent MongoDB from returning more results than required for processing.

3. Java and Springboot pagination with skip and limit

Let’s use the skip and limit method to implement pagination in MongoDB. If you want to retrieve a large set of data from MongoDB, instead of getting it all together, retrieve data in a smaller set.

The below code skips the previously retrieved documents and gets the documents in a smaller subset.

List<String> dataList;
var currentPage = 0;
do {
   log.info("Retrieving data : " + limit);

   dataList = sampleDao.getNames((long) currentPage * limit, limit);
   if (currentPage == 0 && dataList.isEmpty()) {
       break;
   }

   currentPage++;

} while (!dataList.isEmpty());

The below code shows you how to get documents from Springboot using skip and limit with MongoDB query. You can customize it based on your requirements.

public List<String> getNames(long skip, int docs) {
   List<String> docs = new ArrayList<>(); 
   try {
       var mongoTemplate = mongoTemplateAdapter.getTemplate();
       var query = new Query();
       query.with(Sort.by(Sort.Direction.ASC, '_id');
       query.skip(skip);
       query.limit(docs);
       docs = mongoTemplate.find(query, String.class);
       return docs;
   } catch (Exception ignored) {

   }
}

4. Alternate Pagination approach

The above native approach is not efficient when we have huge documents in the collection. Suppose you have 100M documents, and you want to get the data from the last offset, MongoDB has to build up the whole dataset and walk from the beginning to the specified offset, this will be low performance. As your offset increases, the performance keeps degrading.

4.1. MongoDB pagination with range queries

Range queries can use indexes to avoid scanning unwanted documents, typically yielding better performance as the offset grows compared to using skip() for pagination.

4.1.1. Java and Springboot pagination with range queries

You can use the below procedure to implement pagination with range queries:

• You can choose a field such as _id which generally changes in a consistent direction over time and has a unique index to prevent duplicate values.
• Query for documents whose field is less than the start value using the $lt (for descending order), $gt (for ascending order) and sort() operators, and
• Store the last-seen field value for the next query.

If you retrieve the ‘_id’ of the last document on the current page, you can use find() with condition instead of skip().

We can rewrite the implementation with skip and limit as below:

String currentID;
List<String> list = Collections.emptyList();
do {
   currentID = list.isEmpty() ? "" : list.get(list.size() - 1)._id;
   list = sampleDao.getNames(currentID, limit);
} while (!list.isEmpty());

public List<String> getNames(String startValue, int docs) {
   List<String> docs = new ArrayList<>(); 
   try {
       var mongoTemplate = mongoTemplateAdapter.getTemplate();
       var query = new Query();
       query.addCriteria(Criteria.where('_id').gt(startValue));
       query.with(Sort.by(Sort.Direction.ASC, '_id');
       query.skip(skip);
       query.limit(docs);
       docs = mongoTemplate.find(query, String.class);
       return docs;
   } catch (Exception ignored) {

   }
}

5. Conclusion

To sum up, we have learned the various approaches to implement Pagination in MongoDB with Java and SpringBoot. You can refer to our GitHub repository for samples.

To learn more about MongoDB, refer to our articles.