Implementing Server-Side Pagination

Introduction

In the realm of software development, implementing server-side pagination is a crucial aspect of optimizing the performance of web applications. In this guide, we will explore a practical implementation of server-side pagination using the popular web framework Express.js and a database ORM called Prisma. With the growing demand for custom software services in various sectors such as healthcare, financial services, and banking, this implementation can significantly enhance the functionality of custom software development tailored to the specific needs of these industries. Let’s delve into the steps to integrate server-side pagination.

In today’s dynamic tech landscape, efficient data management forms the backbone of seamless user experiences within applications. The integration of server-side pagination with filtering and search capabilities significantly enhances data retrieval and processing. This blog will guide you through the process of implementing server-side pagination with filtering and searching in a Node.js application using Express and Prisma, two widely used tools in the Node.js ecosystem.

Creating a Pagination Component in React:

Next, let’s create a React component that will display the paginated data and handle the pagination controls. We’ll use the axios library to make HTTP requests to the backend API.

// PaginationComponent.js
import React, { useState, useEffect } from 'react';
import axios from 'axios';
const PaginationComponent = () => {
  const [currentPage, setCurrentPage] = useState(1);
  const [totalPages, setTotalPages] = useState(0);
  const [products, setProducts] = useState([]);
  const fetchProducts = async (page) => {
    try {
      const response = await axios.get(`/api/products?page=${page}&pageSize=10`);
      const { products, totalPages } = response.data;
      setProducts(products);
      setTotalPages(totalPages);
    } catch (error) {
      console.log(error);
    }
  };
  useEffect(() => {
    fetchProducts(currentPage);
  }, [currentPage]);
  const handlePrevPage = () => {
    if (currentPage > 1) {
      setCurrentPage(currentPage- 1);
    }
  };
  const handleNextPage = () => {
    if (currentPage < totalPages) {
      setCurrentPage(currentPage + 1);
    }
  };
  return (
    <div>
      {/* Display the products */}
      {products.map((product) => (
        <div key={product.id}>{product.name}</div>
      ))}
      {/* Pagination controls */}
      <button onClick={handlePrevPage} disabled={currentPage === 1}>
        Previous Page
      </button>
      <button onClick={handleNextPage} disabled={currentPage === totalPages}>
        Next Page
      </button>
    </div>
  );
};
export default PaginationComponent;

Code Explanation

In the above code, we define the PaginationComponent functional component. It uses the useState hook to manage the currentPage, totalPages, and products state variables. The fetchProducts function is responsible for making the API call to retrieve the paginated data. It uses the axios library to send a GET request to the backend API, passing the current page and page size as query parameters. The retrieved data is then stored in the state variables.

We use the useEffect hook to trigger the fetchProducts function whenever the currentPage changes. This ensures that the component fetches the appropriate data whenever the user navigates to a different page.

The handlePrevPage and handleNextPage functions update the currentPage state, allowing the user to navigate to the previous and next pages. We disable the pagination controls when the user is on the first or last page to prevent invalid navigation.

Steps to Integrate Server-Side Pagination

1. Setting Up the Environment: Ensure that you have Node.js and npm installed on your system. To begin, run the following commands in your terminal to install the necessary packages:

2. Initializing Express and Prisma: Start by importing the required modules and initializing the Express application and Prisma client.

const express = require('express');
const { PrismaClient } = require('@prisma/client');
const prisma = new PrismaClient();
const app = express();

3. Creating the Endpoint: Define an endpoint that handles GET requests for fetching paginated data. Extract the page, pageSize, filter, and searchKeyword query parameters from the request.

app.get('/items', async (req, res) => {
const { page = 1, pageSize = 10, filter = '', searchKeyword = '' } = req.query;
// ... (rest of the code)
})

4. Implementing Filtering and Searching: Utilize the filter and searchKeyword values to customize the Prisma query for filtering and searching. Adjust the where clause accordingly based on the provided filter and search parameters.

let where = {};
if (filter && searchKeyword) {
  where = {
    AND: [
      { category: { contains: filter, mode: 'insensitive' } },
      { name: { contains: searchKeyword, mode: 'insensitive' } },
    ],
  };
} else if (filter) {
  where = {
    category: { contains: filter, mode: 'insensitive' },
  };
} else if (searchKeyword) {
  where = {
    name: { contains: searchKeyword, mode: 'insensitive' },
  };
}

5. Calculating Pagination Parameters: Calculate the offset and take values based on the page and pageSize parameters to determine the subset of data to be retrieved from the database.

const offset = (page - 1) * pageSize;

6. Retrieving Data and Total Count: Fetch the filtered and searched data from the Prisma database using the calculated parameters. Retrieve the total count of items that match the filter and search criteria.

const totalCount = await prisma.item.count({
  where,
});
const items = await prisma.item.findMany({
  where,
  skip: offset,
  take: pageSize,
});

7. Returning the Results: Construct a JSON response containing the current page, page size, total item count, and the retrieved data.

res.json({
  page,
  pageSize,
  total: totalCount,
  data: items,
});

What Users Think About in Terms of Pagination

Most of the time, you as a user will be completely unaware that something is doing pagination — and that’s a good thing! What will notice are pages that take too long to load, a “next” link that feels unresponsive, and an apparent randomization of page counts as you navigate and data transforms under you while looking at a particular set of information.

Proper server-side pagination will vanish into the background, clicking to the next page will feel just as fast and responsive as the first, and it will feel roughly the same whether you are on the second page or the two-hundredth. As such, performance testing a paginated result set should never be done on just the first page — you may find that it takes a long time to generate a result set of a hundred mock records you made to test with, but it performs noticeably worse when your actual product has to render several thousand records for real users.

Why This Is A Problem As Your Product Grows

Your product was small at first, and pagination seemed like a problem to worry about later, once there was an actual reason to have more data than could be displayed at once. In practice, though, retrofitting an existing application with proper pagination is significantly more work than building a paginated API in the first place.

Endpoints that return paged results are not immediately useful for things that expect to receive the full result set — these will need to be rewritten as the product grows, or alternate measures taken in the meantime. Frontends built to assume they can have the full data available at all times, whether to sort, filter, or just display, will need to be adjusted to work with the paged results, or have their logic moved to the backend. None of it is outright impossible, but all of it becomes considerably more tedious for the developers involved.

How Much Flexibility To Offer For An API Is A Tradeoff

When designing an API that will return paged results, you will have to decide to what degree you are going to let the consumer of the API decide how they want the results to be ordered, sorted, and grouped. Too little and you will find yourself limiting every application that uses the API, since they will have to implement their own sorting and filtering to get around your insufficient flexibility.

Too much and you risk performance issues at the database level and have to implement additional safeguards on the API to avoid abuse. A middle ground is best, though where that middle ground is will likely depend on context — if it is an internal API with a known frontend, you can be far more flexible since you control both parts of the equation. If it is a public API, however, you will want to lean towards less flexibility and more documentation for the sake of the developers that will have to work with it.

What It Says About Your Application’s UX

A lot of the time, you will not think about pagination as a UX concern, but it is one nonetheless. Having to deal with the kinds of issues outlined above will make a large number of your users distrustful of your product’s ability to handle information correctly — especially if they suspect, rightly or wrongly, that the data they see is not really the full data set, or that there are internal inconsistencies.

This will often happen when you have dynamic data in the pagination, where items change between pages because new items were added or old ones taken away while the user browsed the data, even if the page numbers themselves are consistent. You should plan for how your application will behave in these edge cases ahead of time — it is easy to think about pagination only in terms of how much data you give per request, but as applications grow, you will find that there are other ways data can change at the database level, and user experience elsewhere will need to change as well.

Role of Database Indexing in Pagination Performance

A nuance that is often overlooked when implementing pagination is that the pagination logic itself is only part of the problem – the database needs to be optimized for the specific queries used as well. If there are no indexes on the columns that are being filtered, searched, or sorted by, each paginated query will require scanning a large amount of data to find the relevant rows.

This can have a significant performance impact, especially for large datasets that are paginated into tens or hundreds of thousands of rows – the exact use case for which pagination is most needed. Adding indexes to columns that are frequently used for filtering or sorting, such as a category column for filtering or a creation timestamp for sorting by newest first, should be one of the highest priorities when implementing pagination. It should be considered as an integral part of the pagination implementation, rather than an afterthought to be addressed later when performance issues arise.

Communicating Pagination State to the User

While the implementation details of the pagination logic are important, its overall presentation to the user should be considered from the perspective of the user experience as well. An often overlooked UX detail in pagination UI is the lack of summary information about the current position in the data set.

Presenting the current results as a range (“Showing 21–40 of 1,204 results”) provides much more useful information to the user than a simple “Next” and “Previous” button. This is especially important in data-heavy applications, where the user needs to know how much data they are working with in order to determine if it makes sense to continue pagination or refine their search/filter criteria to narrow down the results. The combination of a well-designed pagination UI and the overall implementation detailed above will have a significant impact on the user experience of anyone who needs to work with large data sets on a regular basis.

Testing Paginated Elements Properly

The pagination feature may seem simple to implement and test; however, the actual number of edge cases that must be tested is quite large. Testing pagination on a happy path is a trivial task, but a proper testing session would include test cases that would simulate going to the first and the last pages as well as searching for elements with no results.

A thoughtful stress test would attempt to reach nonexistent pages by providing incorrectly encoded strings, negative numbers, and zero as input parameters. Software that is released without proper pagination testing tends to perform poorly and have various issues in production because the developers did not consider some of the edge cases. A checklist of common issues that can be resolved by proper pagination testing can help ensure that a product is not shipped with buggy or inconvenient features related to the pagination mechanism.

Why Consistent Pagination in API Endpoints Is Important

When developing a paginated API resource, a developer might not think about the consequences of their decision regarding the response format. They can write correct code that would return data correctly without considering how different pieces of information about the current page would be organized.

This oversight can lead to additional issues when multiple endpoints, which return data in different formats, must be used in a single project. In such a scenario, the frontend code would become more complicated for no significant practical benefit. A single response structure should be selected for all paginated endpoints, and the developer should document the chosen solution to avoid confusion among their colleagues. This approach will also become critical when new developers, who joined the project, need to update or expand the existing codebase.

Why Improperly Designed Pagination Can Be a Problem

The importance of consistency when designing the pagination structure is immense even for projects that have no intention of supporting multiple clients. The reason for this is the increasing complexity of projects with time. After the initial release of an API, a developer may want to update some of its endpoints or introduce new ones.

If the original API had pagination-related issues or was never properly designed, these updates can cause serious problems. If the API is used in an application as a single point of reference, these issues would only affect a single project. However, if the API is available to the public, the updates would affect all clients that use the affected endpoints.

That is why developers working on public APIs tend to spend more time on the process of designing and testing pagination than those who work on closed projects. The former will have to update all of their clients, while the latter can update only a single application to reflect the new structure. In more complex situations, a developer can use a combination of new API versions and a well-designed pagination system to update the resource without disrupting the existing clients.

What Happens When Pagination Interacts with Caching

Caching responses from paginated endpoints is effective but needs to be done deliberately, to avoid subtle bugs resulting from incorrect assumptions. A cache unaware of pagination may return a page of results that were not requested by the user, or an old query result with applied filters, when a filtered request is made again with different parameters. For example, a user searching products may change the filters between requests, and receive the same results as before, due to a request using the same parameters unknowingly hitting the cache.

On the other hand, the solution to this problem is often a matter of design. The query parameters defining the page, such as the page number and size, should be part of the key used to cache a response, so that different pages are cached separately. This goes for any parameters that affect the result, including the filters, sorting options, or any other request attributes.

The caching layer may need to be instructed to consider two requests with the same endpoint but different query parameters as different requests, to be cached separately. This can be achieved by using the entire query as part of the cache key, or using a hash of the query or serialized request object instead of the URL for the cache key.

The other thing to consider with caching and pagination is the cache lifetime. A page with live information that is frequently updated should have a shorter cache lifetime, than a page with static information that rarely changes. It is important to define appropriate cache lifetimes for different paginated endpoints within an application.

If a single cache lifetime is used for all paginated endpoints, the cached results for rapidly changing data would expire too quickly, while cached results for slowly changing data would be refreshed too often. Another approach is to evict specific pages from the cache when the underlying data changes, instead of waiting for the cache to expire. This technique is particularly useful for the first page of large sets of search results, which is requested most often, and should be refreshed more frequently than the rest.

By designing the caching strategy correctly, an application can provide the benefit of caching, being able to serve pages with results faster, without the user having to wait for the server to respond. At the same time, an improperly designed cache can make paginated queries more difficult to use than they should be, causing the user to question the accuracy of the results due to inconsistencies between pages.

Conclusion

Implementing server-side pagination is a vital technique for optimizing web application performance, especially when dealing with large datasets. By leveraging Express.js and Prisma, we can efficiently manage data retrieval while ensuring a seamless user experience. Consider customizing this implementation to suit your specific project requirements, and remember to prioritize error handling and performance optimization.
By following the steps outlined in this guide, you can integrate server-side pagination seamlessly into your custom software development projects, ensuring smooth data management and retrieval.