How to Add Infinite Scroll to Your Application

Before we begin, let me tell you a bit about 1Click. 1Click is an event registration and management platform that allows users to create an account beforehand and register for any event with just one click. It gets rid of the hassle of endless forms and filling in your details again and again and provides a platform for event managers and attendees to list and register for events.
As a part of the project, we had a “Recommended Events” page that used a recommender system (a machine learning model) that ordered events by how likely you are to be interested in and register for that event based on events you have registered for on the platform in the past.

Here was the problem: The Recommended Events page would take very long to load. This was in part due to the reordering of a massive number of events present in the database based on the user’s predicted likes each time the page was loaded, and also due to the creation and manipulation of these many DOM elements by ReactJS.

A combination of the long loading time and the fact that the user is probably not going to scroll through every event in the database most of the time helped us realise that this is the same problem that many applications also face, whether it’s YouTube recommending videos or Facebook recommending posts; and they’ve all refined a very elegant solution — the (pseudo) infinite scroll.

Thinking about it, we found a simple solution: After the server reordered the events based on the recommender system, we would divide the entire set into lots of equal sizes (say 25 each) to what we call “pages”. We would then only fetch and render the first page on the feed, and when the user scrolls past a particular amount of items, say 90% of the total feed height, we would fetch the next page and append those fetched set of items to the end of the feed, creating the illusion of infinite scroll.

This way, the first few events are loaded in really quickly, and the user can scroll (almost endlessly) to view the next events. They can also scroll back up to view the items they have already seen. So, we implemented it.

So, to do this, this is what we changed on the server side:

In a nutshell, we added a limit of our lot size (here, 25) to the query result along with an offset of the page number times 25. This was done to fetch for example events 25–50 when the page number is given as 1 (Page numbers started at 0 since we had to fetch the beginning of the feed always).

# PREVIOUS - FETCH ALL EVENTS - SLOW :(

@app.route("/api/recommended")
def recommended():
    recommended_events = list(recommend((session.get("user"))["_id"], users_col))
    m = {"$match": {"_id": {"$in": recommended_events}}}
    a = {"$addFields": {"__order": {"$indexOfArray": [recommended_events, "$_id"]}}}
    s = {"$sort": {"__order": 1}}
    recommended_events = events_col.aggregate([m, a, s])
    final = []
    for event in recommended_events:
        final.append(event)
    return parse_json(final), 200

# NEW WITH PAGING - MUCH BETTER!

@app.route("/api/recommended/<int:page>")
def recommended(page):
    recommended_events = list(recommend((session.get("user"))["_id"], users_col))
    m = {"$match": {"_id": {"$in": recommended_events}}}
    a = {"$addFields": {"__order": {"$indexOfArray": [recommended_events, "$_id"]}}}
    s = {"$sort": {"__order": 1}}
    l = {"$limit": 25} # ADDED
    p = {"$skip": page * 25} # ADDED
    recommended_events = events_col.aggregate([m, a, s, p, l])
    final = []
    for event in recommended_events:
        final.append(event)
    return parse_json(final), 200

Making some changes on the front end:

Adding a state to store and update the current page number, and a handleScroll function that attaches to the items’ container div to detect when we have scrolled past 80 or 90% of the height of its contents.

// ...
const [ page, setPage ] = useState(0)
const [ events, setEvents ] = useState([])

useEffect(() => {
    const getRecommended = async () => {
      const res = await AxFetch.get(`/api/recommended/${page}`, { validateStatus: false })
      return res.data;
    }
    getRecommended(page).then((data) => {
        // Appending data to the end of the events array
        setEvents(prevEvents => [...prevEvents, ...data]) 
    })
  }, [page])

const handleScroll = (event) => {
    const { scrollTop, clientHeight, scrollHeight } = event.currentTarget;
    const scrollPercentage = (scrollTop + clientHeight) / scrollHeight;
    if (scrollPercentage > 0.90) { 
      setPage(prevPage => prevPage + 1);
    }
};
// ...

Everything should work fine, right?

Well, as it turned out, it didn’t.

The first (kind of trivial) problem was that for some reason, the first page/set of elements was always loaded in twice, regardless of what we tried. This turned out to be a feature of React’s strict mode where it tries to load in the component, delete it, clean it up, and load it again to ensure that the user will not face any issues due to re-rendering. Since our component wasn’t setting up any websocket connections and since React’s strict mode isn’t used during production, we chose to ignore and turn it off to see the logic working first and fix it later.

The second problem was a lot more interesting. Let’s see if you can notice the issue in this video.

After the user scrolled to nearly the end of the page, events kept on getting added. While we expected 25+25=50 events after reaching the end of the container the first time, we ended up getting 125 events, this meant that either the server was sending too many events (which would be unlikely because we programmed it to send only 25 per request), or for some reason, the scroll handler was being triggered way more times than it needed to — leading to 125 events being added.

Debugging! The Scroll Handler was firing off faster than the events were being fetched, leading to more events being fetched and rendered than needed.

Our second guess was right — It turned out, that the scroll handler was detecting that we were at 90% of the scroll height much faster and many more times before we could fetch and add those events to bring the user out of that “I’m still in the last 10% of scroll height, please request more events” condition.

What if we just changed the condition to request more elements at let’s say 60% of the scroll height? That might work, but it wouldn’t be optimal, we would be making more requests than needed, and in our testing with various conditions (80%,70%,60%,50%, etc.), the scroll problem could not be avoided. We had to find a different way of fixing this.

Wait, there is something else that all apps that have infinite scroll also do…

Photo: LinkedIn Infinite Scroll — Placeholder

On almost all of these apps (YouTube, Facebook, LinkedIn), if you scroll faster than they can load in new items, they usually display these “loading items” that look like items that are about to load and get filled in with content. So that’s what we did.

First, we created a component called a DummyEventCard that looks like something in the gif above. Once the scroll handler was activated, we would first append a bunch of these DummyEventCards to the feed to get the user out of that “I’m still in the last 10% of scroll height, please request more events” condition, a request is then made for the next set of events. Once they arrive, we will replace the DummyEventCards with the actual EventCards for the real events, and voilà.

The final frontend code looks something like this:

function Recommended() {
  const [ page, setPage ] = useState(0)
  const [events, setEvents] = useState([])
  const [isLoadingEvents, setIsLoadingEvents] = useState(false)
  const [continuousLoad, setContinuousLoad] = useState(false)


  const { state } = useAuthContext()

  useEffect(() => {
    const getRecommended = async () => {
      const res = await AxFetch.get(`/api/recommended/${page}`, { validateStatus: false })
      return res.data;
    }
    getRecommended(page).then((data) => {
        setEvents(prevEvents => [...prevEvents, ...data]) // Add events to the end of the array
        setContinuousLoad(false) // Remove the loading/dummy event cards
    })
  }, [page])


  const handleScroll = (event) => {
    const { scrollTop, clientHeight, scrollHeight } = event.currentTarget;
    const scrollPercentage = (scrollTop + clientHeight) / scrollHeight;
    if (scrollPercentage > 0.90) { 
      setContinuousLoad(true) // Load in the dummy event cards
      setPage(prevPage => prevPage + 1); // Request for more pages
    }
  };

  return (
    <>
      {!isLoadingEvents && events.length > 0 ? <div>
        <h1>{state?.user ? "Recommended Events For You" : "Events"}</h1>
        <div onScroll={handleScroll} style={{height:'70vh'}}>
          {events?.map((event, index) => (
            <EventCard key={index} event={event} />
          ))}
          {continuousLoad && <><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /><DummyEventCard /></>}
        </div>
      </div> : <h1>No Recommended Events for now</h1>}
      {isLoadingEvents && <div>Loading</div>}
    </>
  )
}

Here is the final result!

YouTube: Infinite Scroll — Final Showcase

Now that you’ve got the magic of infinite scroll under your belt, go forth and build an app that keeps users glued to their screens (in a good way)! Remember, a captivating feed is just one piece of the puzzle, so keep iterating and innovating to make your app truly shine. But for now, take a moment to appreciate the endless possibilities we’ve just unlocked together!

I hope you enjoyed this article on our adventure for infinite scroll and it helped you understand our thought process and how we build various aspects of applications. Feel free to connect with me on LinkedIn or refer to the final code on GitHub here.

Thanks for reading!