Fetching and rendering records individually or in batches for an unknown total of records

Profile

footer text

Fetching and rendering records individually or in batches for an unknown total of records

I'm currently working on a ruby on rails + react app, fetching and rendering some records with axios and useEffect. It works fine as is, displays the data after fetching all.

How could I fetch/render individually or in controlled batches? Meaning I would like to fetch for example 1 by one or 5 at a time and render after each record/batch is available (so the list would be progressively growing from the user point of view). I'm not looking for either pagination nor infinite scroll.

The total of records varies and is unknown.

  const [isLoading, setIsLoading] = useState(false)

 useEffect(() => {
  setIsLoading(true)

  recordGateway.getRecords(recordId)
    .then(res => setOtherRecords(res.data.other_records))
    .catch(() => setHasError(true))
    .finally(() => (setIsLoading(false)))
  }, [])

   getRecords = (recordId: number) =>
    this.axios.get<{ other_records: OtherRecord[] }>(
      `/api/v1/record_card/${recordId}/other_records`
    )

I'm reading documentation about offset and promises but can't seem to understand it enough to apply it or maybe that's not at all what I should be looking for.

Answers(1) :

I would suggest, you use Promise.allSettled () alongside a state variable to hold the records in the array. You can batch your Axios API calls inside it. The promise will be resolved after all your API calls are complete.

When the user scrolls to the end of the list, you can start a batch request & expand the state array with the batch result.

I'm outlining an example for you below. Adjust it according to your needs.

State

const [records, setRecords] = useState([]);

Batch API calls

useEffect(() => {
  // your other codes
  // ...
  // ...


  Promise.allSettled([
    recordGateway.getRecords(recordId),
    recordGateway.getRecords(recordId),
    recordGateway.getRecords(recordId),
    // add more if you need
  ]).then(results => {
    const fetchedRecords = results.map(r => r.value.data.other_records);

    setRecords([...records, ...fetchedRecords])
  })

  // your other codes
  //...
  //...
}, [])

Comments:

2023-01-17 23:25:14

thank you for your input. Let me ask you for some further clarification: the quote about the scroll, isn't that something like infinite scrolling? And the repetition of recordGateway.getRecords(recordId) means that each is a record that will be fetched, meaning in your example, 4 would be fetched and later on 4 more? Or does it mean that's the full number of records that would be fetched? Because the aim is to eventually fetch all records (total unknown) just not having to wait for all to be fetched before starting the rendering

2023-01-17 23:25:14

@Whoopcg to do infinite scroll, you can use your isLoading state variable. Pass the variable as the dependency to the useEffect. When the user scrolls to a certain position, set the variable to true & inside useEffect use an if statement to run data fetching only when the variable is set to true. After the fetching is complete, reset the variable to false. This way you can go on with your data loading as long as you want to scroll.

2023-01-17 23:25:14

Also when you are defining the isLoading variable, do it like this const [isLoading, setIsLoading] = useState(true). This way you can make sure to load data on the first render too!

2023-01-17 23:25:14

Infinite scroll, as I said, is not currently the aim but nevertheless, thank you for your explanation! That's for sure something I will keep in mind in different scenarios. I'm sorry, I still fail to fully understand your draft code as I'm not sure about the aim of the repetition of ´recordGateway.getRecords(recordId)´ and how does that interfere with fetching in several batches (I was expecting to see some sort of place where the starting point for the next batch would be stored, or something like that)

2023-01-17 23:25:14

Let's say, you have an array of record ids from which you want to fetch only one or by batch of 5. In that case you can loop over the required number of elements and return an array of promise that will be passed to the Promise.allSettled. After the promise settles, you can update the state & do whatever you want to do. As for which records to fetch next, you can keep some time of flag that points to the last record id that was fetched. In next batch you start after that (wishing it helps you!)

Questions & Answers