8 solutions to deal with 100,000 pieces of data returned by the backend at one time

Click on the front-end Q above to pay attention to the official account

Reply to join the group, join the front-end Q technology exchange

Description of the problem

  • Interviewer: The backend returns 100,000 data to you at one time, how do you deal with it?

  • Me: smile crookedly, what the f**k!

Question inspection points

Seemingly nonsensical questions actually test the candidate’s “breadth and depth of knowledge”, although this situation is rarely encountered in work…

  • Examine how the front end handles large amounts of data

  • Investigate candidates for performance optimization for large amounts of data

  • “Examine the way candidates think about dealing with problems”

  • ?…

The complete code will be provided at the end of the article for your better understanding

Use express to create an interface with 100,000 pieces of data

If you are not familiar with express, you can read this full-stack article of the author (and the complete code) when you have time: “Vue + Express + Mysql full-stack project growth Delete, modify, query, page sorting and export table functions》

route.get("/bigData", (req, res) => {
  res.header('Access-Control-Allow-Origin', '*'); // Allow cross domain
  let arr = [] // Define an array to store 100,000 pieces of data
  for (let i = 0; i < 100000; i ++ ) { // loop to add 100,000 pieces of data
    arr. push({
      id: i + 1,
      name: 'name' + (i + 1),
      value: i + 1,
    })
  }
  res.send({ code: 0, msg: 'success', data: arr }) // return 100,000 pieces of data
})

Click the button, send a request, get the data, and render it to the form

The html structure is as follows:

<el-button :loading="loading" @click="plan">Click to request loading</el-button>

<el-table:data="arr">
  <el-table-column type="index" label="order" />
  <el-table-column prop="id" label="ID" />
  <el-table-column prop="name" label="name" />
  <el-table-column prop="value" label="corresponding value" />
</el-table>

data() {
    return {
      arr: [],
      loading: false,
    };
},

async plan() {
    // Send a request, get the data, and assign it to arr
}

Option 1: Render all data directly

If 100,000 pieces of data are requested to be rendered directly, the page will be stuck. Obviously, this method is not advisable

async plan() {
      this.loading = true;
      const res = await axios.get("http://ashhuai.work:10000/bigData");
      this.arr = res.data.data;
      this.loading = false;
}

Scheme 2: Use timer group by batch and heap to render sequentially (timed loading, heap idea)

  • Normally, 100,000 data requests will take between 2 seconds and 10 seconds (it may be longer, depending on the specific content of the data)

  • And this way is, after the front-end requests 100,000 pieces of data, don’t rush to render, first divide the 100,000 pieces of data into piles and batches

  • For example, 10 pieces of data are stored in one pile, then there are 10,000 piles of 100,000 pieces of data

  • Use a timer to render a bunch at a time, and render 10,000 times

  • In this way, the page will not be stuck

Grouping, batching and heaping functions

  • Let’s first write a function to divide 100,000 pieces of data into heaps

  • The so-called splitting is actually “The idea is to intercept data of a certain length at a time”

  • For example, 10 pieces of data are intercepted at a time. Intercept 0~9 for the first time, and intercept 10~19 for the second time.

  • For example, the original data is: [1,2,3,4,5,6,7]

  • Suppose we divide the pile into 3 pieces, then the result is a two-dimensional array

  • That is: [ [1,2,3], [4,5,6], [7]]

  • Then traverse this two-dimensional array to get the data of each item, that is, the data of each pile

  • Then use the timer a little bit, a bunch of assignments to render

Grouping, batching and heaping functions (one pile is divided into 10)

function averageFn(arr) {
  let i = 0; // 1. Start intercepting from the 0th
  let result = []; // 2. Define the result, the result is a two-dimensional array
  while (i < arr.length) { // 6. When the index is equal to or greater than the total length, the interception is complete
    // 3. Start traversing from the first item of the original array
    result.push(arr.slice(i, i + 10)); // 4. From the original 100,000 pieces of data, intercept 10 pieces at a time for heap division
    i = i + 10; // 5. After the 10 pieces of data are intercepted, the next 10 pieces of data will be intercepted, and so on
  }
  return result; // 7. Finally, just throw the result out
}

Create a timer to assign values and render in sequence

For example, we assign and render every second

async plan() {
      this.loading = true;
      const res = await axios.get("http://ashhuai.work:10000/bigData");
      this.loading = false;
      let twoDArr = averageFn(res.data.data);
      for (let i = 0; i < twoDArr. length; i ++ ) {
        // It is equivalent to creating many timed tasks to process in a short period of time
        setTimeout(() => {
          this.arr = [...this.arr, ...twoDArr[i]]; // assignment rendering
        }, 1000 * i); // 17 * i // Pay attention to the set time interval... 17 = 1000 / 60
      }
    },

This method is equivalent to creating many scheduled tasks to process in a short period of time. Too many scheduled tasks consume resources.

In fact, this method has the idea of paging with large data volume

Scheme 3: Use requestAnimationFrame instead of timer for rendering

Regarding the advantages of requestAnimationFrame over timer, Daoists can read this article by the author: “Performance Optimization: Easy to understand and learn requestAnimationFrame and Examples of use scenarios”

“Anyway, when you encounter a timer, you can consider whether it is possible to use the requested animation frame for optimized execution rendering?”

If you use the request animation frame, you need to modify the code writing method. The previous one does not change, but the writing method in the plan method can be changed. Pay attention to the notes:

async plan() {
  this.loading = true;
  const res = await axios.get("http://ashhuai.work:10000/bigData");
  this.loading = false;
  // 1. Divide the large amount of data into heaps
  let twoDArr = averageFn(res.data.data);
  // 2. Define a function that is specially used for assignment rendering (using each item in the two-dimensional array)
  const use2DArrItem = (page) => {
    // 4. From the first item to the last item
    if (page > twoDArr. length - 1) {
      console.log("Every item has been obtained");
      return;
    }
    // 5. Use the method of requesting animation frames
    requestAnimationFrame(() => {
      // 6. Take out one item and splicing one item (concat is also OK)
      this.arr = [...this.arr, ...twoDArr[page]];
      // 7. This item is done, continue to the next item
      page = page + 1;
      // 8. Until the end (recursive call, pay attention to the end condition)
      use2DArrItem(page);
    });
  };
  // 3. Get and render from the first item in the two-dimensional array, the first pile (the index of the first item in the array is 0)
  use2DArrItem(0);
}

Scheme 4: Paging with paging components, paging at the front end (displaying a pile on each page, the idea of dividing into piles)

The author has encountered this method before. The corresponding scenario at that time was that the amount of data was only dozens of pieces, and the backend directly threw dozens of pieces of data to the frontend, and let the frontend paginate.

The reason why the backend doesn’t do pagination is. At that time, he asked for leave temporarily, so he did paging on the front end.

  • In the case of a large amount of data, this method is also a solution

  • The idea is to intercept on the basis of all data

  • The brief code is as follows:

getShowTableData() {
    // Get the interception start index
    let begin = (this. pageIndex - 1) * this. pageSize;
    // Get the interception end index
     let end = this.pageIndex * this.pageSize;
    // Intercept through the index to display
    this.showTableData = this.allTableData.slice(begin, end);
}

For the complete case code, please see the author’s article: “The backend returns all the data at once, let the frontend intercept and display for pagination”

「Actually, this large task is divided into many small tasks. This method, method, and application idea is the method of fragmentation (time). In other scenarios, such as when uploading large files , I also have this kind of thinking, such as a large file of 500MB, split into 50 small files, one is 10MB... As for the article uploaded by the large file, I will write it when the author is free...”

Scheme 5: Table scroll bottom loading (scroll to the bottom, then load a bunch)

The point here is that we need to judge when the scroll bar bottoms out. There are two main ways to judge

  • scrollTop + clientHeight >= innerHeight

  • or

  • new MutationObserver() to observe

At present, the principles of some mainstream plug-ins on the market are roughly these two.

This is an example of the author, using the plug-in v-el-table-infinite-scroll, which is essentially a custom command. Corresponding npm address: www.npmjs.com/package/el-…

Of course, there are other plug-ins, such as vue-scroller, etc.: one meaning, no more details

Note that loading at the bottom is also divided into piles. The 100,000 pieces of data obtained by sending the request are divided into piles, and then every time the bottom is touched, just load a pile

Using el-table-infinite-scroll instruction steps in el-table

Install, pay attention to the version number (distinguish between vue2 and vue3)

“cnpm install \--save [email protected]”

Register to use the directive plugin

// use the infinite scroll plugin
import elTableInfiniteScroll from 'el-table-infinite-scroll';
Vue.use(elTableInfiniteScroll);

Because it is a custom command, it can be written directly on the el-table tag

<el-table
  v-el-table-infinite-scroll="load"
  :data="tableData"
>
  <el-table-column prop="id" label="ID"></el-table-column>
  <el-table-column prop="name" label="name"></el-table-column>
</el-table>

async load() {
    // Bottom loading, display data...
},

Case Code

For the convenience of everyone’s demonstration, here the author directly attaches a case code, pay attention to the “step” comment in it

<template>
  <div class="box">
    <el-table
      v-el-table-infinite-scroll="load"
      height="600"
      :data="tableData"
      border
      style="width: 80%"
      v-loading="loading"
      element-loading-text="The amount of data is too large, the guest officer will wait later..."
      element-loading-spinner="el-icon-loading"
      element-loading-background="rgba(255, 255, 255, 0.5)"
      :header-cell-style="{
        height: '24px',
        lineHeight: '24px',
        color: '#606266',
        background: '#F5F5F5',
        fontWeight: 'bold',
      }"
    >
      <el-table-column type="index" label="order"></el-table-column>
      <el-table-column prop="id" label="ID"></el-table-column>
      <el-table-column prop="name" label="name"></el-table-column>
      <el-table-column prop="value" label="corresponding value"></el-table-column>
    </el-table>
  </div>
</template>

<script>
// heap function
function averageFn(arr) {
  let i = 0;
  let result = [];
  while (i < arr. length) {
    result.push(arr.slice(i, i + 10)); // Intercept 10 slices at a time for heap splitting
    i = i + 10; // These 10 interceptions are finished, and then the next 10 are ready to be intercepted
  }
  return result;
}
import axios from "axios";
export default {
  data() {
    return {
      allTableData: [], // initial request to get all the data
      tableData: [], // data to be displayed
      loading: false
    };
  },
  // The first step is to send a request, get a large amount of data, and convert it into a two-dimensional array, and store it in groups and blocks
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashhuai.work:10000/bigData");
    this.allTableData = averageFn(res.data.data); // Use the heap function to store a two-dimensional array
    // this.originalAllTableData = this.allTableData // You can also save a copy of the original value for future use, whatever works
    this.loading = false;
    // The second step, after the operation is completed, execute the bottom loading method
    this. load();
  },
  methods: {
    // It will be executed once initially, of course it can also be configured so that it will not be executed
    async load() {
      console.log("Automatically execute it multiple times, the first execution will calculate how many times to execute according to the height");
      // The fifth step, loading at the bottom is equivalent to taking out each item of the two-dimensional array for use, and returning to stop when it is used up
      if (this. allTableData. length == 0) {
        console.log("There is no data");
        return;
      }
      // The third step, when loading, take out the first item of the two-dimensional array and splice it into the table data to be displayed
      let arr = this. allTableData[0];
      this.tableData = this.tableData.concat(arr);
      // The fourth step, after splicing and displaying, delete the data of the first item of the two-dimensional array
      this.allTableData.shift();
    },
  },
};
</script>

Scheme 6: Use infinite loading/virtual list for display

What is a virtual list?

  • The so-called virtual list is actually a manifestation of “front-end cover-up”.

  • It seems that all the data has been rendered, but in fact only the “Visible Area” part is rendered.

  • It’s a bit like we watch a movie. When we watch it, we watch it on a movie screen, watching it second by second (non-stop showing)

  • But in fact, the movie is two hours long. If the two-hour movie is spread out, how many movie screens are needed?

  • Similarly, if all 100,000 pieces of data are rendered, how many dom node elements are needed?

  • So we only show the user what he “can see now”

  • If the user wants to fast forward or rewind (pull down the scroll bar or pull up the scroll bar)

  • Then present the corresponding content on the movie screen (presented in the visible area)

  • In this way, it looks like all dom elements and every piece of data are rendered.

Regarding the front-end blindfold method, in specific work, if it can be used skillfully, it will greatly improve our development efficiency

Write a simple dummy list

Here, the author directly uploads the code, and everyone can use it by copying and pasting. The author wrote some notes for everyone to understand. Of course, you can also go to the author’s warehouse to have a look, the GitHub warehouse is at the end of the article

Code

<template>
  <!-- Virtual list container, similar to "window", the height of the window depends on how many pieces of data are displayed at a time
            For example, the window can only see 10 pieces of data, one piece of 40 pixels, and 10 pieces of 400 pixels
            Therefore, the height of the window is 400 pixels, pay attention to enable positioning and scroll bars -->
  <div
    class="virtualListWrap"
    ref="virtualListWrap"
    @scroll="handleScroll"
    :style="{ height: itemHeight * count + 'px' }"
  >
    <!-- placeholder dom element, its height is the total height of all data -->
    <div
      class="placeholderDom"
      :style="{ height: allListData. length * itemHeight + 'px' }"
    ></div>
    <!-- content area, display 10 pieces of data, note that the top value of its positioning changes -->
    <div class="contentList" :style="{ top: topVal }">
      <!-- Each (item) data -->
      <div
        v-for="(item, index) in showListData"
        :key="index"
        class="itemClass"
        :style="{ height: itemHeight + 'px' }"
      >
        {<!-- -->{ item.name }}
      </div>
    </div>
    <!-- loading part -->
    <div class="loadingBox" v-show="loading">
      <i class="el-icon-loading"></i>
       &nbsp; &nbsp;<span>loading...</span>
    </div>
  </div>
</template>
<script>
import axios from "axios";
export default {
  data() {
    return {
      allListData: [], // All data, for example, this array stores 100,000 pieces of data
      itemHeight: 40, // height of each bar (item), such as 40 pixels
      count: 10, // How many pieces of data are displayed on one screen
      start: 0, // index of starting position
      end: 10, // index of end position
      topVal: 0, // The scroll bar of the parent element scrolls, change the value corresponding to the top positioning of the child element, and ensure linkage
      loading: false,
    };
  },
  computed: {
    // Intercept the data to be displayed from all the data allListData showListData
    showListData: function () {
      return this.allListData.slice(this.start, this.end);
    },
  },
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashhuai.work:10000/bigData");
    this.allListData = res.data.data;
    this.loading = false;
  },
  methods: {
    // Scrolling here can add throttling to reduce the trigger frequency
    handleScroll() {
      /**
       * Get how many pixels the scroll bar has scrolled in the vertical direction from Element.scrollTop
       *
       * Divide the scrolling distance by the height of each item, that is, how many items have been scrolled, of course, an integer must be taken
       * Example: Rolling 4 meters, one step length is 0.8 meters, how many steps to roll, 4/0.8 = step 5 (round up for easy calculation)
       *
       * And because we want to display 10 items at a time, we know the start position item, plus the end position item,
       * You can get the interval [start position, start position + size items] == [start position, end position]
       * */
      const scrollTop = this. $refs. virtualListWrap. scrollTop;
      this.start = Math.floor(scrollTop / this.itemHeight);
      this.end = this.start + this.count;
      /**
       * Dynamically change the top value of the positioning to ensure linkage and dynamically display the corresponding content
       * */
      this.topVal = this.$refs.virtualListWrap.scrollTop + "px";
    },
  },
};
</script>
<style scoped lang="less">
// virtual list container box
.virtualListWrap {
  box-sizing: border-box;
  width: 240px;
  border: solid 1px #000000;
  // enable scroll bar
  overflow-y: auto;
  // enable relative positioning
  position: relative;
  .contentList {
    width: 100%;
    height: auto;
    // with absolute positioning
    position: absolute;
    top: 0;
    left: 0;
    .itemClass {
      box-sizing: border-box;
      width: 100%;
      height: 40px;
      line-height: 40px;
      text-align: center;
    }
    // Change the color of odd and even rows
    .itemClass: nth-child(even) {
      background: #c7edcc;
    }
    .itemClass:nth-child(odd) {
      background: pink;
    }
  }
  .loadingBox {
    position: absolute;
    top: 0;
    left: 0;
    right: 0;
    bottom: 0;
    width: 100%;
    height: 100%;
    background-color: rgba(255, 255, 255, 0.64);
    color: green;
    display: flex;
    justify-content: center;
    align-items: center;
  }
}
</style>

Use the vxetable plug-in to realize virtual list

If it is not a list but a table, the author here recommends a useful UI component, vxetable. You can tell from the name that it is doing table-related business. Among them is the virtual list.

Both vue2 and vue3 are supported, and the performance is better. The official said: **virtual scrolling (can support up to 5w columns and 30w rows)* *

powerful!

Official website address: vxetable.cn/v3/#/table/…

Installation code

Pay attention to the installed version, the version used by the author is as follows:

cnpm i xe-utils [email protected] \--save

“main.js”

// use VXETable
import VXETable from 'vxe-table'
import 'vxe-table/lib/style.css'
Vue.use(VXETable)

The code is also very simple, as follows:

<template>
  <div class="box">
    <vxe-table
      border
      show-overflow
      ref="xTable1"
      height="300"
      :row-config="{ isHover: true }"
      :loading="loading"
    >
      <vxe-column type="seq"></vxe-column>
      <vxe-column field="id" title="ID"></vxe-column>
      <vxe-column field="name" title="name"></vxe-column>
      <vxe-column field="value" title="corresponding value"></vxe-column>
    </vxe-table>
  </div>
</template>

<script>
import axios from "axios";
export default {
  data() {
    return {
      loading: false,
    };
  },
  async created() {
    this.loading = true;
    const res = await axios.get("http://ashhuai.work:10000/bigData");
    this.loading = false;
    this. render(res. data. data);
  },
  methods: {
    render(data) {
      this. $nextTick(() => {
        const $table = this. $refs. xTable1;
        $table.loadData(data);
      });
    },
  },
};
</script>

Scheme 7: Open multi-threaded Web Worker for operation

In this case, using Web Worker to open another thread to operate the code logic, the benefit is not particularly large (if the virtual scrolling list plug-in is used)

But it can be regarded as an expanded idea. During the interview, you can talk about it and mention it.

Friends who are not familiar with Web Worker, you can read the author’s previous article: “Performance optimization using vue-worker plug-in (based on Web Worker) to enable multi-threaded computing to improve efficiency”

Plan Eight: Plan ahead and prevent problems before they happen

The following is my humble opinion, for reference only…

  • After all the above solutions have been said, it is not over.

  • In fact, this question not only examines the breadth and depth of the candidate’s knowledge, but also examines the candidate’s way of thinking about dealing with problems, which is especially important!

  • The author has worked as a candidate to apply for a job, and as an interviewer to interview. As far as programmer development work is concerned, if you are not familiar with technical knowledge points, you can learn quickly, such as documents, Google, Baidu, technical exchange groups, and relevant colleagues can provide certain support

  • What’s more important is to take a fancy to the candidate’s way of thinking, thinking mode

  • Just imagine, two candidates are of the same level of strength, but one only knows how to work hard, and does not think about it; while the other is working hard, he will also look up at the stars and analyze how to work hard. Complete tasks cost-effectively, focusing on process and results

  • In this case, which one is more popular?

If the author is a candidate, after talking about the above 7 options, the author will add the eighth option: plan for a rainy day and prevent problems before they happen

Scene simulation

The interviewer randomly looked at my resume in his hand, stroked his beard and screamed strangely: “Boy, the backend wants to return 100,000 pieces of data to you at one time, how do you deal with it?”

I raised my eyebrows and smiled crookedly: “After the above 7 solutions are stated, I think we can solve similar problems fundamentally. That is the eighth solution,“We must plan ahead and prevent problems before they happen” . “

“Oh?” The interviewer was puzzled, and slowly put down my resume: “I would like to hear more about it.”

I answered unhurriedly: “In the specific development work, when we receive a requirement, during the technical review period, we have to discuss with the backend about a more appropriate technical solution. This problem is The backend wants to return me 100,000 pieces of data at one time, the point is not so much data as 100,000 pieces, but why the backend does this?”

The interviewer looked up and listened carefully.

I said word by word: “Except for ** business really need this kind of solution** (if the customer asks, then there is nothing to say, just do it), the backend will do this There are roughly two reasons. The first one is that he doesn’t know much about the limit statement in SQL, but this is basically impossible. The second is that he has something to do, so he wrote it casually. Therefore, it is necessary to communicate with him, from the big data The amount of interface request time is too long, and too many dom element rendering leads to poor performance, and the maintainability of the project. I believe that as long as the communication is correct, this unreasonable situation can be avoided from the root. occur.”

The interviewer suddenly asked slyly: “What if the backend doesn’t give you a pagination after the communication? What should you do? Your communication has no effect! How do you deal with it! People don’t listen to you!” It seems that this question is very tricky , He folded his arms in front of his chest, leaning on the back of the chair, waiting for the awkward smile that was about to bloom on my face that I couldn’t answer.

I snorted coldly in my heart: The little tricks…

I stared into the eyes of the interviewer and said seriously: “If the communication at work is ineffective, it is either my own communication language problem. I will pay attention to this and constantly improve my communication skills and speaking style, or… “

My voice raised three points: “There is a problem with the person I communicated with! He is lazy and slippery at work! He is stubborn! Embarrassing others! He is superior! Self-righteous! In this case, I will find my direct leader to intervene, because This is no longer a matter of project requirements, but a matter of basic quality of employees!”

After a pause for a second, my voice softened a bit: “However, I believe that there is absolutely no such person among our company’s employees. All of them are excellent employees with strong abilities and correct attitudes. After all, our company is in the industry. It has been famous for a long time, and I came here because of it. Are you right?”

Shock flashed in the interviewer’s eyes. He didn’t expect that I would kick the ball to him again. However, in order to maintain his image, he immediately recovered his composure, but his facial muscles were trembling uncontrollably.

I added: “In fact, at work, as a role closer to users, the front-end needs to communicate with colleagues in various positions, such as back-end, product, UI, testing, etc. We need to use reasonable communication methods, To improve work efficiency, complete projects, realize their own value, and create benefits for the company, I think this is what every employee needs to do, and it must be done.”

The interviewer stroked his beard again and screamed strangely: “The boy’s performance is not bad, you have been hired! A monthly salary of 2200, bring your own computer, no company and no money, 007 working system, you can’t steal company snacks, and…”

Me: Ada…

Original link: https://juejin.cn/post/7205101745936416829

Author: Shui Rong Shui Fu

Summary

“Effective communication comes from the thinking mode of solving problems. In most cases, the importance is greater than the current technical knowledge”

  • Website effect demonstration address: http://ashhuai.work:8888/#/bigData

  • GitHub warehouse address: https://github.com/shuirongshuifu/elementSrcCodeStudy

Write at the end

Previous recommendations
Latest, responsive syntactic sugar in Vue is deprecated
The details of async and await devils that you don't know
Hurry up! The Tencent WeChat team is recruiting, and the resume will be sent directly to the interviewer!

at last

Welcome to add me on WeChat, pull you into the technology group, long-term exchange and learning...

Welcome to pay attention to "Front-end Q", study the front-end seriously, and be a professional technical person...

Click to watch and support me