Loading huge content in batches in data tables


Hi guys,
Last day, i was having discussion with my friend over using the Data tables to display listing at admin end. We are currently using data tables as it makes it too easy to use searches, sorting , pagination and many more features on table content.
Our concern was, if there are like 100 thousand records the data table would load all of them at once, which may take time and stop things on front end for a while. So he suggested me to write some custom code but this would increase effort at my end.
Is there a way to load content on batches and link it with pagination in datatables ? Or may be load 10 records at first and then keep loading them at backend without showing any effect on front end. I have writen a code where we use paginaton and on click of pagination we load next set of content. but things get complicated when we use pagination with searches or filters. I somehow managed to do it but still i believe there is something much easier and fast that i am missing there as i am not aware about it. anysort of help or suggstion is welcome.

Thanks for anyone going to put any effort on this.