How Web Scraping is Used to Scrape Website using Infinite Scrolling?
For example, you want to scrape 100 Flipkart products from every category. But this formula will only extract the top 15 products on a page. Flipkart offers a feature called limitless scrolling, which eliminates the need for pagination (such as?page=2,?page=3) in the URL. If it had this functionality, we would have entered the value into a "while loop" and increased the page values as seen below. page_count = 0 while page_count < 5: URL = "http://example.com?page=%d" %(page_count) # scraping code... page_count += 1 So, let's go back to limitless scrolling. The usage of "Ajax" allows any webpage to employ limitless scrolling. However, that ajax request also includes a URL from which the goods on the same page are loaded when scrolling. To look at that URL. Launch Google Chrome and navigate to the page. Then, in the console, right-click and choose LogXMLHttpRequests. Refresh the page and scroll gently now. When new goods are added, you will noti...