Web CrawlerPublished 5 years ago in category News — Comment
Have you ever wondered how Google crawls data? We have! Fortunately we had an opportunity to develop a web crawler for a customer with data extraction. It was a very challenging project but one of the most interesting. Taking care to crawl in parallel, taking care not to suffocate web servers, smartly revisiting only pages, extracting data … very interesting.