Top Web Crawling Libraries for Java

us.codecraft : webmagic-extension

Jul 22, 2021
24 usages
10k stars

us.codecraft : webmagic-core

Jul 22, 2021
36 usages
10k stars
jsoup 1.15.1

jsoup Java HTML Parser · jsoup is a Java library for working with real-world HTML. It provides a very convenient API for fetching URLs and extracting and manipulating data, using the best of HTML5 DOM methods and CSS selectors. jsoup implements the WHATWG HTML5 specification, and parses HTML to the same DOM as modern browsers do.

May 15, 2022
2.1k usages
9.2k stars
crawler4j 4.4.0

edu.uci.ics:crawler4j · Open Source Web Crawler for Java

Mar 26, 2018
7 usages
4.2k stars
nutch 1.18

Apache Nutch · Nutch is an open source web-search software. It builds on Hadoop, Tika and Solr, adding web-specifics, such as a crawler, a link-graph database etc.

Jan 19, 2021
1 usages
2.2k stars

storm-crawler-archetype

Apr 13, 2022
735 stars