Apache StormCrawler · v3.7.0

Every web
has its
spider.

Apache StormCrawler is an open source SDK for building distributed web crawlers based on Apache Storm®. Low-latency, scalable and polite: a collection of reusable components, written mostly in Java, under Apache License v2.

Get started Download 3.7.0

Apache License v2 Java, on Apache Storm®

SEED · START FROM A LIBRARY

Feed it URLs, get a crawler.

Apache StormCrawler is a library and collection of resources that developers leverage to build their own crawlers. The good news: doing so can be pretty straightforward. Seed it with your URLs and let the topology do the rest.

READ THE FULL GUIDE

FETCH · POLITE, AT SCALE

Crawl continuously, not in batches.

URLs are processed as streams across a whole Apache Storm® cluster: add workers, not complexity. Storm's guaranteed processing means no URL is silently lost, and robots.txt, delays and politeness settings are built in from the start.

CONFIGURATION

PARSE · REUSABLE MODULES

Apache Tika®, sitemaps, feeds: plug them in.

Apart from the core components, we provide external resources you can reuse in your project: a ParserBolt that uses Apache Tika® to parse various document formats, sitemap and feed parsers, filters and much more. Pluggable, mostly plain Java.

EXTERNAL MODULES

INDEX · YOUR BACKEND

Straight into OpenSearch®, Apache Solr® or SQL.

Spouts and bolts for OpenSearch®, Apache Solr® and SQL storage ship as external modules, so documents are indexed as they are fetched, whether you crawl a stream of URLs or run large-scale recursive crawls where low latency is required.

STORAGE MODULES

Run your first crawl.

$ mvn archetype:generate \
   -DarchetypeGroupId=org.apache.stormcrawler \
   -DarchetypeArtifactId=stormcrawler-archetype \
   -DarchetypeVersion=3.7.0