Apache StormCrawler 3.x - Documentation

Below is a simple docker-compose.yml configuration to spin up URLFrontier, Zookeeper, Storm Nimbus, Storm Supervisor, and the Storm UI:

Notes:

After setting up your Docker Compose, you should start it up:

Check the logs and see if every service is up and running:

Next, access the Storm UI via http://localhost:8080 and check that a Storm Nimbus as well as a Storm Supervisor is available.

Build the generated archetype by running

This will create a uberjar named ${artifactId}-${version}.jar (matches the artifact id and the version specified during the archetype generation) in your target directory.

Now you are ready to insert your first seeds into URLFrontier. To do so, create a file seeds.txt containing your seeds:

After you have saved it, we need to inject the seeds into URLFrontier. This can be done by running URLFrontiers client:

where seeds.txt is the previously created file containing URLs to inject, with one URL per line.

Now it is time to run our first crawl. To do so, we need to start our crawler topology in distributed mode and deploy it on our Storm Cluster.

where ${NETWORK} is the name of the Docker network of the previously started Docker Compose. You can find this name by running

After running the storm jar command, you should carefully monitor the logs via

as well as the Storm UI. It should now list a running topology.

In the default archetype, the fetched content is printed out to the default system out print stream.

Congratulations! You learned how to start your first simple crawl using Apache StormCrawler.

Feel free to explore the rest of our documentation to build more complex crawler topologies.

The crawl begins with the Frontier, which is responsible for scheduling, prioritization, politeness, and retry logic. URLs are emitted by a Spout (e.g., AggregationSpout, SQLSpout) and partitioned by the URLPartitioner, typically using the host as a key to enforce politeness constraints.

The Fetcher retrieves web resources and emits both the fetched content and associated metadata such as HTTP status codes, headers, and MIME types. Based on the content type, documents are routed to specialized parsers, including SiteMapParser, JSoupParser for HTML content, and ParserBolt (Tika module) for binary formats via Apache Tika.

Parsed content is then sent to the Indexer and persisted by the Storage layer. Throughout the pipeline, fetch and parse outcomes are reported to the StatusUpdater, which feeds URL status information back to the frontier, closing the crawl feedback loop.

URL Filters determine whether a URL should be accepted, rejected, or modified before it is scheduled for fetching. They operate on seed URLs, discovered links, and redirect targets, ensuring that only crawl-worthy URLs enter the frontier.

In Figure 1, URL Filters are conceptually positioned between link discovery and the frontier. Their primary role is to control crawl scope and prevent frontier explosion.

Typical URL Filters include:

By applying these filters early, StormCrawler prevents unnecessary fetches and maintains an efficient, focused crawl.

Parse Filters operate after content has been successfully fetched and parsed. They allow fine-grained control over how extracted data and outgoing links are processed.

Parse Filters are applied within the parsing bolts, following parsing by SiteMapParser, JSoupParser, or ParserBolt (Tika). They can modify extracted text, metadata, and links before the content is indexed or new URLs are emitted.

Common Parse Filters include:

Parse Filters enable domain-specific logic without coupling it directly to the crawler’s core components.

URL Filters focus on deciding what should be crawled, while Parse Filters focus on deciding what should be kept and how it should be interpreted.

The AdaptiveScheduler extends the DefaultScheduler and adjusts the fetch interval based on whether a page has changed since the last fetch (detected via MD5 signature comparison). If the page changed, the interval shrinks toward a minimum; if unchanged, it grows toward a maximum. This avoids unnecessary re-fetches of static pages while keeping dynamic pages fresh.

Configuration example:

The AdaptiveScheduler requires the MD5SignatureParseFilter to be configured so that the signature metadata field is populated.

There are actually two different bolts for fetching the content of URLs:

Both declare the same output:

with the status stream being used for handling redirections, restrictions by robots directives, or fetch errors, whereas the default stream gets the binary content returned by the server as well as the metadata to the following components (typically a parsing bolt).

Both use the same Protocols implementations and URLFilters to control the redirections.

The FetcherBolt has an internal set of queues where the incoming URLs are placed based on their hostname/domain/IP (see config fetcher.queue.mode) and a number of FetchingThreads (config fetcher.threads.number – 10 by default) which pull the URLs to fetch from the FetchQueues. When doing so, they make sure that a minimal amount of time (set with fetcher.server.delay – default 1 sec) has passed since the previous URL was fetched from the same queue. This mechanism ensures that we can control the rate at which requests are sent to the servers. A FetchQueue can also be used by more than one FetchingThread at a time (in which case fetcher.server.min.delay is used), based on the value of fetcher.threads.per.queue.

Incoming tuples spend very little time in the execute method of the FetcherBolt as they are put in the FetchQueues, which is why you’ll find that the value of Execute latency in the Storm UI is pretty low. They get acked later on, after they’ve been fetched. The metric to watch for in the Storm UI is Process latency.

The SimpleFetcherBolt does not do any of this, hence its name. It just fetches incoming tuples in its execute method and does not do multi-threading. It does enforce politeness by checking when a URL can be fetched and will wait until it is the case. It is up to the user to declare multiple instances of the bolt in the Topology class and to manage how the URLs get distributed across the instances of SimpleFetcherBolt, often with the help of the URLPartitioner.

The JSoupParserBolt can be used to parse HTML documents and extract the outlinks, text, and metadata it contains. If you want to parse non-HTML documents, use the Tika-based ParserBolt from the external modules.

This parser calls the URLFilters and ParseFilters defined in the configuration. Please note that it calls MetadataTransfer prior to calling the ParseFilters. If you create new Outlinks in your , you’ll need to make sure that you use MetadataTransfer there to inherit the Metadata from the parent document.

The JSoupParserBolt automatically identifies the charset of the documents. It uses the status stream to report parsing errors but also for the outlinks it extracts from a page. These would typically be used by an extension of AbstractStatusUpdaterBolt and persisted in some form of storage.

StormCrawler can handle sitemap files thanks to the SiteMapParserBolt. This bolt should be placed before the standard ParserBolt in the topology, as illustrated in CrawlTopology.

The reason for this is that the SiteMapParserBolt acts as a filter: it passes on any incoming tuples to the default stream so that they get processed by the ParserBolt, unless the tuple contains isSitemap=true in its metadata, in which case the SiteMapParserBolt will parse it itself. Any outlinks found in the sitemap files are then emitted on the .

The SiteMapParserBolt applies any configured ParseFilters to the documents it parses and, just like its equivalent for HTML pages, it uses MetadataTransfer to populate the Metadata objects for the Outlinks it finds.

The FeedParserBolt handles RSS and Atom feeds. Like the SiteMapParserBolt, it acts as a filter in the topology: it passes non-feed tuples through on the default stream, and only parses tuples whose metadata contains isFeed=true. Discovered feed entries are emitted on the status stream. The bolt is configured with feed.sniffContent (to auto-detect feeds) and feed.filter.hours.since.published (to discard old entries).

The URLPartitionerBolt partitions incoming URLs by host, domain, or IP address (controlled by partition.url.mode). It is typically placed before a fetcher bolt to ensure that URLs destined for the same server are routed to the same bolt instance, enabling effective politeness enforcement. It uses the URLPartitioner utility internally.

The StatusEmitterBolt is an abstract base class for bolts that emit tuples on both the default stream and the status stream. It provides utility methods for emitting status updates (e.g., errors, redirections, discovered URLs). The fetcher and parser bolts (FetcherBolt, SimpleFetcherBolt, JSoupParserBolt, SiteMapParserBolt, FeedParserBolt) all extend this class.

The FileSpout reads seed URLs from UTF-8 text files and loads them into memory. Each line is parsed using StringTabScheme into a URL and optional metadata (tab-separated). This is useful for one-off crawls with a known list of URLs.

The MemorySpout stores URLs in an in-memory priority queue. It is intended for testing and debugging in local mode or with a single worker. It can be used with MemoryStatusUpdater to emulate a recursive crawl without an external storage backend.

The AbstractQueryingSpout is the abstract base class for spouts that query a storage backend for URLs to fetch. External module spouts such as AggregationSpout (OpenSearch), SolrSpout, SQLSpout, and the URLFrontier Spout all extend this class. It handles throttling, acking, and coordination with the URL buffer (urlbuffer.class).

ParseFilters are called from parsing bolts such as JSoupParserBolt and SiteMapParserBolt to extract data from web pages. The extracted data is stored in the Metadata object. ParseFilters can also modify the Outlinks and, in that sense, act as URLFilters.

ParseFilters need to implement the interface ParseFilter, which defines three methods:

Here is the default JSON configuration file for ParseFilters. The configuration allows multiple instances of the same filter class with different parameters and supports complex parameter objects. ParseFilters are executed in the order they appear in the JSON file.

The URL filters can be used to both remove or modify incoming URLs (unlike Nutch where these functionalities are separated between URLFilters and URLNormalizers). This is generally used within a parsing bolt to normalize and filter outgoing URLs, but is also called within the FetcherBolt to handle redirections.

URLFilters need to implement the interface URLFilter which defines a single method:

and inherits a default one from Configurable:

The configuration is done via a JSON file which is loaded by the wrapper class URLFilters. The URLFilter instances can be used directly, but it is easier to use the class URLFilters instead. Some filter implementations can also be configured with the standard configuration mechanism.

Here is an example of a JSON configuration file.

The JSON configuration allows loading several instances of the same filtering class with different parameters and can handle complex configuration objects since it makes no assumptions about the content of the field param. The URLFilters are executed in the order in which they are defined in the JSON file.

When handling HTTP redirects, StormCrawler offers three modes:

The following network protocols are implemented in StormCrawler:

Since #829, the HTTP protocol version used is configurable via http.protocol.versions (see also comments in crawler-default.yaml).

For example, to force that only HTTP/1.1 is used:

The metadata argument to HTTPProtocol.getProtocolOutput() can affect the behavior of the protocol. The following metadata keys are detected by HTTPProtocol implementations and utilized in performing the request:

Example:

Notes:

The politeness of a web crawler is not limited to how frequently it fetches pages from a site, but also in how it identifies itself to sites it crawls. This is done by setting the HTTP header User-Agent, just like your web browser does.

The full user agent string is built from the concatenation of the configuration elements:

Whereas StormCrawler used to provide a default value for these, this is not the case since version 2.11 and you will now be asked to provide a value.

You can specify the user agent verbatim with the config http.agent but you will still need to provide a http.agent.name for parsing robots.txt files.

This is also known as the robots.txt protocol, it is formalised in RFC 9309. Part of what the robots directives do is to define rules to specify which parts of a website (if any) are allowed to be crawled. The rules are organised by User-Agent, with a * to match any agent not otherwise specified explicitly, e.g.:

In the example above the rule allows access to the URLs with the /publications/ path prefix, and it restricts access to the URLs with the /example/ path prefix and to all URLs with a .gif suffix. The "*" character designates any character, including the otherwise-required forward slash.

The value of http.agent.name is what StormCrawler looks for in the robots.txt. It MUST contain only uppercase and lowercase letters ("a-z" and "A-Z"), underscores ("_"), and hyphens ("-").

Unless you are running a well known web crawler, it is unlikely that its agent name will be listed explicitly in the robots.txt (if it is, well, congratulations!). While you want the agent name value to reflect who your crawler is, you might want to follow rules set for better known crawlers. For instance, if you were a responsible AI company crawling the web to build a dataset to train LLMs, you would want to follow the rules set for Google-Extended (see list of Google crawlers) if any were found.

This is what the configuration http.robots.agents allows you to do. It is a comma-separated string but can also take a list of values. By setting it alongside http.agent.name (which should also be the first value it contains), you are able to broaden the match rules based on the identity as well as the purpose of your crawler.

If your Apache StormCrawler topology doesn’t extend org.apache.stormcrawler.ConfigurableTopology, you will need to manually register StormCrawler’s Metadata class for serialization in Storm. For more information on Kryo serialization in Apache Storm, you can refer to the documentation.

To register Metadata for serialization, you’ll need to import org.apache.storm.Config and org.apache.stormcrawler.Metadata. Then, in your topology class, you’ll register the class with:

where conf is your Storm configuration for the topology.

Alternatively, you can specify in the configuration file:

The class MetadataTransfer is an important part of the framework and is used in key parts of a pipeline:

An instance (or extension) of MetadataTransfer gets created and configured with the method:

which takes as parameter the standard Storm .

A MetadataTransfer instance has mainly two methods, both returning Metadata objects:

The former is used when creating Outlinks, i.e., in the parsing bolts but also for handling redirections in the [[FetcherBolt(s)]].

The latter is used by extensions of the AbstractStatusUpdaterBolt class to determine which Metadata should be persisted.

The behavior of the default MetadataTransfer class is driven by configuration only. It has the following options:

Note that the method getMetaForOutlink calls filter to determine which key values to keep.

The values below are used by sub-classes of AbstractIndexerBolt.

This refers to persisting the status of a URL (e.g. ERROR, DISCOVERED etc.) along with something like a nextFetchDate that is being calculated by a Scheduler.

Configures parsing of fetched text and the handling of discovered URIs

Options on how Storm Crawler should handle metadata tracking as well as minimising metadata clashes

Integration with OpenSearch for indexing, URL status persistence, and metrics. See the opensearch module for full details.

Integration with Apache Solr for indexing, URL status persistence, and metrics. See the solr module for full details. Supports both standalone Solr and SolrCloud (via ZooKeeper).

Integration with relational databases via JDBC for URL status persistence, indexing, and metrics. See the sql module for full details and table creation scripts.

Integration with URLFrontier, a language-agnostic API for URL management and scheduling. See the urlfrontier module for full details.

Apache Tika integration for parsing non-HTML documents (PDF, Word, etc.). See the tika module for full details.

Integration with AWS services: CloudSearch for indexing and S3 for content caching. See the aws module for full details.

LLM-based text extraction using OpenAI-compatible APIs (including local models via Ollama). See the ai module for full details.

Browser-based fetching using Playwright for JavaScript-rendered pages. See the playwright module for full details.

Language identification for crawled documents using the lang-detect library. See the langid module source code for details.

The LanguageID parse filter is configured in parsefilters.json:

Reading and writing WARC (Web ARChive) files for archival and replay. See the warc module for full details.

The following options are shared across storage-backed spouts (OpenSearch, Solr, SQL) that extend AbstractQueryingSpout:

URL filters accept, reject, or modify URLs during parsing and redirection handling. To create a custom filter, extend the abstract URLFilter class and implement the filter method.

Register the filter in urlfilters.json:

Filters are executed in the order they appear in the JSON file. Return null to reject a URL, or return the (possibly modified) URL string to accept it.

Parse filters extract data from fetched documents and can modify metadata and outlinks. Extend ParseFilter and implement the filter method.

Register it in parsefilters.json:

If needsDOM() returns false for all configured parse filters, the parsing bolt skips DOM generation entirely, improving performance.

Protocol implementations define how content is fetched. Implement the Protocol interface:

Register it in your configuration YAML:

Use the DelegatorProtocol when you need to route URLs to different protocol implementations based on metadata or URL patterns.

The Playwright protocol exposes a PageAction extension point so you can plug site-specific post-navigate behaviour (tab/accordion expansion, cookie-banner dismissal, infinite-scroll, custom evaluate() calls, screenshotting, …​) into the protocol without subclassing it. Actions are loaded as an ordered chain from a JSON file referenced by playwright.page.actions.config.file and follow the same Configurable lifecycle as URL/parse filters. The chain runs after page.navigate() succeeds and before page.content() is captured, so any DOM mutations land in the rendered content returned by the protocol.

A handful of built-in actions ship with the module — DismissOverlayAction, ExpandClickablesAction, ScrollToBottomAction, EvaluateAction, WaitForSelectorAction, ScreenshotAction. Reach for a custom action when none of those fit. Extend PageAction and implement apply:

Reference the action by its fully-qualified class name in the chain JSON:

Per-action failures in apply() are logged and swallowed by the chain wrapper so one bad action cannot abort the rest. If you need a hard failure on misconfiguration, throw from configure() — that propagates at topology start-up, before any URL is fetched.

For a bolt that emits on the status stream (like fetchers and parsers), extend StatusEmitterBolt:

For a custom spout, extend BaseRichSpout and implement open(), nextTuple(), ack(), fail(), and declareOutputFields(). See FileSpout for a simple example. For spouts backed by a storage layer, extend AbstractQueryingSpout instead.

The FetcherBolt groups incoming URLs into queues based on the fetcher.queue.mode setting:

Each queue enforces a minimum delay between consecutive requests. The delay is determined by:

When robots.txt specifies a Crawl-delay:

StormCrawler fetches and caches robots.txt rules per host. The cache is configured via:

Individual aspects of robot compliance can be toggled:

Each URL has a Status value that drives scheduling:

When a URL receives FETCH_ERROR status, the AbstractStatusUpdaterBolt increments a fetch.error.count counter in the URL’s metadata. Once this counter reaches max.fetch.errors (default: 3), the status is escalated to ERROR.

The DefaultScheduler computes the next fetch date based on status:

On a successful fetch, the error counter and error metadata (error.cause, error.message, error.source) are cleared automatically.

You can set fetch intervals based on metadata key-value pairs:

The FetcherBolt tracks:

The reporting interval is controlled by fetcher.metrics.time.bucket.secs (default: 10 seconds).

To persist metrics, register a MetricsConsumer in your topology configuration:

StormCrawler provides MetricsConsumer implementations for OpenSearch, Solr, and SQL.

The Storm UI (default port 8080) provides real-time visibility into topology health, including execute/process latency, emit counts, and error rates per bolt and spout.

In Flux or topology code, set parallelism for each bolt to control Storm executor allocation. Increase spout parallelism when the frontier is the bottleneck.

For crawls with many concurrent hosts, tune OkHttp’s connection pool:

Keep the pool size (idle + active) under 1000. For large crawls, increase protocol.instances.num to distribute connections across multiple pools.

By default, StormCrawler’s OkHttp protocol trusts all certificates (http.trust.everything: true). For production crawls over untrusted networks, consider setting this to false and configuring a proper trust store.

Basic Authentication is supported via:

These credentials are sent as an Authorization header with every request. For per-site authentication, use metadata-driven headers instead (see [Metadata-dependent Behavior For HTTP Protocols] in the Internals section).

Proxy credentials can be configured via:

For rotating proxies, use the MultiProxyManager with a proxy list file. See the Proxy configuration section for details.