Security

Reporting New Security Problems with Apache StormCrawler

The Apache Software Foundation takes a very active stance in eliminating security problems and denial of service attacks against its products.

We strongly encourage people to report security problems privately using the security mailing list of the ASF Security Team before disclosing them in a public forum.

Please note that the security mailing list should only be used for reporting undisclosed security vulnerabilities and managing the process of fixing such vulnerabilities. We cannot accept regular bug reports or other queries at this address. All mail sent to this address that does not relate to an undisclosed security problem in our source code will be ignored.

The private security mailing address is: security@apache.org

Threat Model and Security Considerations

StormCrawler is designed to operate in trusted environments as part of a distributed Apache Storm® cluster. This document outlines the threat model and key security assumptions to help users understand the secure use and deployment of StormCrawler.

Crawled Content Is Untrusted by Design

StormCrawler is a library for broad web crawling. It fetches URLs it was told to fetch, including URLs discovered in previously fetched pages, and it parses the bytes that come back. Both the URLs and the bytes are chosen by the operators of the sites being crawled, which means attacker-controlled input is not an edge case for a crawler: it is the normal operating condition. Every component that handles a response body, a response header, a redirect target or an extracted outlink is handling hostile data by default.

It follows that consequences of the crawler crawling what it was configured to crawl are not, in themselves, vulnerabilities. If a topology is pointed at a host, fetches a resource from it and stores the result, that is the software working as intended, regardless of what the remote host chose to serve. Content crossing a boundary it was never meant to cross is what the project fixes first: crawled bytes influencing the crawler's own configuration or control data, reaching resources the operator did not expose to the crawl, or causing credentials to be disclosed. Whether such a defect also warrants a published advisory is a separate question, answered under Advisories and Hardening below.

This is the frame for the rest of this section. The controls described below exist so that operators can define where that boundary lies for their deployment. Choosing not to configure them widens the boundary rather than removing it.

Malformed or hostile input that makes a crawl slow, stalls a worker or exhausts its memory is a bug the project fixes, whether it arrives in a response or by way of data the crawl stored earlier. Availability effects of this kind are confined to the crawl the operator chose to run, and the project treats them as robustness rather than as a compromise of the deployment. This is separate from the availability of the Apache Storm® cluster itself, which is the operator's to protect.

Trusted Configuration

The configuration file used by StormCrawler is loaded during topology submission and is treated as a trusted source. It does not involve any user-supplied input at runtime.

If an attacker is able to modify this file, they would already have full access to the system, including:

Securing the configuration file and the environment in which topologies are submitted is essential. However, modification of the file implies full system compromise and is out of scope for runtime protections.

Apache Storm® Cluster Security

StormCrawler runs on an Apache Storm® cluster, which is designed to allow users to:

This model inherently trusts cluster users and assumes they are authorized.

Security Recommendations:

Operator Responsibilities

StormCrawler is a library, not a finished crawler. Several controls that determine what a crawl is allowed to reach ship disabled or permissive, because the library cannot know the scope of the crawl a given operator intends to run. Configuring them is the operator's responsibility.

URL Filtering

The library ships no URL filters at all: urlfilters.config.file is present but commented out in crawler-default.yaml, so a topology built directly on the library will follow every outlink it discovers, unbounded, unless the operator supplies a filter configuration. Projects generated from the StormCrawler Maven archetypes do not have this problem: they set urlfilters.config.file and ship a default-regex-filters.txt as a starting point.

A topology running with no URL filtering at all is not a supported configuration. Operators should define the intended crawl scope explicitly, in terms of hosts, domains and URL patterns, and should treat the filter configuration as a security control rather than a tuning parameter.

Protocol Schemes

The protocols key currently ships as "http,https,file", which registers the file protocol implementation alongside the HTTP ones. Because outlinks are extracted from crawled pages, an enabled file scheme gives crawled content a path to the local filesystem of the worker unless URL filtering prevents it. This is the combination that matters: the file scheme and the absence of URL filters together, not either one alone.

Non-web schemes should be enabled deliberately, for deployments that actually crawl local files, and should be paired with URL filters that constrain which paths may be reached. Operators who do not crawl local content should reduce protocols to "http,https".

Egress Restriction

Regular-expression URL filters operate on the URL string and cannot know what a hostname resolves to. A hostname under the control of a crawled site can resolve to a loopback, link-local or private address, which no pattern on the URL can detect. http.filter.ipaddress.include and http.filter.ipaddress.exclude address this: they are applied after DNS resolution, at connection time, and are the only control in the library that can act on the resolved address. Both ship commented out in crawler-default.yaml, with "localhost,sitelocal" shown as a suggested exclude value.

Operators crawling the public web should set an exclude rule covering loopback and site-local ranges at minimum; operators crawling a known internal estate should prefer an explicit include rule. Note that this filtering is implemented in the OkHttp protocol only. Topologies using other protocol implementations, including the browser-based ones, need to obtain the same restriction from the network layer around the workers.

Resource Limits

http.content.limit defaults to -1, meaning no limit: the size of a fetched document is decided by the remote server. The archetype configurations set it to 65536. Since response bodies are attacker-chosen, an unbounded limit lets a crawled host decide how much memory a worker allocates for a single fetch.

Operators should set a finite http.content.limit appropriate to the content they intend to index. The related http.robots.content.limit defaults to the same value and deserves the same treatment.

Transport and Credentials

http.trust.everything currently defaults to true. With that setting, the OkHttp protocol accepts any server certificate and performs no hostname verification. The rationale is that broad crawling encounters expired, self-signed and otherwise misconfigured certificates constantly, that a crawl aborting on them would be of limited use, and that the fetched bytes are treated as hostile in any case, so certificate validity adds little to how the response is handled.

That rationale covers the content of an ordinary anonymous crawl. It does not cover anything the crawler sends. With certificate and hostname verification disabled, the crawler cannot tell which host it is actually talking to, so any credential, custom header or replayed cookie attached to a request crosses a channel whose far end was never authenticated. Operators who crawl anything authenticated must weigh this, and should set http.trust.everything to false for those crawls.

Credentials configured through http.basicauth.user and http.basicauth.password are turned into a fixed Authorization header that is attached to every request the protocol instance makes, with no host scoping. If such a topology follows an outlink to a third-party host, the credentials go with it. Operators crawling authenticated sites should scope credentials by origin themselves, by running a separate topology per origin with URL filters confining that topology to it, rather than combining authenticated and general crawling in one topology.

Cookie replay is a related risk. http.use.cookies is false by default, and enabling it causes cookies stored from a response to be sent with requests for links found in that response. Operators enabling it on an authenticated crawl should apply the same per-origin separation as for Basic authentication.

Cookie scoping as implemented is not a security boundary. Operators should confine authenticated crawls by topology rather than relying on which cookies are sent where.

Advisories and Hardening

StormCrawler follows the ASF Security Team's policy on when an advisory is issued.

Because StormCrawler is a library rather than a deployed service, the configuration it ships is a starting point and not a secured deployment. Where a weakness is reachable because a control described on this page was not configured, we improve the default and treat the change as hardening. Where the library defeats a control an operator did configure, or discloses a configured secret in a way this page does not describe, we treat it as a vulnerability.

Whether we publish an advisory follows from that classification and not from how much work the fix requires of anyone. A vulnerability in a released artifact gets an advisory even when installing the next release is all an operator has to do, because the point of an advisory is to let operators judge how urgently to upgrade. Where more than an upgrade is needed, such as changing a configuration, inspecting or repairing stored data, or rotating a credential, the advisory says so and the release notes repeat it. A hardening change ships in a normal release with a release note and no advisory.

Log output is not a confidentiality boundary. Enabling DEBUG logging on the protocol implementations can put configured credentials into the logs. Treat DEBUG output from a topology that uses an authenticated proxy or crawls authenticated sites as sensitive, and scope log retention accordingly.

As the ASF policy notes, what counts as a vulnerability depends in part on what a project states as expected behaviour, which is why this page describes the crawler's threat model rather than only its reporting process.

Summary

StormCrawler's security model assumes a trusted deployment environment. Users should:

Asking Questions About Known Security Problems

Questions about:

should be addressed to the dev mailing list.

Known Security Vulnerabilities

No known security vulnerability yet.