Skip to content
Fencer Intel
medium 6.5

GHSA-23j4-mw76-5v7h json

Scrapy allows redirect following in protocols other than HTTP

published 2y ago modified last yr.

cvss
6.5
version 3.1, GitHub
cisa kev
Not listed
not in CISA's catalog
epss
—
no score, EPSS covers CVEs only
public exploits
0
none known

Summary

### Impact

Scrapy was following redirects regardless of the URL protocol, so redirects were working for `data://`, `file://`, `ftp://`, `s3://`, and any other scheme defined in the `DOWNLOAD_HANDLERS` setting.

However, HTTP redirects should only work between URLs that use the `http://` or `https://` schemes.

A malicious actor, given write access to the start requests (e.g. ability to define `start_urls`) of a spider and read access to the spider output, could exploit this vulnerability to: - Redirect to any local file using the `file://` scheme to read its contents. - Redirect to an `ftp://` URL of a malicious FTP server to obtain the FTP username and password configured in the spider or project. - Redirect to any `s3://` URL to read its content using the S3 credentials configured in the spider or project.

For `file://` and `s3://`, how the spider implements its parsing of input data into an output item determines what data would be vulnerable. A spider that always outputs the entire contents of a response would be completely vulnerable, while a spider that extracted only fragments from the response could significantly limit vulnerable data.

### Patches

Upgrade to Scrapy 2.11.2.

### Workarounds

Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.

### References

This security issue was reported by @mvsantos at https://github.com/scrapy/scrapy/issues/457.

Patch / fix

  • pypi scrapy 2.11.2

Is GHSA-23j4-mw76-5v7h in your dependencies? Fencer shows which of your repositories are affected as soon as an advisory lands, and opens the fix as a pull request.

Check with Fencer

CVSS scores

assigned by version score vector
GitHub 3.1 6.5 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:N/A:N

Affected packages

ecosystem package affected fixed in
pypi scrapy
<2.11.2
2.11.2

Classification

Weakness
  • CWE-552 Files or Directories Accessible to External Parties

References

Sources

Fencer Intel keeps one entry per vulnerability: the CVE id wins, then GHSA, and records that alias each other are merged, except that two CVEs always keep separate entries. The same data is available as JSON.