Skip to content
Fencer Intel
medium 4.3

GHSA-jm3v-qxmh-hxwv json

Scrapy's redirects ignoring scheme-specific proxy settings

published 2y ago modified last yr.

cvss
4.3
version 3.1, GitHub
cisa kev
Not listed
not in CISA's catalog
epss
—
no score, EPSS covers CVEs only
public exploits
0
none known

Summary

### Impact

When using system proxy settings, which are scheme-specific (i.e. specific to `http://` or `https://` URLs), Scrapy was not accounting for scheme changes during redirects.

For example, an HTTP request would use the proxy configured for HTTP and, when redirected to an HTTPS URL, the new HTTPS request would still use the proxy configured for HTTP instead of switching to the proxy configured for HTTPS. Same the other way around.

If you have different proxy configurations for HTTP and HTTPS in your system for security reasons (e.g., maybe you don’t want one of your proxy providers to be aware of the URLs that you visit with the other one), this would be a security issue.

### Patches

Upgrade to Scrapy 2.11.2.

### Workarounds

Replace the built-in retry middlewares (`RedirectMiddleware` and `MetaRefreshMiddleware`) and the `HttpProxyMiddleware` middleware with custom ones that implement the fix from Scrapy 2.11.2, and verify that they work as intended.

### References

This security issue was reported by @redapple at https://github.com/scrapy/scrapy/issues/767.

Patch / fix

  • pypi scrapy 2.11.2

Is GHSA-jm3v-qxmh-hxwv in your dependencies? Fencer shows which of your repositories are affected as soon as an advisory lands, and opens the fix as a pull request.

Check with Fencer

CVSS scores

assigned by version score vector
GitHub 3.1 4.3 CVSS:3.1/AV:N/AC:L/PR:L/UI:N/S:U/C:L/I:N/A:N

Affected packages

ecosystem package affected fixed in
pypi scrapy
<2.11.2
2.11.2

References

Sources

Fencer Intel keeps one entry per vulnerability: the CVE id wins, then GHSA, and records that alias each other are merged, except that two CVEs always keep separate entries. The same data is available as JSON.