Skip to content

fix: NO_PROXY should support IPv4, IPv6 and localhost#2659

Merged
lovelydinosaur merged 4 commits intoencode:masterfrom
HearyShen:fix_no_proxy
Apr 19, 2023
Merged

fix: NO_PROXY should support IPv4, IPv6 and localhost#2659
lovelydinosaur merged 4 commits intoencode:masterfrom
HearyShen:fix_no_proxy

Conversation

@HearyShen
Copy link
Contributor

@HearyShen HearyShen commented Apr 14, 2023 •
edited by lovelydinosaur
Loading

Closes #2660

The get_environment_proxies function in _utils.py is responsible for parsing and mounting proxy info from system environment.

httpx/httpx/_utils.py

Lines 229 to 264 in 4b5a92e

def get_environment_proxies() -> typing.Dict[str, typing.Optional[str]]:
"""Gets proxy information from the environment"""
# urllib.request.getproxies() falls back on System
# Registry and Config for proxies on Windows and macOS.
# We don't want to propagate non-HTTP proxies into
# our configuration such as 'TRAVIS_APT_PROXY'.
proxy_info = getproxies()
mounts: typing.Dict[str, typing.Optional[str]] = {}
for scheme in ("http", "https", "all"):
if proxy_info.get(scheme):
hostname = proxy_info[scheme]
mounts[f"{scheme}://"] = (
hostname if "://" in hostname else f"http://{hostname}"
)
no_proxy_hosts = [host.strip() for host in proxy_info.get("no", "").split(",")]
for hostname in no_proxy_hosts:
# See https://curl.haxx.se/libcurl/c/CURLOPT_NOPROXY.html for details
# on how names in `NO_PROXY` are handled.
if hostname == "*":
# If NO_PROXY=* is used or if "*" occurs as any one of the comma
# separated hostnames, then we should just bypass any information
# from HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and always ignore
# proxies.
return {}
elif hostname:
# NO_PROXY=.google.com is marked as "all://*.google.com,
# which disables "www.google.com" but not "google.com"
# NO_PROXY=google.com is marked as "all://*google.com,
# which disables "www.google.com" and "google.com".
# (But not "wwwgoogle.com")
mounts[f"all://*{hostname}"] = None
return mounts

For env no_proxy, according to CURLOPT_NOPROXY explained, it should support domains, IPv4, IPv6 and the localhost. However, current get_environment_proxies function implementation only supports domains correctly as it always adds wildcard * in front of the hostname.

I encountered error when my environment no_proxy includes IPv6 address like ::1. It is wrongly transformed into all://*::1 and causes urlparse error since the _urlparse.py parses the :1 as port.

image

To fix this issue, I looked into this repo and suggest handling the no_proxy hostnames as domains, IPv4, IPv6 and the localhost seperately. I have updated the get_environment_proxies function in _utils.py and tested it.

Replies and discussions are welcomed!

@HearyShen HearyShen mentioned this pull request Apr 14, 2023
Closed
@plucury plucury mentioned this pull request Apr 19, 2023
Closed
1 task
@lovelydinosaur lovelydinosaur added the bug Something isn't working label Apr 19, 2023
@lovelydinosaur
Copy link
Contributor

lovelydinosaur commented Apr 19, 2023

Nicely done, yup.

@lovelydinosaur lovelydinosaur merged commit 15d09a3 into encode:master Apr 19, 2023
This was referenced Apr 20, 2023
Merged
Closed
@fnesveda fnesveda mentioned this pull request May 7, 2023
Closed
@epenet epenet mentioned this pull request May 17, 2023
Merged
@barnabas-kurdi barnabas-kurdi mentioned this pull request Jun 6, 2024
Open
samclearman pushed a commit to titanmsp/httpx that referenced this pull request Apr 26, 2025
@HearyShen @lovelydinosaur @samclearman
fix: NO_PROXY should support IPv4, IPv6 and localhost (encode#2659)
a57ee81 
* fix: NO_PROXY supports IPv4, IPv6 and localhost

* add more tests for test_get_environment_proxies

---------

Co-authored-by: Tom Christie <tom@tomchristie.com>
@Seas0
Copy link

Seas0 commented Nov 3, 2025

This PR introduced a parsing failure (httpx.InvalidURL: Invalid port) when the NO_PROXY environment contains one or more IPv6 Network CIDR notation entries, as in #3221, and have not been fixed for a while. Therefore I'd recommend remove any IPv6 entries in the NO_PROXY environment variable to workaround the bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lovelydinosaur lovelydinosaur lovelydinosaur approved these changes

Assignees

No one assigned

Labels

bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

The get_environment_proxies function in _utils.py does not support IPv4, IPv6 correctly

3 participants

@HearyShen @lovelydinosaur @Seas0