feat: add AsyncFreeProxy with concurrent proxy checking - #63
Open
jundymek wants to merge 1 commit into
Open
Conversation
Add an asyncio variant of FreeProxy that checks proxies concurrently
and returns the first working one, cancelling the remaining checks.
- as_completed with early exit: first successful proxy wins
- Semaphore-based concurrency limit (max_concurrent, default 20)
- per-request timeout via aiohttp.ClientTimeout
- proxy verification through transport.get_extra_info('peername'),
matching the peername check of the sync path
- aiohttp is an optional extra: pip install "free-proxy[async]"
- sync FreeProxy API is unchanged; shared list parsing and filtering
extracted so both paths use one implementation
- raise minimum Python version to 3.9, bump version to 1.3.0
Based on the idea from #39.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #39 — thanks @NerdzzyDev for the idea and the initial work.
What
AsyncFreeProxy, an asyncio variant ofFreeProxybuilt on aiohttp. It checks proxies concurrently and returns the first working one, cancelling the remaining checks. The synchronous API is completely untouched — existing users are unaffected, and a plainpip install free-proxykeeps exactly the same dependencies as today.Design decisions
async_modeflag approach from Add async support #39 was rejected: it makesget()return a string or a coroutine depending on constructor state.pip install "free-proxy[async]". Without it,AsyncFreeProxy()raises a clear error with the install hint. Sync users pay nothing.as_completed+ early exit instead ofgather: we ask "which proxy succeeds first", not "what are all the results". After a winner is found, remaining tasks are cancelled and awaited to settle before the session closes.Semaphore(max_concurrent), default 20, constructor parameter — bounds sockets/file descriptors and is polite to the test URL. The default is validated empirically (see benchmarks).max_concurrentmust be a positive integer (0 would park every task on the semaphore forever).aiohttp.ClientTimeout(total=timeout)— applied inside the semaphore, so queue waiting time never counts against a proxy.transport.get_extra_info('peername')— same guarantee as the sync path'ssock.getpeername(), without the fragile string parsing from Add async support #39.Measured (live run, 2026-08-24)
Proxy list: 100 entries (defaults / https), 200 (US). Single runs on volatile free-proxy lists — treat as orders of magnitude.
max_concurrentsweep (defaults): 5 → 1.46 s, 20 → 0.49 s, 50 → 0.60 s, 100 → 1.45 s. The default of 20 sits in the empirical sweet spot; more concurrency is not faster.The US case is the honest outlier: no US-listed proxy passed the strict check, so the async path scanned all 200 candidates and fell through to the no-country retry round, while the sync path got lucky early. Concurrency helps in proportion to how many dead proxies sit before the first working one; when none works, both paths pay for a full scan. Related observation for a future tweak:
requestscounts itstimeoutper phase (connect + read separately) while the async path uses a strictertotalbudget, so borderline-slow proxies can pass sync and fail async.Testing
IsolatedAsyncioTestCase/AsyncMock— no new dev dependencies.max_concurrentvalidation.Docs
README: async usage section (script / async app / Jupyter),
[async]install,max_concurrentdocumented. CHANGELOG updated for 1.3.0.