Skip to content

feat: add AsyncFreeProxy with concurrent proxy checking - #63

Open
jundymek wants to merge 1 commit into
masterfrom
async-rewrite
Open

feat: add AsyncFreeProxy with concurrent proxy checking#63
jundymek wants to merge 1 commit into
masterfrom
async-rewrite

Conversation

@jundymek

Copy link
Copy Markdown
Owner

Supersedes #39 — thanks @NerdzzyDev for the idea and the initial work.

What

AsyncFreeProxy, an asyncio variant of FreeProxy built on aiohttp. It checks proxies concurrently and returns the first working one, cancelling the remaining checks. The synchronous API is completely untouched — existing users are unaffected, and a plain pip install free-proxy keeps exactly the same dependencies as today.

import asyncio
from fp.fp import AsyncFreeProxy

proxy = asyncio.run(AsyncFreeProxy().get())        # plain script
proxy = await AsyncFreeProxy().get()               # inside an async app / Jupyter

Design decisions

  • Separate class, same method names as coroutines — the httpx/redis convention. Sync vs async is visible in the structure of the code (class name), never in runtime configuration. The async_mode flag approach from Add async support #39 was rejected: it makes get() return a string or a coroutine depending on constructor state.
  • aiohttp is an optional extra: pip install "free-proxy[async]". Without it, AsyncFreeProxy() raises a clear error with the install hint. Sync users pay nothing.
  • as_completed + early exit instead of gather: we ask "which proxy succeeds first", not "what are all the results". After a winner is found, remaining tasks are cancelled and awaited to settle before the session closes.
  • Semaphore(max_concurrent), default 20, constructor parameter — bounds sockets/file descriptors and is polite to the test URL. The default is validated empirically (see benchmarks). max_concurrent must be a positive integer (0 would park every task on the semaphore forever).
  • Per-request timeout via aiohttp.ClientTimeout(total=timeout) — applied inside the semaphore, so queue waiting time never counts against a proxy.
  • Proxy verification parity: the async check verifies the response came over a socket connected to the proxy's IP via transport.get_extra_info('peername') — same guarantee as the sync path's sock.getpeername(), without the fragile string parsing from Add async support #39.
  • Python floor raised to 3.9 (3.8 is EOL; pip serves 1.2.x to 3.8 users automatically). Version bumped to 1.3.0 (additive, semver minor).

Measured (live run, 2026-08-24)

Proxy list: 100 entries (defaults / https), 200 (US). Single runs on volatile free-proxy lists — treat as orders of magnitude.

Case Sync Async (mc=20) Speedup
defaults 9.32 s 0.50 s 18.6x
https=True 2.72 s 0.43 s 6.3x
rand=True 5.01 s 0.39 s 12.8x
timeout=1.0 4.78 s 0.38 s 12.6x
country_id=['US'] 5.02 s 6.18 s 0.8x

max_concurrent sweep (defaults): 5 → 1.46 s, 20 → 0.49 s, 50 → 0.60 s, 100 → 1.45 s. The default of 20 sits in the empirical sweet spot; more concurrency is not faster.

The US case is the honest outlier: no US-listed proxy passed the strict check, so the async path scanned all 200 candidates and fell through to the no-country retry round, while the sync path got lucky early. Concurrency helps in proportion to how many dead proxies sit before the first working one; when none works, both paths pay for a full scan. Related observation for a future tweak: requests counts its timeout per phase (connect + read separately) while the async path uses a stricter total budget, so borderline-slow proxies can pass sync and fail async.

Testing

  • 37 tests, all green; the full sync suite passes unchanged (no behavioral regressions).
  • Async tests use stdlib IsolatedAsyncioTestCase/AsyncMock — no new dev dependencies.
  • Coverage includes: first-success-wins with slow losers proven cancelled by timing assertion, semaphore limit measured via a concurrency counter, the real check path against a mocked aiohttp session (peername match / mismatch / ClientError), the fallback retry round (repeat=True, country filter cleared), missing-aiohttp error, and max_concurrent validation.
  • CI (added in 1.2.3) runs the suite on Python 3.9–3.13; network-dependent tests are skipped on CI.

Docs

README: async usage section (script / async app / Jupyter), [async] install, max_concurrent documented. CHANGELOG updated for 1.3.0.

Add an asyncio variant of FreeProxy that checks proxies concurrently
and returns the first working one, cancelling the remaining checks.

- as_completed with early exit: first successful proxy wins
- Semaphore-based concurrency limit (max_concurrent, default 20)
- per-request timeout via aiohttp.ClientTimeout
- proxy verification through transport.get_extra_info('peername'),
  matching the peername check of the sync path
- aiohttp is an optional extra: pip install "free-proxy[async]"
- sync FreeProxy API is unchanged; shared list parsing and filtering
  extracted so both paths use one implementation
- raise minimum Python version to 3.9, bump version to 1.3.0

Based on the idea from #39.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant