Repository navigation
Conversation
Scholar indexes the published record (venue proceedings, journals) that the arXiv-only alphaxiv backend cannot return, so listing both widens prior-art coverage for the novelty audit. Mirrors the serpbase backend: keyed, silently skipped when the key is absent, key read from serply_api_key or SERPLY_API_KEY. Closes RUC-NLPIR#66
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / 概述
Adds
serplyas an optional keyed search backend: Serply's Google Scholar API,wired into
search.backendsexactly likeserpbase.The motivation is corpus coverage for the novelty audit.
alphaxivis the onlypapers backend today and it is arXiv-only, so prior art that exists solely in
venue proceedings or a journal is invisible to it. Scholar indexes that
published record, so
backends: [alphaxiv, serply]merges and de-duplicatesinto a wider prior-art view without changing anything about how the audit runs.
A live query confirms the gap is real rather than theoretical: the top hit for
"iterative hypothesis generation scientific discovery" is an ACL Anthology
paper with no arXiv mirror in the result set.
Linked issue / 关联 issue
Closes #66
Type of change / 改动类型
fix)feat)docs)refactor)test)chore)How was this tested? / 如何验证?
Baseline numbers below are from
upstream/mainin the same virtualenv, so thecomparison is apples to apples.
The unit tests mock
requests.get; no network is touched. Separately I ran thebackend against the real API through
resolve_backend_names/build_search_backendsto check the response contract:Two things that response settles, and both are pinned by a test:
articles, and the envelope also carries analways-empty
resultslist. Keying the wrong array returns zero hits with noerror, so the array name is passed explicitly rather than guessed.
descriptionis the Scholar byline ("Authors - Venue, Year"), not anabstract. It maps to
snippetsbecause that is the field the contract has,but the docs say plainly what it contains so nobody expects abstract text.
Size / 体量
7 files, +195/-22. Slightly above the merged
serpbasebackend (#65), whichwas 5 files, +133/-21, and the extra is accounted for:
docs/search.zh.mdanddocs/configuration.zh.md(+14/-5 combined), theChinese twins, because [Feature]: add Serply Google Scholar as an optional papers search backend #66 said the docs would be bilingual.
Two deliberate omissions, both easy to add if you would rather have them:
serpbaseto the.zhdocs. It is missing there, but it isnot my change to make in this PR.
README.md/README.zh-CN.md. Neither names any keyedbackend today; both document only the legacy
builtin_backend: none | alphaxivpath, soserplyhas nowhere natural to go. Same forexamples/research_config.example.yaml, which has nosearch.backendsblock.
serpbasewas left out of all three as well.Happy to split the
.zhdocs into a follow-up if you prefer this PR atprecedent size.
Notes
SERPLY_API_KEYis unset: a keyed backend with no key is dropped in
resolve_backend_names,so
backends: [alphaxiv, serply]degrades to[alphaxiv]silently, the sameas
serperandserpbase.SerpBaseBackendinsrc/core/tools/web/backends.pyline for line:same
_SyncBackendbase, same_HTTP_TIMEOUT, sameraise_for_status, same{url, title, snippets}output, same truncation tomax_results. It is aGET with an
X-Api-Keyheader where SerpBase is a POST.serply_api_keyonSearchConfig, falling back toSERPLY_API_KEYvia_WebSearchEnv.User-Agent: Arbor.tests/test_search_backends.py; network mocked.docs/search.md,docs/configuration.mdand their.zhtwins.Checklist / 检查清单
fix(config): ...). / PR 标题遵循 Conventional Commits 规范。python tests/test_*.py) and they pass. / 我已在本地运行相关测试并通过。README.zh-CN.md)。Size for why
serplyhas no place in them today.Disclosure: I work with Serply. Happy to adjust scope, naming, or drop this
entirely if it isn't a direction you want for the project.