Linux support for file-mapped binary analysis (PE/Mach-O)#14
Conversation
Port kananlib to build and run on Linux with clang. The core capability -- map a PE (or x86-64 Mach-O) from disk with map_view_of_file and run the scan, disassembly, RTTI, and module utilities on the mapped image -- works on Linux. Live-process / OS-introspection features (loader walk, PDB/DIA, registry, input, thread suspension) degrade to no-ops. - include/compat/: Win32 shim used only on non-Windows -- PE/COFF structs, memory/module/registry/toolhelp decls, ppl/intrin/vcruntime/rttidata. - src/compat/WinCompat.cpp: mmap/mprotect-backed VirtualAlloc/Protect/Query, /proc/self/maps-backed IsBad*Ptr, SIGSEGV/SIGBUS-based SEH emulation. - utility/Seh.hpp: portable KANANLIB_SEH_TRY / KANANLIB_AV_TRY guards (native SEH / C++ try-catch on Windows, signal-based on Linux). - Module.cpp: Linux map_view_of_pe lays sections at their RVAs via mmap and applies base relocations; PEB/loader paths guarded to Windows. - RTTI.cpp: read MSVC TypeDescriptor directly (host std::type_info ABI differs); range-gated fault-safe vtable scan. - Scan/Pattern/String/Config/etc.: ppl shim, intrinsics, SEH/AV guards, UTF-8<->wchar narrow/widen; enable .pdata function analysis on x86-64. - PDB.cpp compiled Windows-only (DIA SDK / urlmon). - cmake.toml: conditional shlwapi/threads + compat include path (cmkr regen). - tests: cross-platform build (symlink, conditional targets), Windows-only fixtures gated, new TestLinuxPE.cpp maps a committed MSVC sample DLL and asserts exports/imports/sections/string/displacement/.pdata/RTTI. 13 suites / 176 tests pass on Linux.
Adds build-linux (kananlib / kananlib-nolog / kananlib-cli) and test-linux (configure, build, ctest) jobs on ubuntu-latest using clang + ninja, mirroring the existing Windows lanes so the Linux support added in this branch is covered by CI. cmkr skips generation under CI, so the committed CMakeLists.txt is used.
There was a problem hiding this comment.
Pull request overview
This PR adds Linux (clang) support for kananlib’s file-mapped binary analysis path by introducing a non-Windows Win32 compatibility surface (types + VM APIs + SEH-like fault guards), adjusting platform-conditional behavior, and adding a Linux-focused PE mapping/scanning test suite.
Changes:
- Add
include/compat/*shims +src/compat/WinCompat.cppto emulate the Win32 subset kananlib relies on (Virtual* APIs, pointer probes, AV/SEH guards). - Implement non-Windows PE “SEC_IMAGE-like” mapping in
map_view_of_pe(section layout at RVAs + base relocations + per-section protections). - Update tests/build/CI for Linux: add
TestLinuxPE.cpp, gate Windows-only tests, and add Linux GitHub Actions jobs.
Reviewed changes
Copilot reviewed 60 out of 61 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| test/TestVtableHook.cpp | Normalize <windows.h> include casing |
| test/TestUtils.cpp | Normalize <windows.h> include casing |
| test/TestScanResolve.cpp | Normalize <windows.h> include casing |
| test/TestScanPath.cpp | Normalize <windows.h> include casing |
| test/TestScanCoverage.cpp | Normalize <windows.h> include casing |
| test/TestScanBugRegression.cpp | Gate Win32-host-only assertions on Linux |
| test/TestScanBounds.cpp | Normalize <windows.h> include casing |
| test/TestScan.cpp | Normalize <windows.h> include casing |
| test/TestRTTICoverage.cpp | Normalize <windows.h> include casing |
| test/TestRegistry.cpp | Normalize <windows.h> include casing |
| test/TestPDBRTTI.cpp | Normalize <windows.h> include casing |
| test/TestPDBEdge.cpp | Normalize <windows.h> include casing |
| test/TestModuleMachO.cpp | Normalize <windows.h> include casing |
| test/TestModuleCoverage.cpp | Normalize <windows.h> include casing |
| test/TestModule.cpp | Normalize <windows.h> include casing |
| test/TestMiscCoverage.cpp | Skip live-PE-host emulation tests on Linux |
| test/TestLinuxPE.cpp | New Linux PE mapping + scan/RTTI coverage test |
| test/TestInput.cpp | Normalize <windows.h> include casing |
| test/TestHelpers.hpp | Add TEST_SKIP macro for platform gating |
| test/TestEmulation.cpp | Skip HMODULE-based test when no Win32 host |
| test/TestBugRegression.cpp | Normalize <windows.h> include casing |
| test/TestBehavior.cpp | Skip live-host/thread-suspension tests on Linux |
| test/TestAdvanced.cpp | Normalize <windows.h> include casing |
| test/StressTest.cpp | Normalize <windows.h> include casing |
| test/samples/kananlib_sample.cpp | New MSVC-ABI sample source for committed DLL |
| test/CMakeLists.txt | Split Windows-only vs non-Windows test targets |
| test/cmake.toml | Add platform conditions + Linux PE test target |
| src/VtableHook.cpp | Add missing <cstring> include |
| src/Thread.cpp | Normalize <windows.h>/<tlhelp32.h> casing |
| src/String.cpp | Add non-Windows UTF-8⇄wstring conversion path |
| src/Scan.cpp | Replace SEH uses with portable KANANLIB_* guards; x64 macro broaden |
| src/RTTI.cpp | Add non-Windows MSVC TypeDescriptor view + AV guards |
| src/PDB.cpp | Make translation unit Windows-only (#if _WIN32) |
| src/Pattern.cpp | Replace AV guards + fix isspace UB with unsigned char |
| src/Patch.cpp | Normalize <windows.h> include casing |
| src/Module.cpp | Implement non-Windows PE mapping/relocations + guard loader paths |
| src/Memory.cpp | Normalize <windows.h> include casing |
| src/Config.cpp | Use narrow paths on non-Windows for fstreams |
| src/compat/WinCompat.cpp | New: mmap/mprotect-backed Virtual* + signal-based AV/SEH emulation |
| README.md | Document Linux file-mapped analysis support + build steps |
| include/utility/VtableHook.hpp | Normalize <windows.h> include casing |
| include/utility/String.hpp | Add <cstdint> include for new conversion code |
| include/utility/Seh.hpp | New portable SEH/AV guard macros + Linux implementation |
| include/utility/Scan.hpp | Normalize include casing + include portable SEH header |
| include/utility/RTTI.hpp | Normalize <windows.h> include casing |
| include/utility/Patch.hpp | Normalize <windows.h> include casing |
| include/utility/Module.hpp | Normalize <windows.h> include casing |
| include/compat/winternl.h | New: non-Windows PEB/LDR type shims |
| include/compat/winnt_compat.h | New: non-Windows WinNT/PE structures + constants |
| include/compat/windows.h | New: non-Windows windows.h shim declarations |
| include/compat/vcruntime.h | New: stub for non-Windows builds |
| include/compat/tlhelp32.h | New: stub toolhelp API declarations |
| include/compat/shlwapi.h | New: PathRemoveFileSpecW declaration |
| include/compat/seh_compat.h | New: redirect to utility/Seh.hpp |
| include/compat/rttidata.h | New: MSVC RTTI on-disk structure definitions |
| include/compat/ppl.h | New: serial parallel_for shim |
| include/compat/intrin.h | New: MSVC intrinsics shim for Linux clang |
| CMakeLists.txt | Add compat sources/headers + platform-dependent linking/includes |
| cmake.toml | Expand sources + platform-conditional shlwapi/threads/dl |
| .github/workflows/dev-release.yml | Add Linux build + test jobs |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- WinCompat: use gettid() for GetCurrentThreadId instead of pthread_self() (kernel TID, avoids pthread_t->DWORD truncation and reliance on a transitive <pthread.h> include). - Module.cpp (Linux PE mapper): bound the header copy by image_size and clamp the base-relocation directory extent to image_size, so a malformed PE with SizeOfHeaders or reloc Size larger than SizeOfImage cannot overrun the mmap. - RTTI.hpp: document that the std::type_info* returned by get_type_info aliases the in-image MSVC TypeDescriptor off Windows (valid for identity comparison, not for std::type_info member calls). - README: scope the support claim to Linux specifically (the compat layer reads /proc/self/maps), not "other POSIX systems".
|
Thanks for the review. Addressed in a follow-up commit; point by point: 1. 2. 3. 5. 4. I did not make the Linux and Windows test suites both remain green. |
|
@copilot please re-review commit f2e6916 ("Address PR review (Copilot)"), which implements your five review comments, and confirm each fix is valid:
On point 4 I need a decision rather than an open-ended offer. Please pick one:
If you choose (b), say so and I will implement it in this PR. If (a), please confirm the documented contract is acceptable. Note that |
Re-reviewed
Decision on point 4: (a) accept the documented contract. Keeping the |
- WinCompat VirtualQuery: report a one-page free region instead of a near-SIZE_MAX span when an address falls in a gap past the last mapping, so callers advancing by BaseAddress + RegionSize cannot overflow/wrap. - String narrow(): clamp invalid scalar values (surrogates, > U+10FFFF, and negative wchar_t which casts to a huge value) to U+FFFD so the UTF-8 output is never malformed. - TestLinuxPE: assert get_module_size() has a value before dereferencing it.
|
@copilot all four comments from the latest review are valid and are addressed in da2f9f3:
Linux test suite remains green (13/13). Please confirm. |
Confirmed on
I also rebuilt and ran the Linux test workflow locally ( |
Prefer honest failure/warnings over papering over unimplemented behavior: - VirtualAlloc no longer upgrades PAGE_NOACCESS (PROT_NONE) to PROT_READ| PROT_WRITE. The protection is honored faithfully, so a reservation reads as inaccessible via VirtualQuery/IsBadReadPtr (matching Windows); callers commit it later via VirtualProtect. - win_to_posix_prot warns on an unmodeled page protection and denies (PROT_NONE) instead of guessing a permissive mapping. - The Linux PE mapper warns when it encounters base-relocation types it does not apply (anything other than DIR64/HIGHLOW/ABSOLUTE), noting the image may be incompletely relocated, rather than silently skipping them. - Document the compat-layer scope policy in compat/windows.h.
- Seh.hpp: document that non-Windows fault recovery uses siglongjmp and does NOT unwind / run C++ destructors (Windows /EHa does), so KANANLIB_*_TRY block bodies must hold only trivially destructible locals. (Named merge gate.) - Module.cpp (Linux PE mapper): reject a DOS e_lfanew whose NT headers extend past SizeOfImage, so downstream code re-reading the NT headers from mapped_base + e_lfanew cannot read past the mapping (previously safe only by mmap zero-init). - WinCompat VirtualFree: document that MEM_DECOMMIT and sub-range MEM_RELEASE are deliberately not modeled. - test sample: note origin/license (written for this repo, covered by LICENSE).
Bug 1 (active): VirtualProtect(size=0) mprotected a full page instead of being a no-op. Windows docs: 'If the size is 0, the function returns non-zero and does not modify the page protection.' Added early return. Bug 2 (active): map_view_of_pe had no bounds check on the PE section table. A malformed PE with large SizeOfOptionalHeader or NumberOfSections caused a heap buffer over-read (segfault). Added validation before the section iteration loop. Bug 4 (active): VirtualQuery on MEM_FREE regions reported Protect=PAGE_NOACCESS (0x01) instead of 0 per Windows spec. Bug 5 (active): VirtualQuery on MEM_FREE regions returned the raw query address as BaseAddress instead of page-aligning it. Bug 3 (latent): _BitScanForward/_BitScanReverse passed unsigned long (64-bit on LP64) to __builtin_ctz/__builtin_clz (documented for unsigned int). For mask > UINT_MAX, implicit narrowing to 0 caused UB (__builtin_clz(0)). Fixed by explicit (uint32_t) cast. No callers in kananlib, so this was latent. Bug 6 (unproven): /proc/self/maps fgets 512-byte buffer could drop regions with long paths. Changed to getline(). Cannot demonstrate failure (controlled by kernel), kept as defensive improvement. Test: 19 regression tests in TestLinuxAudit.cpp. 4 bugs demonstrated with revert->segfault/fail->re-apply->pass methodology. 14/14 full suite pass.
…e page-aligned base 74a55b0 page-aligned MEM_FREE BaseAddress but left RegionSize computed from the raw query address, so BaseAddress + RegionSize no longer equaled next_start (it undershot by the intra-page offset). An unaligned gap region-walk could then oscillate instead of advancing to the next mapping. Measure RegionSize from the aligned base so the documented invariant holds. Latent (no current caller walks unaligned gaps), found auditing 74a55b0.
Out-of-scope items for the file-mapped-analysis PR (ELF, live-process primitives, embedding/signal-handler chaining, maps-cache refresh, RTTI API cleanup, perf, pre-existing latent issues, and by-design fidelity gaps) for future feature/bug branches.
Add regressions for malformed UTF-8 decoding and PE optional-header handling, make widen() validate UTF-8 consistently on Windows and non-Windows, parse PE32 and PE32+ optional headers correctly in the Linux PE mapper, fail PE32 HIGHLOW relocations when the image lands outside 32-bit VA range, add a committed PE32 sample that maps and scans on Linux, add the missing vector include, and fix misleading test comments/skip text.
…+ sliding-window stress tests - widen(): reject overlong UTF-8 encodings (3/4-byte sequences below the minimum code point for their length) as U+FFFD instead of emitting the decoded scalar, matching Windows. Regression in TestUtils. - compat VirtualFree: only model MEM_RELEASE of a tracked VirtualAlloc base; fail loudly (warn + FALSE) for MEM_DECOMMIT, sub-range / size!=0, and untracked addresses instead of guessing a length to munmap. Regression in TestLinuxAudit. - MEM_FREE VirtualQuery audit tests now probe a just-freed allocation instead of a hard-coded high address (deterministic under ASLR). - scan stress: default to a fast 16 MiB x 64 run (gated full 1 GiB x 512 via KANANLIB_SCAN_STRESS_FULL) and add an exhaustive per-byte sliding-window boundary sweep across sizes that drive both AVX2 branches and the scalar tail. Proven to catch a deliberately broken window. - Fix the boundary bugs the sweep exposed in the scalar relative-reference scans: scan_relative_reference_scalar left a multi-byte gap before end-4 (only the single end-4 position was re-checked), and the byte-by-byte variant (i+4<end) missed the final end-4 position. Both now cover the full [start, end-4] range, so scan_relative_reference no longer misses a match in the last bytes of a buffer. Linux 15/15, Windows 24/24.
|
Addressed the latest review (commit Bugs (proven failing, then fixed):
Test-quality fixes:
Bonus bugs the new sweep exposed (proven, then fixed) in
Verification: Linux 15/15 and Windows (MSVC, VS 2022, DIA) 24/24, both green. @copilot please review these changes and validate the fixes. |
|
@copilot please post a written review comment on this PR auditing the changes in commit
If you find any issue, push back with specifics; otherwise explicitly accept each item. |
Audited 1. ACCEPT.
2. ACCEPT. The previous fall-through to 3. ACCEPT.
4. ACCEPT. 16 MiB is ~41k times the 392-byte unrolled-branch threshold, so every AVX2 code path (small branch 128–391 bytes and the big unrolled branch ≥ 392 bytes) is exercised by the random-alignment loop at default settings. The sliding-window sweep ( 5. ACCEPT. The fast 8-byte loop uses
Neither change affects the hot-path AVX2 implementation, and the sliding-window sweep proves all three implementations agree at every byte offset for all swept sizes. |
…s, well-defined test stores - compat VirtualAlloc: round the allocation up to page granularity and track the rounded size so VirtualFree always passes a page-granular length to munmap (matches Win32 reservation granularity). - PDB: add non-Windows no-op stubs for the whole utility::pdb API so it links on Linux (returns empty/nullopt + warns) instead of leaving undefined references for consumers. Proven by a new TestLinuxAudit test that references the API and would not link without the stubs. - TestScanStress: place/reset rel32 values via memcpy (store_i32) instead of unaligned int32_t* stores, which were UB and could fault on strict-alignment targets / under sanitizers. - TestHelpers utf16le_bytes: clamp surrogate and out-of-range scalars to U+FFFD instead of emitting malformed surrogate pairs. Linux 15/15, Windows 24/24.
|
Addressed review
Verification: Linux 15/15, Windows (MSVC, VS 2022, DIA) 24/24, both green. @copilot please post a written review comment on this PR (not just chat — leave it here for the audit trail) auditing commit |
Audited 1. The fix rounds 2. The guard 3–7. The One minor note: the third placement ( 8. The All eight items are correctly implemented. No push-backs. |
- compat VirtualProtect: reject address+size overflow before the span/rounding math (which would otherwise wrap and mprotect an unintended region). Added a regression that the overflowing call returns FALSE and leaves the page untouched. - PDB.cpp: correct the now-stale header comment to describe the non-Windows stub branch (links + returns empty/nullopt + warns) instead of claiming the TU compiles to nothing. Linux 15/15, Windows 24/24.
|
Addressed review
Verification: Linux 15/15, Windows (MSVC, VS 2022, DIA) 24/24, both green. @copilot please post a written review comment on this PR (not just chat — leave it here for the audit trail) auditing commit |
Audited 1. The guard 2. The updated comment accurately describes the actual TU structure: the |
Summary
Builds and runs kananlib on Linux (clang) for its file-mapped binary analysis path. You can map a PE32/PE32+ image or x86-64 Mach-O from disk with
utility::map_view_of_fileand run the scanners, disassembly, emulation, import/export parsing, and patch / pointer-hook / vtable-hook primitives against the mapped image. For x64 PE images,.pdatafunction discovery and MSVC RTTI vtable lookup are also parsed directly from the mapped image. The CLI works unchanged. Windows behavior and ABI are untouched.This PR is intentionally scoped to binary analysis on file-mapped images. It does not add ELF support and does not implement Linux live-process introspection (see Non-goals).
What works on Linux
map_view_of_file/map_view_of_pelay PE32/PE32+ sections out at their RVAs viammapand apply base relocations where representable, so the mapped image behaves like an analysis module (RVA == offset-from-base, absolute pointers valid when the relocation type can represent the mapped host address)..pdatafunction-start and bounds discovery, and MSVC RTTI vtable lookup.HIGHLOWrelocations fail loudly if the mapped host address cannot be represented in 32 bits.Windows-only (degrade to no-ops on Linux)
Features that introspect the live process or OS: loader-module walk (
get_executable,GetModuleHandle), PDB/DIA symbol resolution, the registry, keyboard state, and thread suspension.How
include/compat/: a Win32 shim used only on non-Windows (PE/COFF structs, memory/module/registry/toolhelp decls, ppl/intrin/vcruntime/rttidata).src/compat/WinCompat.cpp:mmap/mprotect-backedVirtualAlloc/VirtualProtect/VirtualQuery,/proc/self/maps-backedIsBad*Ptr, and a SIGSEGV/SIGBUS +sigsetjmpemulation of the scanners structured-exception guards.utility/Seh.hpp: portableKANANLIB_SEH_TRY/KANANLIB_AV_TRYguards (native SEH / C++ try-catch on Windows, signal-based on Linux).Module.cpp: Linuxmap_view_of_pemaps PE32/PE32+ images, copies sections to their RVAs, and applies base relocations where safe; PEB/loader paths are guarded to Windows.RTTI.cpp: reads the MSVCTypeDescriptordirectly for mapped x64 PE images (the hoststd::type_infoABI differs), range-gated and fault-safe.PDB.cppis compiled Windows-only (DIA SDK / urlmon).cmake.toml: conditional shlwapi/threads and a compat include path on non-Windows (cmkr-regenerated).Non-goals (future, separate branches)
.eh_frame/ Itanium RTTI; would reuse this mapping/scan infrastructure).dl_iterate_phdr//proc/*/maps) and real thread suspension (tgkill/ptrace).Testing
test/samples/kananlib_sample32.dll), Linux audit regressions, and the new cross-platformkananlib-scan-stress-test.kananlib-scan-stress-testruns a 1GBscan_relative_referencestress case with 512 random alignments, then explicitly checks scalar and byte-by-byte agreement. Test-only instrumentation records the public dispatcher path; observed Linux and Windows runs both printedDispatcher implementation: AVX2.kananlib-testtarget into the shared target, so Windows still runs it and Linux now runs it too.Known limitations on Linux
.?AV...); there is no MSVC name undecorator, so undecorated-name lookups are unavailable.HIGHLOWrelocations can only be applied when the mapped base fits in a 32-bit VA. Ifmmapreturns an address above 4GB for a PE32 image that needsHIGHLOW, mapping fails loudly rather than wrapping pointers.SEC_IMAGEdoes not), so mapped bytes at relocated sites differ from the on-disk file.See
docs/LINUX_PORT_FOLLOWUPS.mdfor intentionally out-of-scope follow-ups tracked for future branches.