diff --git a/.github/workflows/coverage.yml b/.github/workflows/coverage.yml index 19bc9a38..60f05d76 100644 --- a/.github/workflows/coverage.yml +++ b/.github/workflows/coverage.yml @@ -12,6 +12,6 @@ jobs: with: python_version: '3.11' coverage_source: 'ovos_utils' - test_path: 'test/' - install_extras: '' + test_path: 'test/unittests' + install_extras: '.[extras]' min_coverage: 0 diff --git a/CHANGELOG.md b/CHANGELOG.md index ed7c503e..d7466aa0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,54 +1,172 @@ # Changelog -## [0.8.5a4](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a4) (2026-03-11) +## [0.13.10a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.10a1) (2026-08-13) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.6a2...0.8.5a4) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.9a2...0.13.10a1) **Merged pull requests:** -- chore: Add comprehensive test suites and documentation [\#362](https://github.com/OpenVoiceOS/ovos-utils/pull/362) ([JarbasAl](https://github.com/JarbasAl)) -- fix: make the stopwatch test less strict [\#360](https://github.com/OpenVoiceOS/ovos-utils/pull/360) ([PureTryOut](https://github.com/PureTryOut)) +- fix: give FakeBus the intent-topic bridge \(RULE 1/RULE 2 parity with MessageBusClient\) [\#417](https://github.com/OpenVoiceOS/ovos-utils/pull/417) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.6a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.6a2) (2026-02-02) +## [0.13.9a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.9a2) (2026-08-01) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.6a1...0.8.6a2) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.9a1...0.13.9a2) **Merged pull requests:** -- chore\(deps\): update actions/setup-python action to v6 [\#353](https://github.com/OpenVoiceOS/ovos-utils/pull/353) ([renovate[bot]](https://github.com/apps/renovate)) -- chore\(deps\): update actions/checkout action to v6 [\#352](https://github.com/OpenVoiceOS/ovos-utils/pull/352) ([renovate[bot]](https://github.com/apps/renovate)) +- docs: rewrite README in Simplified Technical English [\#413](https://github.com/OpenVoiceOS/ovos-utils/pull/413) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.6a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.6a1) (2026-02-02) +## [0.13.9a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.9a1) (2026-07-24) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a3...0.8.6a1) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.8a1...0.13.9a1) **Merged pull requests:** -- fix: prevent handler errors from aborting FakeBus emit [\#358](https://github.com/OpenVoiceOS/ovos-utils/pull/358) ([JarbasAl](https://github.com/JarbasAl)) +- fix: watch files that do not exist yet [\#408](https://github.com/OpenVoiceOS/ovos-utils/pull/408) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a3](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a3) (2025-12-19) +## [0.13.8a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.8a1) (2026-07-24) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a2...0.8.5a3) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.7a2...0.13.8a1) **Merged pull requests:** -- chore\(deps\): update dependency python to 3.14 [\#348](https://github.com/OpenVoiceOS/ovos-utils/pull/348) ([renovate[bot]](https://github.com/apps/renovate)) +- fix: FileEventHandler must only fire for the file it watches [\#406](https://github.com/OpenVoiceOS/ovos-utils/pull/406) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a2) (2025-12-18) +## [0.13.7a2](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.7a2) (2026-07-24) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5a1...0.8.5a2) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.7a1...0.13.7a2) **Merged pull requests:** -- chore: Configure Renovate [\#347](https://github.com/OpenVoiceOS/ovos-utils/pull/347) ([renovate[bot]](https://github.com/apps/renovate)) +- refactor: deprecate create\_self\_signed\_cert [\#404](https://github.com/OpenVoiceOS/ovos-utils/pull/404) ([JarbasAl](https://github.com/JarbasAl)) -## [0.8.5a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.8.5a1) (2025-11-07) +## [0.13.7a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.7a1) (2026-07-23) -[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.4...0.8.5a1) +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.6a1...0.13.7a1) **Merged pull requests:** -- fix: use timezone-aware datetime functions and update scheduler event names [\#343](https://github.com/OpenVoiceOS/ovos-utils/pull/343) ([JarbasAl](https://github.com/JarbasAl)) +- fix: replace removed distutils.spawn with shutil.which \(Python 3.12+\) [\#402](https://github.com/OpenVoiceOS/ovos-utils/pull/402) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.6a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.6a1) (2026-07-23) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.5a1...0.13.6a1) + +**Merged pull requests:** + +- fix: ovos-logs CLI leaves stray file in cwd [\#400](https://github.com/OpenVoiceOS/ovos-utils/pull/400) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.5a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.5a1) (2026-07-16) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.4a1...0.13.5a1) + +**Merged pull requests:** + +- fix: log each unique deprecation warning only once [\#398](https://github.com/OpenVoiceOS/ovos-utils/pull/398) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.4a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.4a1) (2026-07-02) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.3a1...0.13.4a1) + +**Merged pull requests:** + +- fix: FakeBus folds the message session BEFORE handlers, not after [\#396](https://github.com/OpenVoiceOS/ovos-utils/pull/396) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.3a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.3a1) (2026-06-29) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.2a1...0.13.3a1) + +**Merged pull requests:** + +- fix: FakeBus folds the default session like any other \(drop owner-only\) [\#393](https://github.com/OpenVoiceOS/ovos-utils/pull/393) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.2a1) (2026-06-29) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.1a1...0.13.2a1) + +**Merged pull requests:** + +- fix: drop deprecated make\_default from FakeBus default-session sync [\#389](https://github.com/OpenVoiceOS/ovos-utils/pull/389) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.1a1) (2026-06-29) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.13.0a1...0.13.1a1) + +**Merged pull requests:** + +- fix: target a real shape-changing pair in namespace-migration tests [\#390](https://github.com/OpenVoiceOS/ovos-utils/pull/390) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.13.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.13.0a1) (2026-06-27) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.2a1...0.13.0a1) + +**Merged pull requests:** + +- feat: AsyncFakeBus namespace migration + env/config flag parity [\#387](https://github.com/OpenVoiceOS/ovos-utils/pull/387) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.12.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.2a1) (2026-06-27) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.1a1...0.12.2a1) + +**Merged pull requests:** + +- fix: translate mirrored payload onto counterpart topic in FakeBus [\#385](https://github.com/OpenVoiceOS/ovos-utils/pull/385) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.12.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.1a1) (2026-06-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.12.0a1...0.12.1a1) + +**Merged pull requests:** + +- fix: raise ovos-spec-tools floor to 0.10.0a1 for NamespaceTranslator [\#383](https://github.com/OpenVoiceOS/ovos-utils/pull/383) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.12.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.12.0a1) (2026-06-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.2a1...0.12.0a1) + +**Merged pull requests:** + +- feat: FakeBus mirrors the legacy\<-\>ovos.\* namespace migration [\#381](https://github.com/OpenVoiceOS/ovos-utils/pull/381) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.2a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.2a1) (2026-06-20) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.1a1...0.11.2a1) + +**Merged pull requests:** + +- fix: allow json-database 1.x [\#379](https://github.com/OpenVoiceOS/ovos-utils/pull/379) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.1a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.1a1) (2026-05-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.11.0a1...0.11.1a1) + +**Merged pull requests:** + +- fix: standardize\_lang\_tag macro=True preserves region \(restore langcodes semantics\) [\#377](https://github.com/OpenVoiceOS/ovos-utils/pull/377) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.11.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.11.0a1) (2026-05-25) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.10.0a1...0.11.0a1) + +**Merged pull requests:** + +- feat: fakebus Message subclasses ovos\_spec\_tools.Message — no API break [\#375](https://github.com/OpenVoiceOS/ovos-utils/pull/375) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.10.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.10.0a1) (2026-05-22) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.9.0a1...0.10.0a1) + +**Merged pull requests:** + +- feat: migrate ovos-utils onto ovos-spec-tools [\#373](https://github.com/OpenVoiceOS/ovos-utils/pull/373) ([JarbasAl](https://github.com/JarbasAl)) + +## [0.9.0a1](https://github.com/OpenVoiceOS/ovos-utils/tree/0.9.0a1) (2026-05-18) + +[Full Changelog](https://github.com/OpenVoiceOS/ovos-utils/compare/0.8.5...0.9.0a1) + +**Merged pull requests:** + +- feat: AsyncFakeBus alongside FakeBus [\#371](https://github.com/OpenVoiceOS/ovos-utils/pull/371) ([JarbasAl](https://github.com/JarbasAl)) diff --git a/README.md b/README.md index 7dd896e6..1bde44e3 100644 --- a/README.md +++ b/README.md @@ -1,87 +1,107 @@ # OVOS-utils -collection of simple utilities for use across the mycroft ecosystem +`ovos-utils` is a shared utility library for the OpenVoiceOS ecosystem. It provides +logging, process lifecycle management, a testing-friendly fake message bus, event +scheduling, file utilities, network checks, audio playback, and XDG path helpers. +Most OVOS packages, including `ovos-bus-client`, `ovos-config`, and `ovos-workshop`, +depend on it, so most projects get it as a transitive dependency. ## Install ```bash -pip install ovos_utils +pip install ovos-utils ``` -## Commandline scripts -### ovos-logs - Small helper tool to quickly navigate the logs, create slices and quickview errors +## Usage ---------------- -- **ovos-logs slice [options]** +The library exposes many small, independent modules. Import only what you need. +For example, use `FakeBus` to test skill code without a live message bus: - **Slice logs of a given time period. Defaults on the last service start (`-s`) until now (`-u`)** +```python +from ovos_utils.fakebus import FakeBus, FakeMessage - _Different logs can be picked using the `-l` option. All logs will be included if not specified._ - _Optionally the directory where the logs are stored (`-p`) and the file where the slices should be dumped (`-f`) can be specified._ - +bus = FakeBus() - _[ex: `ovos-logs slice`]_ - _Slice all logs from service start up until now._ - - _[ex: `ovos-logs slice -s 17:05:20 -u 17:05:25`]_ - _Slice all logs from 17:05:20 until 17:05:25._ - _**no logs in that timeframe in other present logs_ - Screenshot 2023-12-25 185004 - - _[ex: `ovos-logs slice -s 17:05:20 -u 17:05:25 -l skills`]_ - _Slice skills.log from 17:05:20 until 17:05:25._ - - _[ex: `ovos-logs slice -s 17:05:20 -u 17:05:25 -f ~/testslice.log`]_ - _Slice the logs from 17:05:20 until 17:05:25 on all log files and dump the slices in the file ~/testslice.log (default: `~/slice_.log`)._ - Screenshot 2023-12-25 190732 --------------- +def on_utterance(message): + print(message.data["utterances"]) -- **ovos-logs list [-e|-w|-d|-x] [options]** +bus.on("recognizer_loop:utterance", on_utterance) +bus.emit(FakeMessage("recognizer_loop:utterance", {"utterances": ["hello"]})) +``` + +See [docs/index.md](docs/index.md) for the full module overview, with links to +detailed pages on logging, process utilities, `FakeBus`, and event handling. + +## Command line: ovos-logs + +`ovos-logs` is a helper tool that slices, lists, and reduces OVOS service logs. + +- **`ovos-logs slice [options]`**. Slice logs for a time period. The default + period runs from the last service start (`-s`) until now (`-u`). Pick specific + logs with `-l` (default: all logs). Set the log directory with `-p` and the + output file with `-f`. + + ```bash + ovos-logs slice + # Slice all logs from the last service start until now. + + ovos-logs slice -s 17:05:20 -u 17:05:25 + # Slice all logs between 17:05:20 and 17:05:25. + + ovos-logs slice -s 17:05:20 -u 17:05:25 -l skills + # Slice only skills.log between 17:05:20 and 17:05:25. + + ovos-logs slice -s 17:05:20 -u 17:05:25 -f ~/testslice.log + # Slice logs between 17:05:20 and 17:05:25 into ~/testslice.log. + # Default output file: ~/slice_.log + ``` + +- **`ovos-logs list [-e|-w|-d|-x] [options]`**. List log lines by severity + (error, warning, debug, exception). Specify at least one level. You can combine + several. Set the time range with `-s` and `-u` (default: last service start + until now). Pick specific logs with `-l` (default: all logs). + + ```bash + ovos-logs list -x + # List EXCEPTION-level lines (with tracebacks) from the last service start until now. - **List logs by severity (error/warning/debug/exception). A log level has to be specified - more than one can be listed** + ovos-logs list -w -e -s 20-12-2023 -l bus -l skills + # List WARNING and ERROR lines from bus.log and skills.log since 20 December 2023. + ``` - _A start and end date can be specified using the `-s` and `-u` options. Defaults to the last service start until now._ - _Different logs can be picked using the `-l` option. All logs will be included if not specified._ - _Optionally, the directory where the logs are stored (`-p`) and the file where the slices should be dumped (`-f`) can be passed as arguments._ +- **`ovos-logs reduce [options]`**. Shrink logs to a target size in bytes, or + remove entries before a given date. Pick specific logs with `-l` (default: all + logs). Set the log directory with `-p`. - _[ex: `ovos-logs list -x`]_ - _List the logs with level EXCEPTION (plus tracebacks) from the last service start until now._ - Screenshot 2023-12-25 184321 - - _[ex: `ovos-logs list -w -e -s 20-12-2023 -l bus -l skills`]_ - _List the logs with level WARNING and ERROR from the 20th of December 2023 until now from the logs bus.log and skills.log._ - Screenshot 2023-12-25 173739 ---------------------- + ```bash + ovos-logs reduce + # Shrink all logs to 0 bytes. -- **ovos-logs reduce [options]** - - **Downsize logs to a given size (in bytes) or remove entries before a given date.** - - _Different logs can be included using the `-l` option. If not specified, all logs will be included._ - _Optionally the directory where the logs are stored (`-p`) can be specified._ - - _[ex: `ovos-logs reduce`]_ - _Downsize all logs to 0 bytes_ + ovos-logs reduce -s 1000000 + # Shrink all logs to about 1 MB, keeping the latest entries. - _[ex: `ovos-logs reduce -s 1000000`]_ - _Downsize all logs to ~1MB (latest logs)_ + ovos-logs reduce -d "1-12-2023 17:00" + # Shrink all logs to entries after the given date and time. - _[ex: `ovos-logs reduce -d "1-12-2023 17:00"`]_ - _Downsize all logs to entries after the specified date/time_ + ovos-logs reduce -s 1000000 -l skills -l bus + # Shrink skills.log and bus.log to about 1 MB each. + ``` - _[ex: `ovos-logs reduce -s 1000000 -l skills -l bus`]_ - _Downsize skills.log and bus.log to ~1MB (latest logs)_ +- **`ovos-logs show -l `**. Print the contents of a log file. ---------------------- + ```bash + ovos-logs show -l bus + # Print the contents of bus.log. + ``` -- **ovos-logs show -l [servicelog]** + The logs shown depend on which log files exist in the log folder. - **Show logs** +## Related projects - _[ex: `ovos-logs show -l bus`]_ - _Show the logs from bus.log._ +- [OpenVoiceOS/ovos-bus-client](https://github.com/OpenVoiceOS/ovos-bus-client). The real message bus client that `FakeBus` and `FakeMessage` stand in for during testing. +- [OpenVoiceOS/ovos-config](https://github.com/OpenVoiceOS/ovos-config). Reads and writes `mycroft.conf`, the configuration file used by `LOG`, `PIDLock`, and the network utilities. +- [OpenVoiceOS/ovos-workshop](https://github.com/OpenVoiceOS/ovos-workshop). The skill framework built on `EventContainer`, `RuntimeRequirements`, and the other utilities in this library. - _[ex: wrong servicelog]_ - _**logs shown depending on the logs present in the folder_ +## License +Apache License 2.0. See [LICENSE](LICENSE). diff --git a/docs/events.md b/docs/events.md index f15edbeb..da06b66a 100644 --- a/docs/events.md +++ b/docs/events.md @@ -85,3 +85,6 @@ from ovos_bus_client.apis.events import EventSchedulerInterface | `shutdown()` | Cancel repeating events and clear all registered handlers | `when` may be a `datetime`, or a positive `int`/`float` representing seconds from now. + +--- +[← FakeBus](fakebus.md) · [Home](index.md) · [Utilities →](utilities.md) diff --git a/docs/fakebus.md b/docs/fakebus.md index 49e863c9..a8deeb13 100644 --- a/docs/fakebus.md +++ b/docs/fakebus.md @@ -52,6 +52,65 @@ bus.emit(FakeMessage("recognizer_loop:utterance", {"utterances": ["hello"]})) | `close()` | Calls `on_close()` | | `create_client()` | Returns `self` | +For asyncio-native code, see [`AsyncFakeBus`](#asyncfakebus) below. + +--- + +## `AsyncFakeBus` + +`AsyncFakeBus` — `ovos_utils/fakebus.py:351` + +In-process stand-in for `AsyncMessageBusClient` (from `ovos-bus-client`). Use this when your code is asyncio-native and needs a drop-in fake bus without a WebSocket connection. The API surface mirrors the real async client: coroutine methods keep you inside the event loop, while handler registration stays synchronous to match `pyee` and the real client's contract. + +```python +import asyncio +from ovos_utils.fakebus import AsyncFakeBus, FakeMessage + +async def main(): + bus = AsyncFakeBus() + + received = [] + + def on_ping(message): + received.append(message) + + bus.on("test:ping", on_ping) + + await bus.emit(FakeMessage("test:ping", {"n": 1})) + print(received) # [FakeMessage("test:ping", ...)] + + await bus.close() + +asyncio.run(main()) +``` + +### Coroutine vs sync split + +| Sync (handler registration) | Async (I/O surface) | +|---|---| +| `on(msg_type, handler)` | `connect(*args, **kwargs)` | +| `once(msg_type, handler)` | `close()` | +| `remove(msg_type, handler)` | `emit(message)` | +| `remove_all_listeners(event_name)` | `wait_for_message(message_type, timeout)` | +| | `wait_for_response(message, reply_type, timeout)` | + +### Key Methods + +| Method | Description | Source | +|---|---|---| +| `connect()` | No-op; sets `connected_event` and `started_running = True` | `fakebus.py:409` | +| `close()` | Clears `connected_event`, calls `on_close()` | `fakebus.py:418` | +| `emit(message)` | Injects session, dispatches to `pyee` emitter | `fakebus.py:426` | +| `wait_for_message(message_type, timeout)` | Awaits a single message of that type | `fakebus.py:489` | +| `wait_for_response(message, reply_type, timeout)` | Emits a message and awaits the reply | `fakebus.py:513` | +| `create_client()` | Returns `self` (backwards-compat shim) | `fakebus.py:543` | +| `run_forever()` | Sets `started_running = True` (backwards-compat shim) | `fakebus.py:546` | +| `run_in_thread()` | Calls `run_forever()` (backwards-compat shim) | `fakebus.py:549` | + +### Session Handling + +Session injection side effects are identical to `FakeBus`: `emit()` populates `message.context["session"]` from `SessionManager`, and `on_message()` feeds incoming messages back through `Session.from_message()` / `SessionManager.update()`. Both imports are lazy so the class works without `ovos-bus-client` installed. + --- ## `FakeMessage` @@ -96,3 +155,6 @@ msg = FakeMessage("skill:action", {"key": "value"}, {"session_id": "abc"}) ## `dig_for_message()` Tries to import and call `ovos_bus_client.message.dig_for_message`. Returns `None` if `ovos-bus-client` is not installed. + +--- +[← Process Utilities](process-utils.md) · [Home](index.md) · [Events →](events.md) diff --git a/docs/index.md b/docs/index.md index 1df0d780..2358430f 100644 --- a/docs/index.md +++ b/docs/index.md @@ -11,7 +11,7 @@ Shared utility library used by all OVOS components. Provides logging, process li |---|---| | `ovos_utils.log` | `LOG` — OVOS-wide logging class with optional file rotation | | `ovos_utils.process_utils` | `ProcessStatus`, `RuntimeRequirements`, `PIDLock`, `MonotonicEvent` | -| `ovos_utils.fakebus` | `FakeBus`, `FakeMessage` — in-process bus for testing without a live WebSocket | +| `ovos_utils.fakebus` | `FakeBus`, `AsyncFakeBus`, `FakeMessage` — in-process bus for testing without a live WebSocket | | `ovos_utils.events` | `EventContainer`, `EventSchedulerInterface`, handler wrappers | | `ovos_utils.file_utils` | Resource resolution, vocab loading, `FileWatcher` | | `ovos_utils.network_utils` | `get_ip()`, `is_connected_dns()`, `is_connected_http()`, `check_captive_portal()` | @@ -60,6 +60,6 @@ pip install ovos-utils - [Logging](log.md) — `LOG`, `init_service_logger()`, `log_deprecation()`, `deprecated` decorator - [Process Utilities](process-utils.md) — `ProcessStatus`, `RuntimeRequirements`, `PIDLock`, `MonotonicEvent` -- [FakeBus](fakebus.md) — `FakeBus`, `FakeMessage` — in-process message bus for testing +- [FakeBus](fakebus.md) — `FakeBus`, `AsyncFakeBus`, `FakeMessage` — in-process message bus for testing - [Events](events.md) — `EventContainer`, `EventSchedulerInterface`, handler wrappers - [Utilities](utilities.md) — file, network, sound, threading, XDG helpers diff --git a/docs/log.md b/docs/log.md index 541b81e8..157a2c1b 100644 --- a/docs/log.md +++ b/docs/log.md @@ -141,3 +141,6 @@ Return all configured log directories across all services. ## `get_available_logs(directories) → List[str]` Return a list of log file basenames (e.g. `["audio", "skills", "bus"]`) found in the configured log directories. + +--- +[Home](index.md) · [Process Utilities →](process-utils.md) diff --git a/docs/process-utils.md b/docs/process-utils.md index 4dba1ef1..fdefeddd 100644 --- a/docs/process-utils.md +++ b/docs/process-utils.md @@ -136,3 +136,6 @@ Chainable POSIX signal handler. Each instance installs a user function as the ne ## `reset_sigint_handler()` Reset `SIGINT` to the default Python handler. Needed when starting OVOS services from shell scripts that have modified the signal mask. + +--- +[← Logging](log.md) · [Home](index.md) · [FakeBus →](fakebus.md) diff --git a/docs/utilities.md b/docs/utilities.md index e5d22cc9..d251dcda 100644 --- a/docs/utilities.md +++ b/docs/utilities.md @@ -201,3 +201,6 @@ cache_dir = xdg_cache_home() / "mycroft" # ~/.cache/mycroft | `xdg_data_dirs()` | `XDG_DATA_DIRS` | `[/usr/local/share, /usr/share]` | Environment variable values are only used if they are absolute paths; relative paths fall back to the default. + +--- +[← Events](events.md) · [Home](index.md) diff --git a/ovos_logs_console_script b/ovos_logs_console_script new file mode 100644 index 00000000..e69de29b diff --git a/ovos_utils/bracket_expansion.py b/ovos_utils/bracket_expansion.py index 06304e36..5170a0cd 100644 --- a/ovos_utils/bracket_expansion.py +++ b/ovos_utils/bracket_expansion.py @@ -2,43 +2,34 @@ import re from typing import List, Dict import warnings -from ovos_utils.log import deprecated +from ovos_spec_tools import expand as _spec_expand -def expand_template(template: str) -> List[str]: - def expand_optional(text): - """Replace [optional] with two options: one with and one without.""" - return re.sub(r"\[([^\[\]]+)\]", lambda m: f"({m.group(1)}|)", text) - - def expand_alternatives(text): - """Expand (alternative|choices) into a list of choices.""" - parts = [] - for segment in re.split(r"(\([^\(\)]+\))", text): - if segment.startswith("(") and segment.endswith(")"): - options = segment[1:-1].split("|") - parts.append(options) - else: - parts.append([segment]) - return itertools.product(*parts) - - def fully_expand(texts): - """Iteratively expand alternatives until all possibilities are covered.""" - result = set(texts) - while True: - expanded = set() - for text in result: - options = list(expand_alternatives(text)) - expanded.update(["".join(option).strip() for option in options]) - if expanded == result: # No new expansions found - break - result = expanded - return sorted(result) # Return a sorted list for consistency +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR - # Expand optional items first - template = expand_optional(template) - # Fully expand all combinations of alternatives - return fully_expand([template]) +@deprecated("import 'expand' from 'ovos_spec_tools' instead", + f"{VERSION_MAJOR + 1}.0.0") +def expand_template(template: str) -> List[str]: + """Expand a sentence template to its sample set. + + Resolves ``(a|b)`` alternatives and ``[optional]`` segments; named + ``{slot}`` placeholders are carried through unchanged. The samples are + returned sorted. + + .. deprecated:: + Import :func:`expand` from ``ovos_spec_tools`` directly — it is the + single conformant OVOS-INTENT-1 expander, and what this now delegates + to. A template the specification rejects as malformed (a single-branch + group, an empty sample, a slot-only template, …) raises + :class:`ovos_spec_tools.MalformedTemplate`. + """ + # stacklevel=3: warn() -> expand_template body -> @deprecated wrapper -> caller + warnings.warn("expand_template is deprecated; import 'expand' from " + "'ovos_spec_tools' instead", + DeprecationWarning, stacklevel=3) + return sorted(_spec_expand(template)) def expand_slots(template: str, slots: Dict[str, List[str]]) -> List[str]: @@ -53,7 +44,7 @@ def expand_slots(template: str, slots: Dict[str, List[str]]) -> List[str]: list[str]: A list of all expanded combinations. """ # Expand alternatives and optional components - base_expansions = expand_template(template) + base_expansions = sorted(_spec_expand(template)) # Process slots all_sentences = [] diff --git a/ovos_utils/device_input.py b/ovos_utils/device_input.py index 9c7a008a..f001a9a5 100644 --- a/ovos_utils/device_input.py +++ b/ovos_utils/device_input.py @@ -1,5 +1,5 @@ import subprocess -from distutils.spawn import find_executable +from shutil import which from ovos_utils.gui import is_gui_installed from ovos_utils.log import LOG @@ -9,7 +9,7 @@ class InputDeviceHelper: def __init__(self) -> None: self.libinput_devices_list = [] self.xinput_devices_list = [] - if not find_executable("libinput") and not find_executable("xinput"): + if not which("libinput") and not which("xinput"): LOG.warning("Could not find libinput, input device detection will be inaccurate") # ToDo: add support for discovering the input device based of a connected @@ -68,7 +68,7 @@ def _build_linput_devices_list(self): }) def _get_libinput_devices_list(self): - if find_executable("libinput"): + if which("libinput"): try: self._build_linput_devices_list() except Exception: @@ -98,7 +98,7 @@ def _build_xinput_devices_list(self): self.xinput_devices_list.append(dev) def _get_xinput_devices_list(self): - if find_executable("xinput"): + if which("xinput"): try: self._build_xinput_devices_list() except Exception: @@ -113,7 +113,7 @@ def get_input_device_list(self): return self.libinput_devices_list + self.xinput_devices_list def can_use_touch_mouse(self): - if not find_executable("libinput") and not find_executable("xinput"): + if not which("libinput") and not which("xinput"): # if gui installed assume we have a mouse # otherwise let's assume we are a server or something... return is_gui_installed() diff --git a/ovos_utils/dialog.py b/ovos_utils/dialog.py index d44636da..086003aa 100644 --- a/ovos_utils/dialog.py +++ b/ovos_utils/dialog.py @@ -1,20 +1,31 @@ import os import random import re +import warnings from os.path import join from pathlib import Path from typing import Optional -from ovos_utils.bracket_expansion import expand_template +from ovos_spec_tools import expand + from ovos_utils.file_utils import resolve_resource_file from ovos_utils.lang import translate_word -from ovos_utils.log import LOG, log_deprecation +from ovos_utils.log import LOG, deprecated, log_deprecation +from ovos_utils.version import VERSION_MAJOR class MustacheDialogRenderer: """A dialog template renderer based on the mustache templating language.""" + @deprecated("use the OVOS-INTENT-2 §4.2 dialog renderer in " + "'ovos_spec_tools' ('render' / 'DialogRenderer')", + f"{VERSION_MAJOR + 1}.0.0") def __init__(self): + warnings.warn( + "MustacheDialogRenderer is deprecated; use the OVOS-INTENT-2 §4.2 " + "dialog renderer in 'ovos_spec_tools' ('render' / " + "'DialogRenderer')", + DeprecationWarning, stacklevel=3) self.templates = {} self.recent_phrases = [] @@ -92,7 +103,7 @@ def render(self, template_name, context=None, index=None): line = template_functions[index % len(template_functions)] # Replace {key} in line with matching values from context line = line.format(**context) - line = random.choice(expand_template(line)) + line = random.choice(sorted(expand(line))) # Here's where we keep track of what we've said recently. Remember, # this is by line in the .dialog file, not by exact phrase @@ -104,6 +115,8 @@ def render(self, template_name, context=None, index=None): return line +@deprecated("use 'ovos_spec_tools.LocaleResources' to load .dialog resources", + f"{VERSION_MAJOR + 1}.0.0") def load_dialogs(dialog_dir: str, renderer: Optional[MustacheDialogRenderer] = None) -> \ MustacheDialogRenderer: @@ -116,6 +129,10 @@ def load_dialogs(dialog_dir: str, Returns: a loaded instance of a dialog renderer """ + warnings.warn( + "load_dialogs is deprecated; use 'ovos_spec_tools.LocaleResources' " + "to load .dialog resources", + DeprecationWarning, stacklevel=3) if renderer is None: renderer = MustacheDialogRenderer() @@ -132,6 +149,8 @@ def load_dialogs(dialog_dir: str, return renderer +@deprecated("use the OVOS-INTENT-2 §4.2 dialog renderer in 'ovos_spec_tools' " + "('render' / 'DialogRenderer')", f"{VERSION_MAJOR + 1}.0.0") def get_dialog(phrase: str, lang: str = None, context: Optional[dict] = None) -> str: """ @@ -149,6 +168,10 @@ def get_dialog(phrase: str, lang: str = None, str: a randomized and/or translated version of the phrase """ + warnings.warn( + "get_dialog is deprecated; use the OVOS-INTENT-2 §4.2 dialog renderer " + "in 'ovos_spec_tools' ('render' / 'DialogRenderer')", + DeprecationWarning, stacklevel=3) if not lang: log_deprecation("Expected a string lang and got None.", "0.1.0") try: diff --git a/ovos_utils/fakebus.py b/ovos_utils/fakebus.py index c0287274..5a9a3acc 100644 --- a/ovos_utils/fakebus.py +++ b/ovos_utils/fakebus.py @@ -1,11 +1,39 @@ -import json +import asyncio +import warnings from copy import deepcopy +from os import environ from threading import Event -import warnings + from ovos_utils.log import LOG, log_deprecation +from ovos_spec_tools import NamespaceTranslator +from ovos_spec_tools.intent_topics import (canonical_intent_topic, + intent_topic_counterpart, + is_intent_topic, + legacy_intent_topic) from pyee import EventEmitter +#: Context flag stamped on a twin intent frame, mirroring +#: ``ovos_bus_client.client.client.INTENT_COMPAT_TWIN_KEY``. Its presence +#: means the canonical spelling of this dispatch was already delivered +#: alongside it, so a receiver that understands the bridge must not +#: modernize the twin a second time (that would fire the handler twice). +INTENT_COMPAT_TWIN_KEY = "_intent_compat_twin" + + +def _verbatim_copy(message, topic: str): + """Retopic ``message`` onto ``topic``, carrying its context byte-for-byte. + + Mirrors ``ovos_bus_client.client.client._verbatim_copy``. NOT + ``Message.forward`` -- ``forward()`` re-stamps the session, which for + the default session would replace the carried session with this + process's own and desync ``lang`` / ``active_skills`` between the + canonical frame and its twin. + """ + return message.__class__(topic, data=deepcopy(message.data), + context=deepcopy(message.context)) + + def dig_for_message(): try: from ovos_bus_client.message import dig_for_message as _dig @@ -15,12 +43,60 @@ def dig_for_message(): return None +# sentinel: lets us tell "kwarg not passed" apart from "kwarg passed True/False" +_UNSET = object() + + +def _bus_flag(env_var, config_key, default=True): + """Resolve a boolean bus flag the way ``MessageBusClient._bus_flag`` does. + + Precedence: env var (when set) > ``websocket.`` in ovos_config + > ``default``. The env var wins when set to a truthy/falsy string; ovos_config + is optional, so any failure to read it falls back to ``default``. + + Kept layering-clean: mirrors ``ovos_bus_client.client.client._bus_flag`` + without importing from bus-client (bus-client depends on utils, not vice-versa). + """ + val = environ.get(env_var) + if val is not None: + return val.strip().lower() in ("1", "true", "yes", "on") + try: + from ovos_config import Configuration + return bool(Configuration().get("websocket", {}).get(config_key, default)) + except Exception: + return default + + +def _resolve_bus_flags(kwargs): + """Build the namespace ``NamespaceTranslator`` for a fake bus instance. + + An explicitly-passed ``modernize``/``emit_legacy`` kwarg wins (back-compat for + callers passing ``emit_legacy=True/False``); otherwise the flag is resolved via + env var -> ``websocket.*`` config -> default ``True``, matching the real client. + """ + modernize = kwargs.get("modernize", _UNSET) + if modernize is _UNSET: + modernize = _bus_flag("OVOS_BUS_MODERNIZE", "modernize", default=True) + emit_legacy = kwargs.get("emit_legacy", _UNSET) + if emit_legacy is _UNSET: + emit_legacy = _bus_flag("OVOS_BUS_EMIT_LEGACY", "emit_legacy", default=True) + return NamespaceTranslator(modernize=modernize, emit_legacy=emit_legacy) + + class FakeBus: def __init__(self, *args, **kwargs): self.started_running = False self.session_id = "default" self.ee = kwargs.get("emitter") or EventEmitter() self.ee.on("error", self.on_error) + # mirror MessageBusClient's namespace migration so the test/satellite + # double bridges legacy<->ovos.* topics identically. Flags resolve the + # same way the real client does: explicit modernize=/emit_legacy= kwarg + # wins, else env var -> websocket.* config -> default on. + self._translator = _resolve_bus_flags(kwargs) + self._handler_guards = {} # handler -> shared mirror-guard + self._intent_pair_guards = {} # frozenset({topic, counterpart}) -> shared mirror-guard + self._dedup_registrations = {} # handler -> [(msg_type, wrapped), ...] self.on_open() try: self.session_id = kwargs["session"].session_id @@ -31,12 +107,85 @@ def __init__(self, *args, **kwargs): self.on_default_session_update) def on(self, msg_type, handler): + # wrap handlers on migrated/bridged topics so a handler subscribed to + # both spellings of a mirrored dispatch fires once (the mirror is + # dropped). See _mirror_guard_for for the guard-scope rationale. + guard = self._mirror_guard_for(msg_type, handler) + if guard is not None: + def wrapped(message=None): + if guard(message): + return + return handler(message) + + self.ee.on(msg_type, wrapped) + self._dedup_registrations.setdefault(handler, []).append((msg_type, wrapped)) + return self.ee.on(msg_type, handler) + def _mirror_guard_for(self, msg_type, handler): + """The mirror guard a registration on ``msg_type`` must wrap with. + + Mirrors ``ovos_bus_client.client.client.MessageBusClient._mirror_guard_for``. + Two bridges deliver one logical event twice, and each needs a + different guard SCOPE: + + - **namespace migration** (legacy <-> ``ovos.*``): the guard is per + HANDLER, shared across that handler's registrations, so its legacy + ``on()`` and its ``ovos.*`` ``on()`` dedupe against each other. + - **intent-topic compat** (canonical <-> ``.intent``-suffixed): the + guard is per TOPIC PAIR, shared by every registration on either + spelling. + + The intent guard cannot be keyed by handler: ``ovos-workshop`` + 9.3.2a1+ binds the same skill method to both spellings through a + FRESH wrapper closure per binding, so the two registrations are two + distinct ``handler`` objects and a per-handler guard would hand each + its own private state -- the canonical frame runs one closure, the + twin runs the other, and the skill handler fires twice for a single + dispatch. Keying on the pair collapses them. + """ + counterpart = intent_topic_counterpart(msg_type) + if counterpart is not None: + pair_key = frozenset({msg_type, counterpart}) + guard = self._intent_pair_guards.get(pair_key) + if guard is None: + guard = self._translator.new_mirror_guard() + self._intent_pair_guards[pair_key] = guard + return guard + if self._translator.is_migrated(msg_type): + guard = self._handler_guards.get(handler) + if guard is None: + guard = self._translator.new_mirror_guard() + self._handler_guards[handler] = guard + return guard + return None + + def _release_intent_pair_guard(self, msg_type): + """Drop the pair guard once nothing is registered on either spelling. + + Mirrors ``MessageBusClient._release_intent_pair_guard``. + """ + counterpart = intent_topic_counterpart(msg_type) + if counterpart is None: + return + pair_key = frozenset({msg_type, counterpart}) + for regs in self._dedup_registrations.values(): + if any(ev in pair_key for ev, _ in regs): + return + self._intent_pair_guards.pop(pair_key, None) + def once(self, msg_type, handler): self.ee.once(msg_type, handler) def emit(self, message): + # RULE 2 dedup marker: read it, then POP it before any local + # dispatch. A handler invoked below may call message.forward()/ + # reply() to emit a descendant frame on an UNRELATED topic; those + # deep-copy the whole context, so leaving the marker in place would + # brand that unrelated frame a twin and silently suppress its + # modernization. Mirrors MessageBusClient.on_message's pop-before- + # dispatch ordering. + is_intent_twin = message.context.pop(INTENT_COMPAT_TWIN_KEY, False) if "session" not in message.context: try: # replicate side effects from ovos_bus_client.session import Session, SessionManager @@ -45,12 +194,112 @@ def emit(self, message): message.context["session"] = sess.serialize() except ImportError: # don't care message.context["session"] = {"session_id": self.session_id} + # Fold the incoming message's session onto the SessionManager singleton + # BEFORE running handlers, matching the real MessageBusClient.on_message + # order (receive -> fold -> dispatch to handlers). Folding AFTER handlers + # would wipe any in-place / synced session mutation the handlers made + # (handle_session_sync merging intent_context, handle_add_context injecting + # context frames, ...), because the message's session snapshot was + # stamped at emit-time, before those mutations landed — a spec-violating + # self-broadcast-back wipe. The single-process harness has no separate + # receiver, so it must not re-fold its own emit once the handlers have + # run. + self.on_message(message.serialize()) self.ee.emit("message", message.serialize()) try: self.ee.emit(message.msg_type, message) except Exception as e: LOG.exception(f"Error in event handler for '{message.msg_type}': {e}") - self.on_message(message.serialize()) + # namespace migration: also dispatch the counterpart topic(s) so a + # listener on either namespace receives the event (consumers dedupe). + # the mirrored payload is reshaped into the counterpart topic's shape + # (identity for payload-compatible renames, a per-topic transform for + # shape-changing ones) so a listener on it receives the payload in *its* + # shape -- matching MessageBusClient's bridge. + for topic in self._translator.counterpart_topics(message.msg_type): + try: + translated = self._translator.translate_payload( + from_topic=message.msg_type, to_topic=topic, + data=message.data) + self.ee.emit(topic, message.forward(topic, translated)) + except Exception as e: + LOG.exception(f"Error in counterpart dispatch for '{topic}': {e}") + self._bridge_intent_topic(message, is_twin=is_intent_twin) + + def _bridge_intent_topic(self, message, is_twin=False): + """Legacy <-> canonical intent-topic bridge (RULE 1 + RULE 2). + + Mirrors ``ovos_bus_client.client.client.MessageBusClient``'s + ``_send_legacy_intent_twin`` (RULE 1, send-side) and + ``_modernize_intent_topic`` (RULE 2, receive-side). The real client + splits these across the wire (twin goes out on ``emit()``, the + modernized copy is dispatched locally in ``on_message()``); a + ``FakeBus`` has no separate wire hop, so both rules run inline, + against local listeners only, off of the one message being emitted. + + ``is_twin`` carries the marker decision made in ``emit()``, which + pops :data:`INTENT_COMPAT_TWIN_KEY` off the context BEFORE any + dispatch so it cannot leak onto descendant frames a handler derives + from this one (``message.forward()``/``reply()`` deep-copy context). + The marker is therefore never read from ``message.context`` here -- + only the popped value is trusted. + + RULE 2 first, mirroring the real client's receive order, then + RULE 1 -- so a listener on the canonical topic sees the canonical + dispatch before the legacy twin goes out. + + Note on the capture firehose (deliberate FakeBus-wide convention, + shared with the namespace-migration bridge above): the twin/ + modernized copy is dispatched straight to ``self.ee`` on its own + topic and does NOT re-emit ``"message"``. On the real wire, the + twin is a second frame and therefore fires ``on_message`` (and its + ``"message"`` firehose) a second time in every receiving process; + ``FakeBus`` has no wire hop to put it on, so it keeps the + single-process harness's one-emit-one-capture invariant instead of + reproducing the wire's two-frame shape. + """ + # RULE 2 (receive-side modernize): a suffixed frame WITHOUT the twin + # marker came from an emitter old enough to only put the legacy + # spelling on the bus, so nothing canonical was sent alongside it -- + # a canonical-only listener would never hear it without this. + if self._translator.modernize and not is_twin \ + and is_intent_topic(message.msg_type): + canonical = canonical_intent_topic(message.msg_type) + if canonical != message.msg_type: + try: + self.ee.emit(canonical, _verbatim_copy(message, canonical)) + except Exception as e: + LOG.exception(f"Error in intent modernize dispatch for " + f"'{canonical}': {e}") + + # RULE 1 (send-side twin): every canonical intent dispatch is + # twinned onto its legacy spelling so a listener that only knows + # the old suffixed topic still hears it. An already-suffixed + # dispatch is never twinned (legacy_intent_topic is a no-op on it), + # so the mirror cannot cascade. + # + # On the real wire, this twin goes out MARKED (INTENT_COMPAT_TWIN_KEY), + # so an out-of-process receiver's own RULE 2 knows to skip + # re-modernizing it. FakeBus has no wire hop: RULE 2 above already + # made that call inline for THIS dispatch, so the marker's job is + # already done, and the twin delivered to local listeners here must + # NOT carry it. Carrying it forward would break two things: (a) the + # per-topic-pair mirror guard on ``on()`` fingerprints payload+context, + # so a marked twin would fingerprint differently from the canonical + # dispatch it mirrors and the guard would fail to recognize it as a + # duplicate, double-firing a dual-bound handler; and (b) a handler + # that forwards this frame's context to emit an unrelated topic would + # brand that unrelated frame a twin too (see the marker-leak + # regression test), silently suppressing its own modernization. + if self._translator.emit_legacy and is_intent_topic(message.msg_type): + topic = legacy_intent_topic(message.msg_type) + if topic != message.msg_type: + twin = _verbatim_copy(message, topic) + try: + self.ee.emit(topic, twin) + except Exception as e: + LOG.exception(f"Error in intent twin dispatch for " + f"'{topic}': {e}") def on_message(self, *args): """ @@ -66,9 +315,10 @@ def on_message(self, *args): try: # replicate side effects from ovos_bus_client.session import Session, SessionManager sess = Session.from_message(parsed_message) - if sess.session_id != "default": - # 'default' can only be updated by core - SessionManager.update(sess) + # every session — including the default id — folds onto the singleton + # (value-passing; nothing is owner-only, matching the spec-tools + # SessionManager and the real MessageBusClient) + SessionManager.update(sess) except ImportError: pass # don't care @@ -77,7 +327,10 @@ def on_default_session_update(self, message): from ovos_bus_client.session import Session, SessionManager new_session = message.data["session_data"] sess = Session.deserialize(new_session) - SessionManager.update(sess, make_default=True) + # payload is default_session.serialize() (id == "default"); the + # SessionManager singleton syncs default_session by id, so the + # deprecated make_default flag is not needed. + SessionManager.update(sess) LOG.debug("synced default_session") except ImportError: pass # don't care @@ -135,6 +388,19 @@ def rcv(m): return msg def remove(self, msg_type, handler): + regs = self._dedup_registrations.get(handler) + if regs: + for ev, wrapped in [r for r in regs if r[0] == msg_type]: + try: + self.ee.remove_listener(ev, wrapped) + except Exception: + pass + regs.remove((ev, wrapped)) + if not regs: + self._dedup_registrations.pop(handler, None) + self._handler_guards.pop(handler, None) + self._release_intent_pair_guard(msg_type) + return try: self.ee.remove_listener(msg_type, handler) except Exception: @@ -165,182 +431,352 @@ def close(self): self.on_close() -class _MutableMessage(type): - """ To override isinstance checks we need to use a metaclass """ +# The reference Message envelope lives in ovos-spec-tools (OVOS-MSG-1). +# ovos-utils re-exports it under the historical ``FakeMessage`` name and +# attaches the one legacy convenience method downstream still uses — +# ``publish`` — to the class at import time. ``as_dict`` is now on the +# spec-tools class itself; the ``data['destination']`` promotion the +# old ``reply`` did was always a bug (data is the payload, context owns +# routing) and is gone. +# +# The old ``_MutableMessage`` metaclass / dynamic ``__new__`` indirection +# (which tried to return an ``ovos_bus_client.Message`` at runtime if +# bus-client was installed) is no longer needed: spec-tools is a hard +# dependency, the canonical class is always present, and +# ``ovos-bus-client.Message`` is the **same** class (bus-client attaches +# ``publish`` to it too — both attachments are idempotent). +from typing import Any, Dict, Optional + +from ovos_spec_tools.message import Message as FakeMessage +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR + + +# OVOS-MSG-1 defines forward / reply / response as the three normative +# derivations (§5). ``publish`` is a bus-client tradition outside the +# spec; it survives as an attached method for one more major release so +# downstream consumers can migrate. +_PUBLISH_REMOVAL_VERSION = f"{VERSION_MAJOR + 1}.0.0" + + +@deprecated( + "Message.publish is deprecated; use Message.forward (relay under a " + "new topic, preserves context) or Message.reply (§5.2 swap) — both " + "are OVOS-MSG-1 normative", + _PUBLISH_REMOVAL_VERSION) +def _publish(self, msg_type: str, data: Dict[str, Any], + context: Optional[Dict[str, Any]] = None) -> FakeMessage: + """Relay under a new topic without the §5.2 swap; drop ``target``. + + .. deprecated:: + Not part of OVOS-MSG-1 (the spec defines ``forward`` / + ``reply`` / ``response`` as the only normative derivations). + Slated for removal in the next major; use :meth:`forward` + when you do not want the routing-key swap, or :meth:`reply` + when you do. + """ + import warnings + # stacklevel=3: warn() -> body -> @deprecated wrapper -> caller + warnings.warn( + "Message.publish is deprecated; use Message.forward (no §5.2 " + "swap) or Message.reply (with swap) instead — both are " + "OVOS-MSG-1 normative derivations. ``publish`` will be removed " + f"in ovos-utils {_PUBLISH_REMOVAL_VERSION}.", + DeprecationWarning, stacklevel=3) + context = context or {} + new_context = dict(self.context) + new_context.update(context) + new_context.pop("target", None) + return self.__class__(msg_type, data, new_context) + + +# Attach publish() to the spec-tools Message so the method appears on +# every Message instance regardless of which package the caller imported +# the class from. Idempotent with ovos-bus-client's identical attachment. +FakeMessage.publish = _publish - def __instancecheck__(self, instance): - try: - from ovos_bus_client.message import Message as _MycroftMessage - if isinstance(instance, _MycroftMessage): - return True - except ImportError: - pass - return super().__instancecheck__(instance) +class Message(FakeMessage): + """Deprecated alias for the OVOS-MSG-1 ``Message`` envelope. -# fake Message object to allow usage without ovos-bus-client installed -class FakeMessage(metaclass=_MutableMessage): - """ fake Message object to allow usage with FakeBus without ovos-bus-client installed""" + ``from ovos_utils.fakebus import Message`` is in the wild and stays + importable through one more release. New code should import the + envelope where it lives — :class:`ovos_spec_tools.Message` (or + :class:`ovos_bus_client.Message`, which is a subclass). + """ def __new__(cls, *args, **kwargs): - try: # most common case - from ovos_bus_client import Message as _M - return _M(*args, **kwargs) - except ImportError: - pass - return super().__new__(cls) + warnings.warn( + "ovos_utils.fakebus.Message is deprecated; import " + "ovos_spec_tools.Message (or ovos_bus_client.Message)", + DeprecationWarning, + stacklevel=2, + ) + log_deprecation( + "please import Message from ovos_spec_tools / " + "ovos_bus_client directly", "1.0.0") + return FakeMessage(*args, **kwargs) - def __init__(self, msg_type, data=None, context=None): - """Used to construct a message object - Message objects will be used to send information back and forth - between processes of mycroft service, voice, skill and cli - """ - self.msg_type = msg_type - self.data = data or {} - self.context = context or {} +class AsyncFakeBus: + """In-process stand-in for ``AsyncMessageBusClient``. + + Mirrors the same surface as the real async bus client: ``connect`` / + ``close`` / ``emit`` / ``wait_for_message`` / ``wait_for_response`` are + coroutines; ``on`` / ``once`` / ``remove`` stay synchronous. + + No WebSocket, no thread, no real I/O — every emit dispatches + synchronously through a ``pyee.EventEmitter`` to whatever handlers + are registered. + + Useful both in tests (drop-in for ``AsyncMessageBusClient``) and at + runtime (anywhere a sync component expects the legacy ``FakeBus`` but + the surrounding code is asyncio-native). - def __eq__(self, other): + The session-injection side effects match ``FakeBus`` so multi-turn + flows behave identically. + """ + + def __init__(self, *args, **kwargs): + self.started_running = False + self.session_id = "default" + self.ee = kwargs.get("emitter") or EventEmitter() + self.ee.on("error", self.on_error) + # mirror MessageBusClient's namespace migration (see FakeBus.__init__). + self._translator = _resolve_bus_flags(kwargs) + self._handler_guards = {} # handler -> shared mirror-guard + self._intent_pair_guards = {} # frozenset({topic, counterpart}) -> shared mirror-guard + self._dedup_registrations = {} # handler -> [(msg_type, wrapped), ...] + self.connected_event = asyncio.Event() + self.connected_event.set() + self.on_open() try: - return other.msg_type == self.msg_type and \ - other.data == self.data and \ - other.context == self.context + self.session_id = kwargs["session"].session_id except Exception: - return False + pass # don't care - def serialize(self): - """This returns a string of the message info. + self.on("ovos.session.update_default", + self.on_default_session_update) - This makes it easy to send over a websocket. This uses - json dumps to generate the string with type, data and context + # ------------------------------------------------------------------ + # Handler registration (sync — matches AsyncMessageBusClient) + # ------------------------------------------------------------------ - Returns: - str: a json string representation of the message. - """ - return json.dumps({'type': self.msg_type, - 'data': self.data, - 'context': self.context}) + def on(self, msg_type, handler): + # wrap handlers on migrated/bridged topics so a handler subscribed to + # both spellings of a mirrored dispatch fires once (the mirror is + # dropped) -- same as FakeBus.on / MessageBusClient.on. + guard = self._mirror_guard_for(msg_type, handler) + if guard is not None: + def wrapped(message=None): + if guard(message): + return + return handler(message) + + self.ee.on(msg_type, wrapped) + self._dedup_registrations.setdefault(handler, []).append((msg_type, wrapped)) + return + self.ee.on(msg_type, handler) - @staticmethod - def deserialize(value): - """This takes a string and constructs a message object. + # shared with FakeBus -- same guard-scope rules (see FakeBus._mirror_guard_for) + _mirror_guard_for = FakeBus._mirror_guard_for + _release_intent_pair_guard = FakeBus._release_intent_pair_guard - This makes it easy to take strings from the websocket and create - a message object. This uses json loads to get the info and generate - the message object. + def once(self, msg_type, handler): + self.ee.once(msg_type, handler) - Args: - value(str): This is the json string received from the websocket + def remove(self, msg_type, handler): + regs = self._dedup_registrations.get(handler) + if regs: + for ev, wrapped in [r for r in regs if r[0] == msg_type]: + try: + self.ee.remove_listener(ev, wrapped) + except Exception: + pass + regs.remove((ev, wrapped)) + if not regs: + self._dedup_registrations.pop(handler, None) + self._handler_guards.pop(handler, None) + self._release_intent_pair_guard(msg_type) + return + try: + self.ee.remove_listener(msg_type, handler) + except Exception: + pass - Returns: - FakeMessage: message object constructed from the json string passed - int the function. - value(str): This is the string received from the websocket + def remove_all_listeners(self, event_name): + self.ee.remove_all_listeners(event_name) + + # ------------------------------------------------------------------ + # Lifecycle (async) + # ------------------------------------------------------------------ + + async def connect(self, *args, **kwargs): + """No-op for the fake bus; matches the real client's lifecycle. + + Returns immediately with ``connected_event`` set. """ - obj = json.loads(value) - return FakeMessage(obj.get('type') or '', - obj.get('data') or {}, - obj.get('context') or {}) + self.started_running = True + self.connected_event.set() + return self - def forward(self, msg_type, data=None): - """ Keep context and forward message + async def close(self): + self.connected_event.clear() + self.on_close() - This will take the same parameters as a message object but use - the current message object as a reference. It will copy the context - from the existing message object. + # ------------------------------------------------------------------ + # emit (async) — same dispatch shape as FakeBus.emit + # ------------------------------------------------------------------ - Args: - msg_type (str): type of message - data (dict): data for message + async def emit(self, message): + # RULE 2 dedup marker: pop before any dispatch -- see FakeBus.emit + # for the rationale (a handler-derived forward()/reply() must not + # inherit this frame's twin marker). + is_intent_twin = message.context.pop(INTENT_COMPAT_TWIN_KEY, False) + if "session" not in message.context: + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + sess = SessionManager.sessions.get(self.session_id) or \ + Session(self.session_id) + message.context["session"] = sess.serialize() + except ImportError: # don't care + message.context["session"] = {"session_id": self.session_id} + # Fold BEFORE handlers — see FakeBus.emit for the rationale (a post- + # handler self-broadcast-back fold wipes the handlers' in-place / synced + # session mutations with the stale emit-time snapshot). + self.on_message(message.serialize()) + self.ee.emit("message", message.serialize()) + try: + self.ee.emit(message.msg_type, message) + except Exception as e: + LOG.exception(f"Error in event handler for '{message.msg_type}': {e}") + # namespace migration: also dispatch the counterpart topic(s) with the + # payload reshaped into each counterpart's shape -- same as FakeBus.emit. + for topic in self._translator.counterpart_topics(message.msg_type): + try: + translated = self._translator.translate_payload( + from_topic=message.msg_type, to_topic=topic, + data=message.data) + self.ee.emit(topic, message.forward(topic, translated)) + except Exception as e: + LOG.exception(f"Error in counterpart dispatch for '{topic}': {e}") + self._bridge_intent_topic(message, is_twin=is_intent_twin) + + # ------------------------------------------------------------------ + # Sync helpers used internally — same as FakeBus + # ------------------------------------------------------------------ + + _bridge_intent_topic = FakeBus._bridge_intent_topic - Returns: - FakeMessage: Message object to be used on the reply to the message + def on_message(self, *args): + """Handle an incoming websocket message. + + @param args: + message (str): serialized Message """ - data = data or {} - return FakeMessage(msg_type, data, context=self.context) - - def reply(self, msg_type, data=None, context=None): - """Construct a reply message for a given message - - This will take the same parameters as a message object but use - the current message object as a reference. It will copy the context - from the existing message object and add any context passed in to - the function. Check for a destination passed in to the function from - the data object and add that to the context as a destination. If the - context has a source then that will be swapped with the destination - in the context. The new message will then have data passed in plus the - new context generated. - - Args: - msg_type (str): type of message - data (dict): data for message - context: intended context for new message + if len(args) == 1: + message = args[0] + else: + message = args[1] + parsed_message = FakeMessage.deserialize(message) + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + sess = Session.from_message(parsed_message) + # every session — including the default id — folds onto the singleton + # (value-passing; nothing is owner-only, matching the spec-tools + # SessionManager and the real MessageBusClient) + SessionManager.update(sess) + except ImportError: + pass # don't care + + def on_default_session_update(self, message): + try: # replicate side effects + from ovos_bus_client.session import Session, SessionManager + new_session = message.data["session_data"] + sess = Session.deserialize(new_session) + # payload is default_session.serialize() (id == "default"); the + # SessionManager singleton syncs default_session by id, so the + # deprecated make_default flag is not needed. + SessionManager.update(sess) + LOG.debug("synced default_session") + except ImportError: + pass # don't care + + def on_error(self, error): + LOG.error(error) + + def on_open(self): + pass + + def on_close(self): + pass + + # ------------------------------------------------------------------ + # Waiters (async) + # ------------------------------------------------------------------ + + async def wait_for_message(self, message_type, timeout=3.0): + """Wait for a message of a specific type. + + Arguments: + message_type (str): the message type of the expected message + timeout: seconds to wait before timeout, defaults to 3 Returns: - FakeMessage: Message object to be used on the reply to the message - """ - data = deepcopy(data) or {} - context = context or {} - - new_context = deepcopy(self.context) - for key in context: - new_context[key] = context[key] - if 'destination' in data: - new_context['destination'] = data['destination'] - if 'source' in new_context and 'destination' in new_context: - s = new_context['destination'] - new_context['destination'] = new_context['source'] - new_context['source'] = s - return FakeMessage(msg_type, data, context=new_context) - - def response(self, data=None, context=None): - """Construct a response message for the message - - Constructs a reply with the data and appends the expected - ".response" to the message - - Args: - data (dict): message data - context (dict): message context - Returns - (Message) message with the type modified to match default response + The received message or None if the response timed out """ - return self.reply(self.msg_type + '.response', data, context) + evt = asyncio.Event() + captured = {"msg": None} - def publish(self, msg_type, data, context=None): - """ - Copy the original context and add passed in context. Delete - any target in the new context. Return a new message object with - passed in data and new context. Type remains unchanged. + def _rcv(m): + captured["msg"] = m + evt.set() - Args: - msg_type (str): type of message - data (dict): date to send with message - context: context added to existing context + self.ee.once(message_type, _rcv) + try: + await asyncio.wait_for(evt.wait(), timeout=timeout) + except asyncio.TimeoutError: + pass + return captured["msg"] + + async def wait_for_response(self, message, reply_type=None, timeout=3.0): + """Send a message and wait for a response. + + Arguments: + message (Message): message to send + reply_type (str): the message type of the expected reply. + Defaults to ".response". + timeout: seconds to wait before timeout, defaults to 3 Returns: - FakeMessage: Message object to publish + The received message or None if the response timed out """ - context = context or {} - new_context = self.context.copy() - for key in context: - new_context[key] = context[key] + reply_type = reply_type or message.msg_type + ".response" + evt = asyncio.Event() + captured = {"msg": None} - if 'target' in new_context: - del new_context['target'] + def _rcv(m): + captured["msg"] = m + evt.set() - return FakeMessage(msg_type, data, context=new_context) + self.ee.once(reply_type, _rcv) + await self.emit(message) + try: + await asyncio.wait_for(evt.wait(), timeout=timeout) + except asyncio.TimeoutError: + pass + return captured["msg"] + # ------------------------------------------------------------------ + # Backwards-compat passthroughs so AsyncFakeBus is a drop-in even for + # code paths that still call the threading-era helpers. + # ------------------------------------------------------------------ -class Message(FakeMessage): - """just for compat, stuff in the wild importing from here even with deprecation warnings...""" + def create_client(self): + return self - def __new__(cls, *args, **kwargs): - warnings.warn( - "import from ovos-bus-client directly", - DeprecationWarning, - stacklevel=2, - ) - log_deprecation( - "please import from ovos-bus-client directly! this import has been deprecated since version 0.1.0", "1.0.0") - return FakeMessage(*args, **kwargs) + def run_forever(self): + self.started_running = True + + def run_in_thread(self): + self.run_forever() diff --git a/ovos_utils/file_utils.py b/ovos_utils/file_utils.py index 55bb8fa8..aabc3083 100644 --- a/ovos_utils/file_utils.py +++ b/ovos_utils/file_utils.py @@ -9,7 +9,7 @@ from threading import RLock from typing import Optional, List -from ovos_utils.bracket_expansion import expand_template +from ovos_spec_tools import expand from ovos_utils.log import LOG, log_deprecation @@ -238,7 +238,7 @@ def read_vocab_file(path: str) -> List[List[str]]: for line in voc_file.readlines(): if line.startswith('#') or line.strip() == '': continue - vocab.append(expand_template(line.lower())) + vocab.append(sorted(expand(line.lower()))) return vocab @@ -374,11 +374,22 @@ def __init__(self, files: List[str], callback: callable, self.observer = Observer() self.handlers = [] for file_path in files: - if os.path.isfile(file_path): - watch_dir = dirname(file_path) - else: + if os.path.isdir(file_path): watch_dir = file_path - self.observer.schedule(FileEventHandler(file_path, callback, + # a directory was requested, fire for any file inside it + watched_file = None + else: + # a specific file was requested, only fire for that file. + # the file may not exist yet (eg. a config that hasn't been + # written), watchdog just needs its (existing) parent dir + watch_dir = dirname(file_path) or "." + watched_file = file_path + if not os.path.isdir(watch_dir): + LOG.warning(f"Can't watch '{file_path}', " + f"parent directory '{watch_dir}' " + f"does not exist") + continue + self.observer.schedule(FileEventHandler(watched_file, callback, ignore_creation), watch_dir, recursive=recursive) self.observer.start() @@ -398,17 +409,20 @@ def shutdown(self): from watchdog.events import FileSystemEventHandler class FileEventHandler(FileSystemEventHandler): - def __init__(self, file_path: str, callback: callable, + def __init__(self, file_path: Optional[str], callback: callable, ignore_creation: bool = False): """ Create a handler for file change events - @param file_path: file_path being watched Unused(?) + @param file_path: if set, the single file being watched; events + for any other file in the watched directory are ignored. + If None, this handler is watching a directory and events + for any file inside it are reported. @param callback: function to call on file change with modified file path @param ignore_creation: if True, only track file modification events """ super().__init__() self._callback = callback - self._file_path = file_path + self._file_path = os.path.realpath(file_path) if file_path else None if ignore_creation: self._events = ('modified') else: @@ -416,9 +430,23 @@ def __init__(self, file_path: str, callback: callable, self._changed_files = set() self._lock = RLock() + def _is_watched(self, src_path) -> bool: + """ + Check if a reported event path refers to the file this handler + was asked to watch. Always True in directory-watch mode. + @param src_path: `event.src_path`, str or bytes depending on platform + """ + if self._file_path is None: + return True + if isinstance(src_path, bytes): + src_path = src_path.decode() + return os.path.realpath(src_path) == self._file_path + def on_any_event(self, event): if event.is_directory: return + if not self._is_watched(event.src_path): + return with self._lock: if event.event_type == "closed": if event.src_path in self._changed_files: diff --git a/ovos_utils/geolocation.py b/ovos_utils/geolocation.py index b9b20494..a586a9f6 100644 --- a/ovos_utils/geolocation.py +++ b/ovos_utils/geolocation.py @@ -3,8 +3,9 @@ import requests from requests.exceptions import RequestException, Timeout +from ovos_spec_tools import standardize_lang + from ovos_utils import timed_lru_cache -from ovos_utils.lang import standardize_lang_tag from ovos_utils.log import LOG from ovos_utils.network_utils import get_external_ip, is_valid_ip @@ -234,7 +235,7 @@ def get_ip_geolocation(ip: Optional[str] = None, raise ValueError(f"Invalid IP address: {ip}") # normalize language to expected values by ip-api.com - lang = standardize_lang_tag(lang).split("-")[0] + lang = standardize_lang(lang).split("-")[0] if lang not in ["en", "de", "es", "pt", "fr", "ja", "zh", "ru"]: LOG.warning(f"Language unsupported by ip-api.com ({lang}), defaulting to english") lang = "en" diff --git a/ovos_utils/lang/__init__.py b/ovos_utils/lang/__init__.py index 25c63471..3a5fd316 100644 --- a/ovos_utils/lang/__init__.py +++ b/ovos_utils/lang/__init__.py @@ -1,47 +1,71 @@ +import warnings from os import listdir from os.path import isdir, join from typing import Optional from ovos_utils.file_utils import resolve_resource_file +from ovos_utils.log import deprecated +from ovos_utils.version import VERSION_MAJOR +@deprecated("use 'standardize_lang' from 'ovos_spec_tools' instead", + f"{VERSION_MAJOR + 1}.0.0") def standardize_lang_tag(lang_code: str, macro=True) -> str: - """https://langcodes-hickford.readthedocs.io/en/sphinx/index.html""" + """Normalize a BCP-47 language tag. + + ``macro`` controls **macrolanguage substitution** per + :func:`langcodes.standardize_tag` — it swaps a sublanguage for its + macrolanguage (``cmn`` -> ``zh``, ``nb`` -> ``no``). It does **not** + strip the region: ``"en-US"`` round-trips through both ``macro=True`` + and ``macro=False`` unchanged. + + .. deprecated:: + Use :func:`ovos_spec_tools.standardize_lang` — the conformant OVOS + language-tag normalizer. ``standardize_lang`` always returns the + region-preserving form (it does not take a ``macro`` argument); + if you need macrolanguage substitution, call + :func:`langcodes.standardize_tag` directly. + """ + # stacklevel=3: warn() -> this body -> @deprecated wrapper -> caller + warnings.warn("standardize_lang_tag is deprecated; use 'standardize_lang' " + "from 'ovos_spec_tools' instead", + DeprecationWarning, stacklevel=3) try: - from langcodes import standardize_tag as std - return str(std(lang_code, macro=macro)) - except Exception: - if macro: - return lang_code.split("-")[0].lower() - if "-" in lang_code: - a, b = lang_code.split("-", 1) - return f"{a.lower()}-{b.upper()}" - return lang_code.lower() + from langcodes import standardize_tag + return str(standardize_tag(lang_code, macro=macro)) + except ImportError: + # langcodes is optional. Without it, fall back to the spec-tools + # normalizer (region-preserving). The ``macro`` argument is a + # no-op in this branch because macrolanguage tables live in + # langcodes itself. + from ovos_spec_tools import standardize_lang + return standardize_lang(lang_code) -def get_language_dir(base_path: str, lang: str ="en-US") -> Optional[str]: - """ checks for all language variations and returns best path """ - lang = standardize_lang_tag(lang) +@deprecated("use 'closest_lang' from 'ovos_spec_tools' " + "(or 'ovos_spec_tools.LocaleResources')", + f"{VERSION_MAJOR + 1}.0.0") +def get_language_dir(base_path: str, lang: str = "en-US") -> Optional[str]: + """Return the best-matching ``/`` directory under ``base_path``. - candidates = [] - for f in listdir(base_path): - if isdir(f"{base_path}/{f}"): - try: - from langcodes import tag_distance - score = tag_distance(lang, f) - except Exception: # not a valid language code - continue - # https://langcodes-hickford.readthedocs.io/en/sphinx/index.html#distance-values - # 0 -> These codes represent the same language, possibly after filling in values and normalizing. - # 1- 3 -> These codes indicate a minor regional difference. - # 4 - 10 -> These codes indicate a significant but unproblematic regional difference. - if score < 10: - candidates.append((f"{base_path}/{f}", score)) - if not candidates: + .. deprecated:: + Use :func:`ovos_spec_tools.closest_lang` to resolve a language tag + against the available ones, or :class:`ovos_spec_tools.LocaleResources` + which resolves locale directories itself. + """ + # stacklevel=3: warn() -> this body -> @deprecated wrapper -> caller + warnings.warn("get_language_dir is deprecated; use 'closest_lang' from " + "'ovos_spec_tools' (or 'ovos_spec_tools.LocaleResources')", + DeprecationWarning, stacklevel=3) + from ovos_spec_tools import closest_lang + try: + names = [f for f in listdir(base_path) if isdir(join(base_path, f))] + except (FileNotFoundError, NotADirectoryError): return None - # sort by distance to target lang code - candidates = sorted(candidates, key=lambda k: k[1]) - return candidates[0][0] + # closest_lang accepts a tag distance below 10 — the same threshold this + # used previously (OVOS-INTENT-2 §2.2). + match = closest_lang(lang, names) + return join(base_path, match) if match is not None else None def translate_word(name, lang='en-US'): diff --git a/ovos_utils/log.py b/ovos_utils/log.py index f5941cf6..df389929 100644 --- a/ovos_utils/log.py +++ b/ovos_utils/log.py @@ -258,6 +258,11 @@ def get_logs_config(service_name: Optional[str] = None, return _logs_conf +# Tracks (message, caller) pairs already logged so each unique deprecation +# is only reported once instead of spamming the logs on every call +_logged_deprecations = set() + + def log_deprecation(log_message: str = "DEPRECATED", deprecation_version: str = "Unknown", func_name: str = None, @@ -297,6 +302,11 @@ def log_deprecation(log_message: str = "DEPRECATED", if not name.startswith(origin_module): call_info = f"{name}:{call.lineno}" break + # Only log each unique deprecation (message + caller) once + dedupe_key = (log_message, log_name, call_info) + if dedupe_key in _logged_deprecations: + return + _logged_deprecations.add(dedupe_key) # Explicitly format log to print origin log reference LOG.create_logger(log_name).warning( f"Deprecation version={deprecation_version}. Caller={call_info}. " diff --git a/ovos_utils/log_parser.py b/ovos_utils/log_parser.py index 562ba356..4aa3cb8f 100644 --- a/ovos_utils/log_parser.py +++ b/ovos_utils/log_parser.py @@ -11,7 +11,7 @@ from rich.style import Style from rich.table import Table import pydoc -from combo_lock import ComboLock +from combo_lock import NamedLock try: from ovos_config import Configuration @@ -25,7 +25,7 @@ TIME_FORMAT = '%Y-%m-%d %H:%M:%S.%f' -LOGLOCK = ComboLock("ovos_logs_console_script") +LOGLOCK = NamedLock("ovos_logs_console_script") @dataclass diff --git a/ovos_utils/security.py b/ovos_utils/security.py index b9b74312..459ae0f6 100644 --- a/ovos_utils/security.py +++ b/ovos_utils/security.py @@ -8,7 +8,8 @@ import pexpect -from ovos_utils.log import LOG +from ovos_utils.log import LOG, deprecated +from ovos_utils.version import VERSION_MAJOR try: # pycryptodomex @@ -25,10 +26,21 @@ crypto = None +@deprecated("create_self_signed_cert is unmaintained and generates a " + "1024-bit RSA / SHA-1 certificate that modern OpenSSL " + "(SECLEVEL=2, the Debian/Ubuntu/Fedora default) refuses to load " + "('EE_KEY_TOO_SMALL'). Callers should bundle their own " + "self-signed cert generation (RSA >= 2048, SHA-256, with a " + "Subject Alternative Name) instead of relying on this helper.", + f"{VERSION_MAJOR + 1}.0.0") def create_self_signed_cert(cert_dir, name="jarbas"): """ If name.crt and name.key don't exist in cert_dir, create a new self-signed cert and key pair and write them into that directory. + + .. deprecated:: unmaintained; generates a 1024-bit RSA / SHA-1 + certificate that modern OpenSSL rejects at load time. Bundle your + own self-signed cert generation instead. """ if crypto is None: LOG.error("run pip install pyopenssl") diff --git a/ovos_utils/sound.py b/ovos_utils/sound.py index bf87b648..bfdc640a 100644 --- a/ovos_utils/sound.py +++ b/ovos_utils/sound.py @@ -5,7 +5,7 @@ from os.path import isfile from typing import Optional -from distutils.spawn import find_executable +from shutil import which from ovos_utils.log import LOG @@ -39,28 +39,28 @@ def _find_player(uri): _, ext = os.path.splitext(uri) # scan installed executables that can handle playback - sox_play = find_executable("play") + sox_play = which("play") # sox should handle almost every format, but fails in some urls if sox_play: return sox_play + f" --type {ext} %1" # determine best available player - ogg123_play = find_executable("ogg123") + ogg123_play = which("ogg123") if "ogg" in ext and ogg123_play: return ogg123_play + " -q %1" - pw_play = find_executable("pw-play") + pw_play = which("pw-play") # pw_play handles both wav and mp3 if pw_play: return pw_play + " %1" # wav file if 'wav' in ext: - pulse_play = find_executable("paplay") + pulse_play = which("paplay") if pulse_play: return pulse_play + " %1" - alsa_play = find_executable("aplay") + alsa_play = which("aplay") if alsa_play: return alsa_play + " %1" # guess mp3 - mpg123_play = find_executable("mpg123") + mpg123_play = which("mpg123") if mpg123_play: return mpg123_play + " %1" LOG.error("Can't find player for: %s", uri) @@ -135,14 +135,14 @@ def get_sound_duration(path: str, base_dir: Optional[str] = "") -> float: frames = f.getnframes() rate = f.getframerate() return frames / float(rate) - ffprobe = find_executable("ffprobe") + ffprobe = which("ffprobe") if ffprobe: args = (ffprobe, "-show_entries", "format=duration", "-i", path) popen = subprocess.Popen(args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) popen.wait() output = popen.stdout.read().decode("utf-8") return float(output.split("duration=")[-1].split("\n")[0]) - media_info = find_executable("mediainfo") + media_info = which("mediainfo") if media_info: args = (media_info, path) popen = subprocess.Popen(args, stdout=subprocess.PIPE) diff --git a/ovos_utils/version.py b/ovos_utils/version.py index 3a22553d..684ed0bf 100644 --- a/ovos_utils/version.py +++ b/ovos_utils/version.py @@ -1,8 +1,8 @@ # START_VERSION_BLOCK VERSION_MAJOR = 0 -VERSION_MINOR = 8 -VERSION_BUILD = 5 -VERSION_ALPHA = 4 +VERSION_MINOR = 13 +VERSION_BUILD = 10 +VERSION_ALPHA = 1 # END_VERSION_BLOCK __version__ = f"{VERSION_MAJOR}.{VERSION_MINOR}.{VERSION_BUILD}" + (f"a{VERSION_ALPHA}" if VERSION_ALPHA else "") diff --git a/pyproject.toml b/pyproject.toml index d4534048..597e101a 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -13,7 +13,7 @@ requires-python = ">=3.9" dependencies = [ "pexpect~=4.9", "requests~=2.26", - "json_database~=0.10", + "json_database>=0.10,<2.0.0", "kthread~=0.2", "watchdog", "pyee>=8.0.0", @@ -21,6 +21,7 @@ dependencies = [ "rich-click~=1.7", "rich~=13.7", "python-dateutil", + "ovos-spec-tools>=0.16.1a2", ] [project.urls] @@ -33,7 +34,7 @@ extras = [ "ovos-plugin-manager>=0.0.25", "ovos-config>=0.0.12", "ovos-workshop>=0.0.13", - "ovos_bus_client>=0.0.8", + "ovos_bus_client>=2.6.2a2", "langcodes", "timezonefinder", "oauthlib~=3.2", diff --git a/requirements/extras.txt b/requirements/extras.txt deleted file mode 100644 index 2f77edff..00000000 --- a/requirements/extras.txt +++ /dev/null @@ -1,9 +0,0 @@ -rapidfuzz>=3.6,<4.0 -ovos-plugin-manager>=0.0.25 -ovos-config>=0.0.12 -ovos-workshop>=0.0.13 -ovos_bus_client>=0.0.8 -langcodes -timezonefinder -oauthlib~=3.2 -orjson \ No newline at end of file diff --git a/requirements/requirements.txt b/requirements/requirements.txt deleted file mode 100644 index 49a447aa..00000000 --- a/requirements/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -pexpect~=4.9 -requests~=2.26 -json_database~=0.10 -kthread~=0.2 -watchdog -pyee>=8.0.0 -combo-lock~=0.2 -rich-click~=1.7 -rich~=13.7 \ No newline at end of file diff --git a/test/unittests/test_async_fakebus.py b/test/unittests/test_async_fakebus.py new file mode 100644 index 00000000..a3a1f79d --- /dev/null +++ b/test/unittests/test_async_fakebus.py @@ -0,0 +1,267 @@ +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +"""Tests for AsyncFakeBus. + +Mirrors test_fakebus.py shape but exercises the async surface +(connect / close / emit / wait_for_message / wait_for_response) plus +the sync handler-registration contract that matches +AsyncMessageBusClient. +""" +import asyncio +import unittest + +from ovos_utils.fakebus import AsyncFakeBus, FakeMessage + + +def _run(coro): + """Tiny helper so we can use plain unittest.TestCase.""" + return asyncio.run(coro) + + +class TestAsyncFakeBusLifecycle(unittest.TestCase): + def test_constructs_connected(self): + bus = AsyncFakeBus() + self.assertTrue(bus.connected_event.is_set()) + + def test_session_id_from_kwargs(self): + class _Sess: + session_id = "from-kwarg" + bus = AsyncFakeBus(session=_Sess()) + self.assertEqual(bus.session_id, "from-kwarg") + + def test_connect_is_noop_but_sets_event(self): + bus = AsyncFakeBus() + bus.connected_event.clear() + _run(bus.connect()) + self.assertTrue(bus.connected_event.is_set()) + self.assertTrue(bus.started_running) + + def test_close_clears_connected_event(self): + bus = AsyncFakeBus() + self.assertTrue(bus.connected_event.is_set()) + _run(bus.close()) + self.assertFalse(bus.connected_event.is_set()) + + +class TestAsyncFakeBusHandlerRegistration(unittest.TestCase): + def test_on_then_emit_dispatches(self): + bus = AsyncFakeBus() + seen = [] + bus.on("hello", lambda m: seen.append(m)) + _run(bus.emit(FakeMessage("hello", {"x": 1}))) + self.assertEqual(len(seen), 1) + self.assertEqual(seen[0].msg_type, "hello") + self.assertEqual(seen[0].data["x"], 1) + + def test_once_fires_only_once(self): + bus = AsyncFakeBus() + seen = [] + bus.once("evt", lambda m: seen.append(m)) + _run(bus.emit(FakeMessage("evt"))) + _run(bus.emit(FakeMessage("evt"))) + self.assertEqual(len(seen), 1) + + def test_remove_handler(self): + bus = AsyncFakeBus() + seen = [] + + def cb(m): + seen.append(m) + + bus.on("evt", cb) + bus.remove("evt", cb) + _run(bus.emit(FakeMessage("evt"))) + self.assertEqual(seen, []) + + def test_remove_all_listeners(self): + bus = AsyncFakeBus() + bus.on("evt", lambda m: None) + bus.on("evt", lambda m: None) + bus.remove_all_listeners("evt") + self.assertEqual(bus.ee.listeners("evt"), []) + + def test_remove_unknown_handler_does_not_raise(self): + bus = AsyncFakeBus() + # not registered → silent + bus.remove("evt", lambda m: None) + + +class TestAsyncFakeBusEmit(unittest.TestCase): + def test_emit_injects_session_context_when_missing(self): + bus = AsyncFakeBus() + msg = FakeMessage("hello", {}) + self.assertNotIn("session", msg.context) + _run(bus.emit(msg)) + self.assertIn("session", msg.context) + + def test_emit_dispatches_raw_message_event(self): + bus = AsyncFakeBus() + raws = [] + bus.on("message", lambda raw: raws.append(raw)) + _run(bus.emit(FakeMessage("hello"))) + self.assertEqual(len(raws), 1) + self.assertIn("hello", raws[0]) + + +class TestAsyncFakeBusWaitForMessage(unittest.TestCase): + def test_returns_matched_message_emitted_concurrently(self): + bus = AsyncFakeBus() + + async def scenario(): + async def feed(): + await asyncio.sleep(0.02) + await bus.emit(FakeMessage("ping", {"flood_id": "x"})) + asyncio.create_task(feed()) + got = await bus.wait_for_message("ping", timeout=1.0) + return got + + got = _run(scenario()) + self.assertIsNotNone(got) + self.assertEqual(got.msg_type, "ping") + + def test_returns_none_on_timeout(self): + bus = AsyncFakeBus() + + async def scenario(): + return await bus.wait_for_message("never", timeout=0.05) + + self.assertIsNone(_run(scenario())) + + +class TestAsyncFakeBusWaitForResponse(unittest.TestCase): + def test_default_reply_type_is_msg_type_response(self): + bus = AsyncFakeBus() + + async def scenario(): + # echo the request as .response when the request arrives + def echo(m): + # synchronous dispatch — fire the reply inline + # cannot await here; schedule on the loop instead + asyncio.create_task( + bus.emit(FakeMessage(m.msg_type + ".response", + {"echoed": m.data}))) + bus.on("ask", echo) + return await bus.wait_for_response( + FakeMessage("ask", {"q": 1}), timeout=1.0, + ) + + reply = _run(scenario()) + self.assertIsNotNone(reply) + self.assertEqual(reply.msg_type, "ask.response") + self.assertEqual(reply.data["echoed"], {"q": 1}) + + def test_explicit_reply_type(self): + bus = AsyncFakeBus() + + async def scenario(): + def respond(m): + asyncio.create_task(bus.emit(FakeMessage("pong"))) + bus.on("ping", respond) + return await bus.wait_for_response( + FakeMessage("ping"), reply_type="pong", timeout=1.0, + ) + + reply = _run(scenario()) + self.assertIsNotNone(reply) + self.assertEqual(reply.msg_type, "pong") + + def test_returns_none_on_timeout(self): + bus = AsyncFakeBus() + + async def scenario(): + return await bus.wait_for_response( + FakeMessage("never"), timeout=0.05, + ) + + self.assertIsNone(_run(scenario())) + + +class TestAsyncFakeBusCompatShims(unittest.TestCase): + def test_create_client_returns_self(self): + bus = AsyncFakeBus() + self.assertIs(bus.create_client(), bus) + + def test_run_forever_flips_started_running(self): + bus = AsyncFakeBus() + bus.started_running = False + bus.run_forever() + self.assertTrue(bus.started_running) + + def test_run_in_thread_alias(self): + bus = AsyncFakeBus() + bus.started_running = False + bus.run_in_thread() + self.assertTrue(bus.started_running) + + +class TestAsyncFakeBusNamespaceMigration(unittest.TestCase): + """AsyncFakeBus mirrors FakeBus / MessageBusClient namespace migration.""" + + def test_legacy_emit_reaches_spec_listener(self): + bus = AsyncFakeBus() # both flags default on + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(got, ["ovos.utterance.speak"]) # modernize bridged it + + def test_spec_emit_reaches_legacy_listener(self): + bus = AsyncFakeBus() + got = [] + bus.on("speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("ovos.utterance.speak", {"utterance": "hi"}))) + self.assertEqual(got, ["speak"]) # emit_legacy bridged it + + def test_counterpart_payload_is_translated(self): + # a spec listener on the counterpart of a SHAPE-CHANGING legacy topic + # receives the payload reshaped into ITS shape. detach_intent -> + # ovos.intent.deregister splits "skill:intent" into skill_id/intent_name. + bus = AsyncFakeBus() + got = [] + bus.on("ovos.intent.deregister", lambda m: got.append(dict(m.data))) + _run(bus.emit(FakeMessage("detach_intent", + {"intent_name": "skill.foo:HelloIntent"}))) + self.assertEqual(got, [{"skill_id": "skill.foo", "intent_name": "HelloIntent"}]) + + def test_dual_listener_fires_once(self): + bus = AsyncFakeBus() + calls = [] + handler = lambda m: calls.append(m.msg_type) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(len(calls), 1) # mirror deduped + + def test_distinct_listeners_each_fire_once(self): + bus = AsyncFakeBus() + legacy, spec = [], [] + bus.on("speak", lambda m: legacy.append(1)) + bus.on("ovos.utterance.speak", lambda m: spec.append(1)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual((len(legacy), len(spec)), (1, 1)) + + def test_flags_off_no_bridging(self): + bus = AsyncFakeBus(modernize=False, emit_legacy=False) + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(got, []) + + def test_remove_cleans_up(self): + bus = AsyncFakeBus() + calls = [] + handler = lambda m: calls.append(1) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.remove("speak", handler) + bus.remove("ovos.utterance.speak", handler) + self.assertNotIn(handler, bus._handler_guards) + _run(bus.emit(FakeMessage("speak", {"utterance": "hi"}))) + self.assertEqual(calls, []) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/unittests/test_bracket_expansion.py b/test/unittests/test_bracket_expansion.py index 5d6bcbe1..f3b06bb2 100644 --- a/test/unittests/test_bracket_expansion.py +++ b/test/unittests/test_bracket_expansion.py @@ -35,20 +35,15 @@ def test_expand_slots(self): 'change the brightness to high and color to blue'] self.assertEqual(expanded_sentences, expected_sentences) + def test_malformed_template_raises(self): + # a template whose expansion would yield an empty string is malformed + # (OVOS-INTENT-1 §3.6) — it raises rather than producing '' + from ovos_spec_tools import MalformedTemplate + with self.assertRaises(MalformedTemplate): + expand_template("[(this|that) is optional]") + def test_expand_template(self): # Test for template expansion - templates = [ - "[hello,] (call me|my name is) {name}", - "Expand (alternative|choices) into a list of choices.", - "sentences have [optional] words ", - "alternative words can be (used|written)", - "sentence[s] can have (pre|suf)fixes mid word too", - "do( the | )thing(s|) (old|with) style and( no | )spaces", - "[(this|that) is optional]", - "tell me a [{joke_type}] joke", - "play {query} [in ({device_name}|{skill_name}|{zone_name})]" - ] - expected_outputs = { "[hello,] (call me|my name is) {name}": [ "call me {name}", @@ -60,9 +55,11 @@ def test_expand_template(self): "Expand alternative into a list of choices.", "Expand choices into a list of choices." ], + # an emptied [optional] no longer leaves a double space — + # OVOS-INTENT-1 §4.1 normalizes whitespace to single spaces "sentences have [optional] words ": [ - "sentences have words", - "sentences have optional words" + "sentences have optional words", + "sentences have words" ], "alternative words can be (used|written)": [ "alternative words can be used", @@ -92,12 +89,8 @@ def test_expand_template(self): "do things with style and no spaces", "do things with style and spaces" ], - "[(this|that) is optional]": [ - '', - 'that is optional', - 'this is optional'], "tell me a [{joke_type}] joke": [ - "tell me a joke", + "tell me a joke", "tell me a {joke_type} joke" ], "play {query} [in ({device_name}|{skill_name}|{zone_name})]": [ diff --git a/test/unittests/test_device_input.py b/test/unittests/test_device_input.py index 00e22c17..b39cd226 100644 --- a/test/unittests/test_device_input.py +++ b/test/unittests/test_device_input.py @@ -26,7 +26,7 @@ except ImportError: distutils_stub = types.ModuleType("distutils") spawn_stub = types.ModuleType("distutils.spawn") - spawn_stub.find_executable = lambda x: None + spawn_stub.which = lambda x: None distutils_stub.spawn = spawn_stub sys.modules["distutils"] = distutils_stub sys.modules["distutils.spawn"] = spawn_stub @@ -35,7 +35,7 @@ class TestInputDeviceHelper(unittest.TestCase): """Tests for InputDeviceHelper class.""" - @patch("ovos_utils.device_input.find_executable", return_value=None) + @patch("ovos_utils.device_input.which", return_value=None) def test_init_no_executables(self, mock_find: MagicMock) -> None: """InputDeviceHelper should initialise with empty device lists.""" from ovos_utils.device_input import InputDeviceHelper @@ -43,7 +43,7 @@ def test_init_no_executables(self, mock_find: MagicMock) -> None: self.assertEqual(helper.libinput_devices_list, []) self.assertEqual(helper.xinput_devices_list, []) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") @patch("subprocess.check_output") def test_build_libinput_devices_list(self, mock_output: MagicMock, mock_find: MagicMock) -> None: @@ -63,7 +63,7 @@ def test_build_libinput_devices_list(self, mock_output: MagicMock, self.assertEqual(dev["Device"], "My Keyboard") self.assertIn("keyboard", dev["Capabilities"]) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") @patch("subprocess.check_output") def test_build_libinput_multiple_capabilities(self, mock_output: MagicMock, mock_find: MagicMock) -> None: @@ -82,7 +82,7 @@ def test_build_libinput_multiple_capabilities(self, mock_output: MagicMock, self.assertIsInstance(caps, list) self.assertGreater(len(caps), 1) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") @patch("subprocess.check_output", side_effect=Exception("libinput failed")) def test_get_libinput_devices_exception(self, mock_output: MagicMock, mock_find: MagicMock) -> None: @@ -93,7 +93,7 @@ def test_get_libinput_devices_exception(self, mock_output: MagicMock, result = helper._get_libinput_devices_list() self.assertEqual(result, []) - @patch("ovos_utils.device_input.find_executable", return_value=None) + @patch("ovos_utils.device_input.which", return_value=None) def test_get_libinput_devices_no_executable(self, mock_find: MagicMock) -> None: """_get_libinput_devices_list should return empty list when libinput not found.""" from ovos_utils.device_input import InputDeviceHelper @@ -101,7 +101,7 @@ def test_get_libinput_devices_no_executable(self, mock_find: MagicMock) -> None: result = helper._get_libinput_devices_list() self.assertEqual(result, []) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") @patch("subprocess.check_output") def test_build_xinput_devices_list(self, mock_output: MagicMock, mock_find: MagicMock) -> None: @@ -117,7 +117,7 @@ def test_build_xinput_devices_list(self, mock_output: MagicMock, helper._build_xinput_devices_list() self.assertGreater(len(helper.xinput_devices_list), 0) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") @patch("subprocess.check_output", side_effect=Exception("xinput failed")) def test_get_xinput_devices_exception(self, mock_output: MagicMock, mock_find: MagicMock) -> None: @@ -128,7 +128,7 @@ def test_get_xinput_devices_exception(self, mock_output: MagicMock, result = helper._get_xinput_devices_list() self.assertEqual(result, []) - @patch("ovos_utils.device_input.find_executable", return_value=None) + @patch("ovos_utils.device_input.which", return_value=None) def test_get_xinput_devices_no_executable(self, mock_find: MagicMock) -> None: """_get_xinput_devices_list should return empty list when xinput not found.""" from ovos_utils.device_input import InputDeviceHelper @@ -136,7 +136,7 @@ def test_get_xinput_devices_no_executable(self, mock_find: MagicMock) -> None: result = helper._get_xinput_devices_list() self.assertEqual(result, []) - @mock.patch("ovos_utils.device_input.find_executable") + @mock.patch("ovos_utils.device_input.which") def test_can_use_touch_mouse(self, find_exec: MagicMock) -> None: """can_use_touch_mouse should detect touch/mouse/tablet/pointer/gesture.""" from ovos_utils.device_input import InputDeviceHelper @@ -164,7 +164,7 @@ def test_can_use_touch_mouse(self, find_exec: MagicMock) -> None: dev_input.xinput_devices_list.pop() self.assertFalse(dev_input.can_use_touch_mouse()) - @mock.patch("ovos_utils.device_input.find_executable") + @mock.patch("ovos_utils.device_input.which") def test_can_use_keyboard(self, find_exec: MagicMock) -> None: """can_use_keyboard should detect keyboard devices.""" from ovos_utils.device_input import InputDeviceHelper @@ -192,7 +192,7 @@ def test_can_use_keyboard(self, find_exec: MagicMock) -> None: dev_input.xinput_devices_list.pop() self.assertFalse(dev_input.can_use_keyboard()) - @patch("ovos_utils.device_input.find_executable", return_value=None) + @patch("ovos_utils.device_input.which", return_value=None) @patch("ovos_utils.device_input.is_gui_installed", return_value=True) def test_can_use_touch_mouse_no_executable_gui_installed( self, mock_gui: MagicMock, mock_find: MagicMock) -> None: @@ -202,7 +202,7 @@ def test_can_use_touch_mouse_no_executable_gui_installed( result = helper.can_use_touch_mouse() self.assertTrue(result) - @patch("ovos_utils.device_input.find_executable", return_value=None) + @patch("ovos_utils.device_input.which", return_value=None) @patch("ovos_utils.device_input.is_gui_installed", return_value=False) def test_can_use_touch_mouse_no_executable_no_gui( self, mock_gui: MagicMock, mock_find: MagicMock) -> None: @@ -212,7 +212,7 @@ def test_can_use_touch_mouse_no_executable_no_gui( result = helper.can_use_touch_mouse() self.assertFalse(result) - @patch("ovos_utils.device_input.find_executable") + @patch("ovos_utils.device_input.which") def test_get_input_device_list(self, mock_find: MagicMock) -> None: """get_input_device_list should combine libinput and xinput device lists.""" from ovos_utils.device_input import InputDeviceHelper diff --git a/test/unittests/test_fakebus.py b/test/unittests/test_fakebus.py index 918fa4ff..a94d737d 100644 --- a/test/unittests/test_fakebus.py +++ b/test/unittests/test_fakebus.py @@ -57,9 +57,27 @@ def test_response(self) -> None: self.assertEqual(resp.msg_type, "my.request.response") self.assertEqual(resp.data["result"], "ok") + def test_publish_emits_deprecation_warning(self) -> None: + """``publish`` is not part of OVOS-MSG-1 and is scheduled for + removal — every call must fire a DeprecationWarning.""" + import warnings + msg = self._make_message("pub.type", {}, {"source": "x"}) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + msg.publish("new.type", {"payload": 1}) + deps = [w for w in caught + if issubclass(w.category, DeprecationWarning) + and "publish" in str(w.message)] + self.assertTrue(deps, + "FakeMessage.publish() did not emit a " + "DeprecationWarning") + def test_publish(self) -> None: msg = self._make_message("pub.type", {}, {"target": "skill", "source": "x"}) - published = msg.publish("new.type", {"payload": 1}) + import warnings + with warnings.catch_warnings(): + warnings.simplefilter("ignore", DeprecationWarning) + published = msg.publish("new.type", {"payload": 1}) self.assertNotIn("target", published.context) self.assertEqual(published.data["payload"], 1) diff --git a/test/unittests/test_fakebus_intent_topic_bridge.py b/test/unittests/test_fakebus_intent_topic_bridge.py new file mode 100644 index 00000000..988ef084 --- /dev/null +++ b/test/unittests/test_fakebus_intent_topic_bridge.py @@ -0,0 +1,195 @@ +"""FakeBus mirrors MessageBusClient's legacy<->canonical INTENT-topic bridge +(RULE 1 send-side twin / RULE 2 receive-side modernize), so in-process tests +raw-emitting a legacy ``:IntentName.intent`` topic reach a +canonical-only listener the same way a real websocket deployment does, and +vice-versa. + +Root cause this closes: FakeBus already wired ovos_spec_tools's +NamespaceTranslator for the fixed SpecMessage pairs (speak <-> ovos.utterance.speak +etc, see test_fakebus_namespace_migration.py) but NOT the per-intent +dispatch-topic bridge that ovos_bus_client.client.client.MessageBusClient +applies via ``_send_legacy_intent_twin`` / ``_modernize_intent_topic``. Since +ovos-workshop >= 9.3.11a2 dropped its own dual-bind (only registers the +canonical listener), an in-process test emitting the legacy suffixed topic +directly never reached the handler -- while a real deployment, whose bus +client performs this bridge, dealiased fine. +""" +import asyncio +import unittest +from unittest.mock import patch + +from ovos_utils.fakebus import AsyncFakeBus, FakeBus, Message, INTENT_COMPAT_TWIN_KEY + + +def _run(coro): + return asyncio.run(coro) + + +LEGACY = "myskill.foo:HelloIntent.intent" +CANONICAL = "myskill.foo:HelloIntent" + + +class TestFakeBusIntentTopicBridge(unittest.TestCase): + def test_legacy_emit_reaches_canonical_listener(self): + # RULE 2: a raw legacy-suffixed emit (no bus-client, no twin marker) + # must still fire a canonical-only listener. + bus = FakeBus() # both flags default on + got = [] + bus.on(CANONICAL, lambda m: got.append(m.msg_type)) + bus.emit(Message(LEGACY, {"utterance": "hi"})) + self.assertEqual(got, [CANONICAL]) + + def test_canonical_emit_also_fires_legacy_listener(self): + # RULE 1: every canonical intent dispatch is twinned onto its legacy + # spelling so an old suffix-only listener still hears it. + bus = FakeBus() + got = [] + bus.on(LEGACY, lambda m: got.append(m.msg_type)) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(got, [LEGACY]) + + def test_canonical_emit_twin_not_marked_locally(self): + # On the real wire the RULE-1 twin goes out MARKED so an + # out-of-process receiver's RULE 2 knows to skip re-modernizing it. + # FakeBus has no wire hop: it already made that RULE-2 call inline + # for this dispatch, so the twin delivered to LOCAL listeners must + # NOT carry the marker -- matching the real client, whose receiving + # process pops the marker before any local handler ever sees it + # (client.py:351, before local dispatch). Carrying it into the local + # twin would also break the per-topic-pair mirror guard's + # payload+context fingerprint match (see + # test_no_double_fire_dual_listener) and leak onto any descendant + # frame a handler derives via forward()/reply(). + bus = FakeBus() + got = [] + bus.on(LEGACY, lambda m: got.append(m)) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(len(got), 1) + self.assertNotIn(INTENT_COMPAT_TWIN_KEY, got[0].context) + + def test_no_double_fire_dual_listener(self): + # a handler subscribed to BOTH the legacy and canonical topic must + # not see the same logical dispatch twice -- matching the real + # MessageBusClient, whose per-topic-pair mirror guard (shared by + # every registration on either spelling) drops the twin as a + # re-delivery of the same logical event. ovos-workshop 9.3.2a1+ + # binds the skill method to both spellings via a FRESH wrapper + # closure per registration, so this must hold even though the two + # ``bus.on()`` calls below pass the SAME underlying handler object. + bus = FakeBus() + calls = [] + handler = lambda m: calls.append(m.msg_type) + bus.on(LEGACY, handler) + bus.on(CANONICAL, handler) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(len(calls), 1) + self.assertEqual(calls, [CANONICAL]) + + def test_independent_handlers_legacy_only_starves(self): + # two INDEPENDENT handlers -- one on the canonical topic, one on the + # legacy-only spelling -- share the per-topic-pair guard (it cannot + # be scoped to a single handler, see FakeBus._mirror_guard_for), so + # a canonical emit arms the guard and the legacy-only handler + # starves. This matches real-bus behavior: a process holding both + # handlers is unreachable from a single workshop version, so the + # starvation is an accepted trade-off, not a defect. + bus = FakeBus() + canonical_calls = [] + legacy_calls = [] + bus.on(CANONICAL, lambda m: canonical_calls.append(m.msg_type)) + bus.on(LEGACY, lambda m: legacy_calls.append(m.msg_type)) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(canonical_calls, [CANONICAL]) + self.assertEqual(legacy_calls, []) + + def test_twin_marker_does_not_leak_onto_unrelated_forward(self): + # RULE 2 dedup marker regression: a handler on the LEGACY spelling + # that forwards its received message's context onto an UNRELATED + # suffixed topic must not brand that unrelated frame a twin. The + # marker is popped BEFORE dispatch (mirrors + # MessageBusClient.on_message's pop-before-dispatch ordering), so it + # cannot survive onto a descendant frame created by + # Message.forward(), which deep-copies context. + bus = FakeBus() + seen = [] + other_legacy = "other.skill:OtherIntent.intent" + other_canonical = "other.skill:OtherIntent" + + def legacy_handler(m): + bus.emit(m.forward(other_legacy, {})) + + bus.on(LEGACY, legacy_handler) + bus.on(other_canonical, lambda m: seen.append("canonical-modernized")) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(seen, ["canonical-modernized"]) + + def test_twin_marker_suppresses_rule2_recascade(self): + # if a caller manually emits a message already carrying the twin + # marker (simulating what a real client would receive as the twin + # half of a pair), RULE 2 must not modernize it again -- proving the + # bridge cannot cascade into a modernize/twin loop. + bus = FakeBus() + canonical_hits = [] + bus.on(CANONICAL, lambda m: canonical_hits.append(1)) + msg = Message(LEGACY, {"utterance": "hi"}, + {INTENT_COMPAT_TWIN_KEY: True}) + bus.emit(msg) + self.assertEqual(canonical_hits, []) + + def test_non_intent_topic_untouched(self): + bus = FakeBus() + got = [] + bus.on("my.custom.topic", lambda m: got.append(m.msg_type)) + bus.emit(Message("my.custom.topic", {"x": 1})) + self.assertEqual(got, ["my.custom.topic"]) + # and no stray listeners fired for unrelated suffixed-looking topics + got2 = [] + bus.on("speak", lambda m: got2.append(m.msg_type)) + bus.emit(Message("my.custom.topic", {"x": 1})) + self.assertEqual(got2, []) + + def test_flags_off_no_bridging(self): + # each direction gets its own bus/listener pair: a listener on the + # SAME topic as what's emitted always fires (plain same-topic + # dispatch, unrelated to the bridge) -- only the OTHER namespace's + # listener proves whether bridging happened. + bus1 = FakeBus(modernize=False, emit_legacy=False) + got_canonical = [] + bus1.on(CANONICAL, lambda m: got_canonical.append(1)) + bus1.emit(Message(LEGACY, {"utterance": "hi"})) + self.assertEqual(got_canonical, []) # RULE 2 suppressed + + bus2 = FakeBus(modernize=False, emit_legacy=False) + got_legacy = [] + bus2.on(LEGACY, lambda m: got_legacy.append(1)) + bus2.emit(Message(CANONICAL, {"utterance": "hi"})) + self.assertEqual(got_legacy, []) # RULE 1 suppressed + + def test_already_canonical_not_re_twinned_into_itself(self): + # a topic with no legacy counterpart (canonical == legacy, e.g. a + # non-suffixed topic that is not an intent topic at all) is a no-op. + bus = FakeBus() + calls = [] + bus.on(CANONICAL, lambda m: calls.append(1)) + bus.emit(Message(CANONICAL, {"utterance": "hi"})) + # exactly one direct dispatch; the RULE-1 twin went to LEGACY, not + # back onto CANONICAL, so no double count here. + self.assertEqual(calls, [1]) + + def test_async_fakebus_legacy_emit_reaches_canonical_listener(self): + bus = AsyncFakeBus() + got = [] + bus.on(CANONICAL, lambda m: got.append(m.msg_type)) + _run(bus.emit(Message(LEGACY, {"utterance": "hi"}))) + self.assertEqual(got, [CANONICAL]) + + def test_async_fakebus_canonical_emit_fires_legacy_listener(self): + bus = AsyncFakeBus() + got = [] + bus.on(LEGACY, lambda m: got.append(m.msg_type)) + _run(bus.emit(Message(CANONICAL, {"utterance": "hi"}))) + self.assertEqual(got, [LEGACY]) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/unittests/test_fakebus_namespace_migration.py b/test/unittests/test_fakebus_namespace_migration.py new file mode 100644 index 00000000..3eb4b79b --- /dev/null +++ b/test/unittests/test_fakebus_namespace_migration.py @@ -0,0 +1,150 @@ +"""FakeBus mirrors MessageBusClient's legacy<->ovos.* namespace migration, so +e2e/satellite tests exercise the real cross-namespace behaviour.""" +import asyncio +import unittest +from unittest.mock import patch + +from ovos_utils.fakebus import AsyncFakeBus, FakeBus, Message + + +def _run(coro): + return asyncio.run(coro) + + +class TestFakeBusNamespaceMigration(unittest.TestCase): + def test_legacy_emit_reaches_spec_listener(self): + bus = FakeBus() # both flags default on + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(got, ["ovos.utterance.speak"]) # modernize bridged it + + def test_spec_emit_reaches_legacy_listener(self): + bus = FakeBus() + got = [] + bus.on("speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("ovos.utterance.speak", {"utterance": "hi"})) + self.assertEqual(got, ["speak"]) # emit_legacy bridged it + + def test_dual_listener_fires_once(self): + bus = FakeBus() + calls = [] + handler = lambda m: calls.append(m.msg_type) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(len(calls), 1) # mirror deduped + + def test_distinct_listeners_each_fire_once(self): + bus = FakeBus() + legacy, spec = [], [] + bus.on("speak", lambda m: legacy.append(1)) + bus.on("ovos.utterance.speak", lambda m: spec.append(1)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual((len(legacy), len(spec)), (1, 1)) + + def test_flags_off_no_bridging(self): + bus = FakeBus(modernize=False, emit_legacy=False) + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(got, []) # no translation -> spec listener not reached + + def test_unmapped_topic_untouched(self): + bus = FakeBus() + got = [] + bus.on("my.custom.topic", lambda m: got.append(m.msg_type)) + bus.emit(Message("my.custom.topic", {"x": 1})) + self.assertEqual(got, ["my.custom.topic"]) + + def test_shape_changing_payload_reshaped_for_spec_listener(self): + # a spec listener on the counterpart of a SHAPE-CHANGING legacy topic + # receives the payload in ITS shape, not a verbatim legacy copy. + # detach_intent -> ovos.intent.deregister splits the compound + # "skill:intent" name into skill_id + intent_name. + bus = FakeBus() + got = [] + bus.on("ovos.intent.deregister", lambda m: got.append(dict(m.data))) + bus.emit(Message("detach_intent", {"intent_name": "skill.foo:HelloIntent"})) + self.assertEqual(len(got), 1) + self.assertEqual(got[0], {"skill_id": "skill.foo", "intent_name": "HelloIntent"}) + + def test_shape_changing_payload_reshaped_for_legacy_listener(self): + bus = FakeBus() + got = [] + bus.on("detach_intent", lambda m: got.append(dict(m.data))) + bus.emit(Message("ovos.intent.deregister", + {"skill_id": "skill.foo", "intent_name": "HelloIntent"})) + self.assertEqual(len(got), 1) + # rejoined to the legacy compound shape + self.assertEqual(got[0].get("intent_name"), "skill.foo:HelloIntent") + + def test_payload_compatible_rename_delivered_equivalent(self): + bus = FakeBus() + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(dict(m.data))) + bus.emit(Message("speak", {"utterance": "hi", "lang": "en-us"})) + self.assertEqual(got, [{"utterance": "hi", "lang": "en-us"}]) # identity + + def test_remove_cleans_up(self): + bus = FakeBus() + calls = [] + handler = lambda m: calls.append(1) + bus.on("speak", handler) + bus.on("ovos.utterance.speak", handler) + bus.remove("speak", handler) + bus.remove("ovos.utterance.speak", handler) + self.assertNotIn(handler, bus._handler_guards) + bus.emit(Message("speak", {"utterance": "hi"})) + self.assertEqual(calls, []) + + +class TestFakeBusFlagResolution(unittest.TestCase): + """When the kwarg is omitted, flags resolve via env -> websocket.* config -> + default True, matching MessageBusClient._bus_flag. An explicit kwarg wins.""" + + def _legacy_mirrored(self, bus): + # emit a legacy topic; if emit_legacy bridging is on a spec listener fires + got = [] + bus.on("ovos.utterance.speak", lambda m: got.append(m.msg_type)) + if isinstance(bus, AsyncFakeBus): + _run(bus.emit(Message("speak", {"utterance": "hi"}))) + else: + bus.emit(Message("speak", {"utterance": "hi"})) + return got == ["ovos.utterance.speak"] + + def test_default_true_no_env_mirrors(self): + with patch.dict("os.environ", {}, clear=False): + import os + os.environ.pop("OVOS_BUS_MODERNIZE", None) + os.environ.pop("OVOS_BUS_EMIT_LEGACY", None) + self.assertTrue(self._legacy_mirrored(FakeBus())) + self.assertTrue(self._legacy_mirrored(AsyncFakeBus())) + + def test_env_false_disables_mirror(self): + with patch.dict("os.environ", + {"OVOS_BUS_MODERNIZE": "false", + "OVOS_BUS_EMIT_LEGACY": "false"}): + self.assertFalse(self._legacy_mirrored(FakeBus())) + self.assertFalse(self._legacy_mirrored(AsyncFakeBus())) + + def test_explicit_kwarg_beats_env(self): + # env says off, but an explicit modernize=True kwarg still mirrors + with patch.dict("os.environ", + {"OVOS_BUS_MODERNIZE": "false", + "OVOS_BUS_EMIT_LEGACY": "false"}): + self.assertTrue(self._legacy_mirrored(FakeBus(modernize=True))) + self.assertTrue(self._legacy_mirrored(AsyncFakeBus(modernize=True))) + + def test_explicit_false_kwarg_beats_unset_env(self): + import os + with patch.dict("os.environ", {}, clear=False): + os.environ.pop("OVOS_BUS_MODERNIZE", None) + os.environ.pop("OVOS_BUS_EMIT_LEGACY", None) + # default would mirror; explicit modernize=False suppresses it + self.assertFalse(self._legacy_mirrored(FakeBus(modernize=False))) + self.assertFalse(self._legacy_mirrored(AsyncFakeBus(modernize=False))) + + +if __name__ == "__main__": + unittest.main() diff --git a/test/unittests/test_file_utils.py b/test/unittests/test_file_utils.py index 8a0efd84..c1d8579c 100644 --- a/test/unittests/test_file_utils.py +++ b/test/unittests/test_file_utils.py @@ -1,10 +1,11 @@ +import os import shutil import unittest from os import makedirs from os.path import isdir, join, dirname from threading import Event from time import time -from unittest.mock import Mock +from unittest.mock import Mock, patch class TestFileUtils(unittest.TestCase): @@ -143,6 +144,58 @@ def _on_change(fp): shutil.rmtree(test_dir) + def test_filewatcher_not_yet_existing_file(self): + from ovos_utils.file_utils import FileWatcher + + test_dir = join(dirname(__file__), "test_watch_new") + test_file = join(test_dir, "not_yet.watch") + makedirs(test_dir, exist_ok=True) + self.assertFalse(os.path.isfile(test_file)) + + # a path that doesn't exist yet is still watched in file mode: + # the containing (existing) directory is scheduled, and a + # 'created' event for the not-yet-existing file fires the callback + called = Event() + callback = Mock(side_effect=lambda x: called.set()) + watcher = FileWatcher([test_file], callback) + with open(test_file, 'w+'): + pass + self.assertTrue(called.wait(3)) + callback.assert_called_once_with(test_file) + watcher.shutdown() + + # a different file being created in the same directory must + # NOT fire the callback (file mode still filters to the one path) + called.clear() + callback.reset_mock() + watcher = FileWatcher([test_file], callback) + other_file = join(test_dir, "other.watch") + with open(other_file, 'w+'): + pass + self.assertFalse(called.wait(3)) + callback.assert_not_called() + watcher.shutdown() + + shutil.rmtree(test_dir) + + def test_filewatcher_missing_parent_directory(self): + from ovos_utils.file_utils import FileWatcher + + # if the parent directory of a not-yet-existing file also doesn't + # exist, watchdog can't schedule an observer on it; FileWatcher + # must skip that entry (with a LOG.warning) instead of raising + # an opaque watchdog exception + missing_parent = join(dirname(__file__), "definitely_not_there") + missing_file = join(missing_parent, "cfg.json") + self.assertFalse(isdir(missing_parent)) + + callback = Mock() + with patch("ovos_utils.file_utils.LOG") as mock_log: + watcher = FileWatcher([missing_file], callback) + mock_log.warning.assert_called_once() + self.assertEqual(watcher.observer.emitters, set()) + watcher.shutdown() + def test_file_event_handler(self): from ovos_utils.file_utils import FileEventHandler from watchdog.events import FileCreatedEvent, FileModifiedEvent, FileClosedEvent @@ -169,6 +222,46 @@ def test_file_event_handler(self): handler.on_any_event(FileClosedEvent(test_file)) callback.assert_called_once() + # Test events for a different file in the same directory are ignored + # when watching a specific file (the FileWatcher watches the + # containing directory, so watchdog reports every file inside it) + other_file = join(dirname(__file__), "other.watch") + callback.reset_mock() + handler = FileEventHandler(test_file, callback, True) + handler.on_any_event(FileModifiedEvent(other_file)) + handler.on_any_event(FileClosedEvent(other_file)) + callback.assert_not_called() + # but the watched file itself still fires + handler.on_any_event(FileModifiedEvent(test_file)) + handler.on_any_event(FileClosedEvent(test_file)) + callback.assert_called_once() + + # Test directory-watch mode (file_path=None) reports every file + callback.reset_mock() + handler = FileEventHandler(None, callback, True) + handler.on_any_event(FileModifiedEvent(test_file)) + handler.on_any_event(FileClosedEvent(test_file)) + callback.assert_called_once_with(test_file) + callback.reset_mock() + handler.on_any_event(FileModifiedEvent(other_file)) + handler.on_any_event(FileClosedEvent(other_file)) + callback.assert_called_once_with(other_file) + + # Test two handlers watching different files in the same directory + # each fire only for their own file, not for each other's + callback_a = Mock() + callback_b = Mock() + handler_a = FileEventHandler(test_file, callback_a, True) + handler_b = FileEventHandler(other_file, callback_b, True) + # both handlers watch the same directory, so watchdog delivers + # every event to both of them + for handler in (handler_a, handler_b): + for ev_file in (test_file, other_file): + handler.on_any_event(FileModifiedEvent(ev_file)) + handler.on_any_event(FileClosedEvent(ev_file)) + callback_a.assert_called_once_with(test_file) + callback_b.assert_called_once_with(other_file) + # Test include creation callbacks callback.reset_mock() handler = FileEventHandler(test_file, callback, False) diff --git a/test/unittests/test_lang.py b/test/unittests/test_lang.py index 19cbeb20..2f9cdb73 100644 --- a/test/unittests/test_lang.py +++ b/test/unittests/test_lang.py @@ -24,38 +24,41 @@ class TestStandardizeLangTag(unittest.TestCase): """Tests for standardize_lang_tag.""" - def test_macro_strips_region(self) -> None: - """standardize_lang_tag(macro=True) should return bare language code.""" + def test_macro_preserves_region(self) -> None: + """standardize_lang_tag(macro=True) preserves the region. + + ``macro`` is a langcodes concept — it controls *macrolanguage* + substitution (``cmn`` -> ``zh``, ``nb`` -> ``no``), not region + stripping. ``en-US`` round-trips unchanged.""" from ovos_utils.lang import standardize_lang_tag - # When langcodes not available, falls back to split on '-' - with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("en-US", macro=True) - self.assertEqual(result, "en") + self.assertEqual(standardize_lang_tag("en-US", macro=True), "en-US") + self.assertEqual(standardize_lang_tag("en-us", macro=True), "en-US") def test_non_macro_preserves_region(self) -> None: - """standardize_lang_tag(macro=False) should keep the region part.""" + """standardize_lang_tag(macro=False) preserves the region too — + the difference between macro=True/False is macrolanguage + substitution, not region handling.""" from ovos_utils.lang import standardize_lang_tag - with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("en-us", macro=False) - self.assertEqual(result, "en-US") + self.assertEqual(standardize_lang_tag("en-us", macro=False), "en-US") + + def test_macro_substitutes_macrolanguage(self) -> None: + """With ``macro=True``, langcodes maps a sublanguage onto its + macrolanguage. ``cmn`` (Mandarin) -> ``zh`` (Chinese); + ``macro=False`` keeps the original tag.""" + from ovos_utils.lang import standardize_lang_tag + self.assertEqual(standardize_lang_tag("cmn", macro=True), "zh") + self.assertEqual(standardize_lang_tag("cmn", macro=False), "cmn") - def test_no_region_tag(self) -> None: - """standardize_lang_tag with no '-' should return lowercased tag.""" + def test_fallback_without_langcodes(self) -> None: + """With langcodes unavailable, ``standardize_lang_tag`` falls + back to spec-tools (also region-preserving). ``macro`` is a + no-op in this branch.""" from ovos_utils.lang import standardize_lang_tag with patch.dict("sys.modules", {"langcodes": None}): - result = standardize_lang_tag("EN", macro=False) - self.assertEqual(result, "en") - - def test_with_langcodes_library(self) -> None: - """standardize_lang_tag should call langcodes.standardize_tag when available.""" - mock_langcodes = unittest.mock.MagicMock() - mock_langcodes.standardize_tag.return_value = "en" - - with patch.dict("sys.modules", {"langcodes": mock_langcodes}): - from ovos_utils.lang import standardize_lang_tag - result = standardize_lang_tag("en-US", macro=True) - # Result is whatever langcodes returns - self.assertIsInstance(result, str) + self.assertEqual( + standardize_lang_tag("en-us", macro=True), "en-US") + self.assertEqual( + standardize_lang_tag("EN", macro=False), "en") class TestGetLanguageDir(unittest.TestCase): diff --git a/test/unittests/test_log.py b/test/unittests/test_log.py index f816cf1b..15419dc9 100644 --- a/test/unittests/test_log.py +++ b/test/unittests/test_log.py @@ -184,6 +184,39 @@ def _deprecated_function(test_arg): self.assertIn('version=1.0.0', log_msg, log_msg) self.assertIn('test deprecation', log_msg, log_msg) + @patch("ovos_utils.log.LOG.create_logger") + def test_log_deprecation_dedupe(self, create_logger): + fake_log = Mock() + log_warning = fake_log.warning + create_logger.return_value = fake_log + import ovos_utils.log + from ovos_utils.log import log_deprecation, deprecated + ovos_utils.log._logged_deprecations.clear() + + # Repeated calls from the same caller only log once + for _ in range(10): + log_deprecation("repeated deprecation") + log_warning.assert_called_once() + log_msg = log_warning.call_args[0][0] + self.assertIn('repeated deprecation', log_msg, log_msg) + + # A different message still logs + log_deprecation("other deprecation") + self.assertEqual(log_warning.call_count, 2) + log_msg = log_warning.call_args[0][0] + self.assertIn('other deprecation', log_msg, log_msg) + + # Deprecated decorator is also deduplicated + @deprecated("decorated deprecation", "1.0.0") + def _deprecated_function(): + pass + + for _ in range(10): + _deprecated_function() + self.assertEqual(log_warning.call_count, 3) + log_msg = log_warning.call_args[0][0] + self.assertIn('decorated deprecation', log_msg, log_msg) + @patch("ovos_utils.log.get_logs_config") @patch("ovos_utils.log.LOG") def test_monitor_log_level(self, log, get_config): diff --git a/test/unittests/test_log_parser.py b/test/unittests/test_log_parser.py index 48a02714..46ae51d1 100644 --- a/test/unittests/test_log_parser.py +++ b/test/unittests/test_log_parser.py @@ -412,5 +412,52 @@ def test_reads_log_for_last_load(self) -> None: self.assertGreater(result, datetime.fromtimestamp(0)) +class TestOvosLogsCLINoStrayFiles(unittest.TestCase): + """Regression test: importing/using the ovos-logs CLI must not create + a stray lock file in the current working directory. + + Previously `LOGLOCK = ComboLock("ovos_logs_console_script")` was + instantiated at module import time with a bare relative path, which + made ComboLock create (and leave behind) an empty file named + `ovos_logs_console_script` in whatever directory the process was + started from. + """ + + def test_help_does_not_create_stray_file(self) -> None: + import subprocess + import sys + + with tempfile.TemporaryDirectory() as tmpdir: + before = set(os.listdir(tmpdir)) + subprocess.run( + [sys.executable, "-c", + "import sys; sys.argv=['ovos-logs', '--help']; " + "from ovos_utils.log_parser import ovos_logs; " + "ovos_logs()"], + cwd=tmpdir, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + after = set(os.listdir(tmpdir)) + self.assertEqual(before, after, + f"stray files created in cwd: {after - before}") + + def test_importing_log_parser_does_not_create_stray_file(self) -> None: + import subprocess + import sys + + with tempfile.TemporaryDirectory() as tmpdir: + before = set(os.listdir(tmpdir)) + subprocess.run( + [sys.executable, "-c", "import ovos_utils.log_parser"], + cwd=tmpdir, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + after = set(os.listdir(tmpdir)) + self.assertEqual(before, after, + f"stray files created in cwd: {after - before}") + + if __name__ == "__main__": unittest.main() diff --git a/test/unittests/test_sound.py b/test/unittests/test_sound.py index 4843e985..7e175ba2 100644 --- a/test/unittests/test_sound.py +++ b/test/unittests/test_sound.py @@ -26,7 +26,7 @@ if "distutils" not in sys.modules: distutils_stub = types.ModuleType("distutils") spawn_stub = types.ModuleType("distutils.spawn") - spawn_stub.find_executable = lambda x: None + spawn_stub.which = lambda x: None distutils_stub.spawn = spawn_stub sys.modules["distutils"] = distutils_stub sys.modules["distutils.spawn"] = spawn_stub @@ -62,7 +62,7 @@ def test_no_tts_config(self) -> None: class TestFindPlayer(unittest.TestCase): """Tests for _find_player helper.""" - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_sox_play_found(self, mock_find: MagicMock) -> None: """Should prefer sox play when available.""" mock_find.side_effect = lambda x: "/usr/bin/play" if x == "play" else None @@ -71,7 +71,7 @@ def test_sox_play_found(self, mock_find: MagicMock) -> None: self.assertIsNotNone(result) self.assertIn("play", result) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_ogg_player_preferred_for_ogg(self, mock_find: MagicMock) -> None: """Should prefer ogg123 for .ogg files when sox is unavailable.""" def side_effect(x: str) -> str | None: @@ -86,7 +86,7 @@ def side_effect(x: str) -> str | None: self.assertIsNotNone(result) self.assertIn("ogg123", result) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_pw_play_fallback(self, mock_find: MagicMock) -> None: """Should use pw-play when sox is unavailable and file is not ogg.""" def side_effect(x: str) -> str | None: @@ -99,7 +99,7 @@ def side_effect(x: str) -> str | None: self.assertIsNotNone(result) self.assertIn("pw-play", result) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_paplay_for_wav(self, mock_find: MagicMock) -> None: """Should use paplay for .wav when pw-play and sox unavailable.""" def side_effect(x: str) -> str | None: @@ -112,7 +112,7 @@ def side_effect(x: str) -> str | None: self.assertIsNotNone(result) self.assertIn("paplay", result) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_aplay_for_wav_when_paplay_missing(self, mock_find: MagicMock) -> None: """Should fall back to aplay for .wav when paplay is unavailable.""" def side_effect(x: str) -> str | None: @@ -125,7 +125,7 @@ def side_effect(x: str) -> str | None: self.assertIsNotNone(result) self.assertIn("aplay", result) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") def test_mpg123_for_mp3(self, mock_find: MagicMock) -> None: """Should use mpg123 for mp3 when no other player found.""" def side_effect(x: str) -> str | None: @@ -138,7 +138,7 @@ def side_effect(x: str) -> str | None: self.assertIsNotNone(result) self.assertIn("mpg123", result) - @patch("ovos_utils.sound.find_executable", return_value=None) + @patch("ovos_utils.sound.which", return_value=None) def test_returns_none_when_no_player(self, _mock_find: MagicMock) -> None: """Should return None when no suitable player is found.""" from ovos_utils.sound import _find_player @@ -287,7 +287,7 @@ def test_snd_prefix_resolved(self) -> None: duration = get_sound_duration("snd/test.wav", base_dir=base_dir) self.assertGreater(duration, 0) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") @patch("subprocess.Popen") def test_ffprobe_fallback(self, mock_popen: MagicMock, mock_find: MagicMock) -> None: @@ -311,7 +311,7 @@ def find_side_effect(x: str) -> str | None: finally: os.unlink(fname) - @patch("ovos_utils.sound.find_executable") + @patch("ovos_utils.sound.which") @patch("subprocess.Popen") def test_mediainfo_fallback(self, mock_popen: MagicMock, mock_find: MagicMock) -> None: @@ -337,7 +337,7 @@ def find_side_effect(x: str) -> str | None: finally: os.unlink(fname) - @patch("ovos_utils.sound.find_executable", return_value=None) + @patch("ovos_utils.sound.which", return_value=None) def test_no_tool_raises_runtime_error(self, _mock_find: MagicMock) -> None: """get_sound_duration should raise RuntimeError when no tool is available.""" with tempfile.NamedTemporaryFile(suffix=".mp3", delete=False) as f: