diff --git a/README.md b/README.md index abdb4a6..93d1525 100644 --- a/README.md +++ b/README.md @@ -76,8 +76,6 @@ infrastructure installed first. | CocoaPods | `brew install cocoapods` | | An Expo-based RN app | Expo SDK 52+ recommended (this repo uses SDK 57) | -### Step 0 (bare RN only): add Expo Modules support - Skip this if your app already uses Expo. For a bare React Native app, install the Expo Modules runtime once — it's what provides `requireNativeModule` and the autolinking `PlaudSdk` depends on: @@ -86,6 +84,15 @@ depends on: npx install-expo-modules@latest ``` +### Step 0: Install the skill from this repo + +The Skill has context on the Plaud Embedded plugin to help you implement this +plugin for your react-native app. + +```bash +npx skills add Plaud-AI/embedded-react-native +``` + ### Step 1: copy the module into your app Place the module where Expo autolinking looks — a `modules/` folder at your project root: diff --git a/skills/SKILL.md b/skills/SKILL.md new file mode 100644 index 0000000..f0bf1a7 --- /dev/null +++ b/skills/SKILL.md @@ -0,0 +1,181 @@ +--- +name: setup-plaud-react-native +description: Set up the Plaud SDK Expo/React Native module (BLE connect, on-device recording, file list, audio export) in an existing or new React Native app. Use when a user wants to integrate Plaud's native iOS device SDK into a React Native / Expo app, wire up scan → connect → list → export, or troubleshoot why the module isn't linking or is unavailable at runtime. +--- + +# Setting up the Plaud React Native module + +`plaud-sdk` is a local [Expo module](https://docs.expo.dev/modules/overview/) that bridges +Plaud's precompiled native iOS device SDK into React Native. It exposes BLE scan/connect, +on-device recording events, file listing, and audio export to JavaScript, with a typed event +stream. The module lives at `modules/plaud-sdk/` in this repo; a full reference app is at +`react-native-demo/` (`src/app/index.tsx` is the canonical usage example). + +Use this skill to add the module to an app and get it building on a device. + +## ⚠️ Read these constraints before anything else + +The Plaud frameworks are **arm64, iOS 15+, device-only**. There is **no simulator slice** and +**no Android support**. This dictates the entire workflow: + +- You **must run on a physical iPhone** (`npx expo run:ios --device`), never the simulator. +- You **must use a custom dev build**, not Expo Go (this is custom native code). +- On Android or the simulator, `isAvailable` is `false` and every `PlaudSdk` method rejects. + Guard every call site with `isAvailable` so the app stays functional (just without the SDK) + on those targets. + +If the user is on the simulator or expects Android support, stop and set expectations first — +no amount of setup makes the SDK run there. + +## Prerequisites + +| Tool | Notes | +| --- | --- | +| Node.js | v20+ (v24 used in this repo) | +| Xcode | 16.x+ (26.x used here), with a physical iPhone + Apple ID | +| CocoaPods | `brew install cocoapods` | +| An Expo-based RN app | Expo SDK 52+ (this repo uses SDK 57). Bare RN works after Step 0. | + +The module is built on Expo's module system, so the smoothest path is an Expo (or +Expo-prebuild) app. Bare React Native works too — you just install the Expo Modules +infrastructure first. + +## Setup workflow + +Work through these steps in order. Do not skip the `isAvailable` guard (Step 4) — it is the +difference between an app that degrades gracefully off-device and one that crashes. + +### Step 0 — (bare RN only) add Expo Modules support + +Skip if the app already uses Expo. For a bare React Native app, install the Expo Modules +runtime once — it provides `requireNativeModule` and the autolinking the module depends on: + +```bash +npx install-expo-modules@latest +``` + +### Step 1 — copy the module into the app + +Place it where Expo autolinking looks: a `modules/` folder at the project root. + +```bash +cp -R modules/plaud-sdk /path/to/your-app/modules/plaud-sdk +``` + +Then reference it from the app's `package.json` so Metro and TypeScript resolve the +`plaud-sdk` import to the local folder: + +```jsonc +// your-app/package.json +{ + "dependencies": { + "plaud-sdk": "file:./modules/plaud-sdk" + } +} +``` + +```bash +npm install +``` + +The module's `expo-module.config.json` (which registers `PlaudSdkModule`) is what makes +autolinking pick it up — **no manual native linking, no Podfile edits, no Xcode edits.** The +three Plaud `.xcframework`s in `ios/Frameworks/` are vendored by `PlaudSdk.podspec` and +CocoaPods embeds and code-signs them automatically. + +### Step 2 — declare BLE permissions in `app.json` + +These live in the app config (not the module) so they survive `expo prebuild`. Add them under +`expo.ios.infoPlist`: + +```jsonc +{ + "expo": { + "ios": { + "infoPlist": { + "NSBluetoothAlwaysUsageDescription": "Plaud uses Bluetooth to connect to your recorder and sync recordings.", + "UIBackgroundModes": ["bluetooth-central"] + } + } + } +} +``` + +Without `NSBluetoothAlwaysUsageDescription` the app crashes the moment it touches Bluetooth. +`UIBackgroundModes: ["bluetooth-central"]` keeps BLE alive when backgrounded. + +### Step 3 — generate the native project and build + +```bash +npx expo prebuild -p ios # regenerates ios/ from app.json and runs pod install +npx expo run:ios --device # build + install on a connected iPhone +``` + +Re-run `expo prebuild` after **any** native config change (permissions, bundle id, plugins). +In a bare app that manages `ios/` by hand, run `pod install` from `ios/` instead — autolinking +still discovers the module. + +### Step 4 — use it from JS (guard, init, subscribe, drive, clean up) + +The module is **event-driven**: JS calls (`startScan`, `connectBleDevice`, `getFileList`) kick +off work, and results arrive on the event stream, not as return values. The five-part shape: + +```ts +import { PlaudSdk, isAvailable } from 'plaud-sdk'; + +// 1. GUARD — off-iOS the native module isn't linked. Degrade gracefully. +if (!isAvailable) { /* show a "device required" state; skip SDK calls */ } + +// 2. INIT — once, with a per-user JWT (see references/transcription-and-tokens.md). +await PlaudSdk.initSDK({ + userAccessToken, // per-user Bearer JWT (your backend mints this) + customDomain: 'platform-us.plaud.ai', // domain only, no https:// + userId: 'your-app-user-id', // reused as the default connect deviceToken +}); + +// 3. SUBSCRIBE — this is where results land. +const subs = [ + PlaudSdk.addListener('scanResult', ({ devices }) => {/* show devices */}), + PlaudSdk.addListener('connectState', ({ connected, failed }) => { + if (connected) PlaudSdk.getFileList(); // ask for recordings once connected + }), + PlaudSdk.addListener('fileList', ({ files }) => {/* show recordings */}), + PlaudSdk.addListener('exportProgress', ({ progress, message }) => {/* progress UI */}), +]; + +// 4. DRIVE it. +await PlaudSdk.startScan(); +await PlaudSdk.connectBleDevice({ uuid: device.uuid }); // from a scanResult device +const { outputPath } = await PlaudSdk.exportAudio({ sessionId, format: 'mp3' }); + +// 5. CLEAN UP listeners on unmount. +subs.forEach((s) => s.remove()); +``` + +`react-native-demo/src/app/index.tsx` is a complete, production-shaped version (React state, +error handling, live-recording banners, unpair flow). **Read it before building your own +screen** — it shows the correct event → state wiring for every callback. + +## References + +Pull these in only when the task needs them: + +- **`references/api-reference.md`** — every `PlaudSdk` method, every event and its payload, + and all the TypeScript types. Consult when writing call sites or handling a specific event. +- **`references/transcription-and-tokens.md`** — where the per-user JWT comes from, and the + optional export → upload → transcribe HTTP flow (which is **not** part of the native module + and belongs behind a backend in production). +- **`references/troubleshooting.md`** — symptom → cause table for the common failures + (`isAvailable` false, scan returns nothing, connect fails, pod/build errors). + +## Key facts to keep straight + +- **Never edit `ios/` by hand in an Expo app.** It's generated by `expo prebuild`. Put native + config in `app.json` and regenerate. +- **`customDomain` is domain-only** — `platform-us.plaud.ai`, not `https://platform-us.plaud.ai`. +- **`initSDK` does not mint the token.** The per-user Bearer JWT is an app/backend + responsibility. For local testing, `EXPO_PUBLIC_PLAUD_ACCESS_TOKEN` works (Expo inlines + `EXPO_PUBLIC_*` at build), but those vars are extractable from the bundle — never ship + credentials in the client. +- **`readFile` / `putBinary` do not exist here.** They were Capacitor WKWebView CORS shims. In + RN, read exported files with `expo-file-system` and upload with `fetch`. diff --git a/skills/references/api-reference.md b/skills/references/api-reference.md new file mode 100644 index 0000000..9761bf2 --- /dev/null +++ b/skills/references/api-reference.md @@ -0,0 +1,131 @@ +# Plaud SDK — JS API reference + +The typed native module `PlaudSdk` (from `plaud-sdk`). Source of truth: +`modules/plaud-sdk/src/PlaudSdk.types.ts` and `modules/plaud-sdk/ios/PlaudSdkModule.swift`. + +All methods are iOS-device-only and reject on Android / the simulator. Guard call sites with +`isAvailable`. Methods that fetch data (`startScan`, `getFileList`) resolve immediately and +deliver results later via **events** — the promise resolving means "the request was sent," not +"here's the data." + +## Module exports + +```ts +import { PlaudSdk, isAvailable } from 'plaud-sdk'; +``` + +- `isAvailable: boolean` — `true` only when the native module is linked and callable (physical + iOS device). `false` on Android/simulator, where `PlaudSdk` is a no-op `Proxy` whose methods + reject and whose `addListener` returns a harmless `{ remove() {} }`. +- `PlaudSdk: PlaudSdkModule` — the typed handle (also the default export). + +## Methods + +| Method | Signature | Notes | +| --- | --- | --- | +| `initSDK` | `(o: { userAccessToken: string; customDomain: string; userId?: string }) => Promise` | Call once before anything else. `customDomain` is domain-only (no `https://`). `userId` is reused as the default connect `deviceToken`. Rejects `ERR_PLAUD_ARGS` if token/domain missing. | +| `startScan` | `() => Promise` | Begins BLE scan. Internally waits for CoreBluetooth to reach `.poweredOn` (polls ~18s); emits `scanTimeout` with `reason: 'bluetoothNotPoweredOn'` if it never powers on. Devices arrive via `scanResult`. | +| `stopScan` | `() => Promise` | Stops scanning. | +| `connectBleDevice` | `(o: { uuid?: string; serialNumber?: string; deviceToken?: string }) => Promise` | Connect to a device from a prior `scanResult`. Prefer `uuid`. Must scan first (device objects are cached natively) or it rejects `ERR_PLAUD_UNKNOWN_DEVICE`. Connection result arrives via `connectState`. | +| `disconnect` | `() => Promise` | Disconnect the current device. | +| `depair` | `(o?: { clear?: boolean }) => Promise` | Unpair. `clear` defaults `true` (also clears local pairing state). Result via `depair` event. | +| `isConnected` | `() => Promise<{ connected: boolean }>` | Synchronous-ish status check (this one returns data directly). | +| `getFileList` | `(o?: { startSessionId?: number }) => Promise` | Request the on-device recording list. Results arrive via the `fileList` event. | +| `exportAudio` | `(o: { sessionId: number; format?: PlaudAudioFormat; channels?: number }) => Promise<{ sessionId: number; outputPath: string }>` | Decode a recording to a file in `Documents/PlaudExports`. Resolves with the written path; emits `exportProgress` events along the way. `format` defaults to `'mp3'`. Rejects `ERR_PLAUD_ARGS` (bad sessionId) or `ERR_PLAUD_EXPORT`. | + +`PlaudAudioFormat = 'pcm' | 'mp3' | 'wav' | 'opus'`. + +`exportAudio` returns a raw path; prefix with `file://` if not already present before handing +it to `expo-file-system` / `fetch`. + +## Events + +Subscribe with `PlaudSdk.addListener(name, cb)`, which returns `{ remove() }`. Always remove on +unmount. `addListener`/`removeListener`/`removeAllListeners` come from the Expo `NativeModule` +base and are fully typed. + +| Event | Payload | When | +| --- | --- | --- | +| `scanResult` | `{ devices: PlaudScanDevice[] }` | Devices discovered during a scan (may fire repeatedly with a growing list). | +| `scanTimeout` | `{ reason?: string }` | Scan window ended, or BLE never powered on (`reason: 'bluetoothNotPoweredOn'`). | +| `connectState` | `{ connected: boolean; failed: boolean; state: number }` | Connection state changed. `state`: `1`=connected, `0`=disconnected, `{2,-1,-2}`=failure (`failed: true`). | +| `penState` | `PlaudPenState` | Device status snapshot (privacy, key state, uDisk, tokens). | +| `bind` | `{ sn: string \| null; status: number; protVersion: number }` | Bind/pairing handshake result. | +| `fileList` | `{ files: PlaudFile[] }` | Response to `getFileList`. | +| `exportProgress` | `{ sessionId: number; progress: number; message: string }` | Progress during `exportAudio`. | +| `recordStart` | `PlaudRecordStart` | Device-initiated recording started (physical button / VAD). | +| `recordStop` | `PlaudRecordStop` | Device-initiated recording stopped; includes resulting file info. | +| `recordPause` | `PlaudRecordStop` | Recording paused. | +| `recordResume` | `PlaudRecordResume` | Recording resumed. | +| `depair` | `{ status: number }` | Unpair completed — reset all local device state here. | + +`recordStart`/`recordStop`/etc. are **device-initiated** (the user pressed the button on the +recorder). Refresh the file list on `recordStop` to pick up the new recording. + +## Types + +```ts +interface PlaudScanDevice { + name: string; + uuid: string; // CoreBluetooth peripheral id — use this to connect + serialNumber: string; + rssi: number; + supportWiFi: boolean; +} + +interface PlaudConnectState { + connected: boolean; + failed: boolean; // true for handshake failure (state 2/-1/-2), not a normal disconnect + state: number; +} + +interface PlaudPenState { + state: number; privacy: number; keyState: number; uDisk: number; + findMyToken: number; hasSndpKey: number; deviceAccessToken: number; +} + +interface PlaudFile { + sn: string; + sessionId: number; // identifies the recording for exportAudio + size: number; // bytes + scenes: number; + channels: number; + isOgg: boolean; + isMusic: boolean; + duration: number; // seconds +} + +interface PlaudExportProgress { sessionId: number; progress: number; message: string; } + +interface PlaudRecordStart { + sessionId: number; start: number; status: number; + scene: number; startTime: number; reason: number; +} + +interface PlaudRecordStop { + sessionId: number; reason: number; fileExist: boolean; fileSize: number; +} + +interface PlaudRecordResume { + sessionId: number; start: number; status: number; scene: number; startTime: number; +} +``` + +## Canonical scan → connect → list → export flow + +```ts +PlaudSdk.addListener('scanResult', ({ devices }) => setDevices(devices)); +PlaudSdk.addListener('connectState', ({ connected, failed }) => { + if (connected) PlaudSdk.getFileList(); // load recordings on connect + else if (failed) showError('Connection failed — move closer and retry'); +}); +PlaudSdk.addListener('fileList', ({ files }) => setFiles(files)); + +await PlaudSdk.startScan(); +// user picks a device: +await PlaudSdk.connectBleDevice({ uuid: device.uuid }); +// user picks a file: +const { outputPath } = await PlaudSdk.exportAudio({ sessionId: file.sessionId, format: 'mp3' }); +``` + +See `react-native-demo/src/app/index.tsx` for the full stateful version. diff --git a/skills/references/transcription-and-tokens.md b/skills/references/transcription-and-tokens.md new file mode 100644 index 0000000..17f47ec --- /dev/null +++ b/skills/references/transcription-and-tokens.md @@ -0,0 +1,67 @@ +# Tokens and transcription + +Two things the native module does **not** do — both are your app/backend's responsibility. + +## The per-user access token + +`initSDK` requires a **per-user access token** (a Bearer JWT). The SDK does *not* mint it. + +- **Production:** mint it via Plaud's partner OAuth flow on your backend and hand it to the + client. The same token is used both for `initSDK` and for authenticating file uploads. +- **Local testing:** paste one via `EXPO_PUBLIC_PLAUD_ACCESS_TOKEN`. Expo inlines + `EXPO_PUBLIC_*` at build time. + +```ts +async function getUserAccessToken(): Promise { + const token = process.env.EXPO_PUBLIC_PLAUD_ACCESS_TOKEN; + if (!token) throw new Error('No Plaud access token — wire a mint endpoint or set EXPO_PUBLIC_PLAUD_ACCESS_TOKEN.'); + return token; +} +``` + +> ⚠️ `EXPO_PUBLIC_*` vars are inlined into the JS bundle and are **extractable** from the app. +> Fine for a demo build; never ship credentials in the client. + +## Transcription is plain HTTP, not the native module + +Once a recording is exported to a local file (via `PlaudSdk.exportAudio`), uploading and +transcribing it is ordinary HTTP against the Plaud platform API — nothing to do with the native +bridge. In production this belongs **behind a backend** (it needs the partner API key). The +demo does it client-side for convenience only. + +The full working implementation is `react-native-demo/src/lib/plaud-transcription.ts`. The flow: + +1. **Upload** (Bearer *user* token — the same one passed to `initSDK`): + `generate-presigned-urls` → PUT parts to S3 → `complete-upload` → returns a `DownloadUrl`. +2. **Submit** (`X-Client-Id` / `X-Client-Api-Key` partner headers): + `POST /open/partner/ai/transcriptions/` with `{ file_url }`. +3. **Poll** (partner headers): `GET /open/partner/ai/transcriptions/{id}` until the transcript + text is present or it fails/times out. + +Base URL in the demo: `https://platform-us.plaud.ai/developer/api`. + +### Two different credentials — don't mix them up + +| Call | Auth | +| --- | --- | +| `initSDK` + file upload (presigned URLs, complete-upload) | `Authorization: Bearer ` | +| Submit + poll transcription | `X-Client-Id` + `X-Client-Api-Key` (partner credentials) | + +Partner credentials come from the Plaud Developer Portal: +`https://platform.plaud.ai/developer/portal`. + +### RN-specific upload gotcha + +Don't use `file.slice()` to chunk the upload — on React Native it does `new Blob([bytes])` and +RN's Blob polyfill throws "creating blobs from arraybuffer are not supported." Instead read the +file into a `Uint8Array` (`new Uint8Array(await file.arrayBuffer())`) and PUT typed-array +chunks; RN's networking layer base64-encodes typed-array bodies natively. This is already +handled in `plaud-transcription.ts` — copy that file rather than re-deriving it. + +### Env vars (demo) + +``` +EXPO_PUBLIC_PLAUD_ACCESS_TOKEN= # per-user Bearer JWT for initSDK + upload +EXPO_PUBLIC_PLAUD_CLIENT_ID= # partner X-Client-Id (transcription) +EXPO_PUBLIC_PLAUD_API_KEY= # partner X-Client-Api-Key (transcription) +``` diff --git a/skills/references/troubleshooting.md b/skills/references/troubleshooting.md new file mode 100644 index 0000000..db442b7 --- /dev/null +++ b/skills/references/troubleshooting.md @@ -0,0 +1,73 @@ +# Plaud RN module — troubleshooting + +Symptom → cause → fix. Most problems trace back to the device-only constraint or to skipping a +setup step. + +## `isAvailable` is `false` / every method rejects with "native module is unavailable" + +The native module isn't linked and callable in this context. Causes, most common first: + +- **Running on the simulator or Android.** The frameworks are arm64 iOS-device-only — there is + no simulator slice and no Android build. Run on a physical iPhone: + `npx expo run:ios --device`. This is expected behavior off-device, not a bug — the app + should degrade gracefully. +- **Using Expo Go.** This is custom native code; Expo Go can't load it. Use a custom dev build. +- **Module not autolinked.** Confirm `modules/plaud-sdk/` exists at the app root, `package.json` + has `"plaud-sdk": "file:./modules/plaud-sdk"`, you ran `npm install`, then re-ran + `npx expo prebuild -p ios`. `expo-module.config.json` must be present (it registers + `PlaudSdkModule`). + +## App crashes on launch or when scanning + +Missing `NSBluetoothAlwaysUsageDescription`. iOS hard-crashes any Bluetooth access without a +usage-description string. Add it (and `UIBackgroundModes: ["bluetooth-central"]`) under +`expo.ios.infoPlist` in `app.json`, then `npx expo prebuild -p ios` to regenerate. + +## `startScan` resolves but no `scanResult` ever fires + +- **Bluetooth off or permission denied.** The module waits ~18s for CoreBluetooth to power on, + then emits `scanTimeout` with `reason: 'bluetoothNotPoweredOn'`. Handle that event — prompt + the user to enable Bluetooth and grant permission. +- **No device advertising.** The Plaud recorder must be on and in range. `scanResult` fires + repeatedly with a growing device list; give it a few seconds. + +## `connectBleDevice` rejects with `ERR_PLAUD_UNKNOWN_DEVICE` + +You must **scan before you connect** — the native side caches the `BleDevice` objects from +`scanResult` and looks them up by `uuid`/`serialNumber`. Connect using a `uuid` from a device +that appeared in a `scanResult`, in the same session (don't connect to a hardcoded id). + +## `connectState` reports `failed: true` + +Handshake failure (`state` ∈ {2, -1, -2}), distinct from a normal disconnect (`state` 0). +Usually signal/range or a token mismatch. Move the device closer and retry. Confirm `initSDK` +ran with a valid `userAccessToken` and the `userId`/`deviceToken` that binds the device. + +## `initSDK` rejects `ERR_PLAUD_ARGS` + +`userAccessToken` or `customDomain` is empty. `customDomain` must be **domain-only** — +`platform-us.plaud.ai`, not `https://platform-us.plaud.ai`. + +## `exportAudio` never completes / `ERR_PLAUD_EXPORT` + +- Watch `exportProgress` events to see how far it got. +- The output lands in `Documents/PlaudExports`. The resolved `outputPath` may lack the + `file://` scheme — add it before passing to `expo-file-system` / `fetch`. +- Ensure the device stayed connected throughout; a disconnect mid-export aborts it. + +## Pod install / build failures after adding the module + +- Re-run `npx expo prebuild -p ios` (regenerates `ios/` and runs `pod install`). Never + hand-edit `ios/` in an Expo app — it's generated and your edits get wiped. +- Bare RN app managing `ios/` by hand: run `pod install` from `ios/` directly. +- Confirm CocoaPods is installed (`brew install cocoapods`) and Xcode 16+/iOS 15+ deployment + target. The podspec pins `:ios => '15.1'`. +- The three `.xcframework`s must have copied over with the module (they're large binaries under + `modules/plaud-sdk/ios/Frameworks/`). A shallow copy that dropped them breaks the vendored + frameworks link. + +## Transcription upload throws "creating blobs from arraybuffer are not supported" + +RN's Blob polyfill. Don't use `file.slice()`; PUT `Uint8Array` chunks instead. See +`references/transcription-and-tokens.md` — the demo's `plaud-transcription.ts` already does +this correctly.