Skip to content

Commit ac24bbc

Browse files
fhirschmannclaude
andcommitted
feat: HTTP(S) file sync from a manifest URL (e.g. nginx) to SD
Adds a one-way file sync: the device fetches a JSON manifest from a configurable URL and downloads every listed file that is missing locally or differs in size, streaming each in 1 KB chunks straight to the SD card (never buffering whole files). Additive only — local files are never deleted. Optional HTTP Basic Auth (user/password) is sent to the server, so an nginx auth_basic-protected location works. - New Sync module (src/Sync.{h,cpp}) with a background task, reusable Sync_Trigger() and status/progress polling; runs only while nothing is played from SD and pauses RFID/audio/LED tasks during the sync. - Web: /sync GET (status JSON) + POST (trigger); sync config in settings (syncUrl/syncUser/syncPwd in NVS). - New "Sync" settings sub-tab: manifest URL + user + password (with reveal), a "Sync now" button with live progress, and inline docs that explain how it works and the exact manifest format. i18n de/en/fr. Manifest format: {"version":1,"files":[{"path":"dir/file.mp3","size":123},…]} where path is relative to the manifest's directory (download URL = manifest dir + path, also the SD target path) and size drives the up-to-date check. README updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 5072fc7 commit ac24bbc

8 files changed

Lines changed: 575 additions & 0 deletions

File tree

README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -132,6 +132,7 @@ neon logo that doubles as the SVG favicon ([`7be5254`](../../commit/7be5254)):
132132
| Consolidated **Settings** tab (UniFi-style): WiFi, MQTT, FTP and Bluetooth settings merged with the general settings into one top tab with centered horizontal sub-tabs (General / Buttons / WiFi / MQTT / FTP / Bluetooth), the button-assignment config moved into its own **Buttons** sub-tab, plus an FTP start/stop button in the control tab; also fixes the sleep-timer dropdown staying open on outside-click and the desktop control-tab volume slider rendering at zero width | [`8b01876`](../../commit/8b01876) |
133133
| Battery-backed **RTC** (DS3231): optional `RTC_ENABLE` adds a real-time clock on the external I²C bus so the time stays correct without WiFi/NTP. The RTC stores UTC and seeds the system clock at boot (before WiFi is up); NTP remains the master and disciplines the RTC on every sync and hourly. Current time (and DS3231 die-temperature) shown in the info dialog, published via the MQTT state-topic `rtc` (+ Home Assistant discovery), and the RTC can be re-disciplined from system time via the bindable command **155** (button/RFID modifier) | [`bf90f66`](../../commit/bf90f66) |
134134
| **Status control-LEDs**: the optional control LEDs (`NUM_CONTROL_LEDS`, soldered to the end of the same NeoPixel data line — no extra GPIO) can now mirror a runtime state instead of just showing a static color. Each control LED gets a per-slot function in the LED settings — **Static** (legacy fixed color), **Key lock** (red while controls are locked), **Repeat mode** (green = track, blue = playlist, purple = both), **Bluetooth** (violet = waiting, blue = connected), **Battery level** (state of charge as a green→yellow→red gradient) or **Off**. A master switch mutes all control LEDs (e.g. at night), toggleable via the bindable command **121** (button/RFID modifier) and the MQTT command/state-topic `control_leds` (`ON`/`OFF`, + Home Assistant discovery switch) | [`6712ae1`](../../commit/6712ae1) |
135+
| **HTTP file sync**: pull audio files from a web server (e.g. nginx) onto the SD card. A new **Sync** settings sub-tab takes a manifest URL plus optional HTTP Basic Auth (user/password); the device fetches a JSON manifest (`{"files":[{"path","size"},…]}`) and downloads every listed file that is missing locally or differs in size, streaming each in small chunks straight to SD. Additive only (never deletes), runs in a background task, refuses to start while audio is playing, and shows live progress. The Sync tab documents the manifest format and an nginx setup inline | [`f011377`](../../commit/f011377) |
135136

136137
#### Feature highlights
137138

html/locales/de.json

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
"wifi": "WLAN",
3535
"mqtt": "MQTT",
3636
"ftp": "FTP",
37+
"sync": "Sync",
3738
"bluetooth": "Bluetooth",
3839
"general": "Allgemein",
3940
"settings": "Einstellungen",
@@ -395,6 +396,40 @@
395396
"stopButton": "FTP-Server stoppen"
396397
}
397398
},
399+
"sync": {
400+
"title": "Datei-Synchronisation",
401+
"intro": "Lädt Dateien von einem Webserver (z.B. nginx) auf die SD-Karte. Der ESPuino holt ein <b>JSON-Manifest</b> von der angegebenen URL und lädt alle darin gelisteten Dateien, die lokal fehlen oder eine andere Größe haben. Es wird <b>nur hinzugefügt</b> – lokale Dateien werden nie gelöscht. Die Synchronisation läuft nur, wenn gerade nichts von der SD-Karte abgespielt wird.",
402+
"url": {
403+
"title": "Manifest-URL",
404+
"placeholder": "https://server/audio/manifest.json"
405+
},
406+
"user": {
407+
"title": "Benutzername (optional)",
408+
"placeholder": "leer lassen, wenn kein Login"
409+
},
410+
"pwd": {
411+
"title": "Passwort (optional)",
412+
"placeholder": "Passwort",
413+
"show": "Passwort anzeigen/verbergen"
414+
},
415+
"pwd.hint": "Benutzername und Passwort werden per HTTP Basic Auth an den Server gesendet (z.B. nginx auth_basic).",
416+
"run": {
417+
"title": "Jetzt synchronisieren",
418+
"desc": "Holt das Manifest und lädt fehlende/geänderte Dateien herunter. Je nach Datenmenge kann das dauern.",
419+
"button": "Jetzt synchronisieren"
420+
},
421+
"manifest": {
422+
"title": "Aufbau des Manifests",
423+
"desc": "Das Manifest ist eine JSON-Datei, die alle zu synchronisierenden Dateien auflistet. Lege sie auf dem Webserver ab (die URL oben zeigt darauf):",
424+
"fields": "<b>path</b>: Pfad relativ zum Verzeichnis des Manifests – wird sowohl als Download-URL (Manifest-Verzeichnis + path) als auch als Zielpfad auf der SD-Karte verwendet. <b>size</b>: Dateigröße in Bytes – dient zum Erkennen, ob die lokale Datei aktuell ist.",
425+
"nginx": "Beispiel nginx: lege Audiodateien und <code>manifest.json</code> in ein Verzeichnis, das per <code>location</code> ausgeliefert wird; für Login <code>auth_basic</code> + <code>auth_basic_user_file</code> setzen."
426+
},
427+
"status": {
428+
"syncing": "Synchronisiere… {{percent}}%",
429+
"done": "Synchronisation abgeschlossen",
430+
"failed": "Synchronisation fehlgeschlagen"
431+
}
432+
},
398433
"bt": {
399434
"sink": {
400435
"title": "ESPuino als Bluetooth-Lautsprecher",

html/locales/en.json

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
"wifi": "WiFi",
3535
"mqtt": "MQTT",
3636
"ftp": "FTP",
37+
"sync": "Sync",
3738
"bluetooth": "Bluetooth",
3839
"general": "General",
3940
"settings": "Settings",
@@ -396,6 +397,40 @@
396397
"stopButton": "Stop FTP server"
397398
}
398399
},
400+
"sync": {
401+
"title": "File synchronization",
402+
"intro": "Downloads files from a web server (e.g. nginx) onto the SD card. The ESPuino fetches a <b>JSON manifest</b> from the URL below and downloads every file listed in it that is missing locally or has a different size. It is <b>additive only</b> – local files are never deleted. Syncing only runs while nothing is being played from the SD card.",
403+
"url": {
404+
"title": "Manifest URL",
405+
"placeholder": "https://server/audio/manifest.json"
406+
},
407+
"user": {
408+
"title": "Username (optional)",
409+
"placeholder": "leave empty if no login"
410+
},
411+
"pwd": {
412+
"title": "Password (optional)",
413+
"placeholder": "Password",
414+
"show": "Show/hide password"
415+
},
416+
"pwd.hint": "Username and password are sent to the server via HTTP Basic Auth (e.g. nginx auth_basic).",
417+
"run": {
418+
"title": "Sync now",
419+
"desc": "Fetches the manifest and downloads missing/changed files. May take a while depending on the amount of data.",
420+
"button": "Sync now"
421+
},
422+
"manifest": {
423+
"title": "Manifest format",
424+
"desc": "The manifest is a JSON file listing all files to sync. Place it on the web server (the URL above points to it):",
425+
"fields": "<b>path</b>: path relative to the manifest's directory – used both as the download URL (manifest directory + path) and as the target path on the SD card. <b>size</b>: file size in bytes – used to detect whether the local file is up to date.",
426+
"nginx": "nginx example: put the audio files and <code>manifest.json</code> into a directory served via a <code>location</code> block; for a login add <code>auth_basic</code> + <code>auth_basic_user_file</code>."
427+
},
428+
"status": {
429+
"syncing": "Syncing… {{percent}}%",
430+
"done": "Synchronization complete",
431+
"failed": "Synchronization failed"
432+
}
433+
},
399434
"bt": {
400435
"sink": {
401436
"title": "ESPuino as Bluetooth speaker",

html/locales/fr.json

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
"wifi": "WiFi",
3535
"mqtt": "MQTT",
3636
"ftp": "FTP",
37+
"sync": "Sync",
3738
"bluetooth": "Bluetooth",
3839
"general": "Général",
3940
"settings": "Paramètres",
@@ -388,6 +389,40 @@
388389
"stopButton": "Arrêter le serveur FTP"
389390
}
390391
},
392+
"sync": {
393+
"title": "Synchronisation de fichiers",
394+
"intro": "Télécharge des fichiers depuis un serveur web (p.ex. nginx) vers la carte SD. L'ESPuino récupère un <b>manifeste JSON</b> à l'URL ci-dessous et télécharge tous les fichiers listés qui manquent localement ou ont une taille différente. C'est <b>uniquement additif</b> – aucun fichier local n'est supprimé. La synchronisation ne s'exécute que lorsqu'aucune lecture n'est en cours depuis la carte SD.",
395+
"url": {
396+
"title": "URL du manifeste",
397+
"placeholder": "https://server/audio/manifest.json"
398+
},
399+
"user": {
400+
"title": "Nom d'utilisateur (optionnel)",
401+
"placeholder": "laisser vide si pas de connexion"
402+
},
403+
"pwd": {
404+
"title": "Mot de passe (optionnel)",
405+
"placeholder": "Mot de passe",
406+
"show": "Afficher/masquer le mot de passe"
407+
},
408+
"pwd.hint": "Le nom d'utilisateur et le mot de passe sont envoyés au serveur via HTTP Basic Auth (p.ex. nginx auth_basic).",
409+
"run": {
410+
"title": "Synchroniser maintenant",
411+
"desc": "Récupère le manifeste et télécharge les fichiers manquants/modifiés. Peut prendre du temps selon le volume de données.",
412+
"button": "Synchroniser maintenant"
413+
},
414+
"manifest": {
415+
"title": "Format du manifeste",
416+
"desc": "Le manifeste est un fichier JSON listant tous les fichiers à synchroniser. Placez-le sur le serveur web (l'URL ci-dessus pointe dessus) :",
417+
"fields": "<b>path</b> : chemin relatif au répertoire du manifeste – utilisé à la fois comme URL de téléchargement (répertoire du manifeste + path) et comme chemin cible sur la carte SD. <b>size</b> : taille du fichier en octets – utilisée pour détecter si le fichier local est à jour.",
418+
"nginx": "Exemple nginx : placez les fichiers audio et <code>manifest.json</code> dans un répertoire servi via un bloc <code>location</code> ; pour une connexion, ajoutez <code>auth_basic</code> + <code>auth_basic_user_file</code>."
419+
},
420+
"status": {
421+
"syncing": "Synchronisation… {{percent}}%",
422+
"done": "Synchronisation terminée",
423+
"failed": "Échec de la synchronisation"
424+
}
425+
},
391426
"bt": {
392427
"sink": {
393428
"title": "ESPuino en tant que haut-parleur Bluetooth",

html/management.html

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -859,6 +859,17 @@
859859
border-color: var(--cp-cyan);
860860
}
861861

862+
/* monospace code block used for inline documentation (e.g. the sync manifest example) */
863+
.cp-code-block {
864+
border: 1px solid var(--cp-border);
865+
background: rgba(5, 7, 13, .6);
866+
color: var(--cp-cyan);
867+
padding: .75rem 1rem;
868+
border-radius: 4px;
869+
overflow-x: auto;
870+
font-size: .85rem;
871+
}
872+
862873
#track {
863874
font-family: "Share Tech Mono", monospace;
864875
color: var(--cp-cyan);
@@ -2001,6 +2012,7 @@ <h5 class="modal-title" data-i18n="tools.nvs.erase.title"></h5>
20012012
<a class="nav-item nav-link" id="nav-wifi-tab" data-bs-toggle="tab" data-bs-target="#nav-wifi" href="#nav-wifi" role="tab" aria-controls="nav-wifi"><i class="fas fa-wifi"></i> <span class="d-none d-md-inline" data-i18n="nav.wifi"></span></a>
20022013
<a class="nav-item nav-link" id="nav-mqtt-tab" data-visible="false" data-bs-toggle="tab" data-bs-target="#nav-mqtt" href="#nav-mqtt" role="tab" aria-controls="nav-mqtt"><i class="fas fa-network-wired"></i> <span class="d-none d-md-inline" data-i18n="nav.mqtt"></span></a>
20032014
<a class="nav-item nav-link" id="nav-ftp-tab" data-visible="false" data-bs-toggle="tab" data-bs-target="#nav-ftp" href="#nav-ftp" role="tab" aria-controls="nav-ftp"><i class="fas fa-folder"></i> <span class="d-none d-md-inline" data-i18n="nav.ftp"></span></a>
2015+
<a class="nav-item nav-link" id="nav-sync-tab" data-bs-toggle="tab" data-bs-target="#nav-sync" href="#nav-sync" role="tab" aria-controls="nav-sync"><i class="fas fa-sync"></i> <span class="d-none d-md-inline" data-i18n="nav.sync"></span></a>
20042016
<a class="nav-item nav-link" id="nav-bt-tab" data-visible="false" data-bs-toggle="tab" data-bs-target="#nav-bt" href="#nav-bt" role="tab" aria-controls="nav-bt"><i class="fab fa-bluetooth"></i> <span class="d-none d-md-inline" data-i18n="nav.bluetooth"></span></a>
20052017
</div>
20062018
</nav>
@@ -2733,6 +2745,67 @@ <h5 class="modal-title" data-i18n="tools.nvs.erase.title"></h5>
27332745
</div>
27342746
</div>
27352747
</div>
2748+
<div class="tab-pane fade" id="nav-sync" role="tabpanel" aria-labelledby="nav-sync-tab">
2749+
<div class="container" id="syncConfig">
2750+
<form action="#syncConfig" method="POST" onsubmit="syncSettings(); return false">
2751+
<div class="mb-3 col-md-12">
2752+
<legend data-i18n="sync.title"></legend>
2753+
<p data-i18n="[html]sync.intro"></p>
2754+
<label for="syncUrl" data-i18n="[prepend]sync.url.title">:</label>
2755+
<input type="url" class="form-control" id="syncUrl" data-i18n="[placeholder]sync.url.placeholder"
2756+
name="syncUrl" value="">
2757+
<label for="syncUser" data-i18n="[prepend]sync.user.title">:</label>
2758+
<input type="text" class="form-control" id="syncUser"
2759+
data-i18n="[placeholder]sync.user.placeholder" name="syncUser" value="" autocomplete="off">
2760+
<label for="syncPwd" data-i18n="[prepend]sync.pwd.title">:</label>
2761+
<div class="input-group">
2762+
<input type="password" class="form-control" id="syncPwd"
2763+
data-i18n="[placeholder]sync.pwd.placeholder" name="syncPwd" value="" autocomplete="off">
2764+
<button class="btn cp-reveal-btn" type="button" id="syncPwdToggle" tabindex="-1"
2765+
data-i18n="[title]sync.pwd.show" onclick="togglePasswordField('syncPwd', 'syncPwdToggle')">
2766+
<i class="fas fa-eye"></i>
2767+
</button>
2768+
</div>
2769+
<small class="text-muted" data-i18n="sync.pwd.hint"></small>
2770+
</div>
2771+
<div class="text-center">
2772+
<button type="reset" class="btn btn-warning" data-i18n="reset"></button>&nbsp;<button
2773+
type="submit" class="btn btn-primary" data-i18n="submit"></button>
2774+
</div>
2775+
</form>
2776+
<hr>
2777+
<div class="col-md-12">
2778+
<legend data-i18n="sync.run.title"></legend>
2779+
<div data-i18n="sync.run.desc"></div>
2780+
<br>
2781+
<div id="syncProgress" class="progress upload-progress-bar position-relative">
2782+
<div class="progress-bar" role="progressbar"></div>
2783+
<div
2784+
class="label d-flex position-absolute flex-column align-self-center text-center w-100 text-nowrap text-light">
2785+
</div>
2786+
</div>
2787+
<br>
2788+
<div class="text-center">
2789+
<button type="button" class="btn btn-primary" id="syncRunBtn" onclick="startSync()"
2790+
data-i18n="sync.run.button"></button>
2791+
</div>
2792+
</div>
2793+
<hr>
2794+
<div class="col-md-12">
2795+
<legend data-i18n="sync.manifest.title"></legend>
2796+
<p data-i18n="[html]sync.manifest.desc"></p>
2797+
<pre class="cp-code-block"><code>{
2798+
"version": 1,
2799+
"files": [
2800+
{ "path": "Pumuckl/01.mp3", "size": 4823104 },
2801+
{ "path": "Pumuckl/02.mp3", "size": 5012992 }
2802+
]
2803+
}</code></pre>
2804+
<p data-i18n="[html]sync.manifest.fields"></p>
2805+
<p data-i18n="[html]sync.manifest.nginx"></p>
2806+
</div>
2807+
</div>
2808+
</div>
27362809
<div class="tab-pane fade" id="nav-bt" role="tabpanel" aria-labelledby="nav-bt-tab">
27372810
<div class="container" id="btConfig">
27382811
<form action="#btConfig" method="POST" onsubmit="btSettings('btConfig'); return false">
@@ -4465,6 +4538,13 @@ <h5 class="modal-title" data-i18n="tools.nvs.erase.title"></h5>
44654538
document.getElementById("ftpPwd").value = (ftpSettings.password === "-1") ? "" : ftpSettings.password;
44664539
document.getElementById("ftpPwd").setAttribute('maxlength', ftpSettings.maxPwdLength);
44674540
}
4541+
// http file sync
4542+
let syncSettingsData = settings.sync;
4543+
if (syncSettingsData) {
4544+
document.getElementById("syncUrl").value = syncSettingsData.url || "";
4545+
document.getElementById("syncUser").value = syncSettingsData.username || "";
4546+
document.getElementById("syncPwd").value = syncSettingsData.password || "";
4547+
}
44684548
// mqtt
44694549
let mqttSettings = settings.mqtt;
44704550
if (mqttSettings) {
@@ -4928,6 +5008,68 @@ <h5 class="modal-title" data-i18n="tools.nvs.erase.title"></h5>
49285008
socket.send(myJSON);
49295009
}
49305010

5011+
/* save the HTTP file-sync configuration (manifest URL + optional Basic Auth) */
5012+
async function syncSettings() {
5013+
var myObj = {
5014+
"sync": {
5015+
url: document.getElementById('syncUrl').value,
5016+
username: document.getElementById('syncUser').value,
5017+
password: document.getElementById('syncPwd').value
5018+
}
5019+
};
5020+
await fetch("http://" + host + "/settings", {
5021+
method: "POST",
5022+
headers: { "Content-Type": "application/json" },
5023+
body: JSON.stringify(myObj)
5024+
});
5025+
toaster.info(i18next.t("toast.saved"));
5026+
}
5027+
5028+
let syncPollTimer = null;
5029+
function renderSyncProgress(status, progress, message) {
5030+
const bar = document.querySelector('#syncProgress .progress-bar');
5031+
const label = document.querySelector('#syncProgress .label');
5032+
if (bar) bar.style.width = progress + '%';
5033+
let text = '';
5034+
if (status === 1) text = i18next.t('sync.status.syncing', { percent: progress });
5035+
else if (status === 2) text = i18next.t('sync.status.done') + (message ? ' (' + message + ')' : '');
5036+
else if (status === 3) text = i18next.t('sync.status.failed') + (message ? ': ' + message : '');
5037+
if (label) label.textContent = text;
5038+
}
5039+
5040+
async function pollSync() {
5041+
try {
5042+
const r = await fetch("http://" + host + "/sync");
5043+
const d = await r.json();
5044+
renderSyncProgress(d.status, d.progress, d.message);
5045+
if (d.status === 1) return; // keep polling
5046+
// terminal state reached
5047+
clearInterval(syncPollTimer);
5048+
syncPollTimer = null;
5049+
document.getElementById('syncRunBtn').disabled = false;
5050+
if (d.status === 2) toaster.info(i18next.t('sync.status.done'));
5051+
else if (d.status === 3) toaster.error(i18next.t('sync.status.failed') + (d.message ? ': ' + d.message : ''));
5052+
} catch (e) {
5053+
console.warn('sync poll failed', e);
5054+
}
5055+
}
5056+
5057+
/* kick off a sync and poll the device for progress */
5058+
async function startSync() {
5059+
const btn = document.getElementById('syncRunBtn');
5060+
btn.disabled = true;
5061+
renderSyncProgress(1, 0, '');
5062+
try {
5063+
await fetch("http://" + host + "/sync", { method: "POST" });
5064+
} catch (e) {
5065+
btn.disabled = false;
5066+
toaster.error(i18next.t('toast.error') + ': ' + e.message);
5067+
return;
5068+
}
5069+
if (syncPollTimer) clearInterval(syncPollTimer);
5070+
syncPollTimer = setInterval(pollSync, 1500);
5071+
}
5072+
49315073
/* Turns the FTP start button into a stop button while the server is running */
49325074
function updateFtpButton(running) {
49335075
const btn = document.getElementById('ftpModeBtn');

0 commit comments

Comments
 (0)