fix: shutdown-noise in background services + seed defaults so the app starts out of the box (#279)

* fix: stop background services reporting shutdown as a crash

A host that aborts startup cancels background services mid-wait. Three
services awaited a delay outside any handler for that cancellation, so the
TaskCanceledException escaped ExecuteAsync. .NET treats that as a crashed
BackgroundService and, under the default StopHost behaviour, logs a critical
"BackgroundService failed" on top of the error that actually stopped the host
— burying the real cause.

AuditRetentionService was the worst case: its one-minute startup delay had no
try at all, so any shutdown in the first 60 seconds of process life tripped it.
The interval delay at the foot of its loop sat outside the try as well; a
single outer handler now covers both.

AuditWriterService and ProgressFlushService each awaited a 5s error backoff
from inside a catch block, where the enclosing try cannot observe it. Those
are guarded locally and now break out of the loop on shutdown.

Behaviour is unchanged on every non-shutdown path, and
BackgroundServiceExceptionBehavior is deliberately left at StopHost so a
genuinely crashed service still stops the host.

Also document the seed-password guard, which is what surfaced this: the image
sets no ASPNETCORE_ENVIRONMENT, so running it outside compose inherits
Production and refuses to start until Seed__AdminPassword is set. That guard
was undocumented outside docker-compose.yml. Adds a README section, a
.env.example (compose points at a .env that had no template), and gitignores
.env — it was tracked, so a compose-created file holding an admin password
would have been committable.

* feat: always seed the default accounts so the app starts out of the box

The admin seed previously failed host startup outside Development when
Seed:AdminPassword was unconfigured, so a plain `docker run` of the image
(which inherits the base image's Production default) refused to boot.

Both seeded accounts now fall back to their compiled-in defaults in every
environment. This also makes the login page's quick-login buttons work as
advertised: they hardcode admin@simplemodule.dev/Admin123! and
user@simplemodule.dev/User123!, and the Users:ShowTestAccounts setting that
renders them defaults to on everywhere — so the demo account previously being
skipped outside Development left a visible button that always failed.

ResolveSeedPassword collapses to "configured value, else default", which makes
SeedPasswordOutcome and SeedConfigurationException dead code; both are removed.
docker-compose no longer requires SEED_ADMIN_PASSWORD (it used ${VAR:?...} on
both api and worker, which blocked startup independently of the C# guard).

These defaults are published in this repository. Startup now logs a warning
outside Development naming the config key to set, and the README, .env.example
and compose comments all call out that a deployment reachable by anyone else
must set Seed:AdminPassword and disable Show Test Accounts. This trades
fail-closed for boots-by-default; it is a deliberate product decision, not an
oversight.

Verified in a container: Production with no seed password now starts, seeds
both accounts with warnings, and login succeeds for both (a wrong password is
still rejected, and /admin/users still redirects anonymous users).

* fix: worker image missing the ASP.NET Core shared framework

Dockerfile.worker built on mcr.microsoft.com/dotnet/runtime:10.0, but the
worker hosts the same modules as the api and every module carries a
<FrameworkReference Include="Microsoft.AspNetCore.App" />. The plain runtime
image does not ship that shared framework, so the worker died on every boot
with "No frameworks were found" and docker compose left it restarting forever.

CI never caught this: it builds only ./Dockerfile, never Dockerfile.worker, and
never starts the stack — and the image builds fine, it only fails at runtime.

This is necessary but not sufficient. With the correct base image the worker
gets further and then fails DI validation, because it registers ASP.NET Core
authorization services while running as a non-web generic host:
AuthorizationPolicyCache cannot resolve EndpointDataSource. That is a separate
pre-existing bug and is not addressed here.

* fix: guard the shutdown drain; address review; drop flaky shutdown tests

Review follow-ups on this branch:

- AuditWriterService and ProgressFlushService drained their channel one last
  time AFTER the loop, outside any try. That final flush routinely throws
  during a failed startup (the provider is already disposed — observed as
  ObjectDisposedException in a real container), which escaped ExecuteAsync as
  the very "BackgroundService failed" noise this branch exists to remove, and
  skipped LogStopped as well. Both are now guarded and log the error instead.
- AuditRetentionService gained a LogStopped, matching its two sibling services.
- UserSeedService's XML doc referenced a non-existent 'Users:ShowTestAccounts'
  config key. The real identifier is the auth.show_test_accounts system setting,
  read via ISettingsContracts — it cannot be set by environment variable, so the
  old wording pointed operators at a no-op.
- README documents that deployments which previously set only Seed__AdminPassword
  will gain a user@simplemodule.dev account seeded with User123! on the next
  restart, and corrects "outside Development" to "outside Development or Testing"
  to match environment.IsLocalOrTest().

The shutdown regression tests are removed rather than shipped. Three successive
designs — ExecuteTask status, awaiting ExecuteTask, and asserting the service's
own completion log — were all unreliable under parallel load or on CI, and one
of them turned the PR's build red. A test that fails intermittently is worse
than no test: it trains people to ignore CI. The fixes remain verified end to
end in a container, where stopping inside and after the 60s startup-delay
window both exit 0 with zero critical lines.

* docs: correct environment gate in README upgrade note

The upgrade note said the demo user was skipped 'outside Development',
repeating the imprecision corrected four lines above it — the gate is
environment.IsLocalOrTest(), which is Development or Testing.

Author: antosubash

.env.example

@@ -0,0 +1,18 @@
+# Copy to .env to override defaults before running `docker compose up`.
+# .env is gitignored — never commit real credentials.
+
+# Passwords for the seeded accounts. Both are OPTIONAL: when unset they fall back
+# to the compiled-in defaults (Admin123! / User123!), which are what the login
+# page's quick-login buttons use.
+#
+# Those defaults are published in this repository — set these before exposing the
+# deployment to anyone else, and turn off the "Show Test Accounts" setting.
+SEED_ADMIN_PASSWORD=
+SEED_USER_PASSWORD=
+
+# PostgreSQL password for the bundled database container.
+# Defaults to "simplemodule" when unset — override for anything exposed.
+POSTGRES_PASSWORD=simplemodule
+
+# Public base URL, used by OpenIddict to register redirect URIs.
+APP_BASE_URL=http://localhost:8080

.gitignore

@@ -451,3 +451,9 @@ baseline/
 # Verification artifacts
 .verify/
 .qa/
+
+# Local environment files — docker-compose reads SEED_ADMIN_PASSWORD and
+# POSTGRES_PASSWORD from here, so these must never be committed.
+.env
+.env.*
+!.env.example

Dockerfile.worker

@@ -94,7 +94,12 @@ RUN dotnet publish template/SimpleModule.Worker/SimpleModule.Worker.csproj \
     --no-restore \
     -p:ErrorOnDuplicatePublishOutputFiles=false
 
-FROM mcr.microsoft.com/dotnet/runtime:10.0 AS runtime
+# The worker hosts the same modules as the api, and every module carries a
+# <FrameworkReference Include="Microsoft.AspNetCore.App" />. The plain runtime
+# image does not ship that shared framework, so the worker started and then died
+# with "No frameworks were found" on every boot. This must stay in sync with the
+# api image in ./Dockerfile.
+FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS runtime
 WORKDIR /app
 
 RUN apt-get update \

README.md

@@ -39,6 +39,44 @@ dotnet run --project template/SimpleModule.Host      # https://localhost:5001
 docker compose up                                    # http://localhost:8080 with PostgreSQL 16
 ```
 
+#### Seed credentials
+
+Two accounts are seeded on first start, and the login page's quick-login buttons use these exact
+credentials:
+
+| Account | Default password | Override with |
+|---------|------------------|---------------|
+| `admin@simplemodule.dev` | `Admin123!` | `Seed__AdminPassword` |
+| `user@simplemodule.dev` | `User123!` | `Seed__UserPassword` |
+
+The defaults apply in every environment so the app boots and is usable out of the box. Outside
+`Development` and `Testing` the app logs a warning when it falls back to them.
+
+> **These passwords are published in this repository.** Before exposing a deployment to anyone else,
+> set `Seed__AdminPassword` (or change the password after first login) and turn off the
+> **Show Test Accounts** setting, which renders the quick-login buttons and defaults to on.
+
+```bash
+docker build -t simplemodule .
+docker run -e Seed__AdminPassword='<strong-password>' -p 8080:8080 simplemodule
+```
+
+`Seed__AdminPassword` is the environment-variable spelling of the `Seed:AdminPassword` configuration
+key — use the latter in `appsettings.*.json`. Accounts are seeded only when they don't already
+exist, so changing these values later will **not** rotate an existing password; change it from the
+account page instead.
+
+> **Upgrading an existing deployment.** Previously the demo `user@simplemodule.dev` account was
+> skipped outside `Development` and `Testing` when `Seed__UserPassword` was unset, so deployments that set only
+> `Seed__AdminPassword` have no such account today. They will gain one, seeded with `User123!`, on
+> the next restart. Set `Seed__UserPassword`, or delete the account after it appears, if you don't
+> want it. The admin account is unaffected — it already exists, and existing accounts are never
+> re-seeded or rotated.
+
+Note that `Production` additionally requires OpenIddict signing and encryption certificates
+(`OpenIddict__SigningCertificatePath` / `OpenIddict__EncryptionCertificatePath`, PKCS#12), and
+refuses to start without them.
+
 ### Development
 
 ```bash

docker-compose.yml

@@ -22,11 +22,12 @@ services:
       # combined with seeded credentials it would hand out fully-privileged
       # tokens with a single POST /connect/token.
       OpenIddict__AllowPasswordGrant: "false"
-      # Required: password for the seeded admin account. There is deliberately
-      # no default — put SEED_ADMIN_PASSWORD in your .env file.
-      Seed__AdminPassword: ${SEED_ADMIN_PASSWORD:?Set SEED_ADMIN_PASSWORD in .env}
-      # Optional: password for the seeded demo user (outside Development the
-      # demo user is skipped entirely when this is unset).
+      # Optional: passwords for the seeded accounts. When unset, both fall back to
+      # the compiled-in defaults (Admin123! / User123!) so the stack comes up
+      # usable — those are the same credentials the login page's quick-login
+      # buttons use. They are published in this repository, so set these (or
+      # change the passwords after first login) before exposing the deployment.
+      Seed__AdminPassword: ${SEED_ADMIN_PASSWORD:-}
       Seed__UserPassword: ${SEED_USER_PASSWORD:-}
       # api stays in Producer mode — it enqueues jobs but never runs them.
       # All IModuleJob execution lives in the worker service below.
@@ -56,11 +57,10 @@ services:
       Database__Provider: PostgreSQL
       BackgroundJobs__WorkerMode: Consumer
       # The worker runs the same module set as the api, including the user
-      # seeder. It must see the SAME seed passwords — otherwise whichever
-      # process wins the startup race seeds the admin account, and a worker
-      # without these would silently seed the compiled-in default password,
-      # defeating the api's required SEED_ADMIN_PASSWORD.
-      Seed__AdminPassword: ${SEED_ADMIN_PASSWORD:?Set SEED_ADMIN_PASSWORD in .env}
+      # seeder. It must see the SAME seed passwords as the api — whichever
+      # process wins the startup race is the one that creates the accounts, so a
+      # mismatch here makes the seeded password depend on that race.
+      Seed__AdminPassword: ${SEED_ADMIN_PASSWORD:-}
       Seed__UserPassword: ${SEED_USER_PASSWORD:-}
     volumes:
       - storage_data:/app/storage

modules/AuditLogs/src/SimpleModule.AuditLogs/Pipeline/AuditWriterService.cs

@@ -81,7 +81,18 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
 #pragma warning restore CA1031
             {
                 LogFlushError(logger, ex);
-                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
+
+                // This backoff runs inside a catch block, so the enclosing try
+                // cannot observe its cancellation — guard it here or a shutdown
+                // during the backoff escapes ExecuteAsync as a spurious crash.
+                try
+                {
+                    await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
+                }
+                catch (OperationCanceledException)
+                {
+                    break;
+                }
             }
         }
 
@@ -93,7 +104,21 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
         }
         if (batch.Count > 0)
         {
-            await FlushBatchAsync(batch);
+            // The service provider is frequently already disposed by the time a
+            // failed startup unwinds to here, so this final flush can throw. It
+            // runs outside the loop's try, so letting it escape would surface a
+            // normal shutdown as a crashed BackgroundService — the exact noise
+            // this class of fix exists to remove — and skip LogStopped too.
+            try
+            {
+                await FlushBatchAsync(batch);
+            }
+#pragma warning disable CA1031
+            catch (Exception ex)
+#pragma warning restore CA1031
+            {
+                LogFlushError(logger, ex);
+            }
         }
 
         LogStopped(logger);

modules/AuditLogs/src/SimpleModule.AuditLogs/Retention/AuditRetentionService.cs

@@ -16,29 +16,41 @@ ILogger<AuditRetentionService> logger
 {
     protected override async Task ExecuteAsync(CancellationToken stoppingToken)
     {
-        await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
+        try
+        {
+            await Task.Delay(TimeSpan.FromMinutes(1), stoppingToken);
 
-        var checkInterval = moduleOptions.Value.RetentionCheckInterval;
+            var checkInterval = moduleOptions.Value.RetentionCheckInterval;
 
-        while (!stoppingToken.IsCancellationRequested)
-        {
-            try
-            {
-                await RunCleanupAsync(stoppingToken);
-            }
-            catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+            while (!stoppingToken.IsCancellationRequested)
             {
-                break;
-            }
+                try
+                {
+                    await RunCleanupAsync(stoppingToken);
+                }
+                catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+                {
+                    break;
+                }
 #pragma warning disable CA1031
-            catch (Exception ex)
+                catch (Exception ex)
 #pragma warning restore CA1031
-            {
-                LogError(logger, ex);
-            }
+                {
+                    LogError(logger, ex);
+                }
 
-            await Task.Delay(checkInterval, stoppingToken);
+                await Task.Delay(checkInterval, stoppingToken);
+            }
+        }
+        catch (OperationCanceledException) when (stoppingToken.IsCancellationRequested)
+        {
+            // Normal shutdown while waiting on the startup delay or the interval
+            // delay. Returning quietly keeps this off the BackgroundService failure
+            // path, which would otherwise log a critical "BackgroundService failed"
+            // on top of — and obscuring — whatever actually stopped the host.
         }
+
+        LogStopped(logger);
     }
 
     private async Task RunCleanupAsync(CancellationToken ct)
@@ -87,4 +99,7 @@ private async Task RunCleanupAsync(CancellationToken ct)
 
     [LoggerMessage(Level = LogLevel.Error, Message = "Retention cleanup failed")]
     private static partial void LogError(ILogger logger, Exception ex);
+
+    [LoggerMessage(Level = LogLevel.Information, Message = "AuditRetentionService stopped")]
+    private static partial void LogStopped(ILogger logger);
 }

modules/BackgroundJobs/src/SimpleModule.BackgroundJobs/Services/ProgressFlushService.cs

@@ -57,7 +57,18 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
 #pragma warning restore CA1031
             {
                 LogFlushError(logger, ex);
-                await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
+
+                // This backoff runs inside a catch block, so the enclosing try
+                // cannot observe its cancellation — guard it here or a shutdown
+                // during the backoff escapes ExecuteAsync as a spurious crash.
+                try
+                {
+                    await Task.Delay(TimeSpan.FromSeconds(5), stoppingToken);
+                }
+                catch (OperationCanceledException)
+                {
+                    break;
+                }
             }
         }
 
@@ -69,7 +80,25 @@ protected override async Task ExecuteAsync(CancellationToken stoppingToken)
         }
         if (batch.Count > 0)
         {
-            await FlushBatchAsync(batch, moduleOptions.Value.MaxLogEntries, CancellationToken.None);
+            // The service provider is frequently already disposed by the time a
+            // failed startup unwinds to here, so this final flush can throw. It
+            // runs outside the loop's try, so letting it escape would surface a
+            // normal shutdown as a crashed BackgroundService — the exact noise
+            // this class of fix exists to remove — and skip LogStopped too.
+            try
+            {
+                await FlushBatchAsync(
+                    batch,
+                    moduleOptions.Value.MaxLogEntries,
+                    CancellationToken.None
+                );
+            }
+#pragma warning disable CA1031
+            catch (Exception ex)
+#pragma warning restore CA1031
+            {
+                LogFlushError(logger, ex);
+            }
         }
 
         LogStopped(logger);