Skip to content

update windows prefect services #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LOCEANlloydizard wants to merge 2 commits into
base: main
Choose a base branch
from
Open

update windows prefect services #198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd2993a
Update Windows Prefect service setup
LOCEANlloydizard Sep 7, 2026
3149048
Fix Windows Prefect task supervision
LOCEANlloydizard Sep 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
84 changes: 50 additions & 34 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -283,43 +283,58 @@ Notes:
## Running Local Prefect Services on Windows (Task Scheduler)

To run a local Prefect server and worker as background services on Windows, you can
use PowerShell with Windows Task Scheduler and the provided templates:
use Windows Task Scheduler with the provided templates:

- `src/echodataflow/services/deploy_prefect_server.windows.task.xml`
- `src/echodataflow/services/deploy_prefect_worker.windows.task.xml`

1. Copy and customize the service environment file:
```powershell
New-Item -ItemType Directory -Force "$HOME\.config\echodataflow"
New-Item -ItemType Directory -Force "$HOME\.local\var\log\echodataflow"
1. Configure Prefect for the local server:

Copy-Item src\echodataflow\services\services.env.example_local `
"$HOME\.config\echodataflow\services.env"
```powershell
prefect config set PREFECT_API_URL=http://127.0.0.1:4200/api
```

2. Edit `$HOME\.config\echodataflow\services.env` as needed:
- Adjust `ECHODATAFLOW_ENV`
- Adjust `ECHODATAFLOW_HOME`
- Adjust `ECHODATAFLOW_WORKDIR`
- Adjust `ECHODATAFLOW_LOG_DIR`
- Adjust `MAMBA_BIN`
- Adjust `PREFECT_POOL`
- Adjust `PREFECT_API_URL`
2. Render local Task Scheduler XML files from the templates:

3. Copy and customize the Task Scheduler XML templates:
```powershell
Copy-Item src\echodataflow\services\deploy_prefect_server.windows.task.xml `
"$HOME\.config\echodataflow\prefect-server.task.xml"

Copy-Item src\echodataflow\services\deploy_prefect_worker.windows.task.xml `
"$HOME\.config\echodataflow\prefect-worker.task.xml"
$ConfigDir = "$HOME\.config\echodataflow"
New-Item -ItemType Directory -Force $ConfigDir | Out-Null

$User = whoami
$PrefectExe = (Get-Command prefect).Source
$WorkDir = (Get-Location).Path
$Pool = "local"

$Tasks = @{
"deploy_prefect_server.windows.task.xml" = "prefect-server.task.xml"
"deploy_prefect_worker.windows.task.xml" = "prefect-worker.task.xml"
}

foreach ($Task in $Tasks.GetEnumerator()) {
$Template = "src\echodataflow\services\$($Task.Key)"
$Output = Join-Path $ConfigDir $Task.Value

$Content = Get-Content $Template -Raw
$Content = $Content.Replace("__ECHODATAFLOW_USER__", $User)
$Content = $Content.Replace("__PREFECT_EXE__", $PrefectExe)
$Content = $Content.Replace("__ECHODATAFLOW_WORKDIR__", $WorkDir)
$Content = $Content.Replace("__PREFECT_POOL__", $Pool)
$Content = $Content.Replace('encoding="UTF-8"', 'encoding="UTF-16"')

[System.IO.File]::WriteAllText(
$Output,
$Content,
[System.Text.Encoding]::Unicode
)
}
```

4. Both XML templates are self-contained and read their runtime configuration
from `$HOME\.config\echodataflow\services.env`, so they do not need separate
`deploy_prefect_*.windows.ps1` files or hard-coded repository script paths.
Run these commands from the Echodataflow repository root with the
`echodataflow` environment activated. The templates are kept as UTF-8 in the
repository and rendered as UTF-16 for Windows Task Scheduler.

3. Register the scheduled tasks from an **Administrator PowerShell**:

5. Register the scheduled tasks:
```powershell
schtasks /Create /TN "echodataflow-prefect-server" `
/XML "$HOME\.config\echodataflow\prefect-server.task.xml" /F
Expand All @@ -328,20 +343,24 @@ use PowerShell with Windows Task Scheduler and the provided templates:
/XML "$HOME\.config\echodataflow\prefect-worker.task.xml" /F
```

6. Start the tasks:
4. Start the tasks:

```powershell
schtasks /Run /TN "echodataflow-prefect-server"

Start-Sleep -Seconds 10

schtasks /Run /TN "echodataflow-prefect-worker"
```

7. Check task status:
5. Check task status:

```powershell
schtasks /Query /TN "echodataflow-prefect-server" /V /FO LIST
schtasks /Query /TN "echodataflow-prefect-worker" /V /FO LIST
```

8. Verify the local Prefect server:
6. Verify the local Prefect server:

Open:

Expand All @@ -352,15 +371,12 @@ use PowerShell with Windows Task Scheduler and the provided templates:
The Prefect dashboard should load, and the worker should appear online under
**Work Pools**.

9. To stop and delete the tasks:
7. To stop and delete the tasks:

```powershell
schtasks /End /TN "echodataflow-prefect-worker"
schtasks /End /TN "echodataflow-prefect-server"

schtasks /Delete /TN "echodataflow-prefect-worker" /F
schtasks /Delete /TN "echodataflow-prefect-server" /F
```

## License

Echodataflow is licensed under the open source [Apache 2.0 license](https://opensource.org/license/Apache-2.0).
```
67 changes: 15 additions & 52 deletions src/echodataflow/services/deploy_prefect_server.windows.task.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,14 @@
</LogonTrigger>
</Triggers>

<!-- Run in the interactive user session -->
<!--
Run outside the interactive desktop session.
Values marked with __...__ are replaced during installation.
-->
<Principals>
<Principal id="Author">
<LogonType>InteractiveToken</LogonType>
<UserId>__ECHODATAFLOW_USER__</UserId>
<LogonType>S4U</LogonType>
<RunLevel>LeastPrivilege</RunLevel>
</Principal>
</Principals>
Expand All @@ -33,63 +37,22 @@
<Hidden>false</Hidden>
<ExecutionTimeLimit>PT0S</ExecutionTimeLimit>
<Priority>7</Priority>

<RestartOnFailure>
<Interval>PT1M</Interval>
<Count>999</Count>
</RestartOnFailure>
</Settings>

<!-- Launch the PowerShell bootstrap logic directly -->
<!--
Launch Prefect directly so Task Scheduler supervises
the actual Prefect process tree.
-->
<Actions Context="Author">
<Exec>
<Command>powershell.exe</Command>

<!-- Inline startup script; runtime paths come from services.env -->
<Arguments><![CDATA[
-NoLogo
-NoProfile
-NonInteractive
-ExecutionPolicy Bypass
-Command "& {
$EnvFile = Join-Path $HOME '.config\echodataflow\services.env'

# Load service configuration from the standard user config location.
if (-not (Test-Path $EnvFile)) {
Write-Error ('Missing services env file: ' + $EnvFile)
exit 1
}

Get-Content $EnvFile | ForEach-Object {
if ($_ -match '^\s*([^#][^=]+)=(.*)$') {
$Name = $matches[1].Trim()
$Value = $matches[2].Trim()
Set-Item -Path ('Env:' + $Name) -Value $Value
}
}

$RequiredVars = @(
'ECHODATAFLOW_ENV'
'ECHODATAFLOW_WORKDIR'
'ECHODATAFLOW_LOG_DIR'
'MAMBA_BIN'
)

foreach ($Var in $RequiredVars) {
if (-not (Get-Item -Path ('Env:' + $Var) -ErrorAction SilentlyContinue)) {
Write-Error ('Missing required environment variable: ' + $Var)
exit 1
}
}

New-Item -ItemType Directory -Force -Path $env:ECHODATAFLOW_LOG_DIR | Out-Null
Set-Location $env:ECHODATAFLOW_WORKDIR

& $env:MAMBA_BIN run -n $env:ECHODATAFLOW_ENV `
prefect server start `
--host 127.0.0.1 `
--port 4200
}"
]]></Arguments>
<Command>__PREFECT_EXE__</Command>
<Arguments>server start --host 127.0.0.1 --port 4200</Arguments>
<WorkingDirectory>__ECHODATAFLOW_WORKDIR__</WorkingDirectory>
</Exec>
</Actions>
</Task>
</Task>
69 changes: 16 additions & 53 deletions src/echodataflow/services/deploy_prefect_worker.windows.task.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,22 @@
<Description>Run the local Echodataflow Prefect worker at user logon.</Description>
</RegistrationInfo>

<!-- Start at user logon, after a short delay so the server can start first -->
<!-- Start after a short delay so the Prefect server can start first -->
<Triggers>
<LogonTrigger>
<Enabled>true</Enabled>
<Delay>PT1M</Delay>
</LogonTrigger>
</Triggers>

<!-- Run in the interactive user session -->
<!--
Run outside the interactive desktop session.
Values marked with __...__ are replaced during installation.
-->
<Principals>
<Principal id="Author">
<LogonType>InteractiveToken</LogonType>
<UserId>__ECHODATAFLOW_USER__</UserId>
<LogonType>S4U</LogonType>
<RunLevel>LeastPrivilege</RunLevel>
</Principal>
</Principals>
Expand All @@ -34,63 +38,22 @@
<Hidden>false</Hidden>
<ExecutionTimeLimit>PT0S</ExecutionTimeLimit>
<Priority>7</Priority>

<RestartOnFailure>
<Interval>PT1M</Interval>
<Count>999</Count>
</RestartOnFailure>
</Settings>

<!-- Launch the PowerShell bootstrap logic directly -->
<!--
Launch Prefect directly so Task Scheduler supervises
the actual Prefect process tree.
-->
<Actions Context="Author">
<Exec>
<Command>powershell.exe</Command>

<!-- Inline startup script; runtime paths come from services.env -->
<Arguments><![CDATA[
-NoLogo
-NoProfile
-NonInteractive
-ExecutionPolicy Bypass
-Command "& {
$EnvFile = Join-Path $HOME '.config\echodataflow\services.env'

# Load service configuration from the standard user config location.
if (-not (Test-Path $EnvFile)) {
Write-Error ('Missing services env file: ' + $EnvFile)
exit 1
}

Get-Content $EnvFile | ForEach-Object {
if ($_ -match '^\s*([^#][^=]+)=(.*)$') {
$Name = $matches[1].Trim()
$Value = $matches[2].Trim()
Set-Item -Path ('Env:' + $Name) -Value $Value
}
}

$RequiredVars = @(
'ECHODATAFLOW_ENV'
'ECHODATAFLOW_WORKDIR'
'ECHODATAFLOW_LOG_DIR'
'MAMBA_BIN'
'PREFECT_POOL'
)

foreach ($Var in $RequiredVars) {
if (-not (Get-Item -Path ('Env:' + $Var) -ErrorAction SilentlyContinue)) {
Write-Error ('Missing required environment variable: ' + $Var)
exit 1
}
}

New-Item -ItemType Directory -Force -Path $env:ECHODATAFLOW_LOG_DIR | Out-Null
Set-Location $env:ECHODATAFLOW_WORKDIR

& $env:MAMBA_BIN run -n $env:ECHODATAFLOW_ENV `
prefect worker start `
--pool $env:PREFECT_POOL
}"
]]></Arguments>
<Command>__PREFECT_EXE__</Command>
<Arguments>worker start --pool __PREFECT_POOL__</Arguments>
<WorkingDirectory>__ECHODATAFLOW_WORKDIR__</WorkingDirectory>
</Exec>
</Actions>
</Task>
</Task>
Loading