PingAuthorize policy-test GitHub Actions

GitHub Actions that run PingAuthorize Policy Editor (PAP) policy tests from a manifest file, using the PAP Test Suite and Test Runner REST APIs.

A run:

1. loads and validates the manifest,
2. lists the branch/snapshot's test cases (GET /test-suite/list/TEST_CASE) and resolves the manifest's names/groups/IDs against them,
3. creates a test run (POST /test-runner) and polls it to completion (GET /test-runner/{testRunId}),
4. collects per-test results (GET …/summaries, plus GET …/results/{testCaseId} for failures), and
5. reports: step outputs, a job-summary table, ::error annotations per failed test, a JUnit XML file, and a JSON report — and fails the step if any policy test failed.

Zero runtime dependencies (Node 20+, which all GitHub-hosted runners have).

Actions

Action	Purpose
run-policy-tests	Resolve the manifest, execute the tests on the PAP, report results
validate-manifest	Offline manifest validation (fast PR check; no PAP connection)

This project is designed to be pushed as its own pingauthorize-actions repository, with the actions at the repo root:

# from a consuming repository (the usual case)
uses: your-org/pingauthorize-actions/run-policy-tests@main
# from a workflow inside this repo itself
uses: ./run-policy-tests

Quick start

1. Add a manifest (default path .pingauthorize/policy-tests.json):

{
  "version": 1,
  "source": { "branch": "Banking Policies" },
  "tests": {
    "groups": ["Regression"],
    "testCases": ["Smoke.Token owner can read own accounts"]
  },
  "run": { "timeoutSeconds": 600 }
}

2. Add a workflow (full examples in examples/workflows/):

jobs:
  policy-tests:
    runs-on: ubuntu-latest   # use a self-hosted runner if the PAP isn't internet-reachable
    steps:
      - uses: actions/checkout@v4
      - id: tests
        uses: your-org/pingauthorize-actions/run-policy-tests@main
        with:
          api-url: ${{ vars.PAP_API_URL }}        # e.g. https://pap.example.com:8443/api
          token: ${{ secrets.PAP_TOKEN }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: pingauthorize-test-results
          path: pingauthorize-test-results/

Manifest reference

Schema for editor IntelliSense: schema/policy-test-manifest.schema.json (point $schema at it). Example: examples/policy-tests.manifest.json.

Field	Description
version	Manifest format version; must be 1.
description	Free-text description.
source.branch	Policy branch name or ID to test against.
source.snapshotId	Snapshot ID to test against (instead of branch — exactly one).
tests.all	true → run every test case on the branch/snapshot.
tests.testCases[]	Test case full names (dot-separated path, e.g. "Regression.Admin can read"). A bare name works when unique.
tests.groups[]	Test-suite group full names; selects every test case under the group (matched by fullName prefix "<group>.").
tests.testCaseIds[]	Raw test case UUIDs.
onMissing	"fail" (default) or "skip" — what to do when a selection doesn't resolve.
run.description	Description recorded on the PAP test run (default: repo/run metadata).
run.pollIntervalSeconds	Status-poll interval (default 5).
run.timeoutSeconds	Overall timeout; on timeout the run is ended via DELETE /test-runner/{id} (default 600).
run.failOnNoTests	Fail when zero test cases resolve (default true).

Selections are merged and de-duplicated across testCases, groups, and testCaseIds.

run-policy-tests inputs

Input	Required	Default	Description
api-url	yes	—	Policy Editor API base URL, e.g. https://pap.example.com:8443/api
manifest	no	.pingauthorize/policy-tests.json	Manifest path
token	one of ↓	—	OIDC bearer token (Authorization: Bearer …)
user-id	one of ↑↓	—	Demo/no-auth mode (x-user-id header)
basic-auth	one of ↑	—	username:password (LDAP mode)
branch	no	manifest source.branch	Branch name/ID override
snapshot-id	no	manifest source.snapshotId	Snapshot override
report-dir	no	pingauthorize-test-results	Where the JSON + JUnit reports are written
fail-on-failure	no	true	Fail the step when any test fails
poll-interval-seconds / timeout-seconds	no	manifest values	Polling overrides
run-description	no	repo/run metadata	PAP test-run description
ca-cert	no	—	PEM CA bundle to trust (private CAs)
insecure-skip-tls-verify	no	false	Disable TLS verification (dev only)
max-full-results	no	20	Max failed cases to fetch full decision detail for

Exactly one auth input is required (token, user-id, or basic-auth — user-id may also be combined with a token). Exactly one of branch/snapshot must be in effect after overrides.

Outputs

result (pass/fail), total, passed, failed, test-run-id, deployment-package-id, report-json-path, junit-xml-path.

The JUnit file plugs into reporters like dorny/test-reporter or mikepenz/action-junit-report; the JSON report (policy-test-report.json) carries the full normalized result, including decision values and assertion details for failures.

validate-manifest inputs

Input	Default	Description
manifest	.pingauthorize/policy-tests.json	Manifest path to validate (offline)

Running the script directly (local use / other CI systems)

PAP_API_URL=https://pap.example.com:8443/api \
PAP_USER_ID=admin \
node scripts/run-policy-tests.mjs --manifest .pingauthorize/policy-tests.json

All inputs map to PAP_* environment variables (see parseConfig in scripts/run-policy-tests.mjs). Exit codes: 0 success, 1 policy tests failed, 2 manifest/config error, 3 PAP API/test-run error, 4 unexpected.

PAP API surface used

Call	Used for
GET /test-suite/list/TEST_CASE?branch=…|snapshot=…	Resolving manifest names/groups to test case IDs (paginated page/page-size)
POST /test-runner?branch=…|snapshotId=…	Creating the test run ({description, testCaseIds})
GET /test-runner/{testRunId}	Polling state (Pending → Completed) and pass/fail counts
GET /test-runner/{testRunId}/summaries?start=…&count=…	Per-test-case results incl. assertion results
GET /test-runner/{testRunId}/results/{testCaseId}	Decision detail for failed cases
DELETE /test-runner/{testRunId}	Ending a run that exceeds the timeout

Notes / assumptions:

• The PAP docs document the snapshot query parameter as snapshot but the example URLs use snapshotId; the client sends both for snapshot-based runs, which is harmless on either implementation.
• The branch query parameter accepts a branch name or ID (per the PAP docs), so the manifest can use the human-readable branch name.
• Group selection matches fullName prefixes ("<group>."), so group/test names containing literal dots in a single segment can't be addressed by group selection — use explicit testCases/testCaseIds for those.
• The assertion-result entry shape isn't documented; the reporter extracts common field spellings defensively and falls back to compact JSON.

Network and TLS

The runner must be able to reach the PAP — for PAPs on private networks use a self-hosted runner (the reusable example exposes a runs-on input). For PAPs with a private CA, pass the PEM via ca-cert (it is written to the runner temp dir and exported through NODE_EXTRA_CA_CERTS). insecure-skip-tls-verify exists for dev setups but disables all certificate checking — prefer ca-cert.

Development

npm test       # 35 hermetic tests against an in-process mock PAP (node --test, no deps)
npm run mock   # serve the mock PAP standalone; e.g. npm run mock -- --fail tc-2

The mock (test/mock-pap.mjs) mirrors the documented endpoint shapes, including pagination, both auth modes, the Pending → Completed lifecycle, and failure/timeout behavior. scripts/serve-mock-pap.mjs serves it as a standalone process for manual runs of the runner against a local "PAP".

CI (.github/workflows/ci.yml) runs the unit tests on Node 20/22/24, plus a self-test job that exercises both composite actions end-to-end (uses: ./run-policy-tests etc.) against the standalone mock: manifest validation (accept and reject), a passing run, and failing runs with and without fail-on-failure, asserting step outcomes, outputs, and report files.