Enhance device metrics export workflow to support JSON output and include optional location data

- Updated README to reflect new CSV and JSON output formats and added optional AP floorplan location data.
- Refactored account_credentials.yaml to unify credential structure.
- Modified device_metrics_export.py to fetch and process device location data, including new command line options for output formats.
- Updated requirements.txt to use pycentral version 2.0a22.
- Enhanced utils.py with functions to derive site IDs and fetch device locations.

Author: Karthik Satheesh Kumar

device-metrics-export/README.md

@@ -1,11 +1,11 @@
 # Device Metrics Export
 
-This workflow retrieves comprehensive device information from both HPE Aruba Networking Central and GreenLake Platform (GLP) APIs. It consolidates device attributes, monitoring data, connectivity status, and subscription details into a single CSV file for easy analysis and reporting.
+This workflow retrieves comprehensive device information from both HPE Aruba Networking Central and GreenLake Platform (GLP) APIs. It consolidates device attributes, monitoring data, connectivity status, subscription details, and optional AP floorplan location data into CSV and JSON files for analysis, automation, and reporting.
 
 ## Prerequisites
 
 - Python 3.8 or higher
-- API credentials for HPE Aruba Networking Central & GLP (JSON or YAML format)
+- API credentials for HPE Aruba Networking Central and GLP (JSON or YAML format)
 
 ## Installation
 
@@ -25,93 +25,115 @@ source venv/bin/activate  # On Windows use: venv\Scripts\activate
 pip install -r requirements.txt
 ```
 
-This workflow is tested on the `pycentral` SDK (version: `v2.0a10`). Please check compatibility before executing on older/newer versions as there may be changes.
+This workflow is tested on the `pycentral` SDK (version: `v2.0a22`). Please check compatibility before executing on older/newer versions as there may be changes.
 
 ## Configuration
 
-#### New HPE Aruba Networking Central & GLP Credentials (account_credentials.yaml)
+#### Credentials (account_credentials.yaml)
 
-For API operations in new HPE Aruba Networking Central & GLP:
+This workflow requires API credentials generated for GLP platform as the script makes API calls to GLP & Central. 
 
 ```yaml
-new_central:
-    cluster_name: <cluster-name>  # or base_url: <central-api-base-url>
-    client_id: <new-central-client-id>
-    client_secret: <new-central-client-secret>
-glp:
-    client_id: <glp-client-id>
-    client_secret: <glp-client-secret>
+unified:
+  cluster_name: <cluster-name> # e.g. EU-1, or use base_url: <central-api-base-url>
+  workspace_id: <glp-workspace-id>
+  client_id: <glp-client-id>
+  client_secret: <glp-client-secret>
 ```
+
 **Sample Input:** See [`account_credentials.yaml`](./account_credentials.yaml) in this repository for an example credential file.
 
 > [!TIP]
 > **Where to find these:**
-> - [New Central API Gateway Base URLs](https://developer.arubanetworks.com/new-hpe-anw-central/docs/getting-started-with-rest-apis#api-gateway-base-urls) 
-> - [How to get API Credentials for new Central](https://developer.arubanetworks.com/new-hpe-anw-central/docs/generating-and-managing-access-tokens)
-> - [How to get API Credentials for GLP](https://developer.greenlake.hpe.com/docs/greenlake/guides/public/authentication/authentication#creating-a-personal-api-client)
+> - [Finding your Central API Gateway Base URL](https://developer.arubanetworks.com/new-central/docs/getting-started-with-rest-apis#finding-your-base-url) 
+> - [Generating & Managing Access Token](https://developer.arubanetworks.com/new-central/docs/generating-and-managing-access-tokens)
 
 ## Execution
 
 This workflow is executed by the script device_metrics_export.py and covers the following:
 - Loads and validates the credentials file (JSON/YAML) & establishes a connection to Central & GLP
 - Runs API calls - 
-    - New Central - [Get a list of devices](https://developer.arubanetworks.com/new-central/reference/getdevices)
-    - New Central - [Get a device inventory list](https://developer.arubanetworks.com/new-central/reference/getdeviceinventory)
-    - GLP - [Get Devices managed in a workspace](https://developer.greenlake.hpe.com/docs/greenlake/services/device-management/public/openapi/nbapi-inventory-latest/openapi/devices-v1/getdevicesv1)
-    - GLP - [Get subscriptions of a workspace](https://developer.greenlake.hpe.com/docs/greenlake/services/subscription-management/public/openapi/nbapi-subscription/subscriptions-v1/getsubscriptionsv1)
-- Processes results from the above mentioned APIs and outputs the result in a CSV format for usage.
+  - GLP
+    - [Get Devices managed in a workspace](https://developer.greenlake.hpe.com/docs/greenlake/services/device-management/public/openapi/nbapi-inventory-latest/openapi/devices-v1/getdevicesv1)
+    - [Get subscriptions of a workspace](https://developer.greenlake.hpe.com/docs/greenlake/services/subscription-management/public/openapi/nbapi-subscription/subscriptions-v1/getsubscriptionsv1)
+  - New Central 
+    - [Get a list of devices](https://developer.arubanetworks.com/new-central/reference/getdevices)
+    - [Get a device inventory list](https://developer.arubanetworks.com/new-central/reference/getdeviceinventory)
+    - Optionally, [List devices with location information](https://developer.arubanetworks.com/new-central/reference/getdevicelocationsv1)
+Processes results from the above mentioned APIs and outputs the result in CSV and structured JSON formats.
 
 Example usage:
 
 ```bash
-python device_metrics_export.py -c <credentials_file> --output <output_file>
+python device_metrics_export.py -c <credentials_file> --output <output_file.csv> [--output-json <output_file.json>] [--include-floorplan] [--include-raw-location]
 ```
 
 
 ### Command Line Options
 
-The script accepts the following command line argument
+The script accepts the following command line arguments
 
-| Name        | Type   | Description           | Required |
-|-------------|--------|----------------------|----------|
-| credentials_file | string | Path to file with New Central & GLP credentials (JSON or YAML). Must end in .json, .yaml, or .yml. | Yes      |
-| output         | string | Path to output CSV file. Defaults to device_data.csv. | No       |
+| Name             | Type   | Description                                                                 | Required |
+|------------------|--------|-----------------------------------------------------------------------------|----------|
+| credentials_file | string | Path to credentials (JSON or YAML). Must end in .json, .yaml, or .yml. | Yes      |
+| output           | string | Path to output CSV file. Defaults to device_data.csv.                      | No       |
+| --output-json    | string | Optional path to write structured JSON output (single array). Defaults to same name as CSV with .json extension. | No |
+| --include-floorplan | flag | If set, fetch per-site device locations from the Central and include location fields in output. | No |
+| --include-raw-location | flag | If set, include the Raw Location column in the CSV and JSON output, and also fetch per-site device locations (implies `--include-floorplan`). | No |
 
 ## Output 
 
-A CSV file (default: device_data.csv) containing merged device details from Central and GLP. You can find a sample output file [here](device_data.csv)
-
-CSV column titles (ordered):
-- Serial Number
-- Mac Address
-- Device Name
-- Device Type
-- Device Model
-- Deployment
-- IPv4
-- Firmware Version
-- Site
-- Device Group
-- Status
-- Uptime
-- Last Seen At
-- Config Status
-- Config Last Modified At
-- Subscription Key
-- Subscription Tier
-- Subscription Type
-- Subscription End Time
+This workflow emits both a CSV (traditional tabular output) and a structured JSON file (single JSON array of device objects).
+
+- CSV (default: device_data.csv): tabular rows where missing values are empty strings. The CSV includes a Raw Location column which contains a JSON string (serialized) when location details are present.
+- JSON (default: device_data.json alongside CSV unless overridden): a single JSON array of device objects. Fields with no data are serialized as null in JSON. Location data (Raw Location) is kept as a nested object when available.
+
+CSV / JSON fields (ordered for CSV; JSON objects contain the same keys):
+
+| Field | Description |
+|-------|-------------|
+| Serial Number | Device serial number from Central inventory. |
+| Mac Address | Device MAC address. |
+| Device Name | Device name configured in Central. |
+| Device Type | Device category, such as access point, switch, or gateway. |
+| Device Model | Device hardware model. |
+| Deployment | Deployment mode reported by Central inventory. |
+| IPv4 | Device IPv4 address. |
+| Firmware Version | Software or firmware version reported for the device. |
+| Site | Central site name associated with the device. |
+| Device Group | Central compatible device group name. |
+| Status | Device status from Central inventory. |
+| Uptime | Device uptime converted to a human-readable duration. |
+| Last Seen At | Last time the device was seen, converted to UTC. |
+| Config Status | Device configuration status from monitoring data. |
+| Config Last Modified At | Last configuration update time, converted to UTC. |
+| Subscription Key | GLP subscription key assigned to the device. |
+| Subscription Tier | GLP subscription tier. |
+| Subscription Type | Derived subscription type, such as Foundation or Advanced. |
+| Subscription End Time | GLP subscription end time, converted to UTC. |
+| Floorplan ID | Floorplan identifier from Central device-location data. |
+| Building ID | Building identifier from Central device-location data. |
+| X Coordinate | Device's X coordinate on the floorplan. |
+| Y Coordinate | Device's Y coordinate on the floorplan. |
+| Floor Coordinate Unit | Unit used for floorplan's X/Y coordinates. |
+| Latitude | Device's latitude from location data. |
+| Longitude | Device's longitude from location data. |
+| Raw Location | Full normalized location payload. In CSV this is serialized as a JSON string; in JSON this remains a nested object. |
 
 > [!TIP]
-> **Device Group**: The Device Group field will only be populated for groups that are New Central compatible device groups. Classic Central groups will not appear in this export.
+> The Device Group field will only be populated for groups that are Central compatible device groups. Classic Central groups will not appear in this export.
+
+> Location fields from Floorplan ID through Longitude are populated only when `--include-floorplan` (or `--include-raw-location`) is provided. The Raw Location column is additionally gated behind `--include-raw-location`; passing this flag also implies `--include-floorplan`, so no separate flag is needed. All other location columns remain present with blank/null values when location data is unavailable.
+
 
 If no data is returned from the APIs or post-processing yields an empty list, the script prints "No data to save." 
 
 ## Troubleshooting
 
-- Authentication / tokens: Ensure your credentials file is complete and has valid credentials for GLP & New Central.
+- Authentication / tokens: Ensure your credentials file contains a top-level `unified` section with `workspace_id`, `client_id`, `client_secret`, and `cluster_name` or `base_url`.
 - Empty output: Confirm that devices exist in Central inventory and that GLP subscriptions/devices exist for the account.
-- SDK compatibility: If API calls fail unexpectedly, confirm the installed pycentral version matches tested versions (v2.0a10) or update helpers accordingly.
+- Missing location fields: Confirm APs have `siteId` values and floorplan/device-location data exists for the site. Devices remain in the output with blank/null location fields when location data is unavailable.
+- SDK compatibility: If API calls fail unexpectedly, confirm the installed pycentral version matches tested versions (v2.0a22) or update helpers accordingly.
 
 ## Support
 

device-metrics-export/account_credentials.yaml

@@ -1,7 +1,5 @@
-new_central:
-  cluster_name: <cluster-name> #e.g. EU-1
-  client_id: <new-central-client-id>
-  client_secret: <new-central-client-secret>
-glp:
+unified:
+  cluster_name: <central-cluster-name> #e.g. EU-1
   client_id: <glp-client-id>
   client_secret: <glp-client-secret>
+  workspace_id: <glp-workspace-id>

device-metrics-export/device_metrics_export.py

@@ -1,3 +1,5 @@
+import json
+import os
 from pycentral.new_monitoring import MonitoringDevices
 from pycentral import NewCentralBase
 from pycentral.glp import Devices, Subscriptions
@@ -11,6 +13,8 @@
     processed_data,
     ensure_tokens_available,
     get_all_device_inventory,
+    derive_site_ids_with_aps,
+    fetch_device_locations,
 )
 
 CSV_COLUMNS = [
@@ -36,8 +40,19 @@
     "Subscription Tier",
     "Subscription Type",
     "Subscription End Time",
+    # Location fields (may be empty)
+    "Floorplan ID",
+    "Building ID",
+    "X Coordinate",
+    "Y Coordinate",
+    "Floor Coordinate Unit",
+    "Latitude",
+    "Longitude",
+    "Raw Location",
 ]
 
+LOCATION_FETCH_WORKERS = 3
+
 
 def main():
     args = parse_args()
@@ -66,35 +81,82 @@ def main():
             conn=new_central_conn, select="id,key,endTime,tier"
         ),
     }
-    # Fetch and process data
     raw_data = run_concurrent_tasks(tasks)
-    processed = process_all_data(raw_data)
 
-    # Save results to CSV
+    processed_inventory = process_list(_safe("device_inventory", raw_data), "serialNumber")
+    processed_devices = process_monitoring_data(_safe("monitoring_devices", raw_data))
+    processed_locations = {}
+    if args.include_floorplan or args.include_raw_location:
+        site_ids = derive_site_ids_with_aps(processed_inventory, processed_devices)
+        if site_ids:
+            processed_locations = fetch_device_locations(
+                new_central_conn, site_ids, max_workers=LOCATION_FETCH_WORKERS
+            )
+        else:
+            print("No site IDs with APs found; skipping floorplan/device-location fetch.")
+
+    processed = process_all_data(raw_data, processed_locations)
+
+    # Save results to CSV and structured JSON
     output = processed_data(**processed)
+    effective_columns = (
+        CSV_COLUMNS
+        if args.include_raw_location
+        else [col for col in CSV_COLUMNS if col != "Raw Location"]
+    )
     if output:
-        df = pd.DataFrame(output)
-        df = df.reindex(columns=CSV_COLUMNS, fill_value="")
+        # Prepare CSV rows: ensure Raw Location is serialized to a JSON string
+        csv_rows = []
+        for row in output:
+            r = row.copy()
+            raw = r.get("Raw Location", "")
+            if isinstance(raw, (dict, list)):
+                r["Raw Location"] = json.dumps(raw, ensure_ascii=False)
+            elif raw is None:
+                r["Raw Location"] = ""
+            csv_rows.append(r)
+
+        df = pd.DataFrame(csv_rows)
+        df = df.reindex(columns=effective_columns, fill_value="")
         df = df.sort_values(
             by="Device Type",
             ascending=True,
             na_position="last",
         )
         df.to_csv(args.output, index=False)
+
+        # Determine JSON output path
+        json_path = getattr(args, "output_json", None)
+        if not json_path:
+            base, ext = os.path.splitext(args.output)
+            json_path = f"{base}.json" if ext else f"{args.output}.json"
+
+        # Prepare JSON rows: convert empty-string placeholders to null (None) in JSON
+        json_rows = []
+        for row in output:
+            jr = {}
+            for column in effective_columns:
+                value = row.get(column, "")
+                jr[column] = None if value == "" else value
+            json_rows.append(jr)
+
+        with open(json_path, "w", encoding="utf-8") as jf:
+            json.dump(json_rows, jf, indent=2, ensure_ascii=False)
+
         print(
-            f"{len(output)} Central devices processed. Device data is saved to {args.output}"
+            f"{len(output)} Central devices processed. Device data is saved to {args.output} and {json_path}"
         )
     else:
         print("No data to save.")
 
 
 def parse_args():
     """Parse command line arguments"""
-    parser = argparse.ArgumentParser(description="Device Onboarding")
+    parser = argparse.ArgumentParser(description="Device Metrics Export")
     parser.add_argument(
         "-c",
         "--credentials",
-        help="Credentials file for New Central API (JSON or YAML)",
+        help="Credentials file for New Central API (JSON or YAML). Unified PyCentral credential files are supported.",
         required=True,
         type=validate_file_format,
     )
@@ -105,6 +167,24 @@ def parse_args():
         required=False,
         default="device_data.csv",
     )
+    parser.add_argument(
+        "--include-floorplan",
+        help="Include per-site AP floorplan/location data (optional)",
+        action="store_true",
+        default=False,
+    )
+    parser.add_argument(
+        "--include-raw-location",
+        help="Include the Raw Location column in the output. Also implies --include-floorplan (fetches per-site device locations).",
+        action="store_true",
+        default=False,
+    )
+    parser.add_argument(
+        "--output-json",
+        help="Optional path to write structured JSON output (single array). Defaults to replacing CSV extension with .json",
+        required=False,
+        default=None,
+    )
     return parser.parse_args()
 
 
@@ -119,19 +199,22 @@ def validate_file_format(file_path):
     return file_path
 
 
-def process_all_data(raw_data):
-    """Process raw API data into structured format"""
+def process_all_data(raw_data, processed_locations=None):
+    """Process raw API data into structured format. Accept optional processed_locations map."""
+    processed_inventory = process_list(
+        _safe("device_inventory", raw_data), "serialNumber"
+    )
     return {
         "processed_devices": process_monitoring_data(
             _safe("monitoring_devices", raw_data)
         ),
-        "processed_inventory": process_list(
-            _safe("device_inventory", raw_data), "serialNumber"
-        ),
+        "processed_inventory": processed_inventory,
         "processed_glp_devices": process_glp_device(
-            process_list(_safe("glp_devices", raw_data), "serialNumber")
+            process_list(_safe("glp_devices", raw_data), "serialNumber"),
+            known_serials=processed_inventory,
         ),
         "processed_subs": process_list(_safe("glp_subs", raw_data), "id"),
+        "processed_locations": processed_locations or {},
     }
 
 

device-metrics-export/requirements.txt

@@ -1,2 +1,2 @@
 pandas==2.3.3
-pycentral==2.0a10
+pycentral==2.0a22