|
1 | | -# Python Template |
| 1 | +# dataconnect-library-python |
2 | 2 |
|
3 | | -This repository is meant to establish useful practices for python projects. |
| 3 | +Python SDK for the [Medidata DataConnect](https://github.com/mdsol/dataconnect-library-r) service. |
| 4 | + |
| 5 | +This library is a faithful Python translation of the R `dataconnect` package. |
| 6 | +Existing R users will recognise every public method; the names follow |
| 7 | +Python conventions (`snake_case`, `PascalCase` models). |
| 8 | + |
| 9 | +--- |
| 10 | + |
| 11 | +## Transport note |
| 12 | + |
| 13 | +The DataConnect service uses **Apache Arrow Flight** (gRPC binary protocol), |
| 14 | +**not** a plain REST/HTTP API. `pyarrow.flight` is the primary transport |
| 15 | +dependency. `httpx` is used only for the optional public-IP client- |
| 16 | +identification header that accompanies every Flight call. |
| 17 | + |
| 18 | +--- |
4 | 19 |
|
5 | 20 | ## Installation |
6 | 21 |
|
7 | | -See [CONTRIBUTING.md](./CONTRIBUTING.md) for installation instructions. |
| 22 | +```bash |
| 23 | +pip install dataconnect # core (pyarrow + pydantic + httpx) |
| 24 | +pip install dataconnect[pandas] # + pandas for .to_pandas() on results |
| 25 | +``` |
| 26 | + |
| 27 | +Requires **Python ≥ 3.10**. |
| 28 | + |
| 29 | +--- |
| 30 | + |
| 31 | +## Quick start |
| 32 | + |
| 33 | +```python |
| 34 | +import dataconnect |
| 35 | + |
| 36 | +client = dataconnect.init(token="your-bearer-token") |
| 37 | +# or set DATACONNECT_TOKEN env var and omit token= |
| 38 | + |
| 39 | +# ── Studies ────────────────────────────────────────────────────────────── |
| 40 | +page = client.studies(search_study_name="ACME", page=1, page_size=10) |
| 41 | +for study in page.studies: |
| 42 | + print(study.uuid, study.name) |
| 43 | + for env in study.environments: |
| 44 | + print(" env:", env.uuid, env.name) |
| 45 | + |
| 46 | +# ── Datasets ───────────────────────────────────────────────────────────── |
| 47 | +ds_page = client.datasets( |
| 48 | + study_environment_uuid="env-uuid", |
| 49 | + search_dataset_name="Vitals", |
| 50 | +) |
| 51 | +for ds in ds_page.datasets: |
| 52 | + print(ds.dataset_uuid, ds.dataset_name) |
| 53 | + |
| 54 | +# ── Dataset versions ────────────────────────────────────────────────────── |
| 55 | +versions = client.dataset_versions(dataset_uuid="ds-uuid") |
| 56 | +for v in versions: |
| 57 | + print(v.dataset_version) |
| 58 | + |
| 59 | +# ── Fetch data (lazy) ───────────────────────────────────────────────────── |
| 60 | +frame = client.fetch_data(dataset_uuid="ds-uuid") |
| 61 | +table = frame.head(100).collect() # pyarrow.Table — no download until here |
| 62 | +df = table.to_pandas() # optional: convert to pandas |
| 63 | + |
| 64 | +# ── Dry publish (validate without persisting) ───────────────────────────── |
| 65 | +import pyarrow as pa |
| 66 | + |
| 67 | +data = pa.Table.from_pandas(df) |
| 68 | +result = client.dry_publish( |
| 69 | + project_token="project-token", |
| 70 | + dataset_name="my_dataset", |
| 71 | + key_columns=["subject_id"], |
| 72 | + source_datasets=[], |
| 73 | + data=data, |
| 74 | +) |
| 75 | +print(result.success, result.is_schema_valid) |
| 76 | + |
| 77 | +# ── Publish ─────────────────────────────────────────────────────────────── |
| 78 | +result = client.publish( |
| 79 | + project_token="project-token", |
| 80 | + dataset_name="my_dataset", |
| 81 | + key_columns=["subject_id"], |
| 82 | + source_datasets=[], |
| 83 | + data=data, |
| 84 | +) |
| 85 | +print(result.dataset_uuid, result.dataset_version) |
| 86 | +``` |
| 87 | + |
| 88 | +--- |
| 89 | + |
| 90 | +## R → Python mapping |
| 91 | + |
| 92 | +| R | Python | |
| 93 | +|---|--------| |
| 94 | +| `init(token = "…")` | `dataconnect.init(token="…")` | |
| 95 | +| `client$studies(search_study_name, page, page_size)` | `client.studies(search_study_name=, page=, page_size=)` | |
| 96 | +| `client$datasets(study_environment_uuid, …)` | `client.datasets(study_environment_uuid=, …)` | |
| 97 | +| `client$dataset_versions(dataset_uuid)` | `client.dataset_versions(dataset_uuid=)` | |
| 98 | +| `client$fetch_data(dataset_uuid)` | `client.fetch_data(dataset_uuid=)` | |
| 99 | +| `data$frame %>% head(10) %>% collect()` | `frame.head(10).collect()` | |
| 100 | +| `client$dry_publish(project_token, …)` | `client.dry_publish(project_token=, …)` | |
| 101 | +| `client$publish(project_token, …)` | `client.publish(project_token=, …)` | |
| 102 | +| `DATACONNECT_TOKEN` env var | `DATACONNECT_TOKEN` env var (identical) | |
| 103 | +| `stop("…")` | `raise DataConnectError("…")` | |
| 104 | +| `tryCatch(…)` | `try … except DataConnectError` | |
| 105 | +| `data.frame` | `pyarrow.Table` (call `.to_pandas()` for pandas) | |
| 106 | +| R `NULL` | `None` (`Optional[T]`) | |
| 107 | +| R `c("a","b")` | `["a", "b"]` | |
| 108 | + |
| 109 | +--- |
| 110 | + |
| 111 | +## Error handling |
| 112 | + |
| 113 | +```python |
| 114 | +from dataconnect import ( |
| 115 | + DataConnectError, |
| 116 | + AuthenticationError, |
| 117 | + AuthorizationError, |
| 118 | + NotFoundError, |
| 119 | + ValidationError, |
| 120 | + ServerError, |
| 121 | +) |
| 122 | + |
| 123 | +try: |
| 124 | + page = client.studies() |
| 125 | +except AuthenticationError as e: |
| 126 | + print("Bad token:", e.message) |
| 127 | +except NotFoundError as e: |
| 128 | + print("Not found:", e.message) |
| 129 | +except DataConnectError as e: |
| 130 | + print(f"[{e.error_code}] {e.message}") |
| 131 | + for detail in e.details: |
| 132 | + print(" ", detail.field, detail.message) |
| 133 | +``` |
| 134 | + |
| 135 | +--- |
| 136 | + |
| 137 | +## Dependency injection / testing |
| 138 | + |
| 139 | +Pass a pre-built `pyarrow.flight.FlightClient` to avoid a live server: |
| 140 | + |
| 141 | +```python |
| 142 | +from unittest.mock import MagicMock |
| 143 | +import pyarrow.flight as flight |
8 | 144 |
|
9 | | -## To Dos |
| 145 | +mock_fc = MagicMock(spec=flight.FlightClient) |
| 146 | +client = DataConnectClient( |
| 147 | + host="localhost", port=8815, use_tls=False, |
| 148 | + token="test", flight_client=mock_fc, |
| 149 | +) |
| 150 | +``` |
10 | 151 |
|
11 | | -The following actions are needed when using this as the template for a new python project: |
| 152 | +--- |
12 | 153 |
|
13 | | -- [ ] Update `readme.md` for new project |
14 | | -- [ ] Update `factbook.yaml` for new project |
15 | | -- [ ] Update `pyproject.toml` with relevant `name`, `description`, and `authors` and remove unneeded dep groups |
16 | | -- [ ] Update `python_template` folder to match new project name |
17 | | -- [ ] Remove all example code files (leave `__init__.py` in place) |
18 | | -- [ ] Rename all references to `python-template` and `python_template` to new module/folder names |
19 | | -- [ ] Remove all example test files and all code from `conftest.py` (leave the docstring in place) |
20 | | -- [ ] Add placeholder tests so PR passes |
21 | | -- [ ] Get GH Admins to add mdsol robot key so PR passes |
22 | | -- [ ] Add Artifactory robot account info so PR passes |
| 154 | +## Package layout |
23 | 155 |
|
24 | | -## Contributing |
| 156 | +``` |
| 157 | +dataconnect-library-python/ |
| 158 | +├── dataconnect/ |
| 159 | +│ ├── __init__.py Public API: init(), DataConnectClient, models, errors |
| 160 | +│ ├── _client.py DataConnectClient class — all public methods |
| 161 | +│ ├── _connection.py Flight client factory + FlightCallOptions builder |
| 162 | +│ ├── _datasets.py Studies / datasets / versions via list_flights |
| 163 | +│ ├── _publishing.py dry_publish / publish via do_put |
| 164 | +│ ├── _frame.py DataConnectFrame — lazy head() / collect() |
| 165 | +│ ├── _models.py Pydantic v2 response models |
| 166 | +│ └── _errors.py Exception hierarchy + error parser |
| 167 | +├── tests/ |
| 168 | +│ └── test_dataconnect.py Offline unit tests (mock FlightClient) |
| 169 | +└── pyproject.toml |
| 170 | +``` |
25 | 171 |
|
26 | | -See [CONTRIBUTING](CONTRIBUTING.md). |
| 172 | +--- |
27 | 173 |
|
28 | | -## Contact |
| 174 | +## Development |
29 | 175 |
|
30 | | -See the [factbook](factbook.yaml). |
| 176 | +```bash |
| 177 | +pip install -e ".[dev]" |
| 178 | +pytest |
| 179 | +``` |
0 commit comments