Skip to content

Commit b1330cd

Browse files
varsisvarsis
authored
chore(columns): add documentation on reserved columns (#822)
1 parent 70e46f5 commit b1330cd

1 file changed

Lines changed: 115 additions & 0 deletions

File tree

‎docs/DeveloperManuals/ReservedColumns.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
---
2+
title: "Reserved Columns"
3+
description: >
4+
Reserved column names in DevLake's data models and naming conventions for source-system timestamps.
5+
---
6+
7+
# Reserved Columns
8+
9+
## Summary
10+
11+
DevLake's base model structs (`common.NoPKModel` and `common.Model`) automatically manage certain columns. These columns are **reserved** and must not be used to store data originating from external data sources. This page explains which columns are reserved, why they exist, and what naming conventions to follow when your plugin needs to store timestamps from a data source.
12+
13+
## Reserved Column Names
14+
15+
The following columns are reserved by DevLake:
16+
17+
| Column | Type | Purpose |
18+
| ------------------ | -------------- | ----------------------------------------------------------------------------------- |
19+
| `created_at` | `datetime(3)` | Automatically set to the time the record is **inserted** into DevLake's database. |
20+
| `updated_at` | `datetime(3)` | Automatically set to the time the record is last **updated** in DevLake's database. |
21+
| `_raw_data_params` | `varchar(255)` | Links the record back to its raw-layer source. |
22+
| `_raw_data_table` | `varchar(255)` | Identifies which raw table the record was extracted from. |
23+
| `_raw_data_id` | `bigint` | References the specific row in the raw table. |
24+
25+
The `created_at` and `updated_at` columns are managed by GORM's `autoCreateTime` and `autoUpdateTime` tags on the `NoPKModel` base struct. They reflect **when DevLake processed the record**, not when the record was created or updated in the original data source.
26+
27+
## Why This Matters
28+
29+
If a plugin maps a data source's creation or update timestamp directly to `created_at` or `updated_at`, the value will be silently overwritten by GORM's auto-timestamp behavior. This means:
30+
31+
- The source-system timestamp is lost.
32+
- The column no longer accurately reflects DevLake's own processing time.
33+
- Downstream queries and dashboards that rely on these columns may produce incorrect results.
34+
35+
## Naming Convention for Source Timestamps
36+
37+
When your plugin model needs to store timestamps from an external data source, use a **prefixed column name** that identifies the source system. Here are examples from existing plugins:
38+
39+
### Go (Tool-Layer Models)
40+
41+
```go
42+
// ✅ Correct — use a source-specific prefix
43+
type GithubIssue struct {
44+
ConnectionId uint64 `gorm:"primaryKey"`
45+
GithubId int `gorm:"primaryKey"`
46+
// ...
47+
GithubCreatedAt time.Time // timestamp from GitHub
48+
GithubUpdatedAt time.Time `gorm:"index"` // timestamp from GitHub
49+
common.NoPKModel // provides created_at, updated_at
50+
}
51+
52+
// ❌ Wrong — overwrites DevLake's auto-managed columns
53+
type GithubIssue struct {
54+
ConnectionId uint64 `gorm:"primaryKey"`
55+
GithubId int `gorm:"primaryKey"`
56+
CreatedAt time.Time // conflicts with NoPKModel
57+
UpdatedAt time.Time // conflicts with NoPKModel
58+
common.NoPKModel
59+
}
60+
```
61+
62+
### Go (Domain-Layer Models)
63+
64+
Domain-layer models follow the same rule. Use descriptive names like `CreatedDate`, `UpdatedDate`, or source-prefixed names:
65+
66+
```go
67+
type GitlabDeployment struct {
68+
common.NoPKModel
69+
ConnectionId uint64 `gorm:"primaryKey"`
70+
GitlabId int `gorm:"primaryKey"`
71+
CreatedDate time.Time `json:"created_date"` // from GitLab API
72+
UpdatedDate *time.Time `json:"updated_date"` // from GitLab API
73+
// ...
74+
}
75+
```
76+
77+
### Python (PyDevLake Models)
78+
79+
The same convention applies to Python plugins. The `NoPKModel` base class in PyDevLake provides `created_at` and `updated_at` automatically:
80+
81+
```python
82+
class MyToolIssue(ToolModel):
83+
# ✅ Correct
84+
tool_created_at: datetime # timestamp from the external tool
85+
tool_updated_at: datetime # timestamp from the external tool
86+
87+
# ❌ Wrong — these conflict with NoPKModel
88+
# created_at: datetime
89+
# updated_at: datetime
90+
```
91+
92+
## Common Prefixes Used Across Plugins
93+
94+
| Plugin | Prefix Pattern | Example |
95+
| ------- | ----------------------------- | ------------------------------------------- |
96+
| GitHub | `Github` | `GithubCreatedAt`, `GithubUpdatedAt` |
97+
| GitLab | `Gitlab` | `GitlabCreatedAt`, `GitlabUpdatedAt` |
98+
| Jira | `Created`, `Updated` | `Created`, `Updated` (Jira-specific fields) |
99+
| Generic | `CreatedDate` / `UpdatedDate` | `CreatedDate`, `UpdatedDate` |
100+
101+
Pick whichever prefix is most natural for your data source, but **never** use the bare names `created_at` / `updated_at` (or their Go equivalents `CreatedAt` / `UpdatedAt` without a prefix).
102+
103+
## Quick Checklist
104+
105+
When defining a new tool-layer or domain-layer model:
106+
107+
1. **Embed `common.NoPKModel`** (Go) or extend `NoPKModel` / `ToolModel` (Python) — this gives you `created_at` and `updated_at` for free.
108+
2. **Do not** add your own `CreatedAt` or `UpdatedAt` fields without a source-specific prefix.
109+
3. **Name source timestamps** with a prefix that identifies the data source (e.g., `GithubCreatedAt`) or use a generic alternative like `CreatedDate`.
110+
4. **Document** the meaning of each timestamp field with a comment or `gorm:"comment:..."` tag so that other contributors understand which system the timestamp originates from.
111+
112+
## References
113+
114+
- Base model definition: [`models/common`](https://github.com/apache/incubator-devlake/tree/main/backend/core/models/common)
115+
- [PR #8701](https://github.com/apache/incubator-devlake/pull/8701) — CircleCI column naming fix that prompted this convention to be formally documented.

0 commit comments

Comments
 (0)