feat: Asset's Status #170
Description
Is your feature request related to a problem? Please describe.
There is a need in our setup that requires to store status of an asset.
We have a few job type assets and we want to update their status everytime they start, finished, failed, etc.
We also want to store it in historical/log manner because we want the status revision to be listed out later.
So basically an asset can have multiple statuses which compass can provide the API to store.
When status-es is already populated, we can then:
- show list of statuses when fetching an asset
- enrich lineage graph response to include asset status
Describe the solution you'd like
1. New status on asset package to represent asset's status
type Status struct {
ID string `json:"id"`
AssetURN string `json:"asset_urn"`
Status string `json:"status"`
Metadata map[string]interface{} `json:"metadata"`
CreatedAt time.Time `json:"created_at"`
}
type Asset struct {
...
Statuses []Status `json:"statuses"`
}
New endpoint to create or add a new status
[POST] /v1beta1/asset/{urn}/status
{
"status": "COMPLETED" // can be any value
"timestamp": "2022-09-02T07:09:24" // in utc
"metadata": { // map to store extra information
"foo": "bar",
"foo_number": 8,
"foo_bool": true
}
}
2. Modify lineage to show asset's latest status
Current lineage API response is showing list of relations. To be able to show asset's latest status we need our lineage API to return list of assets instead.
Current response
{
"data": [
{
"source": "my-input-1",
"target": "my-job-1"
},
{
"source": "my-job-1",
"target": "my-output-1"
}
]
}
New response
{
"nodes": [
{
"urn": "my-input-1",
"upstreams": [],
"downstreams": [
{"urn": "my-job-1"},
]
},
{
"urn": "my-job-1",
"latest_status": {
"status": "COMPLETED",
"created_at": "2020-12-12T19:10:10",
"metadata": {
"job_url": "https://myjobrun.com"
}
},
"upstreams": [
{"urn": "my-input-1"},
],
"downstreams": [
{"urn": "my-output-1"},
]
},
{
"urn": "my-output-1",
"upstreams": [
{"urn": "my-job-1"},
],
}
]
}
Decisions
Going with all the approaches but renaming Status to Probe as suggested here.
type Probe struct {
ID string `json:"id"`
AssetURN string `json:"asset_urn"`
Status string `json:"status"`
Metadata map[string]interface{} `json:"metadata"`
Timestamp time.Time `json:"timestamp"`
CreatedAt time.Time `json:"created_at"`
}
type Asset struct {
...
Probes []Probe `json:"probes"`
}
New endpoint to create or add a new status
[POST] /v1beta1/assets/{urn}/probes
{
"status": "COMPLETED" // can be any value
"timestamp": "2022-09-02T07:09:24" // in utc
"metadata": { // map to store extra information
"foo": "bar",
"foo_number": 8,
"foo_bool": true
}
}