feature: Allow different invocation methods of produce data workflow

[jmcgrathTT]

Issue:

Currently, the produce data workflow will only run for completed jobs. While this works functionally, it makes produced data workflow runs more difficult to track since they are not bounded to the original workflow due it's asynchronous nature from the github event trigger.

https://github.com/tenstorrent/tt-mlir/actions/workflows/produce_data.yml

This also causes an issue if your workflow run is dynamically named using run-name: at runtime.

tenstorrent/tt-forge#496 (comment)

Suggested change:

Allow input override of the completed status behavior around this code path. Completed status remains the default.

tt-github-actions/.github/actions/collect_data/src/utils.py

Line 246 in 069f1e7

assert github_job.get("status") == "completed", f"{github_job_id} is not completed"

This way, the produced data workflow can be properly bound to the workflow run that collects them. Users know best when data should be collected; otherwise, they rely on the default completed status.

Example:

https://github.com/tenstorrent/tt-forge/blob/d16b31763ebb257a050913f6e3e494c0e9375dfa/.github/workflows/perf-benchmark.yml#L497-L504

[vmilosevic]

Collect data is designed to work only for completed workflows, after workflow completes and not as a part of a workflow while it is still running.

1. Collecting data as part of workflow will not get data from jobs that are re-triggered in workflow ("Re-run failed jobs")
2. Collecting data relies on workflow.json which is not complete until workflow is done, and can't be used correctly for workflows that are still in progress.
3. Database schema is also designed in a way to expects data for completed workflows

This is not the intended way to use collect data, and will not function properly
https://github.com/tenstorrent/tt-forge/blob/d16b31763ebb257a050913f6e3e494c0e9375dfa/.github/workflows/perf-benchmark.yml#L497-L504

[jmcgrathTT]

Collect data is designed to work only for completed workflows,

Yes, so I want to expand this to allow completed & in-progress workflows to be accepted.

Collecting data relies on workflow.json which is not complete until workflow is done, and can't be used correctly for workflows that are still in progress.

I am not seeing that behavior. Here is a test I did on an in-progress workflow. I am missing something?

tt-github-actions/.github/actions/collect_data/download_workflow_data.sh

Line 60 in 2b95153

gh api /repos/$REPOSITORY/actions/runs/$RUN_ID/attempts/$ATTEMPT_NUMBER > generated/cicd/$RUN_ID/workflow.json

REPOSITORY=tenstorrent/tt-forge
RUN_ID=18569668721
ATTEMPT_NUMBER=1

 gh api /repos/$REPOSITORY/actions/runs/$RUN_ID/attempts/$ATTEMPT_NUMBER | jq '.status'
"in_progress"

Database schema is also designed in a way to expects data for completed workflows

Not seeing any schema enforcement here that would prevent that; regardless, I think we could ignore the in-progress jobs since we would capture all jobs that completed ran before the produce data workflow is invoked, and this behavior can be controlled by the needs in the workflow.

[vmilosevic]

If you look at Pipeline schema you can see the pipeline_end_ts mandatory field that implies that pipeline is completed at a time report is created.
Please revert the changes where data collection is invoked as part of workflow, as this is not the designed usage.
https://github.com/tenstorrent/tt-forge/blob/d16b31763ebb257a050913f6e3e494c0e9375dfa/.github/workflows/perf-benchmark.yml#L497-L504

[jmcgrathTT]

If you look at Pipeline schema you can see the pipeline_end_ts mandatory field that implies that pipeline is completed at a time report is created.

pipeline_end_ts references the last time a workflow run was updated. In other words, it's not tied to any completion event other than the gates we have implemented in code after it's called. There would be no change here.

tt-github-actions/.github/actions/collect_data/src/utils.py

Line 76 in 2b95153

pipeline_end_ts = github_pipeline_json["updated_at"]

Sure, pipeline_end_ts would be off by a few seconds when this invocation method occurs at the end of a workflow, but how big a problem is this? I argue that job_*_ts accuracy matters, and this would have zero effect on that. We can also document and set expectations on how pipeline_end_ts could be affected in this invocation method.

Giving the user control over when data collection is invoked is more critical here. In many cases, collecting every job detail is unnecessary.

This can be solved in a few different ways via an option:

• Ignore itself from the job collection (only works if it's the last job in a workflow)
• Ignore by job names
• Ignore all in-progress jobs

Either way you do it, all completed jobs will be collected regardless.

tt-github-actions/.github/actions/collect_data/src/utils.py

Lines 244 to 246 in 069f1e7

	name = github_job.get("name")
	assert github_job.get("status") == "completed", f"{github_job_id} is not completed"

[vmilosevic]

pipeline_end_ts: datetime = Field(description="Timestamp with timezone when the pipeline execution ended.")

It says "when the pipeline execution ended" clearly, and the concept is to create report after pipeline ended, not while it is still running.

As already said collecting data as part of workflow will not get data from jobs that are re-triggered in workflow ("Re-run failed jobs")

There are multiple examples to how to use data collection properly, even in forge repo, please take a look at reference how this action should be used in other repos

https://github.com/tenstorrent/tt-xla/blob/main/.github/workflows/workflow-run-collect-data.yml
https://github.com/tenstorrent/tt-forge-fe/blob/main/.github/workflows/produce_data.yml
https://github.com/tenstorrent/tt-mlir/blob/main/.github/workflows/produce_data.yml
https://github.com/tenstorrent/tt-shield/blob/main/.github/workflows/produce_data.yml
https://github.com/tenstorrent/tt-forge/blob/main/.github/workflows/produce_data.yml

Help me understand the issues you are having with running the collection after pipeline?

[jmcgrathTT]

Hey, sorry for the late response. I got really bogged down with CIv2 troubleshooting the past few days.

Help me understand the issues you are having with running the collection after pipeline?

For sure! Let's back up a bit.

This issue occurs when you have dynamically named workflows, as mentioned here: #88 (comment). (Note, I did see I put runs-on instead of run-name: above, my fault, and I've corrected that.)

What I've found is that you can't do any pattern matching on the workflow name trigger; only static names are supported. We also can't directly call another nested workflow since GitHub has a workflow depth of 3, so it would appear as Performance benchmark, thus being collected after its run.

Secondly, GitHub does not provide the workflow run ID when dispatching a workflow; it becomes available only after you run a workflow list run. To solve this, I injected a parent_id parameter when calling the workflow dispatch.

Cause of this, we need a dynamic workflow name to enable synchronization between workflow dispatches of the root jobs Daily Releaser & the Docker tests.

Performance benchmark parent_run_id: perf-test-tt-xla-0.5.0.dev20251027-1761538115 

Hope this helps, but I've set up a meeting for tomorrow so we can sync on this just in case.

[jmcgrathTT]

@vmilosevic

I've attempted to add pattern matching to solve the dynamic workflow run-ons: name, but it looks like it won't work.

https://github.com/tenstorrent/tt-forge/actions/runs/18885201953

Here is my branch & commits I've tested with
tenstorrent/tt-forge@main...jmcgrath/1711-produce-data-fails-when-on-perf-benchmark-from-xla

Collecting data as part of workflow will not get data from jobs that are re-triggered in workflow ("Re-run failed jobs")

Also part of our conversation, I re-tested my assumption if: always() will execute on retry attempts, and it works. So if you had a collect-data job on re-runs with if: always(), it would work.

https://github.com/tenstorrent/tt-forge/actions/runs/18864029681/job/53882715707

[jmcgrathTT]

@vmilosevic As we discussed on Wednesday, we didn't want to move forward with this because duplicate records will be created in the database during a re-run, which will affect our query. So while I can get this working on the production data from the Tests below, I would have to overcome that.

However, based on tenstorrent/tt-xla#2017 (comment), I've learned we could use temp/staging tables in Superset to remove these duplicates & also retrieve the latest available job.

If I can create a POC of that in action, would that address all the concerns in this feature?

Test run:
https://github.com/tenstorrent/tt-forge/actions/runs/19083367228/job/54517655525

Test branches:
main...jmcgrathTT:tt-github-actions:feature-Allow-different-invocation-methods-of-produce-data-workflow
tenstorrent/tt-forge@main...jmcgrath/1711-produce-data-fails-when-on-perf-benchmark-from-xla

Test Superset results (with double run):