Vitis Unified Backend

[Tanawin1701d]

Description

VitisUnified backend

Motivation

• The current Vitis backend does not support a complete flow from HLS4ML to bitstream generation and driver deployment (zcu102 for my case)

Summarized features

• Automatic creation of AXI master reader/writer interfaces for HLS4ML kernels.
• Based on the v++ compiler and packaging flow.
• Configuration aligned with the Vitis Unified IDE project structure.
• Seamless integration with Xilinx hardware platforms:
  • the platform is the Xilinx package that contains the hardware structure such as axi interconnect, PS configuration, interrupt(except HLS4ML kernel)
  • Platforms encapsulate hardware structures such as AXI interconnects, PS configuration, and interrupts (excluding the HLS4ML kernel).
  • Xilinx provides platform for some boards integrated in Vitis /tools/Xilinx/Vitis/2023.2/base_platforms
  • Developers can also create custom platforms by following the official tutorial: https://github.com/Xilinx/Vitis-Tutorials/tree/2025.1/Vitis_Platform_Creation/Design_Tutorials/01-Edge-KV260
• Automatic PYNQ driver generation for streamlined deployment.

Type of change

For a new feature or function, please create an issue first to discuss it
with us before submitting a pull request.

Note: Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Tests

• we test with the tiny keras unet model @ test/pytest/test_backend/vitis_unified.py with 4 main aspect

bridge test

• Compare VitisUnified with Vitis
• we check predict file with 100% match

cosimulation

• we use two VitisUnified:
  • first one is used to generate bridge simulation
  • second one is used to generate start cosimulation and get the simulation result from cosimulator
• Compare with 1e-4 acceptable torelant (it comes from dat file rounding a bit)

fifo test optimization

• Procedure is similar to Co-simulation and inspect that there is fifo_depth.json exist

hardware test

• Stress test with 10,000 queries but have only 128 (input) + 128 (output) buffer size to make sure there is no deadlock from autogenerated xilinx platform axi-connection
• the tested was in function test_gen_unified in
  • the test was conducted in zcu102 with pynq framework

test reproduce

• Run pytest on test/pytest/test_backend/vitis_unified.py file
• for hardware test (test_gen_unified), you should specify XPFM_PATH(path to xpfm file) to the correct place.
• if LOG_STD == True, HLS4ML will give the HLS+linker compiling message @ console.
• if not, HLS4ML will give the messages @ <output_project_dir>/<prefix>_err.log or <output_project_dir>/<prefix>_out.log

Test Configuration:

Checklist

• I have read the guidelines for contributing.
• I have commented my code, particularly in hard-to-understand areas.
• I have made corresponding changes to the documentation.
• My changes generate no new warnings. (see the section below)
• I have installed and run pre-commit on the files I edited or added.
• I have added tests that prove my fix is effective or that my feature works.

implementation detail

• This flow of this backend to build the ready to ship file, we should do three flow things
  • file generation(HLS4ML generated file) prepare file for system Generation and pynq driver
  • synthesis Kernel (Synthesis Kernel (v++)) do c-synthesis for HLS4ML model
  • linker (Linker+vivado+Bitfile+hwh)

File structure

template structure

• the tree below is the template file allocate at hls4ml/templates/vitis_unified

├── build_lib_multigraph.sh
├── build_lib.sh
├── driver
│   └── pynq
│       └── pynq_driver.py.hls4ml (template for pynq driver)
├── hls_kernel_config.cfg         (config for HLS4ML model Synthesis)
├── myproject_bridge.cpp          (wrapper for C++ simulation using python .predict())
├── myproject_dm.cpp              (wrapper for HLS4ML model convert axi to axi stream)
├── myproject_dm.h   
├── myproject_test.cpp            (for cosimulation and fifo-optimization)
└── workspace
    ├── projectName
    │   └── vitis-comp.json       (project meta-data used for opening using vitis unified IDE)
    └── sysProj
        ├── buildAcc.sh           (script for linking the kernel with platform)
        └── buildConfig.cfg       (config file for linking progress)

output file structure

├ export                        (ready to ship file placed here!)
│   ├ pynq_driver.py
│   ├ system.bit
│   └ system.hwh
├ firmware
│   ├ <project_name>_dm.cpp     (wrapper for HLS4ML model convert axi to axi stream)
│   ├ <project_name>_dm.h       (the syntesizer not use this but required by cosim and bridge sim)
│   ├ <other files>             (other HLS src file generated from Vitis and vivado backend)
├ unifiedWorkspace              (folder for kernel synthesis and linking progress)
│   ├ linker                    (folder for platform linking project)
│   │   ├ buildAcc.sh           (build script for platform link)
│   │   ├ buildConfig.cfg       (config script for platform link)
│   │   └ <other files>         (file that v++ generated during link the platform)
│   └ <project_name>            (folder for HLS project from HLS4ML model)
│       ├ unifiedPrj            (folder for Vitis HLS internal file)
│       └ vitis-comp.json       (project meta-data used for opening using vitis unified IDE)
├ build_lib.sh                  (build script for bridge simulation)
├ hls_kernel_config_cosim.cfg   (config file for cosim and fifo depth optimization)
├ hls_kernel_config_csim.cfg    (config file for csim )
├ myproject_bridge.cpp          (wrapper for C++ simulation using python .predict())
└ myproject_test.cpp          (for cosimulation and fifo-optimization)

configuration

board='zcu102',
        part=None,
        clock_period=5,
        clock_uncertainty='12.5%',
        io_type='io_stream',
        driver='python',
        input_type='float',
        output_type='float',
        in_stream_buf_size=128,
        out_stream_buf_size=128,
        xpfmPath='/opt/Xilinx/Vitis/2023.2/base_platforms/' 'xilinx_zcu102_base_202320_1/xilinx_zcu102_base_202320_1.xpfm',
        **_,

• input_type and output_type are support only float and double. And it must be match
• {in/out}_stream_buf_size unit is in amount elements of the nnet::array
  xpfmPath

note to developer

• In case, you want to debug the generated HLS project using Vitis unified IDE, you can select the workspace folder at the program at unifiedWorkspace. The IDE will automatically detect your project
• For bridge simulation, if the configuration input_type/output_type was set to type x (double or float), you cannot predict with numpy array with different input/output type
• the depth argument @ axi_master write @ <project_name>_dm.cpp must be match of the array size generated the output array@ ````myproject_test.cpp``` for cosim and csim.
  • if the array allocation is larger than depth, the result will not correct
  • if the array allocation is lower than depth, the result is correct, but the system will throw segment falut error
  • the depth size will not impact the resource usage in hls generation
• The linked Vivado project are at <project_folder>/unifiedWorkspace/linker/_x/link/vivado/vpl/prj
• This backend will reject multigraph feature

note to tutorial

• we provide the tutorial at this repository
  https://github.com/Tanawin1701d/vitisUnifiedTutorial

generated warning

• warning in HLS4ML is only about the unet model that we use in pytest, I think it is not warning in the new backend

WARNING:absl:Skipping variable loading for optimizer 'Adam', because it has 17 variables whereas the saved optimizer has 1 variables. 
WARNING: Config parameter "algorithm" overwrites an existing attribute in layer "up_sampling2d" (Resize)

• for kernel synthesis with Vitis, I think it is general warning such as unused parameter, deprecated pragma, dataflow conflict

[gflengas]

An updated version of the tutorial can be found at Tanawin1701d/vitis_unified_backend_tutorial. I have successfully used it to build and run a Keras model with io_type="io_stream" and strategy = 'Latency' .

[gflengas]

Some issues regarding the Vitis Unified backend:

1. cosim.enable_fifo_sizing=true

The cosim.enable_fifo_sizing=true flag, introduced in 5e87503, appears to cause problems on larger designs when
combined with FIFO depth optimization.

In these cases, co-simulation runs twice:

• once for the hls4ml FIFO depth optimization pass
• once again for the Vitis FIFO sizing pass

The first pass completes successfully, but the second pass hangs at RTL Simulation : 0 / 5 [n/a].

For smaller designs, I did not observe any benefit from enabling cosim.enable_fifo_sizing, although
it also did not cause the hang there. As one example, I tested a ~10k-parameter residual CNN with 6
conv blocks and a final dense softmax across all combinations of:

• FIFO depth optimization on/off
• cosim.enable_fifo_sizing on/off

In that experiment, the best implementation result was unchanged by cosim.enable_fifo_sizing. The
improvement came entirely from the hls4ml FIFO depth optimization pass, while enabling
cosim.enable_fifo_sizing alone did not change the final implemented design.

It may be worth disabling this flag by default, or at least making it optional in the unified flow
for larger designs.

2. Name length issue

This is not a synthesis failure. The flow completes HLS packaging and platform generation, then fails
at the start of the final v++ link step because the generated kernel_name:compute_unit_name
exceeds the Vitis limit of 64 characters.

Example:
ERROR: [v++ 60-1214] The total length of a kernel and its compute unit name cannot exceed 64: 'MADOS_RESNET_project_axi_stream:MADOS_RESNET_project_axi_stream_1'
ERROR: [v++ 60-633] Failed to collect compute units: The kernel and its compute unit name length violation

As a result, the .xo and platform artifacts are generated, but .xclbin / bitstream generation
does not complete.

This may not be strictly a backend-only issue, but the backend should be able to guard against it by
validating or sanitizing generated kernel names before invoking v++.

[Tanawin1701d]

Hi @gflengas , thank you for your feedback.

1. cosim.enable_fifo_sizing=true
   1.1 Regarding the hang on the second run, I suspect a deadlock, although I’m not fully certain yet. After resizing the FIFO depth, the residual connections in the network may introduce head-of-line blocking in the skip dispatch link, which could lead to the deadlock.
   1.2 For the redundant FIFO optimization, I will work on fixing this issue.

2. Name length issue
   2.1 I am currently reproducing this issue to identify where the fix should be applied.

I will keep you updated as I make progress.

[marco66colombo]

Hi! I am testing this VitisUnified PR in the context of an implementation-CI prototype, where the goal is to run a small end-to-end implementation flow and collect the generated reports/artifacts across hls4ml releases.

For context, the branch I am testing from is:
https://github.com/marco66colombo/hls4ml/tree/implementation-ci-vitis-unified

The test file is here

The current test uses keras/KERAS_1layer.json from example-models with:

• backend: VitisUnified
• board: zcu102
• part: xczu9eg-ffvb1156-2-e
• Vitis/Vivado: 2023.2
• io_type='io_stream'
• axi_mode='axi_master'
• Strategy='latency'
• clock_period='10ns'
• input_type='float', output_type='float'
• build args: synth=True, cosim=True, bitfile=True

The flow gets through HLS synthesis, cosimulation, and starts the Vitis hardware link. With zcu102, the platform is found correctly, but the job then appears to sit for a long time at:

Run vpl: Step config_hw_runs: RUNNING...

This keeps repeating and does not seem to progress.

Does anything in these settings look wrong, or should I change some part of the test setup for this flow?

[lolbraa]

@marco66colombo Hi!
I've put some work into getting HLS4ML and Vitis Unified working in a Docker container, and in the effort had a similar problem.
Sifting through the logs you'll maybe find the error similar to what's described here (which is what we encountered).
In our case it was in the project-directory vitis_workspace/system_link/_x/link/vivado/vpl/hs_err_pid20323.log

Our fix was to export the env-variable (LD_PRELOAD=/lib/x86_64-linux-gnu/libudev.so.1) right before running .build() in our notebooks.

[marco66colombo]

Thanks @lolbraa, this worked!
The VitisUnified flow now gets past the VPL step and completes bitstream generation.

[lolbraa]

@marco66colombo Glad our frustrating experiences are not in vain!

While I'm here, I'd like to get some general input from you guys.
We're wrapping up our projecting utilizing this backend for deploying to the Xilinx Kria KV-260. To support the device and our general development environment, we've done some minor changes, among which the most notable is updating .tcl scripts to add support for Vivado/Vitis 2025.2 for the KV260.

Now we're pondering (1) if the changes are welcome upstream, and (2) how we should go about creating a pull request.

Do we just PR into VitisUnifiedClean on Tanawin1701d/hls4ml? And is it a deal-breaker if we haven't setup (or know how to, or have time to learn how to) set up unit tests for CI?
We have successfully synthesized with the 2025.2 script many times, and seeing as it's Vivado-generated (per instructions), we reckon it's all right ;)

[JanFSchulte]

Hey @Tanawin1701d What is the status of this? From the discussion, I think there were a few more updates planned? It would be great to integrate this into a new release to come out for FastML in late August.

[JanFSchulte]

pre-commit.ci autofix

[JanFSchulte]

Looks good to me, but I'm also not a user and can't test it hands-on. Given that we have reports from actual users though, that should be fine. I have a few smaller comments, please have a look at those.

Then I'm mostly interested to know if this is considered ~complete now or if there are any outstanding items. For example, do the FIFO-sizing deadlock and the 64-char kernel name overflow reported by @gflengas still need to be addressed?

[JanFSchulte]

The assert requires >= 1, but the message says we support one output tensor. Should the assert be ==, or the message read "at least one"?

[JanFSchulte]

I think we need to use the hard-coded myproject here instead of {self._get_project_name(model)} to open the template file.

[JanFSchulte]

This seems to discard the arguments passed the this class' __init__ function and instead hard-codes these values. Same for the other driver templates.

[JanFSchulte]

enable_gie and disable_gie seem to set the same value.

[JanFSchulte]

Does that mean that we effectively support models with exactly 1 input and 1 output? That would conflict with other places in the code that suggest >=1 is supported.

[gflengas]

So in regards with the naming issue it has been resolved by nghielme#45 but it has not merged yet.

The fifo-sizing issue has been resolved by some additional changes from this PR Tanawin1701d#3 on top of the previous mentioned.

Tanawin1701d#3 also adds 2 more things on top of the fix:

1. Add set -e to link_system.sh to propagate v++ errors, especially useful for gitlab ci/cd runs.
2. Generate a routed vectorless power report from the implemented DCP. Copies all the relevant reports on the Final Reports folder that was already generated by this backend.

The 2nd one is user specific and could be omitted but i think its helpful.

I have also identify some additional bugs :

1. Vitis Unified AXI-stream wrapper didnt honor user-configured input/output buffer depths. It was defaulting to full-tensor FIFOs creating huge BRAM presure that from my testing wasnt necessary. Fix: gflengas@f4fce86
2. The user clock configuration wasnt propagated and instead was hardcoded. Fix: gflengas@bc9cc1b

Will do one more run overnight to validate these 2 one last time. I have also included the PR#1376 on my test branch, so i will report back tomorrow.

[JanFSchulte]

Thanks for the update. Looks like there's still a few things to integrate, but at least it looks there are fixes available for everything.

The CI tests for this are currently failing since they expect a Vitis/Vivado installation and hardcode the path. That will not work. We do now support running synthesis in CI tests, see for example https://github.com/fastmachinelearning/hls4ml/blob/main/test/pytest/test_keras_api.py#L79, which might need some adaptation for this case, I'm not sure.

The other big missing piece is documentation, of course.

[gflengas]

Quick update: the overnight runs passed. I’ve consolidated all the fixes discussed above on a branch based directly on pr/1376:

vitis-unified-upstream

For documentation, would a dedicated docs/backend/vitis_unified.rst make more sense than extending the existing accelerator docs?

We could also reuse the existing Vitis Unified tutorial material for the proposed Part 5 - FPGA inference in the revamped hls4ml tutorial:

• FDF hls4ml tutorial notebook
• Tanawin's tutorial repo

For the CI tests, I can have a look once I have some free time.