📢 Potential removal of dataframe field in Haystack Document - We need your feedback!

[anakin87]

Hello everyone,
We are considering removing the dataframe field from the Haystack Document dataclass and would appreciate your feedback to understand its impact.

Here's why we're exploring this:

• Text-based formats (e.g., CSV, Markdown) are widely used and work well with LLMs, even for tabular data.
• Pandas DataFrame creates serialization challenges (e.g., with Hayhooks).
• Pandas is a heavy dependency, complicating serverless deployments (like AWS Lambda). Making it optional might simplify these scenarios.
• Supporting dataframe consistently across different Document Stores requires complex workarounds.

💬 To make an informed decision, we'd like to hear from you:

• Are you using the dataframe field? If so, how?
• Are there workflows or use cases where it's indispensable?

Please share your thoughts and experiences in the comments. 🙏

[MrKrabXie]

It doesn't affect me, I think it works.

[Sprizgola]

I've never used it in my projects actually so it won't affect any of my workflows.

[lambda-science]

Never understood the point of Dataframe and blob field to be honest

[kelvin-lc]

I think it‘s nice to remove it. content and blob is enough

[srini047]

Not used though knew it existed.

[jlonge4]

Never used it! I think removal is a go 😎

[OscarIntellico]

I don't use it and it actually prevented me for using hayhooks in a deployment

[PaulBFB]

Haven't used it in any of my 3 previous projects and wasn't planning to

[anakin87]

Thank you everyone for your feedback!

Based on this, we have decided to deprecate and remove the dataframe field in Document.
Here's the gist: it will be deprecated in 2.10.0 and removed in 2.11.0.
Check out this issue for the full plan: #8738

[kaqmak]

Althoug I am late to the discussion, I would like to point out a problem with this:
I am currently using the AzureOCRDocumentConverter to convert pdf documents. Currently this splits it into Documents with text and Documents with data frames. For the data frames, ideally I don't want to split them (in a subsequent Splitter), since I would loose the column information. However, for some long tables, I end up exceeding the max tokens for embedding the (markdown converted) table. In this case, I would split the dataframe into smaller pieces ( a list of smaller data frames, each with the same columns), and then finally convert this into Markdown.
I guess you could do this using pure string manipulations on a Markdown (or CSV) textfile, but it certainly becomes way more complicated.