Reader Annotators

[mehmetbutgul]

Description

spark-nlp has already sparknlp.read.html(),
It returns improper schema to process pipelines,
The plan is to add new parameters such as explode(), filtering parameters, normalization parameters ...
base64...

Preferred Solution

create new transformers for the solution

Additional Context

Need to add this conversion on the transformer side

[yigitgull]

• Reader2Doc
• Reader2Table
• Reader2Image

Features

• Support for .md and .html inputs etc.
• Optional normalization parameter (e.g., cleaning HTML, markdown, etc.).
• Explode (like in SentenceDetector)
• splitStrategy parameter (e.g., page-by-page splitting)

Reader2Doc

• include\exclude sections(tables) etc.

Reader2Table

• Output as structured JSON

Reader2Image

• imageFormatParameter
• outputFormat (e.g., DATE64)