[SPARKNLP-1189] Introducing Phi4 (#14606)

* Add Phi-4 model implementation and tokenizer support

- Introduced `Phi4` class for the state-of-the-art Phi-4 model by Microsoft Research, including methods for encoding, decoding, and tagging.
- Added `Phi4Transformer` for integration with Spark NLP, enabling pretrained model loading and configuration.
- Implemented `Phi4Tokenizer` for byte pair encoding specific to the Phi-4 model.
- Updated `BpeTokenizer` to support the new Phi-4 tokenizer.
- Added tests for `Phi4Transformer` to ensure functionality and performance.
- Updated resource downloader to include `Phi4Transformer` for pretrained model access.

* Enhance documentation for Phi-4 model in `Phi4Transformer.scala`

- Expanded the model description to include detailed information on parameters, intended use, benchmarks, safety, limitations, and usage instructions.
- Improved formatting for better readability and clarity.
- Added references for further information on the Phi-4 model.

* Add Phi-4 Transformer implementation and integration

- Introduced `Phi4Transformer` class for the state-of-the-art Phi-4 model by Microsoft Research, enabling advanced reasoning and NLP tasks.
- Added support for loading pretrained models and configuration parameters.
- Implemented a loader for the Phi-4 model in the internal module.
- Created unit tests to validate the functionality of the `Phi4Transformer`.
- Updated the main `__init__.py` to include the new transformer in the module exports.

* Add documentation and example notebook for Phi-4 Transformer

- Created a detailed markdown file for the `Phi4Transformer`, outlining its features, usage, and pretrained model loading instructions.
- Added a Jupyter notebook demonstrating the integration of the Phi-4 model with Spark NLP and Intel OpenVINO, including installation steps and example code.
- Enhanced the Scala implementation comments for better clarity and formatting.
- Updated the test specifications to reflect changes in the handling of temperature settings during predictions.

Author: prabod

docs/en/transformer_entries/Phi4Transformer.md

@@ -0,0 +1,112 @@
+{%- capture title -%}
+Phi4Transformer
+{%- endcapture -%}
+
+{%- capture description -%}
+Text Generation using Microsoft Phi-4.
+
+Phi4Transformer loads the state-of-the-art Phi-4 model for advanced reasoning, code generation, and general NLP tasks. Phi-4 is a 14B parameter, dense decoder-only Transformer model trained on 9.8T tokens, with a 16K context length, and is best suited for prompts in chat format. The model is designed for high-quality, advanced reasoning, math, code, and general NLP, with a focus on English (multilingual data ~8%).
+
+**Key Features:**
+- 14B parameters, dense decoder-only Transformer
+- 16K context length
+- Trained on 9.8T tokens (synthetic, public domain, academic, Q&A, code)
+- Benchmarks: MMLU 84.8, HumanEval 82.6, GPQA 56.1, DROP 75.5, MATH 80.6
+- Safety alignment via SFT and DPO, red-teamed by Microsoft AIRT
+- Released under MIT License
+
+Pretrained models can be loaded with `pretrained` of the companion object:
+
+```scala
+val phi4 = Phi4Transformer.pretrained()
+     .setInputCols("document")
+     .setOutputCol("generation")
+```
+The default model is `"phi-4"`, if no name is provided.
+
+For available pretrained models please see the [Models Hub](https://huggingface.co/microsoft/phi-4).
+
+To see which models are compatible and how to import them see [Import Transformers into Spark NLP 🚀](https://github.com/JohnSnowLabs/spark-nlp/discussions/5669).
+{%- endcapture -%}
+
+{%- capture input_anno -%}
+DOCUMENT
+{%- endcapture -%}
+
+{%- capture output_anno -%}
+DOCUMENT
+{%- endcapture -%}
+
+{%- capture python_example -%}
+import sparknlp
+from sparknlp.base import *
+from sparknlp.annotator import *
+from pyspark.ml import Pipeline
+
+documentAssembler = DocumentAssembler() \
+    .setInputCol("text") \
+    .setOutputCol("documents")
+
+phi4 = Phi4Transformer.pretrained("phi-4") \
+    .setInputCols(["documents"]) \
+    .setMaxOutputLength(60) \
+    .setOutputCol("generation")
+
+pipeline = Pipeline().setStages([documentAssembler, phi4])
+
+data = spark.createDataFrame([
+    (1, "<|im_start|>system<|im_sep|>\nYou are a helpful assistant!\n<|im_start|>user<|im_sep|>\nWhat is Phi-4?\n<|im_start|>assistant<|im_sep|>\n")
+]).toDF("id", "text")
+
+result = pipeline.fit(data).transform(data)
+result.select("generation.result").show(truncate=False)
+{%- endcapture -%}
+
+{%- capture scala_example -%}
+import spark.implicits._
+import com.johnsnowlabs.nlp.base._
+import com.johnsnowlabs.nlp.annotator._
+import org.apache.spark.ml.Pipeline
+
+val documentAssembler = new DocumentAssembler()
+  .setInputCol("text")
+  .setOutputCol("documents")
+
+val phi4 = Phi4Transformer.pretrained("phi-4")
+  .setInputCols(Array("documents"))
+  .setMaxOutputLength(60)
+  .setOutputCol("generation")
+
+val pipeline = new Pipeline().setStages(Array(documentAssembler, phi4))
+
+val data = Seq(
+  (1, "<|im_start|>system<|im_sep|>\nYou are a helpful assistant!\n<|im_start|>user<|im_sep|>\nWhat is Phi-4?\n<|im_start|>assistant<|im_sep|>\n")
+).toDF("id", "text")
+
+val result = pipeline.fit(data).transform(data)
+result.select("generation.result").show(truncate = false)
+{%- endcapture -%}
+
+{%- capture api_link -%}
+[Phi4Transformer](/api/com/johnsnowlabs/nlp/annotators/seq2seq/Phi4Transformer.html)
+{%- endcapture -%}
+
+{%- capture python_api_link -%}
+[Phi4Transformer](/api/python/reference/autosummary/sparknlp/annotator/seq2seq/phi4_transformer/index.html)
+{%- endcapture -%}
+
+{%- capture source_link -%}
+[Phi4Transformer](https://github.com/JohnSnowLabs/spark-nlp/tree/master/src/main/scala/com/johnsnowlabs/nlp/annotators/seq2seq/Phi4Transformer.scala)
+{%- endcapture -%}
+
+{% include templates/anno_template.md
+  title=title
+  description=description
+  input_anno=input_anno
+  output_anno=output_anno
+  python_example=python_example
+  scala_example=scala_example
+  api_link=api_link
+  python_api_link=python_api_link
+  source_link=source_link
+%}