Add Response Specificity, Engagement, Topic Consistency, and Helpfulness Metrics

[ultimus11]

Summary

This PR introduces four new LLM-based conversational quality metrics to Opik:

1. ResponseSpecificityMetric

   • Scores based on how specific or vague the assistant's reply is.
   • Penalizes generic responses like “It depends,” “I'm not sure,” etc.

2. EngagementScoreMetric

   • Evaluates if the assistant's responses keep the user engaged.
   • Checks for emotional tone, follow-up questions, and curiosity.

3. TopicConsistencyMetric

   • Assesses whether the conversation stays on topic across multiple turns.
   • Identifies derailments or topic shifts in responses.

4. ResponseHelpfulnessMetric

   • Determines how actionable and useful the assistant's responses are.

Implementation Details

• Each metric:

  • Inherits from ConversationThreadMetric
  • Uses LLM-based templates to extract verdicts and reasons
  • Implements both synchronous (score) and async (ascore) scoring
  • Provides robust fallback error handling and JSON schema validation

• Added corresponding:

  • Schema classes (schema.py)
  • Prompt templates (templates.py)
  • Imports in __init__.py
  • Inline docstrings (where applicable)

Why This Is Valuable

These new metrics extend Opik’s ability to evaluate nuanced aspects of assistant conversations, improving its effectiveness in monitoring dialogue quality for chatbots and LLM-based systems.

Notes

• Can extend with additional few-shot examples or fine-tuned prompt logic later if needed.
• Fully modular with async support for scalability.

---

Let me know if maintainers have any specific checklist (e.g., test coverage or changelog updates).

[alexkuzmik]

Hi @ultimus11! You probably forgot to push the implementation:)