Adding ability to enable / disable online evaluation rules #2728
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR adds the ability to enable and disable online evaluation rules through a new boolean enabled field. This allows users to pause rule execution without deleting the rules or setting sampling rates to zero.
- Added
enabledfield to all automation rule models and API classes throughout the stack - Added UI support with a toggle switch in the rules table and filtering capabilities
- Updated evaluation logic to skip disabled rules during trace processing
Reviewed Changes
Copilot reviewed 34 out of 34 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Database migration | Adds enabled column to automation_rules table with default TRUE |
| Frontend types | Adds boolean column type and != filter operator for enabled field |
| Frontend components | Adds enabled column with toggle switch to rules tables |
| Backend models | Updates all automation rule models to include enabled field |
| Backend DAOs | Updates database queries to handle enabled field |
| Backend services | Updates rule evaluation logic to check enabled status before sampling |
apps/opik-frontend/src/components/pages/TracesPage/RulesTab/RulesTab.tsx
Show resolved
Hide resolved
apps/opik-frontend/src/components/pages/OnlineEvaluationPage/OnlineEvaluationPage.tsx
Show resolved
Hide resolved
...ain/java/com/comet/opik/api/resources/v1/events/TraceThreadOnlineScoringSamplerListener.java
Show resolved
Hide resolved
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
2 times, most recently
from
798f5fe to
3f27a00
Compare
SDK E2E Tests Results58 tests 57 ✅ 1m 51s ⏱️ For more details on these failures, see this check. Results for commit bb34eed. ♻️ This comment has been updated with latest results. |
Backend Tests Results 187 files 187 suites 19m 21s ⏱️ Results for commit f1dcd81. ♻️ This comment has been updated with latest results. |
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
8 times, most recently
from
0049ddc to
256f812
Compare
thiagohora
reviewed
Jul 16, 2025
...ain/java/com/comet/opik/api/resources/v1/events/TraceThreadOnlineScoringSamplerListener.java
Outdated
Show resolved
Hide resolved
thiagohora
reviewed
Jul 16, 2025
...d/src/main/java/com/comet/opik/domain/evaluators/LlmAsJudgeAutomationRuleEvaluatorModel.java
Outdated
Show resolved
Hide resolved
thiagohora
reviewed
Jul 16, 2025
...java/com/comet/opik/domain/evaluators/TraceThreadLlmAsJudgeAutomationRuleEvaluatorModel.java
Outdated
Show resolved
Hide resolved
thiagohora
reviewed
Jul 16, 2025
...t/opik/domain/evaluators/TraceThreadUserDefinedMetricPythonAutomationRuleEvaluatorModel.java
Outdated
Show resolved
Hide resolved
thiagohora
reviewed
Jul 16, 2025
...va/com/comet/opik/domain/evaluators/UserDefinedMetricPythonAutomationRuleEvaluatorModel.java
Outdated
Show resolved
Hide resolved
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
from
46870f1 to
7ad3953
Compare
...src/test/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResourceTest.java
Outdated
Show resolved
Hide resolved
.../main/resources/liquibase/db-app-state/migrations/000022_add_enabled_to_automation_rules.sql
Outdated
Show resolved
Hide resolved
andriidudar
previously approved these changes
Jul 17, 2025
apps/opik-frontend/src/components/pages/OnlineEvaluationPage/OnlineEvaluationPage.tsx
Outdated
Show resolved
Hide resolved
apps/opik-frontend/src/components/pages/TracesPage/RulesTab/RulesTab.tsx
Outdated
Show resolved
Hide resolved
andrescrz
previously requested changes
Jul 17, 2025
There was a problem hiding this comment.
This is generally in the right direction, but please address the open comments first.
Additionally, I'll send you the link for our PR guidelines. The Cursor automation is great, but we have the convention of self-reviewing PRs before submitting them to the team.
There are some gaps here that would speed up the review and merge process, if the contributor supervises first the PR.
apps/opik-backend/src/main/java/com/comet/opik/api/evaluators/AutomationRuleEvaluator.java
Outdated
Show resolved
Hide resolved
.../opik-backend/src/main/java/com/comet/opik/api/resources/v1/events/OnlineScoringSampler.java
Outdated
Show resolved
Hide resolved
...ain/java/com/comet/opik/api/resources/v1/events/TraceThreadOnlineScoringSamplerListener.java
Outdated
Show resolved
Hide resolved
.../main/resources/liquibase/db-app-state/migrations/000022_add_enabled_to_automation_rules.sql
Outdated
Show resolved
Hide resolved
.../main/resources/liquibase/db-app-state/migrations/000022_add_enabled_to_automation_rules.sql
Outdated
Show resolved
Hide resolved
...src/test/java/com/comet/opik/api/resources/v1/priv/AutomationRuleEvaluatorsResourceTest.java
Outdated
Show resolved
Hide resolved
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
3 times, most recently
from
bb34eed to
43ecd12
Compare
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
from
43ecd12 to
a516a40
Compare
andriidudar
requested changes
Jul 18, 2025
...pik-frontend/src/components/pages-shared/automations/AddEditRuleDialog/AddEditRuleDialog.tsx
Outdated
Show resolved
Hide resolved
...pik-frontend/src/components/pages-shared/automations/AddEditRuleDialog/AddEditRuleDialog.tsx
Outdated
Show resolved
Hide resolved
apps/opik-frontend/src/components/shared/DataTableHeaders/TypeHeader.tsx
Outdated
Show resolved
Hide resolved
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
4 times, most recently
from
f8ef216 to
a61e014
Compare
YarivHashaiComet
force-pushed
the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
from
a61e014 to
f1dcd81
Compare
andriidudar
approved these changes
Jul 18, 2025
YarivHashaiComet
deleted the
OPIK-2063-supprting-enable-disable-online-evaluations-rules
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding ability to enable & disable online evaluation rules