Skip to content

Commit 5567658

Browse files
authored
Update Image Evals Cookbook w/ Logs info (#1981)
1 parent ad69166 commit 5567658

File tree

1 file changed

+17
-1
lines changed

1 file changed

+17
-1
lines changed

examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -245,7 +245,7 @@
245245
"cell_type": "markdown",
246246
"metadata": {},
247247
"source": [
248-
"To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. While we won't dive into it in this cookbook, EvalsAPI also supports stored completions containing images as a data source. \n",
248+
"To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. Note that EvalsAPI also supports stored completions and responses containing images as a data source. See the [Additional Info: Logs Data Source](#additional-info-logs-data-source) section for more info.\n",
249249
"\n",
250250
"Here's the sampling message input we'll use for this example."
251251
]
@@ -509,6 +509,22 @@
509509
"print(json.dumps(dict(first_item), indent=2, default=str))"
510510
]
511511
},
512+
{
513+
"cell_type": "markdown",
514+
"metadata": {},
515+
"source": [
516+
"## Additional Info: Logs Data Source\n",
517+
"\n",
518+
"As mentioned earlier, EvalsAPI supports logs (i.e., stored completions or responses) containing images as a data source. To use this functionality, change your eval configurations as follows: \n",
519+
"\n",
520+
"Eval Creation\n",
521+
" - set `data_source_config = { \"type\": \"logs\" }`\n",
522+
" - revise templating in `grader_config` to use `{{item.input}}` and/or `{{sample.output_text}}`, denoting the input and output of the log\n",
523+
"\n",
524+
"Eval Run Creation\n",
525+
" - specify the filters in the `data_source` field that will be used to obtain the corresponding logs for the eval run (see the [docs](https://platform.openai.com/docs/api-reference/evals/createRun) for more information)"
526+
]
527+
},
512528
{
513529
"cell_type": "markdown",
514530
"metadata": {},

0 commit comments

Comments
 (0)