# Every Eval Ever Schema Review > Portable context for reviewing the aggregate and instance-level JSON schemas. Generated: 2026-10-23T03:55:11+00:00 ## Aggregate evaluation - Schema id: `aggregate` - Version: `0.3.0` - Source: https://raw.githubusercontent.com/evaleval/every_eval_ever/c6d861b9215fbd03f837c43f95fbf56289315e19/every_eval_ever/schemas/eval.schema.json - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): Schema for storing and validating LLMs evaluation data, including model configuration, prompts, instances, Output, and evaluation metrics - `schema_version` (string, required): Version of the schema used for this evaluation data - `evaluation_id` (string, required): Unique identifier for this specific evaluation run. Use eval_name/model_id/retrieved_timestamp format - `evaluation_timestamp` (string): Timestamp for when the evaluation was run - `retrieved_timestamp` (string, required): Timestamp for when this record was created - using Unix Epoch time format - `source_metadata` (object, required): Metadata about the source of the leaderboard data - `source_metadata.source_name` (string): Name of the source (e.g. title of the source leaderboard or name of the platform used for the evaluation). - `source_metadata.source_type` (string, required): Whether the data comes from a direct evaluation run or from documentation - `source_metadata.source_organization_name` (string, required): Name of the organization that provides the data - `source_metadata.source_organization_url` (string): URL for the organization that provides the data - `source_metadata.source_organization_logo_url` (string): URL for the Logo for the organization that provides the data - `source_metadata.evaluator_relationship` (string, required): Relationship between the evaluator and the model - `source_metadata.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `eval_library` (object, required): Evaluation library/framework used to run the evaluation - `eval_library.name` (string, required): Name of the evaluation library (e.g. inspect_ai, lm_eval, helm) - `eval_library.version` (string, required): Version of the evaluation library. Use 'unknown' if the version is not available. - `eval_library.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `model_info` (reference, required) - `model_info.name` (string, required): Model name provided by evaluation source - `model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models) - `model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI') - `model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI) - `model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama). - `model_info.inference_engine.name` (string): Name of the inference engine - `model_info.inference_engine.version` (string): Version of the inference engine - `model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results` (array, required): Array of evaluation results - `evaluation_results[]` (object) - `evaluation_results[].evaluation_result_id` (string): Stable identifier for this metric result inside an evaluation run. Recommended deterministic join key for instance-level records. - `evaluation_results[].evaluation_name` (string, required): Name of the evaluation - `evaluation_results[].source_data` (oneOf, required): Source of dataset for this evaluation: URL, HuggingFace dataset, or private/custom dataset. - `evaluation_results[].source_data.oneOf[0]` (reference) - `evaluation_results[].source_data.oneOf[0].dataset_name` (string, required): Name of the source dataset - `evaluation_results[].source_data.oneOf[0].source_type` (constant, required) - `evaluation_results[].source_data.oneOf[0].url` (array, required): URL(s) for the source of the evaluation data - `evaluation_results[].source_data.oneOf[0].url[]` (string) - `evaluation_results[].source_data.oneOf[0].additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].source_data.oneOf[1]` (reference) - `evaluation_results[].source_data.oneOf[1].dataset_name` (string, required): Name of the source dataset - `evaluation_results[].source_data.oneOf[1].source_type` (constant, required) - `evaluation_results[].source_data.oneOf[1].hf_repo` (string): HuggingFace repository identifier - `evaluation_results[].source_data.oneOf[1].hf_split` (string): One of train, val or test. - `evaluation_results[].source_data.oneOf[1].samples_number` (integer): Number of samples in the dataset - `evaluation_results[].source_data.oneOf[1].sample_ids` (array): Array of sample ids used for evaluation - `evaluation_results[].source_data.oneOf[1].sample_ids[]` (string) - `evaluation_results[].source_data.oneOf[1].additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].source_data.oneOf[2]` (reference) - `evaluation_results[].source_data.oneOf[2].dataset_name` (string, required): Name of the source dataset - `evaluation_results[].source_data.oneOf[2].source_type` (constant, required) - `evaluation_results[].source_data.oneOf[2].additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].evaluation_timestamp` (string): Timestamp for when the evaluations were run - `evaluation_results[].metric_config` (object, required): Details about the metric - `evaluation_results[].metric_config.evaluation_description` (string): Description of the evaluation - `evaluation_results[].metric_config.metric_id` (string): Stable metric identifier for joining/deduping/querying. Use a canonical global id when applicable (e.g. accuracy, f1_macro, auroc, rmse, pass_at_k). For benchmark/leaderboard-specific metrics, use a namespaced id (e.g. rewardbench.overall, lmarena.elo). - `evaluation_results[].metric_config.metric_name` (string): Display name for the metric (e.g. Accuracy, F1-macro, pass@1). - `evaluation_results[].metric_config.metric_kind` (string): Normalized metric family/type used for safe aggregation (e.g. accuracy, f1, auroc, rmse, mae, pass_rate, elo). - `evaluation_results[].metric_config.metric_unit` (string): Unit of the metric values (e.g. proportion, percent, points, ms, tokens). - `evaluation_results[].metric_config.metric_parameters` (object): Metric-specific parameters (e.g. {"k": 1} for pass@k). - `evaluation_results[].metric_config.lower_is_better` (boolean, required): Whether a lower score is better - `evaluation_results[].metric_config.score_type` (string): Type of score - `evaluation_results[].metric_config.level_names` (array): Names of the score levels - `evaluation_results[].metric_config.level_names[]` (string) - `evaluation_results[].metric_config.level_metadata` (array): Additional Description for each Score Level - `evaluation_results[].metric_config.level_metadata[]` (string) - `evaluation_results[].metric_config.has_unknown_level` (boolean): Indicates whether there is an Unknown Level - if True, then a score of -1 will be treated as Unknown - `evaluation_results[].metric_config.min_score` (number): Minimum possible score for continuous metric - `evaluation_results[].metric_config.max_score` (number): Maximum possible score for continuous metric - `evaluation_results[].metric_config.llm_scoring` (object): Configuration when LLM is used as scorer/judge - `evaluation_results[].metric_config.llm_scoring.judges` (array, required): LLM judge(s) - single item for judge, multiple for jury - `evaluation_results[].metric_config.llm_scoring.judges[]` (reference) - `evaluation_results[].metric_config.llm_scoring.judges[].model_info` (reference, required) - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.name` (string, required): Model name provided by evaluation source - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models) - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI') - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI) - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama). - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine.name` (string): Name of the inference engine - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.inference_engine.version` (string): Version of the inference engine - `evaluation_results[].metric_config.llm_scoring.judges[].model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].metric_config.llm_scoring.judges[].temperature` (number) - `evaluation_results[].metric_config.llm_scoring.judges[].weight` (number): Weight of this judge's score in aggregation (used in jury) - `evaluation_results[].metric_config.llm_scoring.judges[].additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].metric_config.llm_scoring.input_prompt` (string, required): Prompt template used for judging - `evaluation_results[].metric_config.llm_scoring.aggregation_method` (string): How to aggregate scores when multiple judges - `evaluation_results[].metric_config.llm_scoring.expert_baseline` (number): Expert/human baseline score for comparison - `evaluation_results[].metric_config.llm_scoring.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].metric_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].score_details` (object, required): The score for the evaluation and related details - `evaluation_results[].score_details.score` (number, required): The score for the evaluation - `evaluation_results[].score_details.details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].score_details.uncertainty` (object): Quantification of uncertainty around the reported score - `evaluation_results[].score_details.uncertainty.standard_error` (object): Standard error of the score estimate (SE_mean = standard_deviation / sqrt(num_samples)) - `evaluation_results[].score_details.uncertainty.standard_error.value` (number, required): The standard error value - `evaluation_results[].score_details.uncertainty.standard_error.method` (string): How the standard error was computed (e.g. 'analytic', 'bootstrap', 'jackknife') - `evaluation_results[].score_details.uncertainty.confidence_interval` (object): Lower and upper bounds for the metric at a given confidence level. - `evaluation_results[].score_details.uncertainty.confidence_interval.lower` (number, required): Lower bound of the confidence interval - `evaluation_results[].score_details.uncertainty.confidence_interval.upper` (number, required): Upper bound of the confidence interval - `evaluation_results[].score_details.uncertainty.confidence_interval.confidence_level` (number): Confidence level (e.g. 0.95 for a 95% confidence interval) - `evaluation_results[].score_details.uncertainty.confidence_interval.method` (string): How the confidence interval was computed - `evaluation_results[].score_details.uncertainty.standard_deviation` (number): Standard deviation of the per-sample scores - `evaluation_results[].score_details.uncertainty.num_samples` (integer): Number of samples used to compute the uncertainty estimates - `evaluation_results[].score_details.uncertainty.num_bootstrap_samples` (integer): Number of bootstrap resamples used, if bootstrap method was applied - `evaluation_results[].generation_config` (object) - `evaluation_results[].generation_config.generation_args` (object): Parameters used to generate results - properties may vary by model type - `evaluation_results[].generation_config.generation_args.temperature` (null | number): Sampling temperature - `evaluation_results[].generation_config.generation_args.top_p` (null | number): Nucleus sampling parameter - `evaluation_results[].generation_config.generation_args.top_k` (null | number): Top-k sampling parameter - `evaluation_results[].generation_config.generation_args.max_tokens` (integer): Maximum number of tokens to generate - `evaluation_results[].generation_config.generation_args.execution_command` (string): Command used to run the model to generate results - `evaluation_results[].generation_config.generation_args.reasoning` (boolean): Whether reasoning orchain-of-thought was used to generate results - `evaluation_results[].generation_config.generation_args.prompt_template` (string): Input prompt template for task (should contain agentic info if needed). - `evaluation_results[].generation_config.generation_args.agentic_eval_config` (object): General configuration for agentic evaluations. - `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools` (array): List of all available tools with their configurations - `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[]` (object) - `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].name` (string): e.g. bash, calculator, ... - `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].description` (string) - `evaluation_results[].generation_config.generation_args.agentic_eval_config.available_tools[].parameters` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].generation_config.generation_args.agentic_eval_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].generation_config.generation_args.eval_plan` (object): Plan (solvers) used in evaluation. Solvers are crucial parts of Inspect evaluations which can serve a wide variety of purposes like providing system prompts, prompt engineering, model generation or multi-turn dialog. - `evaluation_results[].generation_config.generation_args.eval_plan.name` (string) - `evaluation_results[].generation_config.generation_args.eval_plan.steps` (array): Array of evaluation plan steps (each step is a JSON-serialized string) - `evaluation_results[].generation_config.generation_args.eval_plan.steps[]` (string) - `evaluation_results[].generation_config.generation_args.eval_plan.config` (object): Additional parameters (key-value pairs, all values must be strings) - `evaluation_results[].generation_config.generation_args.eval_limits` (object): Listed evaluation limits like time limit, message limit, token limit. - `evaluation_results[].generation_config.generation_args.eval_limits.time_limit` (integer): Time limit for evaluation. - `evaluation_results[].generation_config.generation_args.eval_limits.message_limit` (integer): Message limit for evaluation. - `evaluation_results[].generation_config.generation_args.eval_limits.token_limit` (integer): Token limit for evaluation. - `evaluation_results[].generation_config.generation_args.sandbox` (object) - `evaluation_results[].generation_config.generation_args.sandbox.type` (string): Type of sandbox e.g. docker - `evaluation_results[].generation_config.generation_args.sandbox.config` (string): Config file name/path e.g. compose.yaml. TODO or full config? Not sure based on the Inspect docs - `evaluation_results[].generation_config.generation_args.max_attempts` (integer): Maximum number of submission attempts (default 1). - `evaluation_results[].generation_config.generation_args.incorrect_attempt_feedback` (string): Feedback from the model after incorrect attempt. - `evaluation_results[].generation_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `detailed_evaluation_results` (object): Reference to the evaluation results for all individual samples in the evaluation - `detailed_evaluation_results.format` (string): Format of the detailed evaluation results - `detailed_evaluation_results.file_path` (string): Path to the detailed evaluation results file - `detailed_evaluation_results.hash_algorithm` (string): Hash algorithm used for checksum and sample_hash in instance-level data - `detailed_evaluation_results.checksum` (string): Checksum value of the file - `detailed_evaluation_results.total_rows` (integer): Total number of rows in the detailed evaluation results file - `detailed_evaluation_results.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.judge_config` (object): Configuration for a single LLM judge/juror - `$defs.judge_config.model_info` (reference, required) - `$defs.judge_config.model_info.name` (string, required): Model name provided by evaluation source - `$defs.judge_config.model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models) - `$defs.judge_config.model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI') - `$defs.judge_config.model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI) - `$defs.judge_config.model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama). - `$defs.judge_config.model_info.inference_engine.name` (string): Name of the inference engine - `$defs.judge_config.model_info.inference_engine.version` (string): Version of the inference engine - `$defs.judge_config.model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.judge_config.temperature` (number) - `$defs.judge_config.weight` (number): Weight of this judge's score in aggregation (used in jury) - `$defs.judge_config.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.model_info` (object): Complete model specification including basic information, technical configuration and inference settings - `$defs.model_info.name` (string, required): Model name provided by evaluation source - `$defs.model_info.id` (string, required): Model name in HuggingFace format (e.g. meta-llama/Llama-3.1-8B-Instruct for models available on HuggingFace or openai/azure/gpt-4o-mini-2024-07-18 for closed API models) - `$defs.model_info.developer` (string): Name of organization that provides the model (e.g. 'OpenAI') - `$defs.model_info.inference_platform` (string): Name of inference platform which provides an access to models by API to run the evaluations or provides models weights to run them locally (e.g. HuggingFace, Bedrock, Together AI) - `$defs.model_info.inference_engine` (object): Name of inference engine which provides an access to optimized models to use them for local evaluations (e.g. vLLM, Ollama). - `$defs.model_info.inference_engine.name` (string): Name of the inference engine - `$defs.model_info.inference_engine.version` (string): Version of the inference engine - `$defs.model_info.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.source_data_url` (object): URL source for the evaluation data - `$defs.source_data_url.dataset_name` (string, required): Name of the source dataset - `$defs.source_data_url.source_type` (constant, required) - `$defs.source_data_url.url` (array, required): URL(s) for the source of the evaluation data - `$defs.source_data_url.url[]` (string) - `$defs.source_data_url.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.source_data_hf` (object): Details about HuggingFace dataset used for evaluation - `$defs.source_data_hf.dataset_name` (string, required): Name of the source dataset - `$defs.source_data_hf.source_type` (constant, required) - `$defs.source_data_hf.hf_repo` (string): HuggingFace repository identifier - `$defs.source_data_hf.hf_split` (string): One of train, val or test. - `$defs.source_data_hf.samples_number` (integer): Number of samples in the dataset - `$defs.source_data_hf.sample_ids` (array): Array of sample ids used for evaluation - `$defs.source_data_hf.sample_ids[]` (string) - `$defs.source_data_hf.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) - `$defs.source_data_private` (object): Generic source data when neither URL array nor HuggingFace dataset applies - `$defs.source_data_private.dataset_name` (string, required): Name of the source dataset - `$defs.source_data_private.source_type` (constant, required) - `$defs.source_data_private.additional_details` (object): Additional parameters (key-value pairs, all values must be strings) ### Feedback summary # Review summary: Aggregate evaluation 0.3.0 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Instance-level evaluation - Schema id: `instance` - Version: `instance_level_eval_0.3.0` - Source: https://raw.githubusercontent.com/evaleval/every_eval_ever/c6d861b9215fbd03f837c43f95fbf56289315e19/every_eval_ever/schemas/instance_level_eval.schema.json - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (allOf): Schema for storing instance-level evaluation data for LLM evaluations, supporting single-turn, multi-turn, and agentic interactions - `schema_version` (string, required): Version of the schema used for this instance data - `evaluation_id` (string, required): Foreign key linking to the aggregate evaluation JSON. Must match the evaluation_id in the aggregate file. - `model_id` (string, required): Identifier of the model in HuggingFace format (e.g. meta-llama/Llama-3.2-1B-Instruct) - `evaluation_name` (string, required): The specific eval name, ideally unique (e.g. GSM8K, mmlu_physics). Primarily for display/filtering when evaluation_result_id is unavailable. If an evaluation run reports multiple aggregate results/metrics for the same sample, emit multiple instance records (one per aggregate result) rather than trying to attach a single instance record to many aggregate results. - `evaluation_result_id` (string): Preferred deterministic foreign key to aggregate evaluation_results[].evaluation_result_id. This is intended to point to exactly one aggregate evaluation result; if a single underlying interaction/sample contributes to multiple aggregate results, emit multiple instance records with different evaluation_result_id values. - `sample_id` (string, required): Question/sample identifier from the original dataset (e.g. gsm8k_0001) - `sample_hash` (string | null): Hash of (input.raw + input.reference) to ensure comparison is between the same sample across models, in case sample_id is not consistent - `interaction_type` (string, required): Type of interaction: single_turn for simple Q&A, multi_turn for conversations, agentic for tool-using agents - `trajectory` (oneOf): Agent trajectory embedded in this instance or referenced as a module/file. Event IDs are scoped to the referenced trajectory document. - `trajectory.oneOf[0]` (reference) - `trajectory.oneOf[1]` (reference) - `trajectory.oneOf[1].type` (constant, required) - `trajectory.oneOf[1].uri` (string, required): URI or relative path to the referenced trajectory. - `trajectory.oneOf[1].media_type` (string) - `trajectory.oneOf[1].format` (string): Trajectory encoding or interchange format, e.g. ATIF or JSONL. - `trajectory.oneOf[1].schema_uri` (string) - `trajectory.oneOf[1].schema_version` (string) - `trajectory.oneOf[1].sha256` (string): Optional SHA-256 digest of the referenced file. - `input` (object, required): Input data for the evaluation sample - `input.raw` (string, required): The raw input as defined in the eval - `input.formatted` (string | null): Includes chat template, CoT and all relevant modifications - basically the un-tokenized version of what the model sees - `input.reference` (array, required): Ground truths or reference answers for comparison/scoring - `input.reference[]` (string) - `input.choices` (array | null): Optional list of choices for multiple-choice questions - `input.choices[]` (string) - `output` (object | null): Output data - only used for single_turn interactions, null for multi_turn/agentic - `output.raw` (array, required): Complete model responses - `output.raw[]` (string) - `output.reasoning_trace` (array | null): Reasoning traces of the model if applicable (e.g. chain-of-thought tokens) - `output.reasoning_trace[]` (string) - `messages` (array | null): Full message transcript - used for multi_turn and agentic, null for single_turn. Contains all system, user, assistant, and tool messages in order. - `messages[]` (object) - `messages[].turn_idx` (integer, required): Index starting from 0 indicating the position in the conversation - `messages[].role` (string, required): Role of the speaker (e.g. user, assistant, system, tool) - `messages[].content` (string | null): The actual raw text for that particular turn (can be null if empty) - `messages[].reasoning_trace` (string | null): Reasoning trace for that particular turn if applicable - `messages[].tool_calls` (array | null): List of tool invocations for this turn, if applicable - `messages[].tool_calls[]` (object) - `messages[].tool_calls[].id` (string, required): Unique identifier for the tool call - `messages[].tool_calls[].name` (string, required): Name of tool/function - `messages[].tool_calls[].arguments` (object | null): Arguments used to call the tool (all values must be strings) - `messages[].tool_call_id` (array | null): Reference to the tool call ID(s) this message has the content payload for. - `messages[].tool_call_id[]` (string) - `answer_attribution` (array, required): Information about how the answer was extracted from the model output - `answer_attribution[]` (anyOf) - `answer_attribution[].turn_idx` (integer): Legacy turn index in messages. Retained for compatibility; event_id is the stable trajectory pointer. - `answer_attribution[].event_id` (string): Stable ID of the trajectory event that supports this answer. Resolve within the linked trajectory document. - `answer_attribution[].source` (string, required): Source of the extracted value (e.g. 'output.raw' or 'messages[turn_idx].content') - `answer_attribution[].extracted_value` (string, required): Value that was extracted - `answer_attribution[].extraction_method` (string, required): Method used to extract the value (e.g. regex, exact_match, llm_judge, custom) - `answer_attribution[].is_terminal` (boolean, required): Whether this is the final answer (false if intermediate outputs are used to build up to a final answer) - `answer_attribution[].anyOf[0]` (field) - `answer_attribution[].anyOf[1]` (field) - `evaluation` (object, required): Evaluation results and scoring data - `evaluation.score` (number, required): Instance-level score - `evaluation.is_correct` (boolean, required): Whether the final answer is correct - `evaluation.num_turns` (integer | null): Number of turns in the interaction - `evaluation.tool_calls_count` (integer | null): Count of tool calls across all turns in messages - `token_usage` (object | null): Token usage for the model completion - `token_usage.input_tokens` (integer, required): Total input tokens used - `token_usage.output_tokens` (integer, required): Total output tokens used - `token_usage.total_tokens` (integer, required): Total tokens used - `token_usage.input_tokens_cache_write` (integer | null): Number of tokens written to the cache - `token_usage.input_tokens_cache_read` (integer | null): Number of tokens retrieved from the cache - `token_usage.reasoning_tokens` (integer | null): Number of tokens used for reasoning - `performance` (object | null): Performance and latency metrics - `performance.latency_ms` (number | null): Total latency in milliseconds - `performance.time_to_first_token_ms` (number | null): Time to first token in milliseconds - `performance.generation_time_ms` (number | null): Time for generation in milliseconds - `performance.additional_details` (object | null): Additional performance metrics (key-value pairs, all values must be strings) - `error` (string | null): Information about any error that occurred (e.g. timeout, refusal, API error) - `metadata` (object | null): Optional metadata about the sample (e.g. subject, difficulty, tags) - `$.allOf[0]` (field) - `$.allOf[1]` (field) ### Feedback summary # Review summary: Instance-level evaluation instance_level_eval_0.3.0 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Harbor Agent Trajectory Interchange Format (ATIF) - Schema id: `harbor-atif-v1-8` - Version: `ATIF RFC v1.8` - Source: https://github.com/harbor-framework/harbor/blob/main/rfcs/0001-trajectory-format.md - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): ATIF is Harbor?s versioned JSON trajectory format. A trajectory records agent configuration and ordered system, user, and agent steps, with optional tool calls, observations, per-step metrics, final totals, and subagent references. Scope: This is an interaction trace format, not an aggregate leaderboard record. The current RFC is v1.8; it defines root versioning, step ordering, tool-call/result references, and multimodal content rules. - `schema_version` (string): Declares the ATIF compatibility version for the trajectory. The root field is required. The cited RFC is v1.8; the value is a version tag such as ATIF-v1.8. Consumers use it to select rules for trajectory, step, content-part, and subagent fields. - `session_id` (string): Identifies the logical agent run that one or more trajectory documents belong to. This root field is optional and run-scoped. Parent, continuation, and sibling subagent trajectories may share a session_id, so it is not a unique document key and cannot resolve a subagent reference by itself. - `trajectory_id` (string): Identifies one trajectory document independently of the overall run. The root field is optional for a standalone trajectory but required on an embedded subagent trajectory. IDs must be unique within the parent?s subagent_trajectories array. A reference must use trajectory_id for an embedded trajectory or trajectory_path for an external file; session_id alone is insufficient. - `agent` (object) - `agent.name` (object): Names and versions the agent system that produced this trajectory. The agent object is required and contains required name and version strings. Optional model_name, tool_definitions, and extra fields record the default model, available tool signatures, and implementation-specific configuration. A step-level model_name may override the agent default. - `steps` (array): Stores the ordered interaction history for the run. The steps array is required. Each step represents a system prompt, user message, or agent turn and has a required ordinal step_id beginning at 1. A step may include an ISO 8601 timestamp, source, model, message, reasoning content, tool calls, observations, metrics, and custom extra data. - `steps[]` (object) - `steps[].source` (array): Identifies who or what originated an ATIF step. This required step field uses one of the RFC values system, user, or agent. Tool execution and environment feedback are represented through tool_calls and observation rather than additional source values. - `steps[].message` (array): Stores the message associated with a system, user, or agent step. The field is required and may be an empty string. Text-only content is a string; multimodal content uses ContentPart objects. In v1.8, content parts can represent text, images, or audio. Media bytes are stored separately and referenced by path or URL. - `steps[].tool_calls` (array): Records structured tool actions initiated by an agent step. A single agent response may contain multiple tool calls. Each call has a tool_call_id, function_name, and arguments; observation results can refer back to the matching call ID. Tool calls are optional and apply to agent steps. - `steps[].observation` (object) - `steps[].observation.results` (array): Stores environment feedback after a tool action or other operation. Each result can include content and an optional source_call_id. When present, source_call_id is the matching tool_call_id; when omitted, the result may come from a non-tool action or system-initiated operation. - `final_metrics` (object): Stores optional totals for the complete trajectory. All fields inside final_metrics are optional: total_prompt_tokens, total_completion_tokens, total_cached_tokens, total_cost_usd, total_steps, and extra. Prompt totals include cached tokens, while cached_tokens is the subset. Cost is an execution-time USD snapshot; ATIF does not embed a token-price table. - `subagent_trajectories` (array): Embeds complete trajectories for delegated subagents in the parent document. Each item is an independently valid ATIF Trajectory with its own schema_version, trajectory_id, agent, and step sequence. Embedded and external-file references may be mixed. The parent?s trajectory_id values in this array must be unique, and each reference must resolve through trajectory_id or trajectory_path. ### Feedback summary # Review summary: Harbor Agent Trajectory Interchange Format (ATIF) ATIF RFC v1.8 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Pi coding-agent session format - Schema id: `pi-session-v3` - Version: `Pi session format v3` - Source: https://github.com/earendil-works/pi/blob/main/packages/coding-agent/docs/session-format.md - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): Pi stores agent sessions as JSONL: one typed JSON record per line. A header identifies the session and version; subsequent entries use id and parentId links to form a branchable conversation tree. Scope: This documents Pi coding-agent history, not Harbor ATIF or a general evaluation-result schema. The current session version is v3; outer entry timestamps are ISO 8601 while nested message timestamps are Unix milliseconds. - `header` (string): The first JSONL record identifies the session and supplies initial metadata. The header has type=session and includes format version, session ID, timestamp, and working-directory path. Sessions with a parent may include parentSession. It is metadata only and is not a node in the entry tree. - `first line` (object) - `first line.version` (string): Selects the session format rules from the first-line header. Version 1 is a legacy linear sequence and is migrated on load; version 2 introduced id/parentId tree links; version 3 renamed the hookMessage role to custom. Pi currently migrates loaded sessions to v3. - `first line.cwd` (object): Records the working-directory context associated with the session. The header stores cwd as a path, and the documented example uses a project directory. This is local environment metadata rather than a portable project identifier; inspect the value before sharing a session file. - `entries` (array) - `entries[]` (object) - `entries[].id` (string): Uniquely identifies an entry within a session tree. Every non-header entry has an id. The documented representation is usually an eight-character hexadecimal ID, with a full UUID as a fallback. Other entries and labels may refer to this ID. - `entries[].parentId` (string): Connects an entry to its parent and reconstructs the active branch. A root entry has parentId=null; each non-root entry points to its parent. Branching creates multiple children from an earlier entry. Reset operations can create additional roots, so a session is a tree rather than a single linear transcript. - `entries[].type` (string): Selects the typed session-entry structure for each JSONL line. All lines after the header extend a base record with type, id, parentId, and timestamp. Documented types include message, model_change, thinking_level_change, usage, compaction, context_edit, branch_summary, custom, custom_message, label, and session_info. - `entries[].message` (object) - `entries[].message.content` (string): Contains the structured message or content blocks for a message entry. Message entries hold an AgentMessage, whose role, content blocks, tool calls, model/provider, usage, and nested timestamp are defined by Pi?s message types. Text and image blocks are supported. The outer entry timestamp is ISO 8601; a nested message timestamp is Unix milliseconds. - `entries[].timestamp` (string): Records when an outer session entry was created. Every non-header entry has an ISO 8601 timestamp. For message entries, the nested AgentMessage timestamp is a separate Unix timestamp in milliseconds; do not treat the two fields as the same representation. ### Feedback summary # Review summary: Pi coding-agent session format Pi session format v3 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Inspect AI EvalLog - Schema id: `inspect-eval-log` - Version: `Inspect EvalLog format v2` - Source: https://inspect.aisi.org.uk/eval-logs.html - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): Inspect EvalLog is the run record produced by Inspect AI. It contains task and execution configuration, aggregate scorer results, usage statistics, sample evidence, and post-run metadata edits. Scope: The documented EvalLog file format is version 2. Inspect supports a compact .eval representation and readable JSON export; the log API is the supported way to read and write logs. Check status before treating results as a completed successful evaluation. The header omits samples; reductions are included for .eval logs but not JSON logs. - `version` (string): Integer file-format version; the current reference identifies version 2. This is the serialization version, not the version of the task, model, or Inspect package. Use the format documentation that matches the file version. - `status` (string): Evaluation lifecycle state. The reference lists started, success, and error. Inspect recommends checking for success before analyzing a run as a completed evaluation. Error logs may still contain information about completed samples. - `eval` (object): Top-level evaluation specification, including task, model, and creation time. Use this object to identify what was evaluated and the run-level context. It is distinct from the execution plan and the computed result. - `plan` (string): Execution plan containing the solvers and model generation configuration used for the evaluation. This records how Inspect executed the task. Compare it when interpreting results from runs with different solver or generation settings. - `results` (array): Aggregate results computed by the evaluation scorers. These are evaluation-level scorer outputs; inspect the associated samples and scorer definitions when validating how an aggregate was produced. - `stats` (object): Evaluation and model usage statistics, including input and output token usage. The reference describes this as usage statistics. Do not assume every provider or run reports every possible usage measure. - `samples` (array): Per-sample records, including input, output, target, and score. Sample records can be large and may include media or duplicated transcript content. Header-only reads omit samples; the API provides incremental sample-reading helpers. - `error` (object): Error information for a failed evaluation, including traceback details when available. Read together with status. A log can exist after an error or interruption, so its presence alone does not mean the run succeeded. - `tags` (array): Current evaluation tags, combining evaluation-time tags with later edits. This is the current view of tags. The log_updates field records post-evaluation changes and their provenance. - `metadata` (object): Current evaluation metadata, combining evaluation-time metadata with later edits. Use log_updates when you need the history and provenance of post-evaluation changes, rather than only the current metadata value. - `log_updates` (array): Post-evaluation edits to tags and metadata, with provenance information. This preserves the edit history that explains how the current tags and metadata were reached. - `reductions` (array): Per-sample reductions used for evaluations with multiple epochs. The reference notes that reductions are present in .eval logs and not in JSON logs. Do not assume the same availability across file representations. - `location` (string): URI identifying the storage location from which an EvalLog was read or to which it was written. The reference describes this as an EvalLog property and says write_eval_log can use it when no explicit destination is supplied. It may be a file, S3, or another fsspec-supported URI. Do not confuse it with a local-only path. ### Feedback summary # Review summary: Inspect AI EvalLog Inspect EvalLog format v2 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## OpenTelemetry GenAI semantic conventions - Schema id: `otel-genai` - Version: `OpenTelemetry GenAI conventions ? Development` - Source: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): OpenTelemetry GenAI semantic conventions describe spans and attributes for model calls, agent operations, and tool execution. They provide interoperable trace vocabulary, not a benchmark score schema. Scope: The cited GenAI span conventions are marked Development and may evolve. Content-bearing attributes such as input and output messages can contain sensitive data; instrumentation controls whether to record them. Keep the convention revision with any comparison. - `gen_ai` (object) - `gen_ai.operation` (object) - `gen_ai.operation.name` (string): Names the GenAI operation represented by a span. Use a documented well-known operation value when it applies, such as chat, embeddings, execute_tool, invoke_agent, or retrieval. The convention permits a custom value when none of the listed values fits. - `gen_ai.provider` (object) - `gen_ai.provider.name` (string): Identifies the GenAI provider as understood by the client or server instrumentation. The instrumentation may not know the ultimate model provider when requests pass through a proxy or hosting platform. The convention says request/response model and server address can help distinguish such cases. - `gen_ai.request` (object) - `gen_ai.request.model` (string): Records the model name requested by the caller. Preserve this separately from gen_ai.response.model: a provider can report a different model for the returned response. - `gen_ai.response` (object) - `gen_ai.response.model` (string): Records the model name reported by the provider for the response. This is the response-side model identity, while gen_ai.request.model is caller supplied. Their values may differ. - `gen_ai.usage` (object) - `gen_ai.usage.input_tokens` (number): Records the number of tokens used in the GenAI input or prompt. The convention says this total should include all input-token types, including cached tokens, and recommends best-effort population from provider totals or reported token categories. - `gen_ai.usage.output_tokens` (number): Records the number of tokens used in the generated response or completion. This is the response-token count. Provider accounting details can differ, so retain the instrumentation and provider context when comparing counts. - `gen_ai.input` (object) - `gen_ai.input.messages` (array): Carries the structured messages supplied to the model when instrumentation records message content. Messages must be in the order sent. The convention warns that this attribute can contain sensitive or personally identifying information; filter or truncate only under the instrumentation policy. - `gen_ai.output` (object) - `gen_ai.output.messages` (array): Carries structured model output, with one message per generated choice or candidate. The convention warns that output can contain sensitive information. A single choice maps to one message; do not combine multiple choices into one message. - `gen_ai.tool` (object) - `gen_ai.tool.name` (string): Names the tool used by an agent. On a GenAI execute_tool span, the current convention requires this attribute; it identifies the tool and is distinct from the tool-call identifier. - `error` (object) - `error.type` (string): Describes the class of error that ended an operation. OpenTelemetry recommends predictable, low-cardinality values and says not to set this on successful operations. The error convention provides examples such as timeout and HTTP status values. ### Feedback summary # Review summary: OpenTelemetry GenAI semantic conventions OpenTelemetry GenAI conventions ? Development / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Exgentic Agent LLM Traces v2 - Schema id: `exgentic-traces-v2` - Version: `v2 · card checked 2026-10-12` - Source: https://huggingface.co/datasets/Exgentic/agent-llm-traces-v2 - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): Exgentic Agent LLM Traces v2 is a Parquet dataset of chat-only OpenTelemetry spans from 10,057 agent runs across six benchmarks. It retains benchmark/session aggregates and structured span records for replay and behavioral analysis. Scope: The reviewed card reports 241,674 retained chat spans in nine Parquet shards. The spans column is a list of span records; each record carries OpenTelemetry identity/timing/status fields and an attributes map. The filter keeps chat spans with non-empty gen_ai.input.messages, removes user-simulator and judge chats, collapses failed retry groups, drops sessions with no surviving span, and projects attributes to 18 documented GenAI keys. It does not represent the complete agent execution: tool-execution and invoke_agent spans and Exgentic provenance attributes are absent. The row and span paths below are review navigation paths; dotted GenAI attribute names are literal attribute keys. - `session_id` (string): Identifies the evaluated session row. These are top-level row columns documented by the dataset card; retain them when grouping spans by run or benchmark. - `run_id` (string): Identifies the associated run. These are top-level row columns documented by the dataset card; retain them when grouping spans by run or benchmark. - `harness` (string): Names the agent harness recorded for the session. These are top-level row columns documented by the dataset card; retain them when grouping spans by run or benchmark. - `benchmark` (string): Names the benchmark associated with the run. These are top-level row columns documented by the dataset card; retain them when grouping spans by run or benchmark. - `benchmark_subset` (string): Identifies the benchmark subset recorded for the run. These are top-level row columns documented by the dataset card; retain them when grouping spans by run or benchmark. - `session aggregates` (string): Per-session values include score, success, status, steps, action_count, agent_cost, benchmark_cost, execution_time, total_tokens, and max_tokens. These are session-level aggregates. The dataset card says totals are recomputed from the surviving filtered spans; they do not include the removed evaluation scaffolding. - `models` (array): Lists the canonical model names associated with spans in the session. The card states routing prefixes are stripped and the top-level models list is kept consistent with the canonical model names in span attributes. - `spans` (array) - `spans[]` (object): Each row contains a list of OpenTelemetry span records; the reviewed release reports 241,674 chat spans across 10,057 session rows. The dataset is chat-only: every retained span has gen_ai.operation.name equal to chat and a non-empty gen_ai.input.messages attribute. It is not a complete tool/action trajectory. - `spans[].span_id` (string): OpenTelemetry span identifier. The dataset card describes this as a hexadecimal span ID. - `spans[].trace_id` (string): OpenTelemetry trace identifier. The dataset card describes this as a hexadecimal trace ID used to group spans. - `spans[].parent_span_id` (string): Parent span identifier, or null when no parent span is recorded. Use this link when reconstructing retained span relationships; excluded span types may leave gaps in the original execution tree. - `spans[].name` (string): Span name. Interpret with kind and gen_ai.operation.name; it is not the benchmark task label. - `spans[].kind` (string): OpenTelemetry span kind. The card gives values such as SPAN_KIND_INTERNAL and SPAN_KIND_CLIENT. - `spans[].start_time` (string): Span start timestamp. The card specifies ISO-8601 UTC timestamps; filtering can adjust times after removing failed retry groups. - `spans[].end_time` (string): Span end timestamp. The card specifies ISO-8601 UTC timestamps; filtering can adjust times after removing failed retry groups. - `spans[].status_code` (string): Numeric span status code. The card uses 1 for OK and 2 for ERROR. - `spans[].status_message` (string): Error message when the span status is ERROR. The card documents this field as present for status_code equal to 2. - `spans[].attributes` (string): Map of retained GenAI attributes for the span. The attributes are projected to the 18 keys listed by the dataset card. Dotted names below are literal OpenTelemetry attribute keys, not nested JSON objects. - `spans[].attributes.gen_ai` (object) - `spans[].attributes.gen_ai.operation` (object) - `spans[].attributes.gen_ai.operation.name` (string): Operation name; the dataset filter keeps chat spans, so the retained value is chat. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.request` (object) - `spans[].attributes.gen_ai.request.model` (string): Canonical model name requested for the call; routing prefixes are removed. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.request.temperature` (string): Request temperature, stored as a floating-point value. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.request.max_tokens` (string): Requested maximum token count, stored as an integer. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.request.stop_sequences` (string): List of requested stop sequences. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.response` (object) - `spans[].attributes.gen_ai.response.model` (string): Canonical model name reported for the response. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.response.id` (string): Provider response identifier. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.response.finish_reasons` (string): List of response finish reasons. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.usage` (object) - `spans[].attributes.gen_ai.usage.input_tokens` (string): Input-token usage as an integer. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.usage.output_tokens` (string): Output-token usage as an integer. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.input` (object) - `spans[].attributes.gen_ai.input.messages` (array): JSON-encoded OpenTelemetry-shaped input messages; non-empty for retained chat spans. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.output` (object) - `spans[].attributes.gen_ai.output.messages` (array): JSON-encoded OpenTelemetry-shaped output messages. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.output.type` (string): Requested output type when recorded. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.tool` (object) - `spans[].attributes.gen_ai.tool.definitions` (string): JSON-encoded tool definitions available to the model call. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.system_instructions` (string): System instructions associated with the request. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.provider` (object) - `spans[].attributes.gen_ai.provider.name` (string): GenAI provider name. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.gen_ai.conversation` (object) - `spans[].attributes.gen_ai.conversation.id` (string): Conversation identifier. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].attributes.error` (object) - `spans[].attributes.error.type` (string): Error class for failed calls when present. This is one of the 18 documented retained attribute keys. The dataset card lists the value type and notes where the key is JSON text, a list, or set only on failed calls. - `spans[].resource_attributes` (string): JSON-encoded resource-attribute map. These describe telemetry resource context in the span record; the dataset card represents this field as JSON text. - `spans[].events` (array): JSON-encoded span-event list. Events are retained as JSON text. The card notes events can include gen_ai.evaluation.result scoring data. - `coverage limits` (string): The published v2 filter removes user_simulator and judge chat spans, failed retry groups, empty-input chat spans, sessions with no remaining spans, and exgentic.* provenance attributes. These omissions are part of the published data recipe. Do not infer that a missing tool action, evaluator span, or provenance value did not exist in the full source trace. ### Feedback summary # Review summary: Exgentic Agent LLM Traces v2 v2 · card checked 2026-10-12 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Epoch AI benchmark data catalog - Schema id: `epoch-benchmark-data` - Version: `Live catalog ? checked 2026-10-12` - Source: https://epoch.ai/benchmarks/use-this-data - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): Epoch AI publishes benchmark data through downloadable CSV/ZIP files, interactive tables, and a Python client backed by the Airtable API. Scope: The page exposes separate internal-run and external-run tables. CSV is convenient but does not preserve entity relationships; the Epoch AI Python client accesses the Airtable API and preserves them. Epoch AI data is offered under CC BY with attribution, while benchmark questions and answers belong to their creators and externally sourced data retains its original license. - `benchmark identity` (string): Identifies benchmark records and their associated benchmark information in Epoch AI benchmarking data. Read the benchmark entry together with its source metadata and the version or date of the downloaded data; the hub provides a CSV/ZIP snapshot and interactive table rather than one universal benchmark-result schema. - `model identity` (string): Associates reported benchmark information with model records. Use the model record and its source references to disambiguate similarly named model releases; the download snapshot may change as Epoch AI updates its data. - `result` (string): Represents benchmark performance records as published by Epoch AI. Preserve the metric, value, unit, and benchmark context as supplied. The use-data page describes a data hub and does not define a single score scale for all benchmark rows. - `run provenance` (string): Separates Epoch AI internal runs from external benchmark runs. The page provides separate internal-run and external-run tables. Retain this distinction when comparing or attributing a result. - `relationships` (string): Links benchmark, model, run, and result entities in the source data. The Epoch AI Python client preserves relationships through the Airtable API; the CSV download does not preserve those relationships in the same way. - `license and attribution` (string): Records the source and licensing context needed when reusing benchmark data. Epoch AI states that its data may be used, distributed, and reproduced with source and author credit under CC BY. Benchmark questions and answers remain their creators’ property, and external records retain their original licensing. ### Feedback summary # Review summary: Epoch AI benchmark data catalog Live catalog ? checked 2026-10-12 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## TraceLab coding-agent trace dataset - Schema id: `tracelab` - Version: `Dataset release v0.0.1` - Source: https://github.com/uw-syfi/TraceLab/blob/main/README.md - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): TraceLab releases a sanitized normalized JSONL dataset and a DuckDB copy of Claude Code and Codex coding-agent workload traces, with analysis tooling. Scope: Release v0.0.1 documents 357,161 LLM-round rows, 432,510 tool records, and 43 pseudonymous users. Each JSONL row is one LLM invocation. Public sanitization pseudonymizes identifiers, removes local paths and serialized tool inputs, and keeps structural summaries; this is an observational workload dataset, not a task pass/fail benchmark. The public trace dataset is licensed CC BY 4.0; the project code is Apache 2.0. - `task and sample identity` (string): The normalized round records use provider, session_id, and round_id as the merge identity; trace_key is also present in the row structure. These identifiers relate invocation rows to sessions and rounds. The README describes coding-agent workload traces; it does not define a benchmark task/sample ID or a pass/fail outcome for each round. - `trajectory events` (array): timing_events is an ordered list of trace-observed events for a model invocation; documented event names include user_message, tool_result, reasoning, text, tool_call, and Codex usage_report. Preserve event order and timestamps when analyzing timing. In sanitized public traces, event text and sensitive local context may be absent or reduced. - `evaluation outcome` (string): The source describes coding-agent workload traces and does not define success/failure labels or human quality ratings for each invocation. - `primary source` (string): Links the project README, release assets, and dataset paper for the normalized trace format and corpus. Pin the release version when reproducing counts or analyses. The latest-release alias can point to a newer dataset than v0.0.1. - `tool records` (array): tools contains nested records with tool_name, tool_call_id, emitted_at, input_chars, result_chars, tool_wall_latency_ms, tool_internal_latency_ms, is_error, and result_at. The public dataset does not retain complete tool input or output. input_chars and result_chars are length summaries; tool_internal_latency_ms can be null, while tool_wall_latency_ms is based on observed event times. - `tool continuation metadata` (string): Codex exec_command/write_stdin records can include command_status, command_exit_code, and continuation_of_tool_call_id when the link is available. These are conditional Codex-specific details, not fields guaranteed on every provider or tool record. - `token accounting` (number): The row structure includes input/output token counts and a cache-prefix split; the README defines input_tokens_total as prefix_tokens plus newly_append_tokens. The README notes provider-specific accounting: Claude cache-creation input tokens are copied from usage data, while the corresponding Codex value is null. Keep these components distinct when comparing token totals. - `sanitized command metadata` (object): Sanitized command-launch calls can retain executables, executable_parse_status, executable_parse_reason, and command_skeleton. These summaries are extracted before command input is removed. The README says non-reviewed executable names are pseudonymized as custom_N labels in the sanitized trace. ### Feedback summary # Review summary: TraceLab coding-agent trace dataset Dataset release v0.0.1 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## TRAIL annotated trace-debugging dataset - Schema id: `trail` - Version: `TRAIL dataset ? 148 traces` - Source: https://huggingface.co/datasets/PatronusAI/TRAIL - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): TRAIL (Trace Reasoning and Agentic Issue Localization) is a benchmark for identifying and localizing errors in annotated AI-agent execution traces. Scope: The dataset card describes 148 traces: 118 from GAIA and 30 from SWE-Bench, with 1,987 OpenTelemetry spans. It reports 841 annotated errors, while 575 spans exhibit at least one error; these are different counts. Annotations include span IDs, error categories, supporting evidence, descriptions, and Low/Medium/High impact levels. The card says the data is primarily text-only and notes category imbalance. Although the card lists an MIT license, access is gated and its terms prohibit resharing outside a gated or private Hugging Face repository; it also prohibits training systems intended to automate human evaluation. - `task and sample identity` (string): The dataset contains 118 GAIA traces and 30 SWE-Bench traces, totaling 148 annotated agent executions. The source benchmarks and their task domains differ: the card describes GAIA as open-world search and SWE-Bench as software engineering. Keep this composition visible when interpreting aggregate results. - `trajectory events` (array): Traces are represented as OpenTelemetry spans collected using OpenInference; the card reports 1,987 spans across the dataset. The card describes text-only input and output focus. This is trace-debugging data, not a general multimodal agent log format. - `evaluation outcome` (string): The dataset contains span-level error annotations and trace-level ratings for instruction adherence, plan optimality, security, and reliability. The card reports 841 annotated errors and 575 spans with at least one error. It describes annotation fields as span ID, error category, evidence, description, and impact level; do not treat the two counts as interchangeable. - `primary source` (string): Links the Hugging Face dataset card, the official project repository, and the paper. The card lists MIT, but its access gate requires agreement to terms that prohibit resharing outside a gated or private Hugging Face repository and prohibit training automated human evaluators. Review those terms before use. ### Feedback summary # Review summary: TRAIL annotated trace-debugging dataset TRAIL dataset ? 148 traces / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## ClawBench task formats and run artifacts - Schema id: `clawbench` - Version: `ClawBench V1/V2 task and trace formats` - Source: https://github.com/TIGER-AI-Lab/ClawBench - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): ClawBench evaluates browser agents on everyday tasks using task definitions, request interception, run recordings, and an agentic evaluator. Scope: The current repository lists 152 shipping V1 tasks and 129 V2 tasks across live websites; its note explains the paper counts are two higher because two ASPCA tasks were removed after publication. Each task is defined in task.json and uses an eval_schema for request interception. The five synchronized evidence layers are session replay, action screenshots, browser actions, HTTP traffic, and agent messages. Interception is separate from post-session task scoring, and these artifacts are not one generic trajectory schema. - `session replay` (string): The session-replay layer is a local recording.mp4 or a provider-hosted recording URL in run-meta.json, depending on the browser runtime. The repository describes local or Kernel H.264 video and Browserbase Session Inspector URLs. The five layers are synchronized; when interception fires, the run artifacts are frozen at that point. - `screenshots` (string): Throttled timestamped PNG screenshots are captured after browser actions. Interpret screenshots with the corresponding browser actions and timestamps from the same run. - `http and browser actions` (string): The recording pipeline includes HTTP traffic and browser actions as separate evidence layers. The task eval_schema configures interception using a URL pattern and HTTP method, with optional body filters; the repository task guide is the source for supported matching fields. - `agent messages` (array): One of the five run-recording evidence layers is agent messages. This layer preserves agent-side messages associated with a run; it complements browser and network evidence rather than replacing it. - `artifact relationships` (string): A run is recorded across five evidence layers for a task and model execution. Keep task, model, harness, corpus version, and scoring rubric together when comparing runs; the project reports scores by corpus, harness, and rubric. - `primary source` (string): Links the project repository, task-format guide, paper, and data releases. The task-format guide documents task.json fields including instruction, eval_schema, time_limit, and extra_info. eval_schema supports URL/method matching and optional request filters; it is an interception rule, not itself a success score. - `task` (object) - `task.json` (object) - `task.json.instruction` (string): The task prompt sent to the agent. The source task guide defines this as the instruction for a test case. - `task.json.eval_schema` (object): Request-interceptor configuration used to identify a matching outgoing request. The documented required fields are url_pattern and method; body supports exact key-value filters, and the repository also documents supported query-parameter filters. The interceptor rule is distinct from the later evaluator score. - `task.json.eval_schema.url_pattern` (string): Regular expression matched against request URLs using re.search(). This pattern is part of the deterministic request-interception configuration; it is not a URL the agent must necessarily visit. - `task.json.eval_schema.method` (string): HTTP method required for a matching request. The task guide marks this field as required for interception. - `task.json.time_limit` (number): Maximum permitted task runtime, expressed in minutes in the task example and docs. The task guide describes this as the maximum run time; it is part of task configuration, not a model latency score. - `task.json.extra_info` (array) - `task.json.extra_info[]` (object): Optional supplementary files referenced by path and description. The task guide says referenced files are copied into /my-info/ and described in the agent prompt. ### Feedback summary # Review summary: ClawBench task formats and run artifacts ClawBench V1/V2 task and trace formats / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## HAL Reliability - Schema id: `hal-reliability` - Version: `Dashboard methodology ? checked 2026-10-12` - Source: https://hal.cs.princeton.edu/reliability/methodology/ - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): HAL Reliability reports task accuracy alongside consistency, predictability, and robustness, and reports safety separately. The methodology defines the aggregation rules and interpretation of each metric. Scope: HAL is a metric methodology and dashboard, not an evaluation-file or trajectory format. Dashboard scores are normalized to [0, 1] with higher values better; safety is excluded from the overall reliability aggregate. - `accuracy` (string): Measures the fraction of benchmark tasks completed correctly. HAL reports task accuracy beside reliability so success rate remains visible, but accuracy alone does not measure repeatability, confidence quality, perturbation robustness, or safety. Compute correct tasks divided by total tasks. - `reliability` (number): Averages consistency, predictability, and robustness. The methodology normalizes scores to [0, 1] with higher values better. Safety is deliberately excluded because it measures behavioral-constraint violations rather than task-performance reliability. - `consistency_agg` (string): Measures whether outcomes, action behavior, and resource use repeat across independent runs on the same tasks. HAL gives equal weight to outcome, trajectory, and resource categories. The two trajectory submetrics split the trajectory category evenly; the formula is C_out/3 + (C_traj_d + C_traj_s)/6 + C_res/3. - `outc` (string): Measures agreement among repeated binary success/failure outcomes for each task. For task t, p-hat is the empirical success rate over K repeated runs. C_out,t=(2p-hat?1)^2 equals 1 when all runs agree, whether they all succeed or all fail, and 0 when half succeed and half fail; HAL averages this across tasks. - `traj_d` (string): Measures similarity in action-frequency distributions across successful trajectories, ignoring action order. For each pair of successful trajectories, HAL computes Jensen?Shannon divergence between their action distributions and uses its square root as the Jensen?Shannon distance. The score is one minus the mean pairwise distance. Failed trajectories are excluded. - `traj_s` (string): Measures similarity in the order of actions across successful trajectories. HAL computes normalized Levenshtein/edit distance for each pair: one minus edit distance divided by the longer sequence length. It averages over pairs of successful trajectories, so higher values mean closer action ordering. - `res` (string): Measures run-to-run variation in resource consumption. HAL computes a coefficient of variation for each available resource type, including cost, time, API calls, action count, errors, and per-call latency, then applies exp(?mean CV). Unlike other consistency metrics, this is not conditioned on task success. - `conf` (string): Measures stability of self-reported confidence across repeated runs of the same task. HAL uses exp(?CV_conf), with CV equal to standard deviation divided by mean confidence. The methodology computes this diagnostic but excludes it from the consistency aggregate. - `predictability_agg` (string): Measures how well self-reported confidence predicts binary task outcomes. HAL uses the normalized Brier score as the predictability aggregate. Calibration and AUROC are diagnostics and are not averaged into it; the methodology says the Brier score captures both calibration and discrimination. - `cal` (string): Measures whether predicted confidence agrees with observed accuracy. HAL computes expected calibration error using ten equal-width confidence bins, then reports one minus ECE so higher is better. Each bin?s absolute accuracy-confidence gap is weighted by that bin?s share of predictions. - `auroc` (string): Measures whether confidence ranks correct outcomes above incorrect outcomes across thresholds. This is the discrimination diagnostic: it tests ranking quality, not whether a stated confidence value is numerically calibrated. HAL reports it beside calibration; 0.5 is chance-level ranking and 1.0 is perfect separation in the ROC interpretation. - `brier` (string): Measures squared confidence error against binary correctness outcomes. HAL defines the normalized score as one minus the mean squared difference between confidence c_i and binary outcome y_i. A perfect score is 1; this value is also the complete predictability aggregate, while calibration and AUROC remain diagnostic submetrics. - `risk_coverage` (string): Measures whether confidence-based deferral can reduce errors among predictions the agent keeps. HAL sorts tasks by confidence and measures error risk as coverage changes. The score normalizes excess area under the risk-coverage curve against random ordering; it is computed as a diagnostic and excluded from predictability. - `robustness_agg` (string): Averages accuracy retention under fault, structural, and prompt perturbations. Each component compares accuracy under one controlled perturbation with baseline accuracy and is capped at 1. The three components have equal weight; a score of 1 means no measured accuracy degradation under the tested conditions. - `fault` (string): Measures accuracy retention when tool or API failures are injected. HAL divides fault-condition accuracy by baseline accuracy and caps the ratio at 1. This reports degradation under the tested fault setup; it does not claim robustness to every possible failure. - `struct` (string): Measures accuracy retention when task-input format or structure changes. HAL divides structurally perturbed accuracy by baseline accuracy and caps the ratio at 1. The score describes only the structural changes used in the evaluation. - `prompt` (string): Measures accuracy retention when task instructions are rephrased or varied. HAL divides prompt-variation accuracy by baseline accuracy and caps the ratio at 1. The score describes the tested prompt variants, not arbitrary changes in wording. - `safety_agg` (number): Measures constraint compliance together with the severity of detected violations. An LLM judge reviews trajectories for behavioral-constraint violations. HAL defines low, medium, and high severity weights as 0.25, 0.50, and 1.00, then computes one minus violation probability times mean severity conditional on violation. A score of 1 means no violations were detected. Safety is reported separately from reliability. - `harm` (string): Measures violation severity conditional on tasks where a violation occurred. For each task with violations, HAL takes the maximum severity weight and averages those maxima only over tasks with a violation; the score is one minus that conditional mean. If no tasks have violations, the score is 1. This represents severity separately from violation frequency. - `comp` (string): Measures the fraction of tasks with no detected constraint violation. HAL computes one minus the number of tasks with violations divided by total tasks. A task counts as violated if the judge identifies a constraint breach; conditional severity is represented separately by S_harm. ### Feedback summary # Review summary: HAL Reliability Dashboard methodology ? checked 2026-10-12 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## AssistantBench - Schema id: `hal-assistantbench` - Version: `HAL leaderboard · 2026-10-13` - Source: https://hal.cs.princeton.edu/assistantbench - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL AssistantBench leaderboard. The Results view contains the full row listing and links to run trace samples. - `rank` (integer): Position shown for this result in the HAL leaderboard snapshot. - `scaffold` (string): The agent, framework, or scaffold label HAL associates with this result. - `model` (string): The model label HAL reports for this result, including any date or provider qualifier. - `verified` (boolean): HAL's verification flag for this leaderboard row; this is a source status, not a score. - `accuracy` (string): Accuracy exactly as displayed by HAL. Annotations and uncertainty text are kept as part of the value. - `cost_usd` (string): Total cost in US dollars as displayed by HAL; any uncertainty or range text is preserved. - `runs` (string): Number of runs HAL reports for this result, preserved in its displayed form. - `trace_url` (string): External link to the trace archive associated with this row; the linked archive may be encrypted. ### Feedback summary # Review summary: AssistantBench HAL leaderboard · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## CORE-Bench Hard - Schema id: `hal-corebench-hard` - Version: `HAL leaderboard · 2026-10-13` - Source: https://hal.cs.princeton.edu/corebench_hard - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL CORE-Bench Hard leaderboard. The Results view contains the full row listing and links to run trace samples. - `rank` (integer): Position shown for this result in the HAL leaderboard snapshot. - `scaffold` (string): The agent, framework, or scaffold label HAL associates with this result. - `model` (string): The model label HAL reports for this result, including any date or provider qualifier. - `verified` (boolean): HAL's verification flag for this leaderboard row; this is a source status, not a score. - `accuracy` (string): Accuracy exactly as displayed by HAL. Annotations and uncertainty text are kept as part of the value. - `cost_usd` (string): Total cost in US dollars as displayed by HAL; any uncertainty or range text is preserved. - `runs` (string): Number of runs HAL reports for this result, preserved in its displayed form. - `trace_url` (string): External link to the trace archive associated with this row; the linked archive may be encrypted. ### Feedback summary # Review summary: CORE-Bench Hard HAL leaderboard · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## GAIA - Schema id: `hal-gaia` - Version: `HAL leaderboard · 2026-10-13` - Source: https://hal.cs.princeton.edu/gaia - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL GAIA leaderboard. The Results view contains the full row listing and links to run trace samples. - `rank` (integer): Position shown for this result in the HAL leaderboard snapshot. - `scaffold` (string): The agent, framework, or scaffold label HAL associates with this result. - `model` (string): The model label HAL reports for this result, including any date or provider qualifier. - `verified` (boolean): HAL's verification flag for this leaderboard row; this is a source status, not a score. - `accuracy` (string): Accuracy exactly as displayed by HAL. Annotations and uncertainty text are kept as part of the value. - `level_1_accuracy` (string): Accuracy HAL reports for GAIA Level 1 tasks. - `level_2_accuracy` (string): Accuracy HAL reports for GAIA Level 2 tasks. - `level_3_accuracy` (string): Accuracy HAL reports for GAIA Level 3 tasks. - `cost_usd` (string): Total cost in US dollars as displayed by HAL; any uncertainty or range text is preserved. - `runs` (string): Number of runs HAL reports for this result, preserved in its displayed form. - `trace_url` (string): External link to the trace archive associated with this row; the linked archive may be encrypted. ### Feedback summary # Review summary: GAIA HAL leaderboard · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Reliability · GAIA - Schema id: `hal-reliability-gaia` - Version: `HAL Reliability · 2026-10-13` - Source: https://hal.cs.princeton.edu/reliability/benchmark/gaia/ - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL Reliability · GAIA dashboard. Values preserve the metric strings reported by HAL. - `rank` (integer) - `agent` (string) - `accuracy` (string) - `reliability_overall` (string) - `consistency_agg` (string) - `consistency_outcome` (string) - `consistency_trajectory_distribution` (string) - `consistency_trajectory_sequence` (string) - `consistency_resource` (string) - `predictability_agg` (string) - `predictability_calibration` (string) - `predictability_auroc` (string) - `predictability_brier` (string) - `robustness_agg` (string) - `robustness_fault` (string) - `robustness_structural` (string) - `robustness_prompt` (string) - `safety_agg` (string) - `safety_harm` (string) - `safety_compliance` (string) ### Feedback summary # Review summary: Reliability · GAIA HAL Reliability · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Reliability · τ-bench airline - Schema id: `hal-reliability-taubench-airline` - Version: `HAL Reliability · 2026-10-13` - Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline/ - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL Reliability · τ-bench airline dashboard. Values preserve the metric strings reported by HAL. - `rank` (integer) - `agent` (string) - `accuracy` (string) - `reliability_overall` (string) - `consistency_agg` (string) - `consistency_outcome` (string) - `consistency_trajectory_distribution` (string) - `consistency_trajectory_sequence` (string) - `consistency_resource` (string) - `predictability_agg` (string) - `predictability_calibration` (string) - `predictability_auroc` (string) - `predictability_brier` (string) - `robustness_agg` (string) - `robustness_fault` (string) - `robustness_structural` (string) - `robustness_prompt` (string) - `safety_agg` (string) - `safety_harm` (string) - `safety_compliance` (string) ### Feedback summary # Review summary: Reliability · τ-bench airline HAL Reliability · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Reliability · τ-bench airline (original) - Schema id: `hal-reliability-taubench-airline-original` - Version: `HAL Reliability · 2026-10-13` - Source: https://hal.cs.princeton.edu/reliability/benchmark/taubench_airline_original/ - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A row from the HAL Reliability · τ-bench airline (original) dashboard. Values preserve the metric strings reported by HAL. - `rank` (integer) - `agent` (string) - `accuracy` (string) - `reliability_overall` (string) - `consistency_agg` (string) - `consistency_outcome` (string) - `consistency_trajectory_distribution` (string) - `consistency_trajectory_sequence` (string) - `consistency_resource` (string) - `predictability_agg` (string) - `predictability_calibration` (string) - `predictability_auroc` (string) - `predictability_brier` (string) - `robustness_agg` (string) - `robustness_fault` (string) - `robustness_structural` (string) - `robustness_prompt` (string) - `safety_agg` (string) - `safety_harm` (string) - `safety_compliance` (string) ### Feedback summary # Review summary: Reliability · τ-bench airline (original) HAL Reliability · 2026-10-13 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## Online Mind2Web - Schema id: `hal-online-mind2web` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/online_mind2web - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: Online Mind2Web HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## SciCode - Schema id: `hal-scicode` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/scicode - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: SciCode HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## ScienceAgentBench - Schema id: `hal-scienceagentbench` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/scienceagentbench - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: ScienceAgentBench HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## SWE-bench Verified Mini - Schema id: `hal-swebench-verified-mini` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/swebench_verified_mini - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: SWE-bench Verified Mini HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## TAU-bench Airline - Schema id: `hal-taubench-airline` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/taubench_airline - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: TAU-bench Airline HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## USACO - Schema id: `hal-usaco` - Version: `HAL trace sample · 2026-10-19` - Source: https://hal.cs.princeton.edu/usaco - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: USACO HAL trace sample · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section. ## All 9 trace samples - Schema id: `hal-trace-inventory` - Version: `HAL trace variables · 2026-10-19` - Source: https://huggingface.co/datasets/agent-evals/hal_traces - Current-version comments: 0 - Prior-version comments retained: 0 ### Schema outline - `$` (object): A normalized field-path inventory derived from one selected HAL trace sample per benchmark. It reports observed structural metadata only; raw trace contents and literal example values are omitted. - `path` (string) - `occurrences` (integer) - `type_counts` (object) - `per_sample` (object) - `string_length` (object) - `number_range` (object) - `object_key_count` (object) - `array_item_count` (object) - `null_count` (integer) - `dynamic_keys` (object) ### Feedback summary # Review summary: All 9 trace samples HAL trace variables · 2026-10-19 / Entire schema 0 open comments and 0 resolved comments. ## Open issues No open comments in this section.