POST
/
api
/
repos
/
:namespace
/
:repo_name
/
evaluations
/
:resource
curl --request POST \
  --url https://hub.oxen.ai/api/repos/:namespace/:repo_name/evaluations/:resource

Parameters:

  • name: evaluation name
  • prompt: the prompt you want to use for the evaluation
  • type: evaluation type, currently only (“text” and “embeddings”) are supported
  • model: model to use for evaluation (supported models: “gpt-4o”, “gpt-4o-mini”, “text-embedding-3-small”, “text-embedding-3-large”)
  • is_sample: run a sample instead of a whole dataset
  • target_column: column to write the results to
  • target_branch: branch to write the results to
  • auto_commit: automatically commit the results to the target branch
  • commit_message: the commit message to use when auto_committing

To create a repo, a valid token in the header is needed.

​
Example Request

Replace :namespace, :repo_name, and :resource with the appropriate values.

For example if the file you want to process is at:

https://oxen.ai/ox/customer-intents/main/data.parquet

The parameters should be:

  • :namespace -> ox
  • :repo_name -> customer-intents
  • :resource -> main/data.parquet (combination of branch name and file_name)

and the full URL for the POST request should be:

https://hub.oxen.ai/api/repos/ox/customer-intents/evaluations/main/data.parquet

curl -X POST -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
    https://hub.oxen.ai/api/repos/:namespace/:repo_name/evaluations/:resource \
    --data '{
    "name": "My Awesome Evaluation",
    "prompt": "Classify the following text into one of the following categories: [intent_1, intent_2, intent_3]\n{my_text_column_name}",
    "type": "text",
    "model": "gpt-4o-mini",
    "is_sample": false,
    "target_column": "prediction",
    "target_branch": "api-results-branch",
    "auto_commit": true,
    "commit_message": "test commit message"
}
'

​
Example Response

{
    "evaluation": {
        "closing_commit": null,
        "commit_id": "e6c2c7192f620617",
        "commit_started_at": null,
        "completed_at": null,
        "conflict_target_branch": null,
        "created_by": {
            "id": "4fbfe7e0-6d65-4ccb-86f5-ec8093c7ddac",
            "image": null,
            "name": "Ox",
            "username": "ox"
        },
        "credits_per_token": 6e-7,
        "error_message": null,
        "id": "cadbecba-25fb-4e5d-8637-4d85af81d186",
        "inserted_at": "2024-10-12T21:38:44Z",
        "is_sample": false,
        "master_id": "cadbecba-25fb-4e5d-8637-4d85af81d186",
        "model": "gpt-4o-mini",
        "name": "My Awesome Evaluation",
        "parent_id": null,
        "progress": {
            "processed": 0,
            "total": 0
        },
        "prompt": "Classify the following text into one of the following categories: [intent_1, intent_2, intent_3]\n{my_text_column_name}",
        "queued_at": "2024-10-12T21:38:44Z",
        "repository_id": "c769cfa0-043c-4ee4-8919-052bc7808d60",
        "resource": {
            "path": "data.parquet",
            "version": "main"
        },
        "sample_count": 10,
        "started_at": null,
        "status": "queued",
        "target_branch": "api-results-branch",
        "target_column": "prediction",
        "target_path": null,
        "tokens_used": 0,
        "type": "text",
        "workspace_id": "89179fc2-d058-45e9-81d3-b9bc5ca8b00a"
    },
    "status": "success",
    "status_message": "resource_found"
}