How to build a LLM as a judge evaluation workflow.

⚖️ LLM as a Judge Evaluation

Oxen.ai

Oxen.ai is a developer platform for fine-tuning and evaluating models on your own data.

🐂 What is Oxen?

Oxen.ai exposes endpoints and a chat interface to test out a variety of models with a unified interface.

⚡️ Model Inference

Oxen.ai lets you go from datasets to custom models with a few clicks.

🦾 Fine-Tuning

📊 Datasets

Oxen.ai allows you to find the best model and prompt for your use case.

🔬 Evaluations

Oxen.ai is built on top of a blazing fast data version control system that allows you to version, branch, and share datasets, model weights, and experiments with your team.

💾 Version Control

How to install the Oxen client, server, or python package.

⚒️ Installation

If you are familiar with git, oxen should be an easy learning curve.

💻 Command Line Interface

Learn how to get started with the oxenai python package.

🐍 Python

`oxen-server` is the storage backend for Oxen. It is where the merkle tree, commit history, and other metadata is stored.

📡 Oxen Server

🔥 CLI Performance

Quickly find changes in your datasets with Oxen.ai

Dataset Diffs

Oxen.ai gives you the flexibility to attach metadata to files to make them more discoverable and useful.

File Metadata

Workspaces allows you to interact with your data without having to download the entire dataset locally.

Workspaces

Welcome to [Oxen.ai](http://Oxen.ai) Feature Updates! We send out an email blast with all new/updated features, bug fixes, and more relating to your Oxen experience. If you want to opt in (and get cool invites to our [weekly arXiv Dives](https://www.oxen.ai/blog?tag=arxiv-dives)), [click here](https://www.oxen.ai/community)! This page will have a complete update with demos, photos, and top feature updates/bug squashes.

🐂 Feature Updates

🐍 Here lies detailed Python module documentation

All Modules

Repositories

Endpoints

List Namespaces

Get Namespace

Create a new repository

Get repository

Update repository

Delete repository

Transfer repo namespace

Resolve repository API URL

List Repositories

Get commit

List parent commits

List commit history

List branches

Get a branch if it exists, create a new one if not

Get branch

Update branch

Delete a branch

get branch and resource path

List directory for resource

Get a file for resource

Get meta data for a file resource

Get Data Frame

Index Dataframe for Querying

Diff entries

Diff file

Diff commits

Diff entries in dir

Merge commits or branches

Query Workspace Data Frame

Index Data Frame for Querying

Enable Nearest Neighbor Search

Add File to Workspace

List Workspaces

Create Workspace

Fork repo

Version

Health

Documentation

Blog

GitHub

Python API

HTTP API

Sign Up

Support

Oxen.ai lets you spin up a Python Notebook on a GPU in seconds.

📓 Notebooks

Explore, process, and version data with your favorite tools

🗺️ Explore, Process, and Version Data

Rate examples from your dataset and write them back to a data frame before committing.

🏷️ Build a Custom Labeling Tool

How to compute vector embeddings for a text dataset on a GPU.

🔎 Compute Text Embeddings

Build and version a synthetic dataset to train a model on

🧪 Generate Synthetic Datasets

How to train an LLM on your own data in a Marimo Notebook.

🏋️‍♀️ Fine-Tune an LLM

How to build a human in the loop evaluation workflow.

🕵️‍♂️ Evaluation w/ Human in the Loop

How to automate workflows by running a notebook from the command line.

💻 Running Notebooks as Scripts

Get Repository

Create Repository

Update Repository

Delete Repository

Get Branches

Create Branch

List Commits

List Files and Directories

Get File Metadata

Download File

Get a data frame from a repository. See [Workspaces](/http-api/workspaces/index_dataframe) for more information on how to edit and query.

Get Data Frame (Read Only)

Workspaces enable the ability to work on the remote repository as if it were local.

Index a data frame into a workspace so that it can be queried with SQL and nearest neighbor search.

Index Data Frame

Query a data frame in a workspace with SQL.

Query Data Frame

Create an embeddings index in a workspace so that it can be used for nearest neighbor search.

Get Started

Developer Tools

Other Concepts

Release Notes

⚖️ LLM as a Judge Evaluation