How to build a LLM as a judge evaluation workflow.

⚖️ LLM as a Judge Evaluation

Oxen.ai

Oxen is a lightning fast data version control system for structured and unstructured machine learning datasets.

🐂 What is Oxen?

How to install the Oxen client, server, or python package.

⚒️ Installation

🐮 Learn The Basics

If you are familiar with git, oxen should be an easy learning curve.

💻 Command Line Interface

Learn how to get started with the oxenai python package.

🐍 Python

`oxen-server` is the storage backend for Oxen. It is where the merkle tree, commit history, and other metadata is stored.

📡 Oxen Server

🔥 Performance

Oxen.ai allows you to view, edit, and query your data in a beautiful interface.

🔎 Explore Datasets

Edit data on Oxen.ai or build your own workflows with the Python Library.

🏷️ Labeling Workflows

OxenHub enables your team to version, branch, and share datasets with your team.

🤝 Collaboration

Quickly find changes in your datasets with Oxen.ai

Dataset Diffs

Oxen.ai helps you compare results from your machine learning models.

Comparing Models

Managing and validating data schemas is an important part of making robust data pipelines.

Schemas

After you push your data to an [oxen-server](/getting-started/oxen-server), you can use remote repositories to interact with your data.

Remote Repositories

Workspaces allows you to interact with your data without having to download the entire dataset locally.

Workspaces

Welcome to [Oxen.ai](http://Oxen.ai) Feature Updates! We send out an email blast with all new/updated features, bug fixes, and more relating to your Oxen experience. If you want to opt in (and get cool invites to our [weekly arXiv Dives](https://www.oxen.ai/blog?tag=arxiv-dives)), [click here](https://www.oxen.ai/community)! This page will have a complete update with demos, photos, and top feature updates/bug squashes.

🐂 Feature Updates

🐍 Here lies detailed Python module documentation

All Modules

Repositories

Endpoints

List Namespaces

Get Namespace

Create a new repository

Get repository

Update repository

Delete repository

Transfer repo namespace

Resolve repository API URL

List Repositories

Get commit

List parent commits

List commit history

List branches

Get a branch if it exists, create a new one if not

Get branch

Update branch

Delete a branch

get branch and resource path

List directory for resource

Get a file for resource

Get meta data for a file resource

Get Data Frame

Index Dataframe for Querying

Diff entries

Diff file

Diff commits

Diff entries in dir

Merge commits or branches

Query Workspace Data Frame

Index Data Frame for Querying

Enable Nearest Neighbor Search

Add File to Workspace

List Workspaces

Create Workspace

Fork repo

Version

Health

Documentation

Blog

GitHub

Python API

HTTP API

Sign Up

Support

Oxen.ai lets you spin up a Python Notebook on a GPU in seconds.

🍃 Marimo Notebooks

Explore, process, and version data with your favorite tools

🗺️ Explore, Process, and Version Data

Rate examples from your dataset and write them back to a data frame before committing.

🏷️ Build a Custom Labeling Tool

How to compute vector embeddings for a text dataset on a GPU.

🔎 Compute Text Embeddings

Build and version a synthetic dataset to train a model on

🧪 Generate Synthetic Datasets

🏋️‍♀️ Fine-Tune an LLM

How to build a human in the loop evaluation workflow.

🕵️‍♂️ Evaluation w/ Human in the Loop

🏃‍♂️ Run Models On Your Data

Sentiment Analysis is the process of categorizing the overall feeling and opinion of a piece of text. Usually it is broken down into multiple categories such as positive, negative, and neutral.

🧐 Sentiment Analysis

Text translation is the process of translating text into another language. This process has never been easier with LLMs commonly supporting multiple languages such as Spanish, French, and Italian.

📝 Text Translation

**Retrieval-Augmented Generation (RAG)** is the process of retrieving information that the AI does not have in its original training data to answer a question. To set up RAG, the first step is to embed the data you would like to retrieve. Basically, convert the text, image, or other data to a number the AI knows how to find and retrieve when asked a question which relates to the text, image, etc.

🔢 RAG: Generating Embeddings

**Retrieval-Augmented Generation (RAG)** is the process of retrieving information that the AI does not have in its training data to answer a question. RAG is usually used to answer customer service questions, analyze legal documents, answer internal business questions, and much more.

🧠 RAG: Extracting Answers

It can be expensive and time-consuming to collect or label data. Synthetic Data Generation is the process of using a strong model to generate data for you, skipping a lot of manual labor.

🤖 Synthetic Data Generation

Get Repository

Create Repository

Update Repository

Delete Repository

Get Branches

Create Branch

List Commits

List Files and Directories

Get File Metadata

Download File

Get a data frame from a repository. See [Workspaces](/http-api/workspaces/index_dataframe) for more information on how to edit and query.

Get Data Frame (Read Only)

Workspaces enable the ability to work on the remote repository as if it were local.

Index a data frame into a workspace so that it can be queried with SQL and nearest neighbor search.

Index Data Frame

Query a data frame in a workspace with SQL.

Query Data Frame

Create an embeddings index in a workspace so that it can be used for nearest neighbor search.

Get Started

Key Features

Other Concepts

⚖️ LLM as a Judge Evaluation