Data Frames
Oxen provides a powerful data frame library under the hood that allows you to interact with tabular data.
Whether it is csv, parquet, or line delimited json, it is useful to store your training data in data frames that we can filter, aggregate, slice and dice.
To follow along with the examples below feel free to grab the example data from our public SpamOrHam repository.
mkdir spam-or-ham
cd spam-or-ham
oxen download datasets/SpamOrHam data.tsv
Look At Your Data
Oxen comes with a convenient df subcommand to view your data frames on disk. This is useful for quickly inspecting your data before you start modifying it.
$ oxen df data.tsv
shape: (4_774, 2)
+-----------+---------------------------------+
| dcategory | text |
| --- | --- |
| str | str |
+-----------+---------------------------------+
| ham | Go until jurong point, crazy..… |
| ham | Ok lar... Joking wif u oni... |
| spam | Free entry in 2 a wkly comp to… |
| ham | U dun say so early hor... U c … |
| ham | Nah I dont think he goes to u… |
| … | … |
| ham | Well, im glad you didnt find … |
| ham | Guy, no flash me now. If you g… |
| spam | Do you want a New Nokia 3510i … |
| ham | Mark works tomorrow. He gets o… |
| ham | Keep ur problems in ur heart, … |
+-----------+---------------------------------+
Upload Your Data
Then add the data to a repository of your own so that you can modify it. You can do this in the UI, Python, or CLI.
If you have pushed to the Oxen Hub, you can view, edit, and query your data directly using the UI.
Editing Data Frames
Oxen allows you to interact with data frames that are not downloaded to your local machine. This can be useful for data collection, labeling workflows, or quickly inspecting data without having to download it.
Once you have pushed your data to an Oxen repository, Oxen exposes a CRUD interface to interact with the rows.
from oxen import DataFrame
# Connect to the data frame
df = DataFrame("my-username/spam-or-ham", "data.tsv")
# Add a row
row_id = df.insert_row({"category": "spam", "message": "CLICK HERE TO WIN INSTANTLY."})
# Get a row by id
row = df.get_row_by_id(row_id)
print(row)
# Update a row
row = df.update_row(row_id, {"category": "new_category"})
print(row)
# Delete a row
df.delete_row(row_id)
# Commit the changes
df.commit("Update label")
All of these operations are exposed over HTTP, so you are not limited to using the Python library. Check out all our HTTP reference docs to see how to interact with your data programatically.
Local Data Frames
oxen df
Oxen has a convenient df (short for “Data Frame”) command to deal with tabular data. This example data has 10,000 rows and 6 columns of bounding boxes around cats or dogs. The shape hint at the top of the output can be useful for making sure you are transforming the data correctly.
oxen df train.csv
shape: (9_000, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
|-------------------------+-------+--------+--------+--------+--------|
| images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 |
| images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 |
| images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000431980.jpg ┆ dog ┆ 98.3 ┆ 110.46 ┆ 42.69 ┆ 26.64 |
| images/000000071025.jpg ┆ cat ┆ 55.33 ┆ 105.45 ┆ 160.15 ┆ 73.57 |
| images/000000518015.jpg ┆ cat ┆ 43.72 ┆ 4.34 ┆ 72.98 ┆ 129.1 |
| images/000000171435.jpg ┆ dog ┆ 22.86 ┆ 100.03 ┆ 125.55 ┆ 41.61 |
+-------------------------+-------+--------+--------+--------+--------+
Oxen uses a combination of polars and duckdb under the hood, and uses the Apache Arrow data format to provide powerful cross application functionality.
Useful Commands
There are many ways you might want to view, transform, and filter your data on the command line before committing to the version of the dataset.
To quickly see all the options on the df command you can run oxen df --help.
Convert Dataset Format
The --output option is handy for quickly transforming data files between data formats on disk. Some formats like parquet and arrow are more efficient for data different tasks, but are not human readable like tsv or csv. Data format is always a trade off you’ll have to decide on for your application.
Oxen currently supports these file extensions: csv, tsv, parquet, arrow, json, jsonl.
oxen df train.csv -o train.parquet
shape: (9_000, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
|-------------------------+-------+--------+--------+--------+--------|
| images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 |
| images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 |
| images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000431980.jpg ┆ dog ┆ 98.3 ┆ 110.46 ┆ 42.69 ┆ 26.64 |
| images/000000071025.jpg ┆ cat ┆ 55.33 ┆ 105.45 ┆ 160.15 ┆ 73.57 |
| images/000000518015.jpg ┆ cat ┆ 43.72 ┆ 4.34 ┆ 72.98 ┆ 129.1 |
| images/000000171435.jpg ┆ dog ┆ 22.86 ┆ 100.03 ┆ 125.55 ┆ 41.61 |
+-------------------------+-------+--------+--------+--------+--------+
Writing "train.parquet"
SQL Query
Oxen has a powerful SQL query engine built in to the CLI. You can run SQL queries on your data frames with the --sql flag.
oxen df train.csv --sql 'SELECT * FROM df WHERE label = "dog"'
shape: (4_860, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+--------+--------+--------+--------+
| images/000000128154.jpg ┆ dog ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 |
| images/000000544590.jpg ┆ dog ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 |
| images/000000236841.jpg ┆ dog ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000055645.jpg ┆ dog ┆ 8.67 ┆ 122.36 ┆ 60.22 ┆ 99.24 |
| images/000000094271.jpg ┆ dog ┆ 47.6 ┆ 115.26 ┆ 111.57 ┆ 102.27 |
| images/000000041257.jpg ┆ dog ┆ 6.81 ┆ 117.29 ┆ 207.06 ┆ 86.08 |
| images/000000321014.jpg ┆ dog ┆ 51.86 ┆ 61.18 ┆ 166.26 ┆ 63.11 |
+-------------------------+-------+--------+--------+--------+--------+
Text2SQL
If you are too lazy to write SQL queries, Oxen also has a powerful text2sql engine built in to the CLI. You can run text2sql queries on your data frames with the --text2sql flag. This uses an LLM to convert natural language queries to SQL queries. This can be useful for quickly querying data frames without having to remember SQL syntax.
oxen df train.csv --text2sql 'show me all the rows where the label is dog'
shape: (4_860, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+--------+--------+--------+--------+
| images/000000128154.jpg ┆ dog ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 |
| images/000000544590.jpg ┆ dog ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 |
| images/000000236841.jpg ┆ dog ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000055645.jpg ┆ dog ┆ 8.67 ┆ 122.36 ┆ 60.22 ┆ 99.24 |
| images/000000094271.jpg ┆ dog ┆ 47.6 ┆ 115.26 ┆ 111.57 ┆ 102.27 |
| images/000000041257.jpg ┆ dog ┆ 6.81 ┆ 117.29 ┆ 207.06 ┆ 86.08 |
| images/000000321014.jpg ┆ dog ┆ 51.86 ┆ 61.18 ┆ 166.26 ┆ 63.11 |
+-------------------------+-------+--------+--------+--------+--------+
NOTE: The text2sql engine is still in development and may not work for all queries. It also requires you to have an Oxen.ai API key setup.
Randomize
Often you will want to randomize data before splitting into train and test sets, or even just to peek at different data values.
oxen df train.csv --randomize
shape: (9_000, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+--------+--------+--------+--------+
| images/000000124002.jpg ┆ cat ┆ 82.92 ┆ 8.31 ┆ 108.31 ┆ 158.48 |
| images/000000207597.jpg ┆ dog ┆ 75.64 ┆ 3.65 ┆ 125.47 ┆ 218.19 |
| images/000000113810.jpg ┆ cat ┆ 104.34 ┆ 44.65 ┆ 119.66 ┆ 159.42 |
| images/000000340160.jpg ┆ dog ┆ 79.78 ┆ 89.31 ┆ 127.1 ┆ 103.66 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000310573.jpg ┆ dog ┆ 102.55 ┆ 91.48 ┆ 42.24 ┆ 52.18 |
| images/000000162801.jpg ┆ cat ┆ 112.96 ┆ 75.05 ┆ 57.38 ┆ 98.19 |
| images/000000544117.jpg ┆ dog ┆ 108.16 ┆ 124.28 ┆ 11.08 ┆ 64.58 |
| images/000000283210.jpg ┆ dog ┆ 49.37 ┆ 40.01 ┆ 174.43 ┆ 182.0 |
+-------------------------+-------+--------+--------+--------+--------+
View Schema
Oxen automatically detects and versions the schema of your data frame. See the schema docs for more information on the power of Oxen schemas.
To view a data frame’s schema in full, you can use the --schema flag to display the full schema of this data frame.
oxen df train.csv --schema
+--------+-------+
| column | dtype |
+----------------+
| file | str |
|--------+-------|
| label | str |
|--------+-------|
| min_x | f64 |
|--------+-------|
| min_y | f64 |
|--------+-------|
| width | f64 |
|--------+-------|
| height | f64 |
+--------+-------+
View Specific Columns
Maybe you have many columns, and only need to work with a few. You can specify column names in a comma separated list with --columns.
oxen df train.csv --columns 'file,label'
shape: (9_000, 2)
+-------------------------+-------+
| file ┆ label |
| --- ┆ --- |
| str ┆ str |
+-------------------------+-------+
| images/000000128154.jpg ┆ cat |
| images/000000544590.jpg ┆ cat |
| images/000000000581.jpg ┆ dog |
| images/000000236841.jpg ┆ cat |
| … ┆ … |
| images/000000431980.jpg ┆ dog |
| images/000000071025.jpg ┆ cat |
| images/000000518015.jpg ┆ cat |
| images/000000171435.jpg ┆ dog |
+-------------------------+-------+
Concatenate (vstack)
Maybe you have filtered down data, and want to stack the data back into a single frame. The --vstack option takes a variable length list of files you would like to concatenate.
oxen df train.csv --filter 'label-dog' -o /tmp/dogs.parquet
oxen df train.csv --filter 'label-cat' -o /tmp/cats.parquet
oxen df /tmp/cats.parquet --vstack /tmp/dogs.parquet -o annotations/data.parquet
Take Indices
Sometimes you have a specific row or set of rows of data you would like to look at. This is where the --take option comes in handy.
oxen df train.csv --take '1,13,42'
shape: (3, 6)
+-------------------------+-------+-------+-------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+-------+-------+--------+--------+
| images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000279829.jpg ┆ cat ┆ 30.01 ┆ 13.58 ┆ 82.51 ┆ 176.39 |
| images/000000209289.jpg ┆ dog ┆ 72.75 ┆ 42.06 ┆ 111.52 ┆ 153.09 |
+-------------------------+-------+-------+-------+--------+--------+
Add Column
Your data might not match the schema of a data frame you want to combine with, in this case you may need to add a column to match the schema. You can do this and project default values with --add-col 'col:val:dtype'
oxen df train.csv --add-col 'is_cute:unknown:str'
shape: (9_000, 7)
+-------------------------+-------+--------+--------+--------+--------+---------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height ┆ is_cute |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ str |
+-------------------------+-------+--------+--------+--------+--------+---------+
| images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 ┆ unknown |
| images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 ┆ unknown |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 ┆ unknown |
| images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 ┆ unknown |
| … ┆ … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000431980.jpg ┆ dog ┆ 98.3 ┆ 110.46 ┆ 42.69 ┆ 26.64 ┆ unknown |
| images/000000071025.jpg ┆ cat ┆ 55.33 ┆ 105.45 ┆ 160.15 ┆ 73.57 ┆ unknown |
| images/000000518015.jpg ┆ cat ┆ 43.72 ┆ 4.34 ┆ 72.98 ┆ 129.1 ┆ unknown |
| images/000000171435.jpg ┆ dog ┆ 22.86 ┆ 100.03 ┆ 125.55 ┆ 41.61 ┆ unknown |
+-------------------------+-------+--------+--------+--------+--------+---------+
Add Row
Sometimes it can be a pain to append data to a data file without writing code to do so. The --add-row option makes it as easy as a comma separated list and automatically parses the data to the correct dtypes.
oxen df train.csv --add-row 'images/my_cat.jpg,cat,0,0,0,0'
shape: (9_001, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+--------+--------+--------+--------+
| images/000000128154.jpg ┆ cat ┆ 0.0 ┆ 19.27 ┆ 130.79 ┆ 129.58 |
| images/000000544590.jpg ┆ cat ┆ 9.75 ┆ 13.49 ┆ 214.25 ┆ 188.35 |
| images/000000000581.jpg ┆ dog ┆ 49.37 ┆ 67.79 ┆ 74.29 ┆ 116.08 |
| images/000000236841.jpg ┆ cat ┆ 115.21 ┆ 96.65 ┆ 93.87 ┆ 42.29 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000071025.jpg ┆ cat ┆ 55.33 ┆ 105.45 ┆ 160.15 ┆ 73.57 |
| images/000000518015.jpg ┆ cat ┆ 43.72 ┆ 4.34 ┆ 72.98 ┆ 129.1 |
| images/000000171435.jpg ┆ dog ┆ 22.86 ┆ 100.03 ┆ 125.55 ┆ 41.61 |
| images/my_cat.jpg ┆ cat ┆ 0.0 ┆ 0.0 ┆ 0.0 ┆ 0.0 |
+-------------------------+-------+--------+--------+--------+--------+
Unique
Oxen can efficiently compute all the unique values given a column name, or comma separated list of column names.
oxen df train.csv --unique "file"
oxen df train.csv -u "file,label"
Sort
Sorting can be achieved with the sort flag. For example you may want to find the largest bounding boxes by sorting on the height column.
oxen df train.csv --sort "height"
shape: (9_000, 6)
+-------------------------+-------+--------+--------+--------+--------+
| file ┆ label ┆ min_x ┆ min_y ┆ width ┆ height |
| --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- |
| str ┆ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 |
+-------------------------+-------+--------+--------+--------+--------+
| images/000000580919.jpg ┆ dog ┆ 61.28 ┆ 88.31 ┆ 2.71 ┆ 1.83 |
| images/000000577310.jpg ┆ dog ┆ 132.25 ┆ 193.86 ┆ 3.28 ┆ 1.95 |
| images/000000393384.jpg ┆ dog ┆ 138.85 ┆ 89.89 ┆ 1.25 ┆ 2.11 |
| images/000000477398.jpg ┆ dog ┆ 185.11 ┆ 195.93 ┆ 2.51 ┆ 2.6 |
| … ┆ … ┆ … ┆ … ┆ … ┆ … |
| images/000000069205.jpg ┆ dog ┆ 0.0 ┆ 0.0 ┆ 224.0 ┆ 224.0 |
| images/000000554737.jpg ┆ cat ┆ 0.0 ┆ 0.0 ┆ 224.0 ┆ 224.0 |
| images/000000213819.jpg ┆ cat ┆ 8.32 ┆ 0.0 ┆ 207.77 ┆ 224.0 |
| images/000000397212.jpg ┆ cat ┆ 0.36 ┆ 0.0 ┆ 115.5 ┆ 224.0 |
+-------------------------+-------+--------+--------+--------+--------+
Reverse
You can also reverse the order of a data table. By default --sort sorts in ascending order, but can be reversed with the --reverse flag.
oxen df train.csv --reverse
shape: (7_128, 2)
+-------------------------+----------------+
| file ┆ count('label') |
| --- ┆ --- |
| str ┆ u32 |
+-------------------------+----------------+
| images/000000315555.jpg ┆ 19 |
| images/000000016950.jpg ┆ 19 |
| images/000000244933.jpg ┆ 17 |
| images/000000113762.jpg ┆ 14 |
| … ┆ … |
| images/000000026942.jpg ┆ 1 |
| images/000000491845.jpg ┆ 1 |
| images/000000536154.jpg ┆ 1 |
| images/000000559557.jpg ┆ 1 |
+-------------------------+----------------+