Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.oxen.ai/llms.txt

Use this file to discover all available pages before exploring further.

Say you are working in a repository with over 100GB of data. For a variety of reasons, you may only want to interact with a subset of this data. It is expensive to clone the whole history. Maybe you want to add a single row to a dataset or upload a set of 10 images. You may only need a small subset of the data downloaded to view it locally before sending back up changes. In these cases, it doesn’t make sense to download the entire repository history locally. Instead, you can use a combination of tools that oxen provides to only interact with the data you need. Oxen has three main ways of interacting with subsets of your data.
  1. Remote Workspaces - Interact with your data all server side, no files are downloaded locally.
  2. Partial Clones - Clone a subtree of the data in your repository to a local working directory.
  3. Download Read Only - Download a read only copy of the subset to your local machine.
Each of these methods has it’s own benefits and trade offs. We will go over each of them in more detail below.

​
Remote Workspaces

The concept of a remote workspace in Oxen is akin to having a working directory on the remote server where you can stage changes before committing. This allows you to perform multiple write operations without requiring a commit message for each one. Data written to workspaces is not ephemeral, as it does survive a server restart, but it is not committed to the history of the repository, so may be deleted without a trace. Remote Workspace Under the hood, a workspace is tied to a commit id and an origin branch. This means whatever changes you make will always be with respect to the commit you created the workspace off of. If the remote branch advances, you may have to resolve merge conflicts before committing.

​
Instantiating a Workspace

A workspace is created off of a remote repository and a branch name. The branch name is just a convenience for the user to create a workspace on the underlying commit id. 
from oxen import RemoteRepo
from oxen import Workspace

repo = RemoteRepo("ox/CatDogBBox")
workspace = Workspace(repo, "add-images")
If no branch name is provided, the workspace will be created off of the default branch (usually main). 
from oxen import RemoteRepo
from oxen import Workspace

repo = RemoteRepo("ox/CatDogBBox")
workspace = Workspace(repo)

​
Adding Files

When adding data, it is always a good idea to create a branch for the changes you are about to make. This will allow you to commit changes without affecting the default branch.

​
Creating a Branch

from oxen import RemoteRepo

repo = RemoteRepo("ox/CatDogBBox")
# Create the branch on the remote
repo.create_branch("add-images")
# Make sure you are pointing to this branch
repo.checkout("add-images")

​
Uploading Files

Workspaces allow you to upload files without immediately committing them. Think of this as a staging area where you can upload the data, and then batch commit when you are ready. 
from oxen import RemoteRepo
from oxen import Workspace

repo = RemoteRepo("ox/CatDogBBox")
workspace = Workspace(repo, "add-images")
workspace.add("/path/to/image.png")
status = workspace.status()
print(status.added_files())

​
Removing Uploaded Files

If you accidentally add file from the remote workspace and want to remove it, no worries, you can unstage it with oxen remote rm --staged. 
from oxen import RemoteRepo
from oxen import Workspace

repo = RemoteRepo("ox/CatDogBBox")
workspace = Workspace(repo, "add-images")
workspace.rm("image.jpg")
status = workspace.status()
print(status.added_files())

​
Commit Changes

When you are confident in the changes you have made, you can commit the changes to the remote workspace. This will create a new commit on the remote branch. 
from oxen import RemoteRepo
from oxen import Workspace

repo = RemoteRepo("ox/CatDogBBox")
workspace = Workspace(repo, "add-images")
workspace.commit("adding an image", "add-images")
🎉 You have now committed data to the remote branch without cloning the full repo. Note: If the remote branch cannot be merged cleanly, the remote commit will fail, and you will have to resolve the merge conflicts with some more advanced commands which we will cover later.

​
Partial Clones

The first command line parameter you should be aware of is the --filter flag. This flag is inclusive for the paths you want to clone. 
oxen clone https://hub.oxen.ai/ox/Flowers --filter "images/roses"
This will clone all the data under the images/roses directory into a local working directory. Under the hood, it also creates a .oxen directory which contains the merkle tree for the cloned data, and content addressable copies of each file in the subtree. Partial Clone You can also specify a depth parameter to control how deep the clone is. If you have many nested subdirectories, you can use the --depth flag to limit how deep the clone goes. 
oxen clone https://hub.oxen.ai/ox/Flowers --filter "." --depth 1
Note that full clones and partial clones end up using ~2x the storage. This is because the clone contains the merkle tree for the cloned data, and content addressable copies of each file in the subtree.

​
Download Read Only

If you have no intention of making any changes to the data, the easiest way to interact with a subset is to download a read only copy. This can be done with the oxen download command. 
oxen download ox/Flowers images/roses
Under the hood, this command does not download any of the history, content addressed version files, or other metadata. It simply downloads the data unpacks it to a local directory. Download Read Only This is the most efficient way to download data if you are simply going to read the data or throw it away later.