Skip to main content

​
oxen.oxen_fs

​
OxenFS Objects

class OxenFS(fsspec.AbstractFileSystem)
OxenFS is a filesystem interface for Oxen repositories that implements the fsspec protocol. This allows you to interact with Oxen repositories using familiar filesystem operations and integrate with other compatible libraries like Pandas.

​
Basic Usage

​
Creating a Filesystem Instance

import oxen

# For Oxen Hub repositories
fs = oxen.OxenFS("ox", "Flowers")

# For local oxen-server
fs = oxen.OxenFS("ox", "test-repo", host="localhost:3000", scheme="http")

​
Reading Files

with fs.open("data/train.csv") as f:
    content = f.read()

​
Writing Files

You must have write access to the repository to write files. See: https://docs.oxen.ai/getting-started/python#private-repositories OxenFS will automatically commit the file to the repository when the context is exited (or the file is closed some other way). New directories are automatically created as needed. 
# Write with custom commit message
with fs.open("data/test.txt", mode="wb", commit_message="Added test.txt") as f:
    f.write("Hello, world!")

# You can also set/update the commit message inside the context
with fs.open("data/test.txt", mode="wb") as f:
    f.commit_message = "Updated test.txt"
    f.write("Hello, world again!")

​
Writing file objects

If you’re integrating Oxen in a situation where you already have a file object, you can save it to your repo by using shutil.copyfileobj like this: 
import shutil

file_object_from_somewhere = open("data.csv")

with fs.open("train/data.csv", mode="wb") as output_file:
    output_file.commit_message = "Copy from a file object"
    shutil.copyfileobj(file_object_from_somewhere, output_file)

​
Integration with Third Party Libraries (Pandas, etc.)

OxenFS works seamlessly with Pandas and other fsspec-compatible libraries using the URL format: oxen://namespace:repo@revision/path/to/file

​
Reading Data

These will work with Pandas {to,from}_{csv,parquet,json,etc.} functions. 
import pandas as pd

# Read parquet directly from Oxen repository
df = pd.read_parquet("oxen://openai:gsm8k@main/gsm8k_test.parquet")

​
Writing Data

# Write DataFrame directly to Oxen repository
df.to_csv("oxen://ox:my-repo@main/data/test.csv", index=False)

​
Notes

  • Only binary read (“rb”) and write (“wb”) modes are currently supported
    • But writing will automatically encode strings to bytes
  • Does not yet support streaming files. All operations use temporary local files.

​
__init__

def __init__(namespace: str,
             repo: str,
             host: str = "hub.oxen.ai",
             revision: str = "main",
             scheme: str = "https",
             **kwargs)
Initialize the OxenFS instance. Arguments:
  • namespace - str The namespace of the repository.
  • repo - str The name of the repository.
  • host - str The host to connect to. Defaults to ‘hub.oxen.ai’
  • revision - str The branch name or commit id to checkout. Defaults to ‘main’
  • scheme - str The scheme to use for the remote url. Default: ‘https’

​
ls

def ls(path: str = "", detail: bool = False)
List the contents of a directory. Arguments:
  • path - str The path to list the contents of.
  • detail - bool If True, return a list of dictionaries with detailed metadata. Otherwise, return a list of strings with the filenames.

​
OxenFSFileWriter Objects

class OxenFSFileWriter()
A file writer for the OxenFS backend. This is normally called through OxenFS.open() or fsspec.open().

​
write

def write(data: str | bytes)
Write string or binary data to the file.

​
flush

def flush()
Flush the file to disk.

​
tell

def tell()
Return the current position of the file.

​
seek

def seek(offset: int, whence: int = os.SEEK_SET)
Seek to a specific position in the file.

​
commit

def commit(commit_message: Optional[str] = None)
Commit the file to the remote repo.

​
close

def close()
Close the file writer. This will commit the file to the remote repo.