Managing and validating data schemas is an important part of making robust data pipelines.
⚠️ This is a work in progress. Do not rely on being stable yet. ⚠️
When you add a dataframe to Oxen, it automatically detects and versions the schema of any tabular data. This is done by using Polars under the hood to infer the column names and datatypes.
To list all the schemas that have been detected and committed, you can use the oxen schemas
subcommand.
The schema detection is done on a per file basis. This means that if you have a directory of csv or parquet files, each file will have its own schema.
To view a specific schema, you can pass in a schema hash, name, or path to the oxen schemas
command.
Schemas are automatically detected when you add csv
, tsv
, jsonl
, parquet
, and arrow
files to Oxen. Before a schema is committed, you can see the detected schemas in the oxen status
command.
To view more detailed information about the detected schema, use the --staged
flag on the oxen schemas
command.
To view how Polars interprets the schema before adding the file, you can use the oxen df
command with the --schema
flag.
You can also add additional information to the schema. This is useful if you want to provide context about the data for a UI, data fetching, or any other reason.
Notice the empty column metadata
in the schema above. You can add arbitrary JSON blobs to the schema itself, as well as each column.
Metadata may provide useful information for your end application:
At the root of each schema is an Optional<json::Value>
metadata value. This is useful for adding information about the schema itself. For example, you can add a description of the schema or a json blob that gives context to a data renderer.
You will see the additional metadata listed above the schema if it is added.
You can also add metadata to specific columns. Say you wanted to add information to the file
column about the root directory of the images, you could do the following:
The -c
flag stands for column
and the -m
flag stands for metadata
. The metadata is a JSON blob that can be used to store any information you want.
The OxenHub UI uses schema metadata to render more complex datatypes in the UI. For example viewing inline images directly in a dataframe.
TODO: Add image of OxenHub UI.
TODO: Pushing logic is not working, unless you have changed the dataframe.
Schemas changes will not be saved until you commit them.
To view the schemas staged for commit, you can use the --staged
flag.
You can then commit the schema to the dataframe with the commit
subcommand.
These changes are persistent across commits and will be carried forward.
It is nice to have human readable names to refer to schemas by. Use the oxen schemas name
command to name a schema.
If you have accidentally staged a schema, you can remove it with the oxen schemas rm
command.
Oxen.ai can render images through the webhub if you add the proper schema metadata.
Under the hood this applied a metadata blob to the column, telling Oxen to render an image. More verbosely it would look like:
Oxen.ai can render links to other files through the webhub if you add the proper schema metadata.
Under the hood this applied a metadata blob to the column, telling Oxen to render an link. More verbosely it would look like:
Managing and validating data schemas is an important part of making robust data pipelines.
⚠️ This is a work in progress. Do not rely on being stable yet. ⚠️
When you add a dataframe to Oxen, it automatically detects and versions the schema of any tabular data. This is done by using Polars under the hood to infer the column names and datatypes.
To list all the schemas that have been detected and committed, you can use the oxen schemas
subcommand.
The schema detection is done on a per file basis. This means that if you have a directory of csv or parquet files, each file will have its own schema.
To view a specific schema, you can pass in a schema hash, name, or path to the oxen schemas
command.
Schemas are automatically detected when you add csv
, tsv
, jsonl
, parquet
, and arrow
files to Oxen. Before a schema is committed, you can see the detected schemas in the oxen status
command.
To view more detailed information about the detected schema, use the --staged
flag on the oxen schemas
command.
To view how Polars interprets the schema before adding the file, you can use the oxen df
command with the --schema
flag.
You can also add additional information to the schema. This is useful if you want to provide context about the data for a UI, data fetching, or any other reason.
Notice the empty column metadata
in the schema above. You can add arbitrary JSON blobs to the schema itself, as well as each column.
Metadata may provide useful information for your end application:
At the root of each schema is an Optional<json::Value>
metadata value. This is useful for adding information about the schema itself. For example, you can add a description of the schema or a json blob that gives context to a data renderer.
You will see the additional metadata listed above the schema if it is added.
You can also add metadata to specific columns. Say you wanted to add information to the file
column about the root directory of the images, you could do the following:
The -c
flag stands for column
and the -m
flag stands for metadata
. The metadata is a JSON blob that can be used to store any information you want.
The OxenHub UI uses schema metadata to render more complex datatypes in the UI. For example viewing inline images directly in a dataframe.
TODO: Add image of OxenHub UI.
TODO: Pushing logic is not working, unless you have changed the dataframe.
Schemas changes will not be saved until you commit them.
To view the schemas staged for commit, you can use the --staged
flag.
You can then commit the schema to the dataframe with the commit
subcommand.
These changes are persistent across commits and will be carried forward.
It is nice to have human readable names to refer to schemas by. Use the oxen schemas name
command to name a schema.
If you have accidentally staged a schema, you can remove it with the oxen schemas rm
command.
Oxen.ai can render images through the webhub if you add the proper schema metadata.
Under the hood this applied a metadata blob to the column, telling Oxen to render an image. More verbosely it would look like:
Oxen.ai can render links to other files through the webhub if you add the proper schema metadata.
Under the hood this applied a metadata blob to the column, telling Oxen to render an link. More verbosely it would look like: