Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(docs): Add new docs site #161

Merged
merged 2 commits into from
Oct 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,7 @@
/target

# macOS
.DS_Store
.DS_Store

# Docs build
.docs
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
![Icechunk logo](icechunk_logo.png)

Icechunk is a transactional storage engine for Zarr designed for use on cloud object storage.
<!--home-start-->

Let's break down what that means:

Expand Down Expand Up @@ -35,7 +36,7 @@ Icechunk aspires to support the following core requirements for stores:

This Icechunk project consists of three main parts:

1. The [Icechunk specification](spec/icechunk_spec.md).
1. The [Icechunk specification](spec/icechunk-spec.md).
1. A Rust implementation
1. A Python wrapper which exposes a Zarr store interface

Expand All @@ -51,10 +52,9 @@ Governance of the project will be managed by Earthmover PBC.

We recommend using Icechunk from Python, together with the Zarr-Python library

> [!WARNING]
> Icechunk is a very new project.
> It is not recommended for production use at this time.
> These instructions are aimed at Icechunk developers and curious early adopters.
!!! warning "Icechunk is a very new project."
It is not recommended for production use at this time.
These instructions are aimed at Icechunk developers and curious early adopters.

### Installation and Dependencies

Expand Down Expand Up @@ -142,8 +142,8 @@ This is just an alias for
cargo test --all
```

> [!TIP]
> For other aliases see [Justfile](./Justfile).
!!! tip
For other aliases see [Justfile](./Justfile).

## Snapshots, Branches, and Tags

Expand All @@ -169,7 +169,7 @@ Tags are appropriate for publishing specific releases of a repository or for any
## How Does It Work?

> [!NOTE]
> For more detailed explanation, have a look at the [Icechunk spec](spec/icechunk_spec.md)
> For more detailed explanation, have a look at the [Icechunk spec](spec/icechunk-spec.md)

Zarr itself works by storing both metadata and chunk data into a abstract store according to a specified system of "keys".
For example, a 2D Zarr array called myarray, within a group called mygroup, would generate the following keys.
Expand Down
1 change: 1 addition & 0 deletions docs/.env.sample
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
MKDOCS_GIT_COMMITTERS_APIKEY=
79 changes: 79 additions & 0 deletions docs/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# imported files
docs/icechunk-python/examples
docs/icechunk-python/notebooks
docs/spec.md

# C extensions
*.so

# Build
.site

.env
# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
node_modules/
parts/
sdist/
var/
package*.json
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover
.hypothesis/

# Translations
*.mo

# Scrapy stuff:
.scrapy

# PyBuilder
target/

# IPython Notebook
.ipynb_checkpoints

# pyenv
.python-version

# virtualenv
venv/
ENV/

# MkDocs documentation
site*/
41 changes: 41 additions & 0 deletions docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# Icechunk Documentation Website

Built with [MkDocs](https://www.mkdocs.org/) using [Material for MkDocs](https://squidfunk.github.io/mkdocs-material/).

## Developing

### Prerequisites

This repository uses [Poetry](https://python-poetry.org/) to manage dependencies

1. Install dependencies using `poetry install`

### Running

1. Run `poetry shell` from the `/docs` directory
2. Start the MkDocs development server: `mkdocs serve`

!!! tip
You can use the optional `--dirty` flag to only rebuild changed files, although you may need to restart if you make changes to `mkdocs.yaml`.

### Building

1. Run `mkdocs build`

Builds output to: `icechunk-docs/.site` directory.

## Dev Notes

#### Symlinked Files

Several directories and files are symlinked into the MkDocs' `/docs`[^1] directory in order to be made available to MkDocs. Avoid modifying them directly:
* `/docs/icechunk-python/examples/`
* `/docs/icechunk-python/notebooks/`
* `/docs/spec.md`

These are also ignored in `.gitignore`

!!! tip
See [icechunk-docs/macros.py](./macros.py) for more info.

[^1] : Disambiguation: `icechunk/docs/docs`
81 changes: 81 additions & 0 deletions docs/docs/icechunk-python/developing.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Icechunk Python

Python library for Icechunk Zarr Stores

## Getting Started

Activate the virtual environment:

```bash
python3 -m venv .venv
source .venv/bin/activate
```

Install `maturin`:

```bash
pip install maturin
```

Build the project in dev mode:

```bash
maturin develop
```

or build the project in editable mode:

```bash
pip install -e icechunk@.
```

!!! NOTE
This only makes the python source code editable, the rust will need to be recompiled when it changes

Now you can create or open an icechunk store for use with `zarr-python`:

```python
from icechunk import IcechunkStore, StorageConfig
from zarr import Array, Group

storage = StorageConfig.memory("test")
store = await IcechunkStore.open(storage=storage, mode='r+')

root = Group.from_store(store=store, zarr_format=zarr_format)
foo = root.create_array("foo", shape=(100,), chunks=(10,), dtype="i4")
```

You can then commit your changes to save progress or share with others:

```python
store.commit("Create foo array")

async for parent in store.ancestry():
print(parent.message)
```

!!! tip
See [`tests/test_timetravel.py`](https://github.com/earth-mover/icechunk/blob/main/icechunk-python/tests/test_timetravel.py) for more example usage of the transactional features.


## Running Tests

You will need [`docker compose`](https://docs.docker.com/compose/install/) and (optionally) [`just`](https://just.systems/).
Once those are installed, first switch to the icechunk root directory, then start up a local minio server:
```
docker compose up -d
```

Use `just` to conveniently run a test
```
just test
```

This is just an alias for

```
cargo test --all
```

!!! tip
For other aliases see [Justfile](./Justfile).
66 changes: 66 additions & 0 deletions docs/docs/icechunk-python/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# Icechunk Python

### Installation and Dependencies

Icechunk is currently designed to support the [Zarr V3 Specification](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html).
Using it today requires installing the [still unreleased] Zarr Python V3 branch.

To set up an Icechunk development environment, follow these steps

Activate your preferred virtual environment (here we use `virtualenv`):

```bash
python3 -m venv .venv
source .venv/bin/activate
```

Alternatively, create a conda environment

```bash
mamba create -n icechunk rust python=3.12
conda activate icechunk
```

Install `maturin`:

```bash
pip install maturin
```

Build the project in dev mode:

```bash
cd icechunk-python/
maturin develop
```

or build the project in editable mode:

```bash
cd icechunk-python/
pip install -e icechunk@.
```

!!! warning
This only makes the python source code editable, the rust will need to be recompiled when it changes

### Basic Usage

Once you have everything installed, here's an example of how to use Icechunk.

```python
from icechunk import IcechunkStore, StorageConfig
from zarr import Array, Group

# Example using memory store
storage = StorageConfig.memory("test")
store = await IcechunkStore.open(storage=storage, mode='r+')

# Example using file store
storage = StorageConfig.filesystem("/path/to/root")
store = await IcechunkStore.open(storage=storage, mode='r+')

# Example using S3
s3_storage = StorageConfig.s3_from_env(bucket="icechunk-test", prefix="oscar-demo-repository")
store = await IcechunkStore.open(storage=storage, mode='r+')
```
1 change: 1 addition & 0 deletions docs/docs/icechunk-python/reference.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
::: icechunk
Loading
Loading