Skip to content

Commit efcf7d0

Browse files
committed
added mkdocs
1 parent 9e5bdd0 commit efcf7d0

File tree

7 files changed

+207
-0
lines changed

7 files changed

+207
-0
lines changed

docs/about.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# About
2+
3+
## Changelog
4+
See `CHANGES.md`.
5+
6+
## Reporting issues
7+
Open an issue at https://github.com/deepesdl/deep-code/issues.
8+
9+
## Contributions
10+
PRs are welcome. Please follow the code style (black/ruff) and add tests where relevant.
11+
12+
## Development install
13+
```bash
14+
pip install -e .[dev]
15+
pytest
16+
pytest --cov=deep-code
17+
black .
18+
ruff check .
19+
```
20+
21+
## Documentation commands (MkDocs)
22+
```bash
23+
pip install -e .[docs] # install mkdocs + theme
24+
mkdocs serve # live preview at http://127.0.0.1:8000
25+
mkdocs build # build site into site/
26+
mkdocs gh-deploy --clean # publish to GitHub Pages
27+
```
28+
29+
## License
30+
MIT License. See `LICENSE`.

docs/cli.md

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# CLI
2+
3+
## Generate configs
4+
Create starter templates for both workflow and dataset:
5+
6+
```bash
7+
deep-code generate-config # writes to current directory
8+
deep-code generate-config -o ./configs # custom output folder
9+
```
10+
11+
## Publish metadata
12+
Publish dataset, workflow, or both (default is both) to the target environment:
13+
14+
```bash
15+
deep-code publish dataset.yaml workflow.yaml # production (default)
16+
deep-code publish dataset.yaml workflow.yaml -e staging # staging
17+
deep-code publish dataset.yaml -m dataset # dataset only
18+
deep-code publish workflow.yaml -m workflow # workflow only
19+
deep-code publish --dataset-config ./ds.yaml --workflow-config ./wf.yaml -m all
20+
```
21+
22+
Options:
23+
- `--environment/-e`: `production` (default) | `staging` | `testing`
24+
- `--mode/-m`: `all` (default) | `dataset` | `workflow`
25+
- `--dataset-config` / `--workflow-config`: explicitly set paths and bypass auto-detection
26+
27+
## How publishing works
28+
1. Reads your configs and builds dataset STAC collections plus variable catalogs.
29+
2. Builds workflow and experiment OGC API Records.
30+
3. Forks/clones the target metadata repo (production, staging, or testing), commits generated JSON, and opens a pull request on your behalf.

docs/configuration.md

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
# Configuration
2+
3+
## Dataset config (YAML)
4+
```yaml
5+
dataset_id: your-dataset.zarr
6+
collection_id: your-collection
7+
osc_themes: [cryosphere]
8+
osc_region: global
9+
dataset_status: completed # or ongoing/planned
10+
documentation_link: https://example.com/docs
11+
access_link: s3://bucket/your-dataset.zarr
12+
```
13+
14+
## Workflow config (YAML)
15+
```yaml
16+
workflow_id: your-workflow
17+
properties:
18+
title: "My workflow"
19+
description: "What this workflow does"
20+
keywords: ["Earth Science"]
21+
themes: ["cryosphere"]
22+
license: proprietary
23+
jupyter_kernel_info:
24+
name: deepesdl-xcube-1.8.3
25+
python_version: 3.11
26+
env_file: https://example.com/environment.yml
27+
jupyter_notebook_url: https://github.com/org/repo/path/to/notebook.ipynb
28+
contact:
29+
- name: Jane Doe
30+
organization: Example Org
31+
links:
32+
- rel: about
33+
type: text/html
34+
href: https://example.org
35+
```
36+
37+
More templates and examples live in `dataset_config.yaml`, `workflow_config.yaml`, and `example-config/`.

docs/examples.md

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
# Examples
2+
3+
- Templates: `dataset_config.yaml`, `workflow_config.yaml`
4+
- Example configs: `examples/example-config/`
5+
- Notebooks on publishing: `examples/notebooks`

docs/getting-started.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# Getting Started
2+
3+
When working with cloud platforms like DeepESDL, workflow outputs typically live in S3 object storage. Before the project ends or once datasets and workflows are finalized, move the datasets into the ESA Project Results Repository (PRR) and publish the workflows (Jupyter notebooks) to a publicly accessible GitHub repository. The notebook path becomes an input in the dataset config file.
4+
5+
Use the EarthCODE Project Results Repository to publish and preserve outputs from ESA-funded Earth observation projects. It is professionally maintained, FAIR-aligned, and keeps your results findable, reusable, and citable for the long term—no storage, operations, or access headaches.
6+
7+
To transfer datasets into the ESA PRR, contact the DeepESDL platform team at [[email protected]](mailto:[email protected]).
8+
9+
In the near future, `deep-code` will include built-in support for uploading your results to the ESA PRR as part of the publishing workflow, making it seamless to share your scientific contributions with the community.
10+
11+
## Requirements
12+
- Python 3.10+
13+
- GitHub token with access to the target EarthCODE metadata repo.
14+
- Input configuration files.
15+
- Datasets which needs to be published is uploaded to S3 like object storage and made publicly accessible.
16+
17+
## Install
18+
```bash
19+
pip install deep-code
20+
```
21+
22+
## Authentication
23+
The CLI or the Python API reads GitHub credentials from a `.gitaccess` file in the directory where you run the command:
24+
25+
1. **Generate a GitHub Personal Access Token (PAT)**
26+
27+
1. Navigate to GitHub → Settings → Developer settings → Personal access tokens.
28+
2. Click “Generate new token”.
29+
3. Choose the following scopes to ensure full access:
30+
- repo (Full control of repositories — includes fork, pull, push, and read)
31+
4. Generate the token and copy it immediately — GitHub won’t show it again.
32+
33+
2. **Create the .gitaccess File**
34+
35+
Create a plain text file named .gitaccess in your project directory or home folder:
36+
37+
```
38+
github-username: your-git-user
39+
github-token: your-personal-access-token
40+
```
41+
Replace your-git-user and your-personal-access-token with your actual GitHub username and token.
42+
43+

docs/index.md

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Overview
2+
3+
`deep-code` is a lightweight Python CLI and API that publishes DeepESDL datasets and workflows as EarthCODE Open Science Catalog metadata. It can generate starter configs, build STAC collections and OGC API records, and open pull requests to the target EarthCODE metadata repository (production, staging, or testing).
4+
5+
## Features
6+
- Generate starter dataset and workflow YAML templates.
7+
- Publish dataset collections, workflows, and experiments via a single command.
8+
- Build STAC collections and catalogs for Datasets and their corresponding variables automatically from the dataset metadata.
9+
- Build OGC API records for Workflows and Experiments from your configs.
10+
- Flexible publishling targets i.e production/staging/testing EarthCODE metadata repositories with GitHub automation.
11+
12+
```mermaid
13+
%%{init: {'flowchart': {'nodeSpacing': 110, 'rankSpacing': 160}, 'themeVariables': {'fontSize': '28px', 'lineHeight': '1.6em'}}}%%
14+
flowchart LR
15+
subgraph User
16+
A["Config files<br/>(dataset.yaml, workflow.yaml)"]
17+
B["deep-code CLI<br/>(generate-config, publish)"]
18+
end
19+
20+
subgraph App["deep-code internals"]
21+
C["Publisher<br/>(mode: dataset/workflow/all)"]
22+
D["STAC builder<br/>OscDatasetStacGenerator"]
23+
E["OGC record builder<br/>OSCWorkflowOGCApiRecordGenerator"]
24+
F["GitHubAutomation<br/>(fork, clone, branch, PR)"]
25+
end
26+
27+
subgraph Output
28+
G["Generated JSON<br/>collections, variables,<br/>workflows, experiments"]
29+
H["GitHub PR<br/>(prod/staging/testing repo)"]
30+
I["EarthCODE Open Science Catalog"]
31+
end
32+
33+
A --> B --> C
34+
C --> D
35+
C --> E
36+
D --> G
37+
E --> G
38+
G --> F --> H --> I
39+
```

docs/python-api.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Python API
2+
3+
`deep_code.tools.publish.Publisher` is the main entry point.
4+
5+
```python
6+
from deep_code.tools.publish import Publisher
7+
8+
publisher = Publisher(
9+
dataset_config_path="dataset.yaml",
10+
workflow_config_path="workflow.yaml",
11+
environment="staging",
12+
)
13+
14+
# Generate files locally (no PR)
15+
publisher.publish(write_to_file=True, mode="all")
16+
17+
# Or open a PR directly
18+
publisher.publish(write_to_file=False, mode="dataset")
19+
```
20+
21+
Other utilities:
22+
- `deep_code.tools.new.TemplateGenerator` for programmatic template generation.
23+
- `deep_code.utils.dataset_stac_generator.OscDatasetStacGenerator` and `deep_code.utils.ogc_record_generator.OSCWorkflowOGCApiRecordGenerator` for lower-level metadata building.

0 commit comments

Comments
 (0)