Skip to content

Commit 6beeb33

Browse files
update links for gpu blog post to v1.0 (#788)
* update links to v1.0 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
1 parent fd6b25b commit 6beeb33

File tree

2 files changed

+4
-6
lines changed

2 files changed

+4
-6
lines changed

src/posts/gpu-pipeline/index.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -107,7 +107,7 @@ During the hackathon, we tested the following strategies to improve the data loa
107107

108108
The copy of the ERA5 dataset we were using initially had a suboptimal chunking scheme of `{'time': 10, 'channel': C, 'height': H, 'width': W}`, which meant that a minimum of 10 time steps of data was being read even if we only needed 2 consecutive time steps.
109109
We decided to rechunk the data to align with our access pattern of 1-timestep at a time, while reformating to Zarr format 3.
110-
The full script is available [here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/blob/main/rechunk/era5_rechunking.ipynb), with the main code looking like so:
110+
The full script is available [here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/blob/v1.0/rechunk/era5_rechunking.ipynb), with the main code looking like so:
111111

112112
```python
113113
import xarray as xr
@@ -198,7 +198,7 @@ With nvCOMP, all steps of data loading including reading from disk, decompressio
198198

199199
To unlock this, we would need zarr-python to support GPU-based decompression codecs, with one for Zstandard (Zstd) currently being implemented in [this PR](https://github.com/zarr-developers/zarr-python/pull/2863).
200200

201-
We tested the performance of GPU-based decompression using nvCOMP with Zarr-Python 3 and KvikIO, and compared it to CPU-based decompression using [this data reading benchmark here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/blob/main/benchmark/era5_zarr_benchmark.py).
201+
We tested the performance of GPU-based decompression using nvCOMP with Zarr-Python 3 and KvikIO, and compared it to CPU-based decompression using [this data reading benchmark here](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/v1.0/benchmarks/era5_zarr_benchmark.py).
202202

203203
Here are the results:
204204

@@ -220,7 +220,7 @@ Ideally, we want to minimize idle time on both the CPU and GPU by overlapping th
220220

221221
To address this inefficiency, we adopted [NVIDIA DALI (Data Loading Library)](https://docs.nvidia.com/deeplearning/dali/user-guide/docs/index.html), which provides a flexible, GPU-accelerated data pipeline with built-in support for asynchronous execution across CPU and GPU stages. DALI helps reduce CPU pressure, enables concurrent preprocessing, and increases training throughput by pipelining operations.
222222

223-
First, we began with a minimal example in the [zarr_DALI directory](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/main/zarr_DALI) with short, contained examples of a DALI pipeline loading directly from Zarr stores. This example shows how to build a custom DALI `pipeline` that uses an `ExternalSource` operator to load batched image data from a Zarr store and transfer them directly to GPU memory using CuPy arrays.
223+
First, we began with a minimal example in the [zarr_dali_example directory](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/v1.0/zarr_dali_example) with short, contained examples of a DALI pipeline loading directly from Zarr stores. This example shows how to build a custom DALI `pipeline` that uses an `ExternalSource` operator to load batched image data from a Zarr store and transfer them directly to GPU memory using CuPy arrays.
224224

225225
In short, to use DALI with Zarr for data loading, you need to:
226226

@@ -273,7 +273,7 @@ output = pipe.run()
273273
images_gpu, labels_gpu = output
274274
```
275275

276-
Next, checkout the [end-to-end example](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/main/zarr_ML_optimization) directory, where we showed how to use DALI to load data from Zarr stores, preprocess it on the GPU, and feed it into a PyTorch model for training.
276+
Next, checkout the [end-to-end example](https://github.com/pangeo-data/ncar-hackathon-xarray-on-gpus/tree/v1.0/zarr_ML_optimization) directory, where we showed how to use DALI to load data from Zarr stores, preprocess it on the GPU, and feed it into a PyTorch model for training.
277277

278278
Profiling results show that the DALI pipeline enables efficient overlap of CPU and GPU operations, significantly reducing GPU idle time and boosting overall training throughput.
279279

src/posts/pangeo-ml-ecosystem-2023/index.md

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,13 +94,11 @@ Lastly, we highlighted some of the high-level Pangeo ML libraries enabling user
9494
## Where to learn more
9595

9696
- Educational resources:
97-
9897
- [Project Pythia Cookbooks](https://cookbooks.projectpythia.org)
9998
- [GeoSMART Machine Learning Curriculum](https://geo-smart.github.io/mlgeo-book)
10099
- [University of Washington Hackweeks as a Service](https://guidebook.hackweek.io)
101100

102101
- Pangeo ML Working Group:
103-
104102
- [Monthly meetings](https://pangeo.io/meeting-notes.html#working-group-meetings)
105103
- [Discourse Forum](https://discourse.pangeo.io/tag/machine-learning)
106104

0 commit comments

Comments
 (0)