Tool for merging NetCDF output files

[AlexGuntner]

Dear all,

I am currently running some simulations and have chosen to use the checkpoint restart option for some cases. After the initial run, I obtain a file named sampling00000.nc, and upon restarting from a checkpoint, a new file such as sampling25000.nc is generated.

Is there a way to overwrite the initial sampling file when using checkpoint restarts? If not, do you have any available scripts or utilities that can be used to merge the two sampling files?

Best regards

[hgopalan]

Overwriting is an option we can discuss internally and see if we want to add a flag for it but will be a while before it happens.

The easiest way to merge the netcdf files is to use xarray or netCDF4 in python. You can ask chatGPT to write the code for you. What is it that you are trying to do?

[AlexGuntner]

Hi @hgopalan,

I'm looking to create an animation of my simulation using the animate class from the AMR-Wind frontend. However, I currently have two separate sampling files, which is making things a bit tricky.

I tried asking ChatGPT, but since I'm not very familiar with NetCDF4 files, I haven’t had much luck. When I try to inspect the files, I only see a 'time(num_time_steps)' variable. I assume the actual domain data and variables I need are stored in subgroups, but I’m not sure how to access them properly.

[hgopalan]

Run ncdump -h nameoffile

It will give you the name of the sub-group. The sub-group name is usually the name you assign in lablels in the input file.

[hgopalan]

from netCDF4 import Dataset
import numpy as np

Open the NetCDF file

file_path = 'your_file.nc' # Replace with your actual file path
nc_file = Dataset(file_path, 'r')

Navigate to the subgroup 'mast1'

mast1_group = nc_file.groups['mast1']

Access the 'velocityx' variable in the subgroup

velocityx_var = mast1_group.variables['velocityx']

Convert the variable to a NumPy array

velocityx_array = np.array(velocityx_var[:])

Close the NetCDF file

nc_file.close()

Now velocityx_array contains the data as a NumPy array

print(velocityx_array.shape) # Optional: Check the shape of the array

Check if this code snipped works.

[lawrenceccheung]

Hi @AlexGuntner,

You can use the catNC utility https://github.com/Exawind/amr-wind-frontend/blob/main/utilities/catNC.py to merge netcdf sampling files. In its simplest form, you can use it like this:

catNC.py  sampling00000.nc  sampling25000.nc -o out.nc

Also, if you're making movies using the amr-wind-frontend, you don't actually need to merge the netcdf files together before generating the movie. If you're using the instantaneousplanes option in the postprocessing engine like in this notebook, then you can run through each sampling file and generate the images, then in the last one add the animate command, as in the following yaml section:

instantaneousplanes:
  - name: HHplot1
    ncfile: post_processing/turbhh_00000.nc
    # [some other options]
    savefile: movieframes/NREL5MW_wake_z90_{time:0.1f}.png
  - name: HHplot2
    ncfile: post_processing/turbhh_25000.nc
    # [some other options]
    savefile: movieframes/NREL5MW_wake_z90_{time:0.1f}.png
    animate:
      name: NREL5MW_movie.mp4
      fps: 5
      imagefilename: movieframes/NREL5MW_wake_z90_{time:0.1f}.png

Lawrence

[AlexGuntner]

Thank you both for your help—everything is solved now!

Best regards,

Alex