How to write a shell script to search specific files and combine them together to a single file?

Author: Y. Chen
Last revision: 2020/03/19, J. Lathière

Sometimes, you may need some works to combine the ORCHIDEE results from the specific output frequency to a single file for the post analysis.

Here, I made a simple bash script for your reference so that you can combine the several files to one single file.

The commands, ncrcat, find, sort and export, will be used. See the following script for the detail.

The script below combines output files (daily temp, monthly output) from 1981-2000. You can change the path or year as you need.

#!/bin/bash

# search file names and rebuild
# set up working years
year=(1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000)

# set the experiment name
exp="BCLb"

# set the file path for searching files 
in_dir_path="/ccc/store/cont003/dsm/p529luy/IGCM_OUT/LMDZOR/PROD/amip/"${exp}"/ATM/Output/DA"

#echo $in_dir_path
# move path to search the files
cd $in_dir_path

# start the loop for searching the files...
for index in {0..19}
do
    echo "Search files on year: ${year[$index]}"
    # use the "find" and "sort" command to find the files 
    # and sort by the numeric number appearing in the filenames 
    inputfilename=( $(find -name "${exp}_${year[$index]}*" | sort -n) )
    echo "PROCESSING input files: "${inputfilename[@]}

    # make an output file name based on the working "exp" and "year" 
    export outputfilename="${exp}_${year[$index]}.nc"

    # use "ncrcat" command to re-create(combine) the nc file based on the searched inputfilename    
    ncrcat ${inputfilename[@]} /ccc/store/cont003/dsm/cheny/Daily/${exp}/$outputfilename

    echo "PLEASE CHECK output file: "$outputfilename
done 

echo "FINISHED THE COMBINE files job.. (AH-JA...)"$outputfilename