Tuesday, February 6, 2018

Install WRF

After days of frustration, I finally get WRF running in my MacBook. Thanks to the help from wrfhelp@ucar.edu
$ brew install NetCDF
🍺  /usr/local/Cellar/netcdf/4.6.0: 84 files, 6MB
$ brew install mpich
🍺  /usr/local/Cellar/mpich/3.2.1_1: 887 files, 14.2MB
$ brew install zlib
Warning: zlib 1.2.11 is already installed
$ brew install libpng
Warning: libpng 1.6.34 is already installed
$ brew install jasper
Warning: jasper 2.0.14 is already installed
However, NetCDF 4.6.0 doesn’t work for WRF. I don’t know the exact reason. One possible reason is After 4.2 (2012), the Fortran netCDF library has been split off into an independent source distribution, intended to be built as a separate library, after the C library is built and installed. This separation simplifies the building and use of the C and Fortran netCDF libraries and allows them to evolve independently.
However, even if you install netCDF-Fortran 4.4.4, WRF still doesn’t work. My workable solution is based on netCDF 4.1.3, downloaded from http://www2.mmm.ucar.edu/wrf/OnLineTutorial/compilation_tutorial.php#STEP2
The tricky part has something to do with the compatibility with Mac OS high Sierra. You have to add #include <config.h>to first line of load.c file.
mkdir Build_WRF
cd ~/Downloads/Build_WRF
mkdir LIBRARIES
export DIR=~/Downloads/Build_WRF/LIBRARIES
export CC=gcc
export CXX=g++
export FC=gfortran
export FCFLAGS=-m64
export F77=gfortran
export FFLAGS=-m64
# NetCDF
cd netcdf-4.1.3
subl ncgen3/load.c
# add "#include <config.h>" to fist line
./configure --prefix=$DIR/netcdf --disable-dap \
     --disable-netcdf-4 --disable-shared
make  
make install  
export PATH=$DIR/netcdf/bin:$PATH
export NETCDF=$DIR/netcdf

Library compatibility test

NETCDF=/usr/local/Cellar/netcdf/4.6.0
cd /Downloads/WRFV3/TESTS/Fortran_C_NETCDF_MPI_tests
cp ${NETCDF}/include/netcdf.inc .
gfortran -c 01_fortran+c+netcdf_f.f
gcc -c 01_fortran+c+netcdf_c.c
gfortran 01_fortran+c+netcdf_f.o 01_fortran+c+netcdf_c.o \
     -L${NETCDF}/lib -lnetcdff -lnetcdf
./a.out
# SUCCESS test 1 fortran + c + netcdf
mpif90 -c 02_fortran+c+netcdf+mpi_f.f
mpicc -c 02_fortran+c+netcdf+mpi_c.c
mpif90 02_fortran+c+netcdf+mpi_f.o \
02_fortran+c+netcdf+mpi_c.o \
     -L${NETCDF}/lib -lnetcdff -lnetcdf
mpirun ./a.out
# SUCCESS test 2 fortran + c + netcdf + mpi

compile WRF

cd ~/Downloads/WRFV3
 ./configure
checking for perl5... no
checking for perl... found /usr/bin/perl (perl)
** WARNING: No path to NETCDF and environment variable NETCDF not set.
** would you like me to try to fix? [y]
$ y
Enter full path to NetCDF include directory on your system
$ /Downloads/Build_WRF/LIBRARIES/netcdf/include
Please select from among the following Darwin ARCH options:
Enter selection [1-36] : 15
Compile for nesting? (0=no nesting, 1=basic, 2=preset moves, 3=vortex following) [default 0]: 1

$ ./compile em_b_wave >& compile.log
$ subl compile.log
# Executables successfully built

other things learned

default make install path is at /usr/local/bin
Brew install path is at /usr/local/Cellar/
Port install path is at /usr/opt/
see recent changes: ls -lrt
brew install on Centos: http://linuxbrew.sh/
install pip on Centos: sudo yum install -y python-pip
check yum-installed package location: rpm -ql package-name

make install latest netcdf-c

cd /Users/jiang/Downloads/netcdf-c-4.6.0
./configure --prefix=/usr/local
make check
make install

make install lastest netcdf-fortran

cd /Users/jiang/Downloads/netcdf-fortran-4.4.4
NCDIR=/usr/local/Cellar/netcdf/4.6.0
CC=/usr/local/cc
FC=/usr/local/fortran
export LD_LIBRARY_PATH=${NCDIR}/lib
NFDIR=/usr/local
CPPFLAGS=-I${NCDIR}/include LDFLAGS=-L${NCDIR}/lib \
./configure --prefix=${NFDIR}

Tricks for netcdf on high serria

Yuchao,
\1. I did see that it said there was a problem building the executables, but typically there are errors listed further up in the file to help us to determine why the executables weren’t built. We look for errors with a capital ‘E’ (i.e., Error). All others listed are not relevant. However, again, if you don’t have netcdf built correctly, WRF will not build either.
\2. Have you tried to use our compilation tutorial web page to get the libraries installed correctly on your system?
This helps you to verify that everything is okay in your environment, and then to get the libraries installed correctly, and then test that they are compatible with each other, and with your compilers. We recommend following it word-for-word. Even if you have already built some of the libraries, it is recommended to start fresh in a clean directory and rebuild them as indicated in the page, in order to ensure a greater chance of a successful WRF build. If you follow these directions and are unable to get netcdf installed, then you will need to talk to your systems administrator to see what the problem is. We are aware of a netcdf installation problem on Mac OS High Sierra (which is why I asked), if you are using the Clang version of gfortran. I will paste below what another user sent us to solve this problem. However, beyond this problem, it will have to be solved by your systems administrator at your institute. Here is the High Sierra solution. We have not actually tested this here because we aren’t able to repeat the problem on our systems, but in case it helps you…
Some cursory googling suggests that the issue is in MacOS’s default clang compiler: https://www.unidata.ucar.edu/software/netcdf/docs/known_problems.html#clang-ncgen3
**
I’ve found two ways to work around this, one is a band-aid and I think the other is an actual fix. The band-aid is the one suggested at that link: you add the line
#include <config.h>
above line 8 of ncgen3/load.c. This lets clang compile correctly. However, it leaves open the question of why the heck clang is being used at all, given that we told it to use gcc.
**
Well, it turns out that even if you’ve used homebrew to install the latest gcc, as I have, the installed gcc-7 and g++-7 don’t get automatically symlinked to gcc and g++ the way it automatically symlinks gfortran-7 to gfortran. Thus, if you set CC=gcc, the XCode-installed /usr/bin/gcc is called, and it apparently just points back to clang. I should have caught this by checking the versions of all the compilers, but I only did gfortran —version since that’s the compiler I’ve had the most configuration issues with, and I glossed over the possibility that XCode was still first in line for C and C++.
**
So, I’ve had success with two solutions/workarounds. If the user doesn’t want to override XCode for some reason, they can specify CC=gcc-7 and CXX=g++-7, and that allows netcdf to compile (I haven’t tested farther than that, because I prefer the second solution). If they want to use the newer GCC for everything, they can just go to /usr/local/bin and symlink gcc-7 to gcc and g++-7 to g++, and if their path is set up correctly these should then be first priority. I’m currently in the process of going through the full compilation process to see if any new issues crop up, but this should actually help with stability given that the first time through I was compiling with gfortran 7.3 but gcc 4.whatever_Apple_gives_me. Obviously this only applies to folks that use Homebrew but should be similar for macports folks.
~wrfhelp~

Monday, January 29, 2018

Install Centos and CRTM

Refresher (Last time I used Linux is 4 months ago and I almost forgot everything)
  • robo vm is 3 GB .vmdk file, password: robo-nd
  • open terminal: ctrl + option + t
main take-away:
  1. use stable version of VirtualBox(older version)
  2. use ready-to-use version of centos VM. So you have GUI and Firefox.

install on Mac OS, fail

cd config-setup
chmod u+x gfortran.setup  # change user access to write
./gfortran.setup   # source some shell code
source config-setup/gfortran-debug.setup
## following is error, setenv may be linux command
-bash: setenv: command not found
-bash: setenv: command not found
-bash: setenv: command not found
-bash: setenv: command not found
cp Makefile.in Makefile # **.in is for linux
make
./config.status --recheck
make: ./config.status: No such file or directory
make: *** [config.status] Error 1

install on Ubuntu, fail

cd config-setup
chmod u+x gfortran-debug.setup
cd ..
./configure --prefix=/bin/
# configure ...
make
# recipe for target "CRTM_CloudCover_Efine.o" failed
# recipe for target "all" failed
make check
# Error 2
gfortran --version
# GNU Fortran (Ubuntu 5.4.0)  5.4.0
gcc --version
# gcc (Ubuntu 5.4.0)  5.4.0

install on Centos 7 DVD ISO, but no GUI

install virtual box on macOS High Sierra

www.centos.org/download/
University of Oklahoma Open Source Software Mirror: http://mirror.oss.ou.edu/
Virtual Box 5.2.6 doesn’t work on macOS High Sierra. Use VirtualBox 5.2.4. https://www.virtualbox.org/wiki/Download_Old_Builds_5_2
Installation failure? System configuration -> Seurity & Privacy -> authroize changes ( the lock sign)

install centos on virtual box

Installing Centos 7 on VirtualBox: https://www.youtube.com/watch?v=O1nF7HQAJGw
Virtual box - new -> name: centos7 -> next -> 2048 MB->create a virtual hard disk now -> Hard disk file type: VDI -> dynamically allocated
Localhost login: root
Password: wnixxxxxxx

add user

adduser yuchao
passwd yuchao
usermod -aG wheel username
CentOS 使用 yum,Ubuntu 使用 apt-get,近些年又出现了一些与发行版无关的第三方包管理器,比如 Linuxbrew.
The problem is no GUI, so inconvenient to use. May try everything ISO next time.

install on Centos, ready-to-use VirtualBox by OSBoxes, success!

Centos 7-1708, password: osboxes.org
transfer file?
df -h
ping 10.205.224.189
scp jiang@10.205.224.180:./*.sh .
# connection refused
directly download from official website
sudo yum install gcc-gfortran
gfortran --version
# GNU Fortran (GCC) 4.8.5 (Red Hat 4.8.5-16)
gcc --version
# gcc (GCC) 4.8.5 (Red Hat 4.8.5-16)
cd config-setup
chmod u+x gfortran-debug.setup
cd ..
./configure --prefix=/bin/
make
make check
sudo make install
make clean

Wednesday, January 3, 2018

car hood won't open

understand the problem

Yesterday was extremely cold, all day below -5 C. Then my car battery failed. When I tried to open the hood to give it a jump, the hood latch release failed.
This seems a simple mechanical problem. The latch was cleverly designed. After one evening’s study, I had a clear understanding of the problem. And it is very interesting and deserves a blog post.
There are actually 2 latches:
Each latch has its own spring. You can see how the Primary latch works by the following 2 short videos:
What to do if you have a Stuck Hood Latch or hood won’t open: https://youtu.be/mCQCatoG9Rc?t=30s
How to fix a hood latch that wont close || 1997 Chevy Lumina:https://youtu.be/z6CSHeuH-oE?t=8m7s
Or
All you need is a long stick to push the metal along the direction of latch release cable:
Hooray!

Everything is well explained in this awesome video

What if your car’s Hood won’t Open?: https://www.youtube.com/watch?v=xMrjwUfdiiE

Buy parts

back up plan, AAA

$52 per year, 5 miles of towing: https://ok.aaa.com/non-member
promo code: 52WBOS
aaa.com/stop

gcc and make

refresh C grammar

check C compiler:
gcc -v
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 9.0.0 (clang-900.0.39.2)
Target: x86_64-apple-darwin17.3.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin
If you use Mac OS X, the easiest way to obtain GCC is to download the Xcode development environment from Apple’s website and follow the simple installation instructions. Once you have Xcode setup, you will be able to use GNU compiler for C/C++.
#include <stdio.h>
int main(int argc, char** argv) {
    printf("argument count: %d \n",argc);
    for (int i= 0; i< argc; i++){
        printf("argument vector %d is: %s.\n ", i,argv[i]);
    }
    printf("Hello, World!\n");
    return 0;
}
argv and argc are how command line arguments are passed to main() in C and C++.
argc will be the number of strings pointed to by argv. This will (in practice) be 1 plus the number of arguments, as virtually all implementations will prepend the name of the program to the array.
The variables are named argc (argument count) and argv (argument vector) by convention, but they can be given any valid identifier: int main(int num_args, char** arg_strings) is equally valid.
When you output, always define the format you want.
Compile:
$ gcc hello.c  # default output file is a.out
$ gcc -o hello hello.c # define output file as hello
$ ./a.out
Hello, World!
Codes:
#include <stdio.h>
int main() {
   printf("Storage size for int : %d \n", sizeof(int)); //4, the exact size of a type/variable depends on a specific platform
   return 0;
}

Make grammar

why makefile?
gcc main.cpp hello.cpp factorial.cpp -o hello
This command generates hello binary. In this example, we have only four files and we know the sequence of the function calls. Hence, it is feasible to type the above command and prepare a final binary. However, for a large project where we have thousands of source code files, it becomes difficult to maintain the binary builds.
To prepare to use make, you must write a file called the makefile that describes the relationships among files in your program and provides commands for updating each file.
It would be tricky to write a grammar for make, since the grammar is extremely context-dependent.
Names of these variables originate from names of the corresponding tools. Usually, the meaning of these abbreviations is the following:
  • CC stands for “C compiler” (in GCC abbreviation it is also treated as “compiler collection”).
  • LD is a linker (comes from “link editor” or from “loader”).
These are also commonly used in makefiles (see Implicit variables chapter of GNU Make manual):
  • CPP stands for “C preprocessor”
  • CXX is a C++ compiler
  • AS is an assembly language compiler
  • AR is an archive-maintaining program

difference between g++ and gcc

GCC: GNU Compiler Collection, Referrers to all the different languages that are supported by the GNU compiler.
gcc: GNU C Compiler
g++: GNU C++ Compiler
The main differences:
gcc will compile: .c/.cpp files as C and C++ respectively.

g++ will compile: .c/.cpp files but they will all be treated as C++ files.

Also if you use g++ to link the object files it automatically links in the std C++ libraries (gcc does not do this).

gcc compiling C files has less predefined macros.

gcc compiling .cpp and g++ compiling *.c/.cpp files has a few extra macros.

Sunday, December 24, 2017

Read grb2 file in Shell, Python,R and Matlab

GRIB

GRIB, or GRIdded Binary, is a concise data format by the World Meteorological Organization to store weather data.
GRIB files are a collection of self-contained records of 2D data, and the individual records stand alone as meaningful data, with no references to other records or to an overall schema. So collections of GRIB records can be appended to each other or the records separated.
The latest standard is GRIB2.
Why GRIB2?
  • compression rate. File size is 2 or 3 times smaller than NetCDF file.
  • Models are getting bigger faster than the bandwidth increases.
  • Packing is transparent to the user: self-explanatory.
Problems with GRIB: No way in GRIB to describe a collection of GRIB records.
  • Each record is independent, with no way to reference the GRIB writer’s intended schema
  • No foolproof way to combine records into the multidimensional arrays from which they were derived.
Because of such problems, any Python/R/Matlab API or Panoply reading grb2 file will lose the metadata of the records and each record is comined in a random way. Consequently, multiple records are blended with each other and become useless. So the most reliable way is still use wgrb2 in shell or embedbed shell command in C++ to extract the exact records you need in your intended order.

shell: wgrib2 C library

port install will automatically install 24 dependecies including glib2,hdf5, jasper, jpeg, netcdf, openssl, proj, zlib.
Download macport from https://www.macports.org/ and install.
sudo port install wgrib2
wgrib2 gfs_4_20170830_1800_024.grb2  # print out all inv rows
wgrib2 gfs_4_20170830_1800_024.grb2 -s | grep ":PRES:surface" | wgrib2 -i gfs_4_20170830_1800_024.grb2 -text data1.txt  # output surface pressure
wgrib2 gfs_4_20170830_1800_024.grb2 -s | grep ":PRES:surface" | wgrib2 -i gfs_4_20170830_1800_024.grb2 -netcdf pres1.nc  #
wgrib trick: www.ftp.cpc.ncep.noaa.gov/wd51we/wgrib/tricks.wgrib

pygrib

This module is a python interface to the GRIB API C library from the European Centre for Medium-Range Weather Forecasts (ECMWF)
conda install -c conda-forge pygrib
import pygrib
# Library not loaded: @rpath/libpng16.16.dylib
# Reason: Incompatible library version: pygrib.cpython-35m-darwin.so requires version 51.0.0 or later, but libpng16.16.dylib provides version 49.0.0

pynio

python 3 is not compatible with pynio.
conda create --name=py2 python=2.7 anaconda
conda install -c conda-forge pynio
still incompatible
import Nio
# dlopen(/Users/jiang/anaconda3/envs/py2/lib/python2.7/site-packages/PyNIO/nio.so, 2): Library not loaded: @rpath/libgssapi_krb5.2.2.dylib
# Reason: image not found

GRIB-API

The ECMWF GRIB-API is an application program interface accessible from C, FORTRAN and Python programs developed for encoding
Please note that GRIB-API support is being discontinued at the end of 2018.

ecCodes

I couldn’t find a python library to call it.

R: gdal

The Geospatial Data Abstraction Library (GDAL) is a computer software library) for reading and writing raster and vector geospatial data formats,
install.packages("sp")
install.packages("rgdal")
folder = "/Users/jiang/data/goesr/20170831CONUS"
file = "gfs_4_20170830_1800_024.grb2"
setwd(folder)
grib <- readGDAL(file)  # Large SaptialGridDataFrame, 827 Mb
# however, this is not useful because the extracted records are not in the same order with the .inv file
extract grb2 file to nc file, unpack and plot
folder = "/Users/jiang/data/gfs"
file = "gfs_4_20170830_1800_024.grb2"
setwd(folder)
# system("wgrib2 -s gfs_4_20170830_1800_024.grb2 | grep :TMP:  | wgrib2 -i gfs_4_20170830_1800_024.grb2 -netcdf TMP.nc",intern=T) # run once or run in shell
library(ncdf4)
nc_version()  # "ncdf4_1.16_20170401"
temp <- nc_open("TMP.nc")
t2m.mean <- ncvar_get(temp,"TMP_2maboveground")
x <- ncvar_get(temp,"longitude")  #720
y <- ncvar_get(temp,"latitude")   #361
# dim(temp)  # 720,361
# c(nrow(temp),ncol(temp)) # 720, 361
library(maps)
library(fields)
day="DIA"
#png(filename="gfs_t2m.png",width=1215,height=607,bg="white")
rgb.palette <- colorRampPalette(c("snow1","snow2","snow3","seagreen","orange","firebrick"), space = "rgb")#colors
image.plot(x,y,t2m.mean,col=rgb.palette(200),main=as.expression(paste("GFS 24hr Average 2M Temperature",day,"00 UTC",sep="")),axes=T,legend.lab="o C")
require(maptools)
data(wrld_simpl)
plot(wrld_simpl, add = TRUE)

Matlab

It actually uses 2 free toolboxes:
  1. NCtoolbox
  2. M_MAP
clc
clear
mydate='20171224';
url = ['http://nomads.ncep.noaa.gov:9090/dods/wave/nww3/nww3', mydate,'/nww3',mydate,'_00z'];
nco=ncgeodataset(url);  % Instantiate the data set
sort(nco.variables);  % list variables
lon=nco{'lon'}(:);
lat=nco{'lat'}(:);
waveheight=nco{'htsgwsfc'}(1,:,:);% surface 
% significant height of wind waves
waveheight = squeeze(waveheight); % squeeze shape

% Plot the field using M_MAP.  Start with setting the map
% projection using the limits of the lat/lon data itself:
m_proj('miller','lat',[min(lat(:)) max(lat(:))],'lon',[min(lon(:)) max(lon(:))]);
% Next, plot the field using the M_MAP version of pcolor.
m_pcolor(lon,lat,waveheight);
shading flat;
% Add a coastline and axis values.
m_coast('patch',[.7 .7 .7]);
m_grid('box','fancy');
% Add a colorbar and title.
colorbar;
title('Example 1: WAVEWATCH III Significant Wave Height from NOMADS');

Others:

Monday, December 4, 2017

DVD, Core Meteorology

A series of DVD, director: Ron Meyer. Runtime: 30 minutes.
For grade 7 - College

Atmosphere

It is the atmosphere that creates relatively constant daytime versus nighttime temperatures, like the temperatures found on the earth. In our solar system, no atmosphere means no possibility of life.
We live on the earth. More precisely, we live in the troposphere of the earth, like fish in the water.
The stratosphere is home to the jet stream. Also home to Ozone.
The atmosphere is a delicate dynamic balance. A balance absolutely critical for all life on the planet.
heat conduction: the pan doesn’t move.
Remarkably, the most important part of this story for us is that the troposphere is heated from the bottom to the top. In spite of being the furthest from the sun, the air closest to the ground is warmer than the air 5 miles closer to the sun.
The heat move around the atmosphere is called weather. All weather happens in the lower atmosphere.

weather

3 factor affect weather prediction:
  1. initial state has some gap information
  2. model is not perfect
  3. the principle of time and size

climate

Climate is a pattern over time.
Human is adapted to these stable climate patterns as well.

NG- Natural disasters

1993 Storm of the Century

3 different weather models predict different storm track.
No matter how well we can predict this, we can’t stop it from occurring. If we had control of it, the meteorologist would make more money than professional sports players make.
I may lose a few toes, not a big deal… you are in good spirit.

2004 Indian Ocean earthquake and tsunami

Earthquake is actually good to release earth’s internal energy. A solution is to create a small man-made earthquake to defuse a big one.
Experimenting with mother nature is a very difficult thing. We may trigger a bigger earthquake than the one we were trying to prevent.
Inducing quakes is an inexact science.
A 30 billion question: how often does a city like New Orleans get hit by a Category Five hurricane? Experiments show New Orleans is living in its borrowing time. A major hurricane is overdue.
Project Stormfury is trying to intervene the hurricane, but failed.
The benefit of a hurricane is redistributing heat in the atmosphere.

Tornado intercept

Tim Samra died in 2013 in Oklahoma, the first known death as a storm chaser.
Tim Samaras did not seem startled by the question from his love child, Matt Winter. “Matt,” he replied, “Kathy’s a strong woman. She understands this is my passion. And if something happened to me, she’d move on.”