6.9.3. Compiling Sandia OpenSHMEM for CN5000
SOS provides the low-level OpenSHMEM library plus the shmem_*_perf micro-benchmark suite. Build it against libfabric (OFI) for CN5000:
git clone https://github.com/Sandia-OpenSHMEM/SOS.git # 4aa91d7f is the latest known to work commit # sos-tests are downloaded separately and need to be integrated
wget https://github.com/openshmem-org/tests-sos/archive/refs/tags/v1.5.3.tar.gz -O tests-sos-1.5.3.tar.gz
cd SOS
git checkout 4aa91d7f
cd modules/
tar xzf ../../tests-sos-1.5.3.tar.gz
rmdir tests-sos
ln -s tests-sos-1.5.3 tests-sos
cd ..
./autogen.sh
./configure --prefix=<sosinstdir> \
--with-ofi=/usr \
--enable-pmi-simple \
--enable-completion-polling \
--enable-thread-completion \
--enable-error-checking \
--enable-hard-polling \
--enable-manual-progress \
--enable-ofi-manual-progress
make -j install
Important
SOS must use PMI-simple (provided by Hydra’s mpiexec.hydra), not PMIx. If OpenMPI is present on $PATH/$LD_LIBRARY_PATH, remove it before configuring so PMIx is not auto-detected.
6.9.3.1. Building the SOS Performance Benchmark Suite
The shmem_*_perf benchmarks (bandwidth, bidirectional bandwidth, latency, atomics) live in the modules/tests-sos submodule:
export PATH=<sosinstdir>/bin:$PATH
export LD_LIBRARY_PATH=<sosinstdir>/lib:$LD_LIBRARY_PATH
cd modules/tests-sos
./autogen.sh
./configure --prefix=<sosinstdir> \
--enable-lengthy-tests
cd test/performance/shmem_perf_suite
make -j \
shmem_bibw_atomics_perf shmem_bibw_get_perf shmem_bibw_get_perf_nb \
shmem_bibw_nb_atomics_perf shmem_bibw_put_perf shmem_bibw_put_perf_nb \
shmem_bw_atomics_perf shmem_bw_get_perf shmem_bw_get_perf_nb \
shmem_bw_nb_atomics_perf shmem_bw_put_perf shmem_bw_put_perf_nb \
shmem_latency_get_perf shmem_latency_get_perf_nb \
shmem_latency_put_perf shmem_latency_put_perf_nb
cp shmem_*perf* <sosinstdir>/bin/