Difference between revisions of "HowTo:petsc"
(→Running ADF from a command line) |
(→Submitting (parallel) ADF jobs) |
||
Line 105: | Line 105: | ||
The "First Block" creates a parallel vector x and defines which parts of the vector reside on which process. The PETSc specific communicator PETSC_COMM_WORLD is used for this. The "Second Code Block" then assigns specific values to the elements. This is done on the '''global''' version of the vector, and PETSc takes care of the details, for instance which process handles which element. After that, the vector is "assembled", i.e. distributed among the processes. This involves two function calls (Begin and End) to make it possible to "do other things" in the meantime. Just make sure it does not involve the vector. Finally (in the "Third Block"), we print out the vector and get rid of it. After that, we finalize PETSc usage and return. | The "First Block" creates a parallel vector x and defines which parts of the vector reside on which process. The PETSc specific communicator PETSC_COMM_WORLD is used for this. The "Second Code Block" then assigns specific values to the elements. This is done on the '''global''' version of the vector, and PETSc takes care of the details, for instance which process handles which element. After that, the vector is "assembled", i.e. distributed among the processes. This involves two function calls (Begin and End) to make it possible to "do other things" in the meantime. Just make sure it does not involve the vector. Finally (in the "Third Block"), we print out the vector and get rid of it. After that, we finalize PETSc usage and return. | ||
− | == | + | ==How do I compile a PETSc program?== |
+ | Due to the large number of PETSc specific variables and options, it is best to do compilation using a makefile. This allows to apply all these setting through an "include" statement. Here is a bare-bones makefile without any additional optimization or special options that will compile the above sample program: | ||
− | + | <pre> | |
+ | CFLAGS = | ||
− | + | include ${PETSC_DIR}/conf/variables | |
+ | include ${PETSC_DIR}/conf/rules | ||
− | + | sample : sample.o chkopts | |
− | + | -${CLINKER} -o sample sample.o ${PETSC_VEC_LIB} | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
</pre> | </pre> | ||
− | + | The CFLAGS variable allows to pass additional compiler options, but we don't do that in our case. The include statements pull in the necessary options and variables for the compiler and "teaches" the make facility of how to get from PETSc-based C code to properly compiled object files. When executed through the make command, the code is compiled and then linked with the proper libraries: | |
− | + | <pre> | |
+ | hasch@swlogin1$ make sample | ||
− | + | /opt/openmpi-1.8/bin/mpicc -o sample.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 \ | |
+ | -I/opt/petsc/3.5.2/include -I/opt/petsc/3.5.2/include -I/opt/openmpi-1.8/include `pwd`/sample.c | ||
− | + | /opt/openmpi-1.8/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o sample sample.o \ | |
− | + | -Wl,-rpath,/opt/petsc/3.5.2/lib -L/opt/petsc/3.5.2/lib -lpetsc -Wl,-rpath,/usr/lib64/atlas -L/usr/lib64/atlas -llapack -lf77blas \ | |
− | + | -latlas -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/opt/openmpi-1.8/lib -L/opt/openmpi-1.8/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 \ | |
− | + | -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/opt/openmpi-1.8/lib \ | |
− | + | -L/opt/openmpi-1.8/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/openmpi-1.8/lib \ | |
− | + | -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_cxx -lstdc++ -Wl,-rpath,/opt/openmpi-1.8/lib -L/opt/openmpi-1.8/lib \ | |
− | + | -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 \ | |
− | + | -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -ldl -Wl,-rpath,/opt/openmpi-1.8/lib -lmpi -lgcc_s -lpthread -ldl | |
− | + | </pre> | |
− | + | An impressive list of options is revealed which you really don't want to type in manually. This produces an executable "sample" and an object file "sample.o". | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | |||
− | |||
== Licensing == | == Licensing == | ||
Revision as of 16:30, 30 May 2016
Contents
PETSc
This is a short help file on using the scalable suite for the solution of scientific problems based on partial differential equations. The use and application of PETSc requires careful study of the manual. This is meant as a basic introduction to the local usage of PETSc on servers and clusters at the Centre for Advanced Computing.
FeaturesFrom the PETSc web page: "PETSc, pronounced PET-see (the S is silent), is a suite of data structures and routines for the scalable (parallel) solution of scientific applications modeled by partial differential equations. It supports MPI, shared memory pthreads, and GPUs through CUDA or OpenCL, as well as hybrid MPI-shared memory pthreads or MPI-GPU parallelism." (Remark: Since HPCVL does presently not operate any GPUs the corresponding features are not implemented in our version). "PETSc is intended for use in large-scale application projects, and is easy to use for beginners. Its careful design allows advanced users to have detailed control over the solution process. PETSc includes a large suite of parallel linear and non-linear equation solvers and ODE integrators that are easily used in application codes written in C, C++, and Fortran. PETSc provides many of the mechanisms needed within parallel application codes, such as simple parallel matrix and vector assembly routines that allow the overlap of communication and computation. In addition, PETSc includes support for parallel distributed arrays useful for finite difference methods." Here is a list of features, also from the webpage:
Location of the program and setupThe present version of PETSc is 3.5.2 (gcc compiler suite, OpenMPI 1.8). The libraries and executables in the PETSc package are located in the directory /opt/petsc. Note that PETSc was compiled with the OpenMPI 1.8 implementation of MPI and should be used together with that implementation. The compilation used the gcc (Gnu C-compiler) version 4.4.7 which is the system compiler. PETSc is likely to work fine with newer version of that compiler but should not be combined with other compilers or MPI implementations on our systems. It is ** not ** required to sign a statement if you want to use PETSc, as it is distributed under the Gnu Public license. However, this license does not entitle you to use PETSc for commercial purposes. The setup for PETSc is done, just like with most other software on our systems, through "usepackage". Just issue the command use petsc which will set the PETSC_DIR environment variable and add the binary directory to the path. You can do this also manually (for bash): export PETSC_DIR="/opt/petsc/3.5.2" export PETSC_ARCH = "" export PATH="/opt/petsc/3.5.2/bin:"$PATH Scratch filesOne of the settings is the environment variable SCM_TMPDIR which is required to redirect the temporary files that ADF uses to the proper scratch space, presently /scratch/hpcXXXX where hpcXXXX stands for your username. If for some reason ADF does not terminate normally (e.g. a job gets cancelled), it leaves behind large scratch files which you may have to delete manually. To check if such files exist, type ls -lt /scratch/hpcXXXX Usually the scratch files are in sub-directories that start with kid_. Once you have determined that the scratch files are no longer needed (because the program that used them is not running any more), you can delete them by typing rm -r /scratch/hpcXXXX/kid_* Cleaning up the scratch space is the user's responsibility. If it is not done regularly, it can cause jobs to terminate, and much work to be lost. |
Usage in programmingPETSc provides a programming framework that lets you solve rather complex scientific problems with a minimum of coding. It is essential to study the User's Manual thoroughly to learn how to write programs that use PETSc. #include <petscvec.h> #undef __FUNCT__ #define __FUNCT__ "main" int main(int argc,char **argv) { PetscErrorCode ierr; PetscMPIInt rank; PetscInt i,N; PetscScalar one = 1.0; Vec x; PetscInitialize(&argc,&argv,(char*)0,help); ierr = MPI_Comm_rank(PETSC_COMM_WORLD,&rank);CHKERRQ(ierr); /* First block */ ierr = VecCreate(PETSC_COMM_WORLD,&x);CHKERRQ(ierr); ierr = VecSetSizes(x,rank+1,PETSC_DECIDE);CHKERRQ(ierr); ierr = VecSetFromOptions(x);CHKERRQ(ierr); ierr = VecGetSize(x,&N);CHKERRQ(ierr); ierr = VecSet(x,one);CHKERRQ(ierr); /* Second Block */ for (i=0; i<N-rank; i++) { ierr = VecSetValues(x,1,&i,&one,ADD_VALUES);CHKERRQ(ierr); } ierr = VecAssemblyBegin(x);CHKERRQ(ierr); ierr = VecAssemblyEnd(x);CHKERRQ(ierr); /* Third Block */ ierr = VecView(x,PETSC_VIEWER_STDOUT_WORLD);CHKERRQ(ierr); ierr = VecDestroy(&x);CHKERRQ(ierr); ierr = PetscFinalize(); return 0; } The include statement at the top pulls in the definitions for parallel vector operations. It also implicitly adds basic headers necessary for using PETSc. Many of the variable used (ierr, rank, ...) are declared as PETSc-specific datatypes. These are opaque, i.e. they are used only as arguments in PETSc function calls. Typically, the return value of all functions is an error variable, so that calls have the form
First we need to initialize PETSc and determine the "rank", i.e. the number of processors involved. The "First Block" creates a parallel vector x and defines which parts of the vector reside on which process. The PETSc specific communicator PETSC_COMM_WORLD is used for this. The "Second Code Block" then assigns specific values to the elements. This is done on the global version of the vector, and PETSc takes care of the details, for instance which process handles which element. After that, the vector is "assembled", i.e. distributed among the processes. This involves two function calls (Begin and End) to make it possible to "do other things" in the meantime. Just make sure it does not involve the vector. Finally (in the "Third Block"), we print out the vector and get rid of it. After that, we finalize PETSc usage and return. How do I compile a PETSc program?Due to the large number of PETSc specific variables and options, it is best to do compilation using a makefile. This allows to apply all these setting through an "include" statement. Here is a bare-bones makefile without any additional optimization or special options that will compile the above sample program: CFLAGS = include ${PETSC_DIR}/conf/variables include ${PETSC_DIR}/conf/rules sample : sample.o chkopts -${CLINKER} -o sample sample.o ${PETSC_VEC_LIB} The CFLAGS variable allows to pass additional compiler options, but we don't do that in our case. The include statements pull in the necessary options and variables for the compiler and "teaches" the make facility of how to get from PETSc-based C code to properly compiled object files. When executed through the make command, the code is compiled and then linked with the proper libraries: hasch@swlogin1$ make sample /opt/openmpi-1.8/bin/mpicc -o sample.o -c -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 \ -I/opt/petsc/3.5.2/include -I/opt/petsc/3.5.2/include -I/opt/openmpi-1.8/include `pwd`/sample.c /opt/openmpi-1.8/bin/mpicc -fPIC -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g3 -O0 -o sample sample.o \ -Wl,-rpath,/opt/petsc/3.5.2/lib -L/opt/petsc/3.5.2/lib -lpetsc -Wl,-rpath,/usr/lib64/atlas -L/usr/lib64/atlas -llapack -lf77blas \ -latlas -lX11 -lpthread -lssl -lcrypto -Wl,-rpath,/opt/openmpi-1.8/lib -L/opt/openmpi-1.8/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 \ -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_usempi -lmpi_mpifh -lgfortran -lm -lmpi_cxx -lstdc++ -Wl,-rpath,/opt/openmpi-1.8/lib \ -L/opt/openmpi-1.8/lib -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/opt/openmpi-1.8/lib \ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -lmpi_cxx -lstdc++ -Wl,-rpath,/opt/openmpi-1.8/lib -L/opt/openmpi-1.8/lib \ -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.4.7 \ -L/usr/lib/gcc/x86_64-redhat-linux/4.4.7 -ldl -Wl,-rpath,/opt/openmpi-1.8/lib -lmpi -lgcc_s -lpthread -ldl An impressive list of options is revealed which you really don't want to type in manually. This produces an executable "sample" and an object file "sample.o". LicensingADF is a licensed program. The license held by the Centre for Advanced Computing is limited to our computers at our main site. That means that any of our users can use the program on our machines (but nowhere else), whether they are located at Queen's or not. We require users of ADF to sign a statement in which they state that they are informed about the terms of the license to be included in the Gaussian user group named "adf". Please fax the completed statement to (613) 533-2015 or scan/email to cac.admin@queensu.ca. Help
|