Assignment: Parallel viewshed with OpenMP

Use OpenMP to parallelize your code for computing viewsheds developed for the previous assignment. Run a succint experimental analysis of your code using Bowdoin's computing grid and report the speedup as the number of threads/cores increases. Comment on your findings.

Outline

The interface: can be the same as before, but allow the user to specify the number of threads on the command line:
```
./parviewshed tester.asc testervis.asc 5 5 7
```
This computes the viewshed of point (5,5) on elevation grid tester.asc and writes the output viewhed in the grid testervis.asc. The program will request 7 threads.
Outline: Three are three high-level components:
- read the elevation grid in memory
- call the function that computes the viewshed
- write the viewshed grid to a file
Your code may be organized a little differently, but these three parts sholud be there.
To start, add support to time each component separately, and print out the times. When running your code you should see something like this:
```
./parviewshed usadem2.asc usadem2vis.asc 2000 2000 10 
reading usadem2.asc:           xxxx seconds 
computing viewshed(2000,2000): xxxx seconds
writing to grid usadem2vis.asc:xxxx seconds
total: xxxx seconds 
```
The goal of this assigment is to parallelize the function that computes the viewshed. Note that the function is a loop so in principle OpenMP can handle it---you will need to decide what variables are private and shared. You will also need to rewrite your viewshed function so that there are no ways to break out of the function early (any possible execution of the loop should reach point A).
```
void compute_viewshed (Grid eg, Grid vg, int vprow, int vpcol) {
     ...
     for (i=0; i< eg.nrows; i++)  {
         for (j=0; j< eg.ncols; j++) {

	 set (vg, i, j) = is_visible(eg, vprow, vpcol, i, j); 
 
         }//for j  
     }//for i
    //point A
}
```
Feel free to use this opportunity to clean up and simplify your code! For example, I would rewrite the is_visible() function so that it handles everything, and so that the compute_viewshed() function looks pretty much as above.
Testing correctness: Make sure that your code runs correctly (there are no race conditions) by running on several datasets, with several viewpoints, several times, and with different number of threads. Visualize the output each time.

Assessing performance on small datasets: Pick a dataset and a viewpoint on which the viewshed computation takes a few seconds. Keep these fixed. Use your laptop, a desktop in the lab or one of the linux public machines (dover, foxcroft). Run your code and write down the timing of the viewshed function (which you timed separately) as the number of threads goes from 1 to N, where N depends on your platform. N should be about how many cores you have on the machine. You could also let N go beyond the number of cores and see how that impacts the running time.
If you run on dover, the number of cores seems to be 24 (note: check with DJ).
At the end you should have a table that lists, for a given dataset and vp, the timings with various N.
Use the insight to optimize your code. For example, if you don't notice a decent speedup when N increases, you should probably go back and think.
Note: The timings on a desktop or on a public machine (dover) are not accurate because other processes may interfere with yours. If you run on dover/foxcroft, keep your experiments small, not more than a few seconds ---- public machines are not meant to be used by compute-intensive processes.

Once your viewshed is computed in parallel and you have some preliminary numbers on performance, look into parallelizing the other major sections of the code (reading the input grid and writing the output grid).

Now that everything is under control and there are no bugs and no race conditions, you are ready to run experiments on the grid: To assess performance more accurately and on the larger datasets, you will run the same sort of experiments as above but, instead of using your laptop or dover/foxcroft, you will use Bowdoin's computing grid (see below). Pick at least two datasets: a small one (e.g. kaweah), and a large one (e.g. usadem2), pick a viewpoint approximately in the middle, and record the timings as N increases from 1.

Summarize your tables and findings in a brief report, formatted in the same style as the previous ones.

Bowdoin HPC grid

The Bowdoin Computing Grid (also known as "The Grid") is a group of Linux servers which appears as one big, multiprocessor server that can run parallel jobs concurrently. As of Fall 2016, the Grid has 796 CPU cores. Check out Bowdoin's HPC website.

The servers in the grid are not interactive machines --- you cannot interact with them the same as you interact with dover and foxcroft, or with your laptop. The Grid is setup to run batch jobs only (not interactive and/or GUI applications).

The servers on the Grid run the Sun Grid Engine (SGE), which is a software environment that coordinates the resources on the grid. The grid has a headnode which accepts jobs, puts them in a waiting queue until they can be run, sends them to the computational node(s) on the grid to run them, manages them while they run, and notifies the owner when the job is finished. This headnode is a machine called moosehead. To interact with The Grid you need to login to the Grid headnode "moosehead.bowdoin.edu" via an SSH client program.

ssh moosehead.bowdoin.edu

Moosehead is an old server which was configured to run the Sun Grid Engine and do whatever a headnode is supposed to do: moosehead accepts jobs, puts them in a queue until they can be executed, sends them to an execution machine, manages them during execution, and logs the record of their execution when they are finished.

Moosehead runs linux so in principle you can run on it anything that you could run on dover. However DJ (the sysadmin, and Director of Bowdoin's HPC Grid) asks that you don't. Moosehead is an old machine. Use it only to submit jobs to the grid and to interact with the grid. Do the compiling, developing and testing somewhere else (e.g. on dover).

The Grid uses the same shared filespace as all of the Bowdoin Linux machines, so you can access the same home directory and data space as with dover or foxcroft (if you need to transfer files from a machine that is not a part of the Bowdoin network, use scp from your machine to dover or foxcroft first).

Running jobs on the grid

Below I am including a summary of the commands we'll use to start. For more detailed information on how to interact with the grid check out the website maintained by DJ, which should be be your go-to page.

To submit to the grid you have two options:

Use hpcsub.
Create a script and use qsub.

Submit using hpcsub.

The command hpcsub will allow you to submit any single commands to the grid. For example:

ssh moosehead
cd [directory-where-your-code-is-compiled]
hpcsub -pe smp 8 -cmd [your-code] [arguments to pass to the program]

The arguments -pe smp 8 are optional (but, if you are running OpenMP code, you should use them). They specify that your code is to be run in the SMP environment, with 8 cores (here 8 is only an example, it can be any number you want).

For example, if I want to run hellosmp that we talked about in class (which you can find here) using 8 CPU cores in the SMP environment, I would do:

ssh moosehead
[ltoma@moosehead:~]$ pwd
/home/ltoma
[ltoma@moosehead:~]$ cd public_html/teaching/cs3225-GIS/fall16/Code/OpenMP/
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ ls
example1.c  example2.cpp  example3.c  example4.c   hellosmp  hellosmp.c  hellosmp.h  hellosmp.o Makefile
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ hpcsub -pe smp 8 -cmd hellosmp
Submitting job using:
qsub -pe smp 8 hpc.10866
Your job 236150 ("hpc.10866") has been submitted

The headnode puts this job in the queue and starts looking for 8 cores that are free. When 8 cores become available, it assigns these 8 cores to your job. While your job is running no other job can use the 8 cores that it got assigned---- they are exclusively yours while your job runs. To check the jobs currently in the queue, do:

qstat

To check on all jobs running on the cluster, type

qstat -u "*"

For a full listing of all jobs on the cluster, type

qstat -f -u "*"

To display list of all jobs belonging to user foo, type

qstat -u foo

After I submit a job I usually check the queue:

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236150 0.00000 hpc.10866  ltoma        qw    10/12/2016 15:53:20                                    8        
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236150 0.58278 hpc.10866  ltoma        r     10/12/2016 15:53:27 all.q@moose15                      8

Note how the job initially shows as "qw" (queued and waiting) and then changes to "r" (running).

When the job is done you will get an email. If you list the files, you will notice a new file called "hpc.[job-number].xxx". This file represents the standard output for your job ---- all the print commands are redirected to this file.

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ 
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ ls
example1.c  example2.cpp  example3.c  example4.c   hellosmp  hellosmp.c  hellosmp.h  hellosmp.o   hpc.10866.o236150  Makefile

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ cat hpc.10866.o236150
I am thread 1. Hello world!
I am thread 2. Hello world!
I am thread 7. Hello world!
I am thread 0. Hello world!
I am thread 5. Hello world!
I am thread 6. Hello world!
I am thread 4. Hello world!
I am thread 3. Hello world!

Submit using `qsub`.

A more general way to submit jobs is via a script. You will need to create a script to run your programs on the grid. A sample script myscript.sh might look like this:

#!/bin/bash
#$ -cwd
#$ -j y
#$ -S /bin/bash
#$ -M (my_login_name)@bowdoin.edu -m b -m e

./hellosmp

To submit your job to the grid you will do:

ssh moosehead 
cd [folder-containing-myscript.sh]
qsub myscript.sh

Example:

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$  cat myscript.sh 
#!/bin/bash
#$ -cwd
#$ -j y
#$ -S /bin/bash
#$ -M ltoma@bowdoin.edu -m b -m e

#./hellosmp 
./example1
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qsub myscript.sh
Your job 236154 ("myscript.sh") has been submitted
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236154 0.00000 myscript.s ltoma        qw    10/12/2016 16:00:17                                    1        
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236154 0.50500 myscript.s ltoma        r     10/12/2016 16:00:27 all.q@moose22                      1        
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$

Note how your job went from "qw" to not in the queue (basically it ran and finished so fast that we could not see it).

Each job creates a file by appending the job number to the script. In our case this is a file called "myscript.sh.o[job-number]". These .o* file will be the equivalent to what you would see on the console if running the program interactively.

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ ls 
example1             example3.c           hellosmp             hellosmp.o             myscript.sh          
example1.c           example4.c           hellosmp.c            example2.cpp         hello 
hellosmp.h           hpc.10866.o236150    Makefile   myscript.sh.o236154  
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ cat myscript.sh.o236154 
Hello World from thread 0
There are 1 threads


There are 1 threads

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$

Looking at the output we see that example1 was run with just one thread. That's because when we submitted we did not specify that we wanted SMP and how many threads we wanted, so we got whatever the default is (which is no threads). When running OpenMP code you need to submit using arguments -pe smp [numberthreads]. For example:

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qsub -pe smp 8 myscript.sh
Your job 236155 ("myscript.sh") has been submitted
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat 
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ cat myscript.sh.o23615
Hello World from thread 5
Hello World from thread 1
Hello World from thread 0
There are 8 threads
Hello World from thread 6
Hello World from thread 7
Hello World from thread 2
Hello World from thread 4
Hello World from thread 3


There are 1 threads

Ah, that's better.

Using a machine exclusively

If you run a job in the SMP environment requesting, say, 8 cores, the headnode will look for one machine that has 8 core available. The other cores on the same machine may be already in use, or if not, may be given to other jobs in the future. Since all cores on a machine share the memory and also some caches, there will be some competition among the threads. The timing of your job will be impacted by what else is running on the same machine.

If you are running an experimental analysi sand youc are about the timings, you want to request that the whole machine is yours, even if your job is only going to use x processors. You can do that by including flag excl=true:

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qsub -l excl=true -pe smp 8 myscript.sh
Your job 236157 ("myscript.sh") has been submitted

[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236157 0.00000 myscript.s ltoma        qw    10/12/2016 16:05:15                                    8        
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
 236157 0.60500 myscript.s ltoma        r     10/12/2016 16:05:27 all.q@moose22                      8        
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$ qstat
[ltoma@moosehead:~/public_html/teaching/cs3225-GIS/fall16/Code/OpenMP]$

Submitting your work

Make a folder called parviewshed in your svn folder on microwave, and update it with your work.

Have fun!

Use piazza for questions and comments and any communication regarding this assignment.