JIVE Wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== CASA VLBI optional software installation ======

Note that support during the workshop will be limited.


===== Jupyter CASA =====

==== Install instructions for Jupyter CASA ====

For parts of the tutorials we have developed Jupyter Notebooks based on the Jupyter CASA environment. This requires singularity or a docker container to run. For MacOS there is only a beta-version of singularity available, so the instructions below are based on docker. This installation is optional, all tutorials can also be done using the stand-alone version of CASA.

These instructions are verified to work on MacOS 10.14 and 10.15. For installation on a Linux system you can follow the same steps, as the docker platform should work the same way.

  - Install Docker for your [[https://docs.docker.com/get-docker/|preferred OS]].
  - Open a terminal window
  - On the command line of the terminal type ''docker pull penngwyn/jupytercasa''
  - Go to the directory where you want to do your data processing, or generate a new one
  - Move your data here and any notebook you already have
  - Run docker command: ''docker run -p 8888:8888 -i -t -v /tmp/.X11-unix:/tmp/.X11-unix -e DISPLAY=$DISPLAY -v <absolute path to your data processing directory>:/home/jupyter/data penngwyn/jupytercasa''
  - Copy the URL displayed in the terminal, paste it into a browser and edit it such that the section between brackets displays only the IP address. It should look something like: ''http://127.0.0.1:888/?token<loads of characters>''. 
  - You can generate a new notebook by clicking the "New" button on the top right in the notebook server page.
  - If you downloaded the tutorial notebook you should see it in the list on your browser and can launch it by clicking on the link once.

==== Notes ====

There is a known issue with X11-forwarding on MacOS. The workaround for this is to slightly modify step 6 above. You first launch the docker, inside the docker you launch the notebook. You can than proceed with step 7 above.

  - Start the docker with ''docker run -p 8888:8888 -i -t -e DISPLAY=$DISPLAY -v <absolute path to your data processing directory>:/home/jupyter/data penngwyn/jupytercasa bash'' 
  - Inside the same terminal window launch the notebook ''xvfb-run jupyter notebook''


If you launch the docker image with the ''-rm'' option (as it suggested in the Github readme), everything in the docker virtual drives will be removed upon shutdown. You probably don't want this to happen after a long day of work, so make sure the mirroring works. \\

The option ''-v'' mirrors a local directory on your laptop to a directory in the docker image. This is required to make the docker environment see your data and to store your notebooks on your local disk for future use. Upon shutdown of the container or notebook your local disks will not be touched.\\

The tutorial notebook is based on the [[https://casaguides.nrao.edu/index.php?title=VLA_Continuum_Tutorial_3C391-CASA5.5.0|VLA tutorial]] and requires the dataset you can download from this page. You can also follow the tutorial in a CASA5.7 installation on your laptop.\\

The notebook runs CASA5.6, but this should not impact the basic functionality used in the tutorials.\\

==== Available Jupyter notebooks ====

For the VLA tutorial there is a version of the notebook available in the [[https://github.com/aardk/jupyter-casa/tree/master/examples|examples]]. To get this to your laptop, in a terminal type ''wget https://github.com/aardk/jupyter-casa/raw/master/examples/vla-cont-tutorial.ipynb''

There is a beta version of the [[https://code.jive.eu/bemmel/EVN_CASA_pipeline/raw/branch/master/EVN_continuum_pyp.ipynb|EVN pipeline notebook]] and an [[https://www.dropbox.com/s/5tbdj90mzihggpr/n14c3_pipe.ipynb?dl=0|example notebook]] based on the DARA and EVN tutorials. Both contain all calibration steps up to the imaging.


===== CASA rPICARD pipeline =====
The [[https://bitbucket.org/M_Janssen/picard|rPICARD]] software is a generic CASA-based VLBI data reduction pipeline. In this workshop, the unique features of rPICARD are used to guide users through the calibration of mm VLBI data. But the pipeline works just as well for lower frequency (e.g., EVN) VLBI data. Just give it a try ;-)

The pipeline can be installed with [[https://singularity.lbl.gov|Singularity]], [[https://www.docker.com|Docker]], or from source on Linux.

=== Using Singularity ===

You will obtain a containerized version of the software, which is great for scientific reproducibility!\\
1) Go to your working directory.\\
2) $ singularity build casavlbi.pipe [[docker://mjanssen2308/casavlbi:latest|docker://mjanssen2308/casavlbi:latest]] \\
3) $ singularity run ./casavlbi.pipe \\
Your are now inside Singularity, where you have access to rPICARD. To go back to your old shell, type\\
$ exit


=== Using Docker ===

Similar to Singularity and just as great for scientific reproducibility. In fact, you will use the same container here but within Docker your are more 'isolated' from your host system compared to Singularity.\\
1) $ docker pull mjanssen2308/casavlbi:latest\\
2) $ docker run --name picard.cont -it --init --env HOME=/data --user $(id -u) -v /etc/passwd:/etc/passwd --network=host -v /tmp/.X11-unix:/tmp/.X11-unix:ro -e DISPLAY=$DISPLAY -v /full/path/to/your/working/dir:/data mjanssen2308/casavlbi:latest\\
!! It seems like the wiki page messes up double dashes in the above command. See the README of https://bitbucket.org/M_Janssen/picard for the correct version of that command which works for copy/paste.\\
(you will have to fill in the correct path for /full/path/to/your/working/dir here)\\
3) $ cd /data\\
Your are now inside Docker in a mirror of your working directory, where you have access to rPICARD. To go back to your old shell, type\\
$ exit


=== Installing from source on Linux ===

(Note that rPICARD runs on CASA6. This will not interfere with your CASA5 installation as you can have multiple versions of CASA on your system.)\\
1)Go to a directory where you install your software and obtain the correct CASA version for the pipeline:  \\
$ wget https://ftp.science.ru.nl/astro/mjanssen/casa-CAS-13094-2.tar.xz  \\
$ tar xvJf casa-CAS-13094-2.tar.xz  \\
2)Git clone the repository (do not download as tarball because the .git folder is needed):  \\
$ git clone https://bitbucket.org/M_Janssen/picard  \\
3)Link the pipeline to the CASA installation:  \\
$ ./picard/setup.py -p casa-CAS-13094-2  \\
The script will walk you through the automatic installation.  \\
4)Add picard to your path:  \\
$ printf '\nexport PATH=$PATH:'"$(pwd)"'/picard/picard\n' >> ~/.bashrc  \\
5)Add picard to your PYTHONPATH:  \\
$ printf '\nexport PYTHONPATH=$PYTHONPATH:'"$(pwd)"'/picard/picard\n' >> ~/.bashrc  \\
$ source ~/.bashrc \\
6)Install [[https://github.com/haavee/jiveplot|jiveplot]] and/or [[https://singularity.lbl.gov|Singularity]] and add it to your path for much  
better and quicker plotting (optional but strongly recommended).

==== How to use the pipeline ====
There is a quick start guide in the [[https://bitbucket.org/M_Janssen/picard/raw/32393f15cd86f83cfc642b4d12fb616aab08f77c/documentation/picard_documentation.pdf|documentation]] and the usage of rPICARD will be explained in the [[https://docs.google.com/presentation/d/16K9azWq2cfnqZWbARiBiXrpTGTdIrqsToC7ZHd3zdFo/edit?usp=sharing| mm-VLBI]] tutorial on Thursday.
In a nutshell, you have to\\
1) Put all relevant files or links to those files (FITS-IDI visibility data or a MeasurementSet, a priori calibration metadata in the ANTAB format, and flag tables) in a working directory.\\
2) Copy a set of input parameters to the working directory ($ cp -r /usr/local/src/picard/input_template/ input) and set the science_target, calibrators_instrphase, calibrators_bandpass, and calibrators_rldly parameters in input/observation.inp. Then, you have to set the array_type and refant parameters in input/array.inp (the meaning of these parameters are described in the files).\\
3) Run the pipeline with\\
$ picard -p -n 4\\
Here, -p specifies to run rPICARD in the current working directory and -n 4 allows rPICARD to use 4 of your CPU cores for parallel data processing.

==== Additional information ====
  * [[https://ui.adsabs.harvard.edu/abs/2019A%26A...626A..75J|Janssen et al., A&A, 626 (2019) A75]]
  * [[https://bitbucket.org/M_Janssen/picard/raw/32393f15cd86f83cfc642b4d12fb616aab08f77c/documentation/picard_documentation.pdf|picard/documentation/picard_documentation.pdf]]