JIVE Wiki

User Tools

Site Tools

Trace:
sfxc-guide

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
sfxc-guide [2023/07/10 14:22] – keimpemasfxc-guide [2023/07/10 14:38] (current) – keimpema
Line 1: Line 1:
-<html> +====== SFXC User's Manual ====== 
-<head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>SFXC User's Manual</title><meta name="generator" content="DocBook XSL Stylesheets V1.78.1"></head><body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="book"><div class="titlepage"><div><div><h1 class="title"><a name="idp4341760"></a><span class="application"><acronym class="acronym">SFXC</acronym></span> User's Manual</h1></div><div><div class="author"><h3 class="author"><span class="firstname">Mark</span> <span class="surname">Kettenis</span></h3><div class="affiliation"><span class="orgname">JIVE<br></span></div></div></div></div><hr></div><div class="toc"><p><b>Table of Contents</b></p><dl class="toc"><dt><span class="chapter"><a href="#idp4347312">1. Running the <span class="application"><acronym class="acronym">SFXC</acronym></span> correlator</a></span></dt><dt><span class="chapter"><a href="#idp4366352">2. The correlator control file</a></span></dt><dt><span class="chapter"><a href="#idp4390448">3. Preparing your VEX file</a></span></dt><dt><span class="chapter"><a href="#idp4406976">4. Post-processing</a></span></dt></dl></div><div class="list-of-examples"><p><b>List of Examples</b></p><dl><dt>1.1. <a href="#idp4360512">Rank file example</a></dt><dt>2.1. <a href="#idp4389168">Control file example</a></dt></dl></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="idp4347312"></a>Chapter 1. Running the <span class="application"><acronym class="acronym">SFXC</acronym></span> correlator</h1></div></div></div><p> +Mark Kettenis (JIVE)
-<span class="application"><acronym class="acronym">SFXC</acronym></span> is an <acronym class="acronym">MPI</acronym> application.  This means that +
-running it is somewhat dependent on the <acronym class="acronym">MPI</acronym> +
-implementation installed on your system.  The instructions here are +
-for OpenMPI, which currently seems to be the most popular Open Source +
-MPI implementation for Linux systems.+
  
-</p><pre class="screen"> +===== Chapter 1. Running the SFXC correlator ===== 
-<code class="prompt">$</code> <span class="command"><strong>mpirun --machinefile <em class="parameter"><code>machinefile</code></em> --rankfile <em class="parameter"><code>rankfile</code></em> --np <em class="parameter"><code>np</code></em> sfxc <em class="parameter"><code>controlfile</code></em> <em class="parameter"><code>vexfile</code></em> </strong></span> +SFXC is an MPI application. This means that running it is somewhat dependent on the MPI implementation installed on your system. The instructions here are for OpenMPI, which currently seems to be the most popular Open Source MPI implementation for Linux systems.
-</pre><p>+
  
-where <em class="parameter"><code>controlfile</code></em> is the name of the correlator +'' 
-control file that describes the correlation +$ **mpirun --machinefile machinefile --rankfile rankfile --np np sfxc controlfile vexfile** 
-parameters, <em class="parameter"><code>vexfile</code></em> is the name of the VEX file +''
-that describes the experiment, <em class="parameter"><code>np</code></em> is the number of +
-MPI processes to start as described by the machine +
-file <em class="parameter"><code>machinefile</code></em> and rank +
-file <em class="parameter"><code>rankfile</code></em>. +
-</p><p> +
-When creating the rank file, there are a few things that need to be +
-taken into account.+
  
-The process with rank 0 becomes the manager process.  Since the manager +where //controlfile// is the name of the correlator control file that describes the correlation parameters, //vexfile// is the name of the VEX file that describes the experiment, //np// is the number of MPI processes to start as described by the machine file //machinefile// and rank file //rankfile//.
-process doesn't really do a lot, there is no point in assigning more than +
-a single slot to it.+
  
-The process with rank 1 becomes the log process.  As with the manager +When creating the rank file, there are a few things that need to be taken into account. The process with rank 0 becomes the manager process. Since the manager process doesn't really do a lot, there is no point in assigning more than a single slot to it. The process with rank 1 becomes the log process. As with the manager process, there is no point in assigning more than a single slot. The process with rank 2 becomes the output process. This process will be able to take advantage of multiple cores, so assigning two slots is a good idea if you expect a significant output data rate. At JIVE we usually run all these processes on the cluster head node.
-process, there is no point in assigning more than a single slot.+
  
-The process with rank 2 becomes the output process.  This process will be +The processes starting at rank 3 become input processes. There will be one input process for each station in the correrlation. When correlating directly from Mark5 disk packs, these processes will need to run on the Mark5s containing the diskpacks for those stations. When correlating from files, these processes will need to run on machines that have access to the data files for these stations. The process with rank 3 will be assigned to the station that comes first when the stations are ordered alphabetically by station code. The process with rank 4 will be assigned to the station that comes second, etc. The input processes do the unpacking and corner turning of the input data, which can be cpu intensive. So assigning multiple slots is a good idea. By default the unpacking happens in two seperate threads, so using two or three slots makes sense.
-able to take advantage of multiple cores, so assigning two slots is a +
-good idea if you expect a significant output data rate.+
  
-At JIVE we usually run all these processes on the cluster head node. +The remainder of the processes will be assigned to correlations processes. A single slot is sufficient for these processes.
-</p><p> +
-The processes starting at rank 3 become input processes.  There will +
-be one input process for each station in the correrlation.  When +
-correlating directly from Mark5 disk packs, these processes will need +
-to run on the Mark5s containing the diskpacks for those stations. +
-When correlating from files, these processes will need to run on +
-machines that have access to the data files for these stations.+
  
-The process with rank 3 will be assigned to the station that comes +=== Example 1.1. Rank file example ===
-first when the stations are ordered alphabetically by station code. +
-The process with rank 4 will be assigned to the station that comes +
-second, etc.+
  
-The input processes do the unpacking and corner turning of the input +<code>
-data, which can be cpu intensive.  So assigning multiple slots is a +
-good idea.  By default the unpacking happens in two seperate threads, +
-so using two or three slots makes sense. +
-</p><p> +
-The remainder of the processes will be assigned to correlations +
-processes.  A single slot is sufficient for these processes. +
-</p><div class="example"><a name="idp4360512"></a><p class="title"><b>Example 1.1. Rank file example</b></p><div class="example-contents"><pre class="programlisting">+
 rank 0=head slot=0 rank 0=head slot=0
 rank 1=head slot=1 rank 1=head slot=1
Line 66: Line 31:
 rank 36=sfxc-a2 slot=7 rank 36=sfxc-a2 slot=7
 rank 37=sfxc-a3 slot=7 rank 37=sfxc-a3 slot=7
-</pre></div></div><br class="example-break"><p> +</code> 
-<span class="application"><acronym class="acronym">SFXC</acronym></span> will automatically generate delay tables using the CALC10 code + 
-that's included in the distribution.  The CALC10 needs some additional +SFXC will automatically generate delay tables using the CALC10 code that's included in the distribution. The CALC10 needs some additional input files to do its work. These are the JPL Solar System Ephemeris (''DE405_le.jpl''), ocean loading information (''ocean.dat'') and antenna tilt (''tilt.dat''). It expects to find these in a directory pointed to by the CALC_DIR environment variable. A copy of ''DE405_le.jpl'' as well as ocean loading and antenna tilt information for many antennas that co-observe with the European VLBI Network (EVN) can be found in ''sfxc/lib/calc10/data'' in the source distribution. 
-input files to do its work.  These are the JPL Solar System Ephemeris + 
-(<code class="filename">DE405_le.jpl</code>), ocean loading information +===== Chapter 2. The correlator control file  ===== 
-(<code class="filename">ocean.dat</code>) and antenna tilt + 
-(<code class="filename">tilt.dat</code>).  It expects to find these in a +The correlator control file uses the JavaScript Object Notation (JSON) format. It is constumary to give these files a ''.ctrl'' extension. 
-directory pointed to by the <code class="envar">CALC_DIR</code> environment + 
-variable.  A copy of <code class="filename">DE405_le.jpl</code> as well as +* ''output_file''\\ 
-ocean loading and antenna tilt information for many antennas that +A string specifying the name of the file to write the correltor output to. It is costumary to give this file a ''.cor'' extension. 
-co-observe with the European VLBI Network (EVN) can be found + 
-in <code class="filename">sfxc/lib/calc10/data</code> in the source +* ''number_channels''\\ 
-distribution. +An integer specifying the number of desired spectral channels in the correlator output. Has to be power of two. 
-</p></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="idp4366352"></a>Chapter 2. The correlator control file</h1></div></div></div><p> + 
-The correlator control file uses the JavaScript Object Notation (JSON) +* ''integr_time''\\ 
-format.  It is constumary to give these files +A floating-point number specifying the integration time in seconds. Will be rounded to the nearest integral microsecond. 
-a <code class="filename">.ctrl</code> extension. + 
-</p><div class="variablelist"><dl class="variablelist"><dt><span class="term"><code class="varname">output_file</code></span></dt><dd><p> +* ''cross_polarize''\\ 
- A string specifying the name of the file to write the +A boolean indicating whether cross hands should be calculated or not. 
- correltor output to.  It is costumary to give this file + 
- a <code class="filename">.cor</code> extension. +* ''stations''\\ 
-      </p></dd><dt><span class="term"><code class="varname">number_channels</code></span></dt><dd><p> +A list of strings specifying the stations that are to be correlated. 
- An integer specifying the number of desired spectral channels + 
- in the correlator output.  Has to be power of two. +* ''data_sources''\\ 
-      </p></dd><dt><span class="term"><code class="varname">integr_time</code></span></dt><dd><p> +An object containing a list of strings for each station specifying the data source locations for each station. Each data source location is specified in the form of a Uniform Resource Identifier (URI). To correlate data from plain files, the standard file scheme can be used. Correlating data directly from Mark5 disk packs is achieved by specifying an appropriate mk5: URI. All URIs for a single station must use the same scheme. Specifying multiple URIs for a single station is currently only supported for the file scheme. 
- A floating-point number specifying the integration time in + 
- seconds.  Will be rounded to the nearest integral microsecond. +* ''start''\\ 
-      </p></dd><dt><span class="term"><code class="varname">cross_polarize</code></span></dt><dd><p> +A string specifying the start time of the correlation. The time should be specified in VEX (####y###d##h##m##s) format representing UTC. For real-time correlation the string "now" can be used, which will instruct the correlator to use the current wall clock time (in UTC) as the start time. 
- A boolean indicating whether cross hands should be calculated + 
- or not. +* ''stop''\\ 
-      </p></dd><dt><span class="term"><code class="varname">stations</code></span></dt><dd><p> +A string specifying the end time of the correlation. The time should be specified in VEX (####y###d##h##m##s) format representing UTC. 
- A list of strings specifying the stations that are to be correlated. + 
-      </p></dd><dt><span class="term"><code class="varname">data_sources</code></span></dt><dd><p> +* ''exper_name''\\ 
- An object containing a list of strings for each station +A string specifying the experiment name. Used for generating and referencing the appropriate delay tables. 
- specifying the data source locations for each station.  Each + 
- data source location is specified in the form of a Uniform +* ''delay_directory''\\ 
- Resource Identifier (URI).  To correlate data from plain +A string specifying the directory in which to store the delay tables. 
- files, the standard <code class="uri">file</code> scheme can be used. +
- Correlating data directly from Mark5 disk packs is achieved by +
- specifying an appropriate <code class="uri">mk5:</code> URI.  All URIs for a +
- single station must use the same scheme.  Specifying multiple +
- URIs for a single station is currently only supported for +
- the <code class="uri">file</code> scheme. +
-      </p></dd><dt><span class="term"><code class="varname">start</code></span></dt><dd><p> +
- A string specifying the start time of the correlation.  The +
- time should be specified in VEX (####y###d##h##m##s) format +
- representing UTC.  For real-time correlation the +
- string <code class="literal">"now"</code> can be used, which will +
- instruct the correlator to use the current wall clock time (in +
- UTC) as the start time. +
-      </p></dd><dt><span class="term"><code class="varname">stop</code></span></dt><dd><p> +
- A string specifying the end time of the correlation.  The +
- time should be specified in VEX (####y###d##h##m##s) format +
- representing UTC. +
-      </p></dd><dt><span class="term"><code class="varname">exper_name</code></span></dt><dd><p> +
- A string specifying the experiment name.  Used for generating +
- and referencing the appropriate delay tables. +
-      </p></dd><dt><span class="term"><code class="varname">delay_directory</code></span></dt><dd><p> +
- A string specifying the directory in which to store the delay tables. +
-      </p></dd></dl></div><p>+
 An example of a control file is given below: An example of a control file is given below:
-</p><div class="example"><a name="idp4389168"></a><p class="title"><b>Example 2.1. Control file example</b></p><div class="example-contents"><pre class="programlisting">+ 
 +=== Example 2.1. Control file example === 
 +<code>
 { {
     "exper_name": "F13C4",      "exper_name": "F13C4", 
Line 154: Line 99:
     "delay_directory": "file:///home/kettenis/test/f13c4/delays"     "delay_directory": "file:///home/kettenis/test/f13c4/delays"
 } }
-</pre></div></div><br class="example-break"></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="idp4390448"></a>Chapter 3. Preparing your VEX file</h1></div></div></div><p> +</code>
-Some information needs to be provided in the VEX file that is +
-typically not emitted by the scheduling software.  It is essential +
-that you have $CLOCK and $EOP blocks.  Some of the tools distributed +
-with SFXC also use the $TAPELOG_OBS block.  We recommend that in the +
-$EOP block you provide entries at a 24 hour intervals and have an +
-additional entry for the day before and the day after the observation. +
-All these blocks need to be properly referenced; from the $GLOBAL +
-block for $EOP and from the $STATION block fot the $CLOCK and $EOP +
-blocks. +
-</p><p> +
-It is important that the description of the data format in the VEX +
-file is correct.  <span class="application"><acronym class="acronym">SFXC</acronym></span> currently supports the Mark4, VLBA, Mark5B +
-and VDIF data format and includes some heuristics to determine the +
-correct data format from the VEX file.  If <span class="application"><acronym class="acronym">SFXC</acronym></span> crashes, seems to +
-hang or complains it cannot find any valid data, please check that the +
-data format description in your VEX file matches reality. +
-</p><div class="variablelist"><dl class="variablelist"><dt><span class="term">Mark4</span></dt><dd><p> +
-        record_transport_type should be set to Mark5A and +
- electronics_rack_type should be set to Mark4 or VLBA4 in the +
- $DAS block; track_frame_format should be set to Mark4 in the +
- $TRACKS section +
-      </p></dd><dt><span class="term">VLBA</span></dt><dd><p> +
- record_transport_type should be set to Mark5A and +
- electronics_rack_type should be set to VLBA in the $DAS block; +
- track_frame_format should be set to VLBA in the $TRACKS +
- section +
-      </p></dd><dt><span class="term">Mark5B</span></dt><dd><p> +
- record_transport_type should be set to Mark5B in the $DAS +
- block, and either a $TRACKS setion should be present and have +
- its track_frame_format keyword set to Mark5B +
- <a href="#ftn.idp4399072" class="footnote" name="idp4399072"><sup class="footnote">[1]</sup></a> +
- , or a $BITSTREAMS block must be present as proposed for the +
- upcoming VEX 2 standard. +
-      </p></dd><dt><span class="term">VDIF</span></dt><dd><p> +
- VEX 1.5 does not provide the means to properly specify VDIF as +
- the recording format.  Current versions of $sfxc; recognize +
- the $THREADS block as proposed for the new VEX 2 standard by +
- Walter Brisken from NRAO. +
- <a href="#ftn.idp4402240" class="footnote" name="idp4402240"><sup class="footnote">[2]</sup></a> +
- However this proposal has been withdrawn in favour of a new +
- $DATASTREAMS block.  The intention is to have <span class="application"><acronym class="acronym">SFXC</acronym></span> recognize +
- $DATASTREAMS blocks once the final VEX 2 standard arrives.  In +
- the meantime a $THREADS block will need to be added, as SCHED +
- doesn't do this. +
-      </p><p> +
- record_transport_type should be set to Mark5C or VDIF in the +
- $DAS block.  If the record_transport_type is set to Mark5C, +
- electronics_rack_type must be WIDAR.  A $THREADS block must be +
- present. +
-      </p></dd></dl></div><p> +
-<span class="application"><acronym class="acronym">SFXC</acronym></span> has been tested extensively with VEX output from (NRAO) SCHED. +
-Your mileage may vary with output from other VLBI scheduling software. +
-</p><div class="footnotes"><br><hr style="width:100; text-align:left;margin-left: 0"><div id="ftn.idp4399072" class="footnote"><p><a href="#idp4399072" class="para"><sup class="para">[1] </sup></a> +
-   SCHED spells this as MARK5B, which is tolerated by <span class="application"><acronym class="acronym">SFXC</acronym></span> +
-   </p></div><div id="ftn.idp4402240" class="footnote"><p><a href="#idp4402240" class="para"><sup class="para">[2] </sup></a> +
-     <a class="ulink" href="https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc#A_61_36THREADS_61_Block" target="_top"><code class="uri">https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc#A_61_36THREADS_61_Block</code></a> +
-   </p></div></div></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="idp4406976"></a>Chapter 4. Post-processing</h1></div></div></div><p> +
-As with the Mark4 hardware correlator, <span class="application"><acronym class="acronym">SFXC</acronym></span> output is converted into +
-an AIPS++/CASA MeasurementSet using the <span class="command"><strong>j2ms2</strong></span> +
-program.  Create a directory with the name of experiment as given in +
-the VEX file.  Copy the VEX file into this directory and rename it to +
-<code class="filename"><em class="parameter"><code>experiment</code></em>.vix</code> +
-where <em class="parameter"><code>experiment</code></em> is again the name of the +
-experiment as given in the VEX file.+
  
-</p><pre class="screen"> +===== Chapter 3. Preparing your VEX file =====
-<code class="prompt">$</code> <span class="command"><strong>j2ms2 <em class="parameter"><code>file</code></em> ...</strong></span> +
-</pre><p>+
  
-where <em class="parameter"><code>file</code></em> is the name of the correlator output +Some information needs to be provided in the VEX file that is typically not emitted by the scheduling software. It is essential that you have $CLOCK and $EOP blocks. Some of the tools distributed with SFXC also use the $TAPELOG_OBS block. We recommend that in the $EOP block you provide entries at a 24 hour intervals and have an additional entry for the day before and the day after the observation. All these blocks need to be properly referenced; from the $GLOBAL block for $EOP and from the $STATION block fot the $CLOCK and $EOP blocks.
-file.  This will produce a MeasurementSet +
-named <code class="filename"><em class="parameter"><code>experiment</code></em>.ms</code>.  It +
-is possible to specify multiple correlator output files on +
-the <span class="command"><strong>j2ms2</strong></span> command line.  The visibilities in these +
-files are simply concatenated and written out into a single +
-MeasurementSet. +
-</p><p> +
-To convert data into FITS-IDI such that it can be read into AIPS, +
-the <span class="command"><strong>tConvert</strong></span> program can be used.+
  
-</p><pre class="screen"> +It is important that the description of the data format in the VEX file is correct. SFXC currently supports the Mark4, VLBA, Mark5B and VDIF data format and includes some heuristics to determine the correct data format from the VEX file. If SFXC crashes, seems to hang or complains it cannot find any valid data, please check that the data format description in your VEX file matches reality. 
-<code class="prompt">$</code> <span class="command"><strong>tConvert <em class="parameter"><code>experiment</code></em>.ms <em class="parameter"><code>experiment</code></em>.IDI </strong></span> + 
-</pre><p>+* ''Mark4''\\ 
 +record_transport_type should be set to Mark5A and electronics_rack_type should be set to Mark4 or VLBA4 in the $DAS block; track_frame_format should be set to Mark4 in the $TRACKS section 
 + 
 +* ''VLBA''\\ 
 +record_transport_type should be set to Mark5A and electronics_rack_type should be set to VLBA in the $DAS block; track_frame_format should be set to VLBA in the $TRACKS section 
 + 
 +* ''Mark5B''\\ 
 +record_transport_type should be set to Mark5B in the $DAS block, and either a $TRACKS setion should be present and have its track_frame_format keyword set to Mark5B [1] , or a $BITSTREAMS block must be present as proposed for the upcoming VEX 2 standard. 
 + 
 +* ''VDIF''\\ 
 +VEX 1.5 does not provide the means to properly specify VDIF as the recording format. Current versions of $sfxc; recognize the $THREADS block as proposed for the new VEX 2 standard by Walter Brisken from NRAO. [2] However this proposal has been withdrawn in favour of a new $DATASTREAMS block. The intention is to have SFXC recognize $DATASTREAMS blocks once the final VEX 2 standard arrives. In the meantime a $THREADS block will need to be added, as SCHED doesn't do this. 
 + 
 +record_transport_type should be set to Mark5C or VDIF in the $DAS block. If the record_transport_type is set to Mark5C, electronics_rack_type must be WIDAR. A $THREADS block must be present. 
 + 
 +SFXC has been tested extensively with VEX output from (NRAO) SCHED. Your mileage may vary with output from other VLBI scheduling software. 
 + 
 + 
 +[1] SCHED spells this as MARK5B, which is tolerated by SFXC 
 + 
 +[2] https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc#A_61_36THREADS_61_Block 
 + 
 +===== Chapter 4. Post-processing ===== 
 + 
 +As with the Mark4 hardware correlator, SFXC output is converted into an AIPS++/CASA MeasurementSet using the j2ms2 program. Create a directory with the name of experiment as given in the VEX file. Copy the VEX file into this directory and rename it to experiment.vix where experiment is again the name of the experiment as given in the VEX file. 
 + 
 +'' 
 +$ **j2ms2 file ...** 
 +'' 
 + 
 +where ''file'' is the name of the correlator output file. This will produce a MeasurementSet named ''experiment.ms''. It is possible to specify multiple correlator output files on the **j2ms2** command line. The visibilities in these files are simply concatenated and written out into a single MeasurementSet. 
 + 
 +To convert data into FITS-IDI such that it can be read into AIPS, the **tConvert** program can be used. 
 + 
 +'' 
 +$ **tConvert experiment.ms experiment.IDI** 
 +'' 
 + 
 +The resulting FITS-IDI can be read directly into AIPS using **FITLD**. 
 + 
 +Note that at JIVE we run some additional post-processing tools on the MeasurementSet before converting data into FITS-IDI. The most important things are: 
 + 
 +Amplitude correction for a-bit data; currently **j2ms2** assumes all data is 2-bit.
  
-The resulting FITS-IDI can be read directly into AIPS using <span class="command"><strong>FITLD</strong></span>. 
-</p><p> 
-Note that at JIVE we run some additional post-processing tools on the 
-MeasurementSet before converting data into FITS-IDI.  The most 
-important things are: 
-</p><div class="itemizedlist"><ul class="itemizedlist" style="list-style-type: disc; "><li class="listitem"><p> 
-Amplitude correction for a-bit data; 
-currently <span class="command"><strong>j2ms2</strong></span> assumes all data is 2-bit. 
-</p></li><li class="listitem"><p> 
 Flagging of delay-rate zero events. Flagging of delay-rate zero events.
-</p></li><li class="listitem"><p>+
 Flagging of data with low weights. Flagging of data with low weights.
-</p></li></ul></div><p> + 
-If you correlate and convert your own data, you may have to take care +If you correlate and convert your own data, you may have to take care of these things when reducing the data in AIPS.
-of these things when reducing the data in AIPS. +
-</p></div></div></body> +
-</html>+
  
sfxc-guide.1688998957.txt.gz · Last modified: 2023/07/10 14:22 by keimpema