This is an old revision of the document!
<HTML> <title>SFXC User's Manual</title><meta name=“generator” content=“DocBook XSL Stylesheets V1.78.1”><div class=“book”><div class=“titlepage”><div><div><h1 class=“title”><a name=“idp4341760”></a><span class=“application”><acronym class=“acronym”>SFXC</acronym></span> User's Manual</h1></div><div><div class=“author”><h3 class=“author”><span class=“firstname”>Mark</span> <span class=“surname”>Kettenis</span></h3><div class=“affiliation”><span class=“orgname”>JIVE<br></span></div></div></div></div><hr></div><div class=“toc”><p><b>Table of Contents</b></p><dl class=“toc”><dt><span class=“chapter”><a href=“#idp4347312”>1. Running the <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> correlator</a></span></dt><dt><span class=“chapter”><a href=“#idp4366352”>2. The correlator control file</a></span></dt><dt><span class=“chapter”><a href=“#idp4390448”>3. Preparing your VEX file</a></span></dt><dt><span class=“chapter”><a href=“#idp4406976”>4. Post-processing</a></span></dt></dl></div><div class=“list-of-examples”><p><b>List of Examples</b></p><dl><dt>1.1. <a href=“#idp4360512”>Rank file example</a></dt><dt>2.1. <a href=“#idp4389168”>Control file example</a></dt></dl></div><div class=“chapter”><div class=“titlepage”><div><div><h1 class=“title”><a name=“idp4347312”></a>Chapter 1. Running the <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> correlator</h1></div></div></div><p> <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> is an <acronym class=“acronym”>MPI</acronym> application. This means that running it is somewhat dependent on the <acronym class=“acronym”>MPI</acronym> implementation installed on your system. The instructions here are for OpenMPI, which currently seems to be the most popular Open Source MPI implementation for Linux systems.
</p><pre class=“screen”>
$
<span class=“command”><strong>mpirun –machinefile <em class=“parameter”>
machinefile
</em> –rankfile <em class=“parameter”>
rankfile
</em> –np <em class=“parameter”>
np
</em> sfxc <em class=“parameter”>
controlfile
</em> <em class=“parameter”>
vexfile
</em> </strong></span> </pre><p>
where <em class=“parameter”>
controlfile
</em> is the name of the correlator control file that describes the correlation parameters, <em class=“parameter”>
vexfile
</em> is the name of the VEX file that describes the experiment, <em class=“parameter”>
np
</em> is the number of MPI processes to start as described by the machine file <em class=“parameter”>
machinefile
</em> and rank file <em class=“parameter”>
rankfile
</em>. </p><p> When creating the rank file, there are a few things that need to be taken into account.
The process with rank 0 becomes the manager process. Since the manager process doesn't really do a lot, there is no point in assigning more than a single slot to it.
The process with rank 1 becomes the log process. As with the manager process, there is no point in assigning more than a single slot.
The process with rank 2 becomes the output process. This process will be able to take advantage of multiple cores, so assigning two slots is a good idea if you expect a significant output data rate.
At JIVE we usually run all these processes on the cluster head node. </p><p> The processes starting at rank 3 become input processes. There will be one input process for each station in the correrlation. When correlating directly from Mark5 disk packs, these processes will need to run on the Mark5s containing the diskpacks for those stations. When correlating from files, these processes will need to run on machines that have access to the data files for these stations.
The process with rank 3 will be assigned to the station that comes first when the stations are ordered alphabetically by station code. The process with rank 4 will be assigned to the station that comes second, etc.
The input processes do the unpacking and corner turning of the input data, which can be cpu intensive. So assigning multiple slots is a good idea. By default the unpacking happens in two seperate threads, so using two or three slots makes sense. </p><p> The remainder of the processes will be assigned to correlations processes. A single slot is sufficient for these processes. </p><div class=“example”><a name=“idp4360512”></a><p class=“title”><b>Example 1.1. Rank file example</b></p><div class=“example-contents”><pre class=“programlisting”> rank 0=head slot=0 rank 1=head slot=1 rank 2=head slot=2,3 rank 3=sfxc-d2 slot=0,1 rank 4=sfxc-d2 slot=2,3 rank 5=sfxc-d3 slot=0,1 rank 6=sfxc-a0 slot=0 rank 7=sfxc-a1 slot=0 … rank 36=sfxc-a2 slot=7 rank 37=sfxc-a3 slot=7 </pre></div></div><br class=“example-break”><p> <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> will automatically generate delay tables using the CALC10 code that's included in the distribution. The CALC10 needs some additional input files to do its work. These are the JPL Solar System Ephemeris (
DE405_le.jpl
), ocean loading information (
ocean.dat
) and antenna tilt (
tilt.dat
). It expects to find these in a directory pointed to by the
CALC_DIR
environment variable. A copy of
DE405_le.jpl
as well as ocean loading and antenna tilt information for many antennas that co-observe with the European VLBI Network (EVN) can be found in
sfxc/lib/calc10/data
in the source distribution. </p></div><div class=“chapter”><div class=“titlepage”><div><div><h1 class=“title”><a name=“idp4366352”></a>Chapter 2. The correlator control file</h1></div></div></div><p> The correlator control file uses the JavaScript Object Notation (JSON) format. It is constumary to give these files a
.ctrl
extension. </p><div class=“variablelist”><dl class=“variablelist”><dt><span class=“term”>
output_file
</span></dt><dd><p>
A string specifying the name of the file to write the
correltor output to. It is costumary to give this file
a <code class="filename">.cor</code> extension.
</p></dd><dt><span class="term"><code class="varname">number_channels</code></span></dt><dd><p>
An integer specifying the number of desired spectral channels
in the correlator output. Has to be power of two.
</p></dd><dt><span class="term"><code class="varname">integr_time</code></span></dt><dd><p>
A floating-point number specifying the integration time in
seconds. Will be rounded to the nearest integral microsecond.
</p></dd><dt><span class="term"><code class="varname">cross_polarize</code></span></dt><dd><p>
A boolean indicating whether cross hands should be calculated
or not.
</p></dd><dt><span class="term"><code class="varname">stations</code></span></dt><dd><p>
A list of strings specifying the stations that are to be correlated.
</p></dd><dt><span class="term"><code class="varname">data_sources</code></span></dt><dd><p>
An object containing a list of strings for each station
specifying the data source locations for each station. Each
data source location is specified in the form of a Uniform
Resource Identifier (URI). To correlate data from plain
files, the standard <code class="uri">file</code> scheme can be used.
Correlating data directly from Mark5 disk packs is achieved by
specifying an appropriate <code class="uri">mk5:</code> URI. All URIs for a
single station must use the same scheme. Specifying multiple
URIs for a single station is currently only supported for
the <code class="uri">file</code> scheme.
</p></dd><dt><span class="term"><code class="varname">start</code></span></dt><dd><p>
A string specifying the start time of the correlation. The
time should be specified in VEX (####y###d##h##m##s) format
representing UTC. For real-time correlation the
string <code class="literal">"now"</code> can be used, which will
instruct the correlator to use the current wall clock time (in
UTC) as the start time.
</p></dd><dt><span class="term"><code class="varname">stop</code></span></dt><dd><p>
A string specifying the end time of the correlation. The
time should be specified in VEX (####y###d##h##m##s) format
representing UTC.
</p></dd><dt><span class="term"><code class="varname">exper_name</code></span></dt><dd><p>
A string specifying the experiment name. Used for generating
and referencing the appropriate delay tables.
</p></dd><dt><span class="term"><code class="varname">delay_directory</code></span></dt><dd><p>
A string specifying the directory in which to store the delay tables.
</p></dd></dl></div><p>
An example of a control file is given below: </p><div class=“example”><a name=“idp4389168”></a><p class=“title”><b>Example 2.1. Control file example</b></p><div class=“example-contents”><pre class=“programlisting”> {
"exper_name": "F13C4",
"cross_polarize": true,
"number_channels": 256,
"integr_time": 1,
"output_file": "file:///home/kettenis/test/f13c4/f13c4_no0023.cor",
"stations": [
"Eb",
"Fd",
"Nl"
],
"data_sources": {
"Nl": [
"file:///scratch/kettenis/f13c4/f13c4_nl_no0023.m5b"
],
"Eb": [
"file:///scratch/kettenis/f13c4/f13c4_eb_no0023.m5b"
],
"Fd": [
"file:///scratch/kettenis/f13c4/f13c4_fd_no0023.m5b"
]
},
"start": "2013y148d10h29m26s",
"stop": "2013y148d10h34m06s",
"delay_directory": "file:///home/kettenis/test/f13c4/delays"
} </pre></div></div><br class=“example-break”></div><div class=“chapter”><div class=“titlepage”><div><div><h1 class=“title”><a name=“idp4390448”></a>Chapter 3. Preparing your VEX file</h1></div></div></div><p> Some information needs to be provided in the VEX file that is typically not emitted by the scheduling software. It is essential that you have $CLOCK and $EOP blocks. Some of the tools distributed with SFXC also use the $TAPELOG_OBS block. We recommend that in the $EOP block you provide entries at a 24 hour intervals and have an additional entry for the day before and the day after the observation. All these blocks need to be properly referenced; from the $GLOBAL block for $EOP and from the $STATION block fot the $CLOCK and $EOP blocks. </p><p> It is important that the description of the data format in the VEX file is correct. <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> currently supports the Mark4, VLBA, Mark5B and VDIF data format and includes some heuristics to determine the correct data format from the VEX file. If <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> crashes, seems to hang or complains it cannot find any valid data, please check that the data format description in your VEX file matches reality. </p><div class=“variablelist”><dl class=“variablelist”><dt><span class=“term”>Mark4</span></dt><dd><p>
record_transport_type should be set to Mark5A and
electronics_rack_type should be set to Mark4 or VLBA4 in the
$DAS block; track_frame_format should be set to Mark4 in the
$TRACKS section
</p></dd><dt><span class="term">VLBA</span></dt><dd><p>
record_transport_type should be set to Mark5A and
electronics_rack_type should be set to VLBA in the $DAS block;
track_frame_format should be set to VLBA in the $TRACKS
section
</p></dd><dt><span class="term">Mark5B</span></dt><dd><p>
record_transport_type should be set to Mark5B in the $DAS
block, and either a $TRACKS setion should be present and have
its track_frame_format keyword set to Mark5B
<a href="#ftn.idp4399072" class="footnote" name="idp4399072"><sup class="footnote">[1]</sup></a>
, or a $BITSTREAMS block must be present as proposed for the
upcoming VEX 2 standard.
</p></dd><dt><span class="term">VDIF</span></dt><dd><p>
VEX 1.5 does not provide the means to properly specify VDIF as
the recording format. Current versions of $sfxc; recognize
the $THREADS block as proposed for the new VEX 2 standard by
Walter Brisken from NRAO.
<a href="#ftn.idp4402240" class="footnote" name="idp4402240"><sup class="footnote">[2]</sup></a>
However this proposal has been withdrawn in favour of a new
$DATASTREAMS block. The intention is to have <span class="application"><acronym class="acronym">SFXC</acronym></span> recognize
$DATASTREAMS blocks once the final VEX 2 standard arrives. In
the meantime a $THREADS block will need to be added, as SCHED
doesn't do this.
</p><p>
record_transport_type should be set to Mark5C or VDIF in the
$DAS block. If the record_transport_type is set to Mark5C,
electronics_rack_type must be WIDAR. A $THREADS block must be
present.
</p></dd></dl></div><p>
<span class=“application”><acronym class=“acronym”>SFXC</acronym></span> has been tested extensively with VEX output from (NRAO) SCHED. Your mileage may vary with output from other VLBI scheduling software. </p><div class=“footnotes”><br><hr style=“width:100; text-align:left;margin-left: 0”><div id=“ftn.idp4399072” class=“footnote”><p><a href=“#idp4399072” class=“para”><sup class=“para”>[1] </sup></a>
SCHED spells this as MARK5B, which is tolerated by <span class="application"><acronym class="acronym">SFXC</acronym></span>
</p></div><div id="ftn.idp4402240" class="footnote"><p><a href="#idp4402240" class="para"><sup class="para">[2] </sup></a>
<a class="ulink" href="https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc#A_61_36THREADS_61_Block" target="_top"><code class="uri">https://safe.nrao.edu/wiki/bin/view/VLBA/Vex2doc#A_61_36THREADS_61_Block</code></a>
</p></div></div></div><div class="chapter"><div class="titlepage"><div><div><h1 class="title"><a name="idp4406976"></a>Chapter 4. Post-processing</h1></div></div></div><p>
As with the Mark4 hardware correlator, <span class=“application”><acronym class=“acronym”>SFXC</acronym></span> output is converted into an AIPS++/CASA MeasurementSet using the <span class=“command”><strong>j2ms2</strong></span> program. Create a directory with the name of experiment as given in the VEX file. Copy the VEX file into this directory and rename it to
<em class="parameter"><code>experiment
</em>.vix</code> where <em class=“parameter”>
experiment
</em> is again the name of the experiment as given in the VEX file.
</p><pre class=“screen”>
$
<span class=“command”><strong>j2ms2 <em class=“parameter”>
file
</em> …</strong></span> </pre><p>
where <em class=“parameter”>
file
</em> is the name of the correlator output file. This will produce a MeasurementSet named
<em class="parameter"><code>experiment
</em>.ms</code>. It is possible to specify multiple correlator output files on the <span class=“command”><strong>j2ms2</strong></span> command line. The visibilities in these files are simply concatenated and written out into a single MeasurementSet. </p><p> To convert data into FITS-IDI such that it can be read into AIPS, the <span class=“command”><strong>tConvert</strong></span> program can be used.
</p><pre class=“screen”>
$
<span class=“command”><strong>tConvert <em class=“parameter”>
experiment
</em>.ms <em class=“parameter”>
experiment
</em>.IDI </strong></span> </pre><p>
The resulting FITS-IDI can be read directly into AIPS using <span class=“command”><strong>FITLD</strong></span>. </p><p> Note that at JIVE we run some additional post-processing tools on the MeasurementSet before converting data into FITS-IDI. The most important things are: </p><div class=“itemizedlist”><ul class=“itemizedlist” style=“list-style-type: disc; ”><li class=“listitem”><p> Amplitude correction for a-bit data; currently <span class=“command”><strong>j2ms2</strong></span> assumes all data is 2-bit. </p></li><li class=“listitem”><p> Flagging of delay-rate zero events. </p></li><li class=“listitem”><p> Flagging of data with low weights. </p></li></ul></div><p> If you correlate and convert your own data, you may have to take care of these things when reducing the data in AIPS. </p></div></div> </HTML>