To: evntech@jb.man.ac.uk
From: szomoru@jive.nl
Date: 12/17/2006 6:57 PM
Subject: EVNtech: e-VLBI 14-12-06

Hi all. A quick report on our latest e-VLBI run.

Last Thursday, 14-12-06, we had a 26 hour e-VLBI session. Participating stations were (as usual) Wb, Jb, Cm, Mc, On and Tr, data rate was 256 Mbps. The session started at 1100 UT with a few hours for testing and setting up, the science part was to start at 1530 UT. Three science projects had been scheduled, all requesting 6 cm e-EVN at 256 Mbps.

The short summary:

Most of the run went extremely smoothly, with many hours of perfect boredom while the data streamed in. During a short period we experienced problems with the connections to Tr and Mc, but these were resolved without losing significant amounts of data. Problems at the correlator were few and quickly corrected, again without losing much time and data.

The slightly longer summary:

During the test phase the clocks were found, and some problems with the polarizations of Cm were corrected. We also established that 256 Mbps could be easily achieved from all stations. On had to be stowed for a short period because of high winds but eventually was able to participate again. While testing we lost connection to Wb several times. This seemed to be caused by a hardware problem in the Mark5 at JIVE, and as we did not want to risk any problems during the science run, we decided to connect Wb to one of the other Mark5s. Unfortunately the connection to Wb uses SX optics (all the others are LX) and is part of a different subnet, which means we had to physically swap the interface board and modify routing tables and such. This, and some real-time high-stress debugging, caused us to lose the first hour of the first science observations. But I do not think we had any choice in the matter.

After these problems were resolved the first run was perfectly uneventful until the end at 0130 UT (9 hours). All stations sent data at 256 Mbps, and the only breaks occurred during re-starts (the run was divided into blocks of roughly two hours). These re-starts were mostly in the middle of slightly longer observations of fringe finders, and as a result hardly any data was lost.

The start of the second project was at 0300 UT (the time in between was filled with fringe finders). This project ran very smoothly as well, until we started losing connection to Tr. The Mark5s on both sides were reset several times, but in spite of this the amount of data that was lost was limited, as we kept the job going with the remaining 5 stations while trying to fix the connection to Tr. At a certain moment we decided to let the job run without Tr until the next “natural” break in the observations, and at that point we reset the whole system and succesfully included Tr again.

The last part started at 0800 UT and lasted until 1300 UT. Here we encountered some problems with Mc, and as a result lost some 40 minutes of data. After this, the project ran without any problems.

Exact statistics of the whole session (percentage of data lost and so on) will follow when Zsolt returns from his vacation.

On the down side, we found that there were problems with at least one of the two subbands of Cm, and possibly with both. This should definitely have been detected during the test phase.

In conclusion, a very succesfull run. As illustration of the strength of e-VLBI, the PI of the second project was present during the observations and processed his data nearly on the fly. A little after the end of the run he received the final data product and found that all of his candidate secondary calibrator sources were detected.

We are also getting a better idea of how to minimize data losses when various problems occur, and I have started summarizing this in an operations check list.

Thanks to all, and well done. Things are looking good and with 512 Mbps coming up and Effelsberg, Yebes and Shanghai coming online, next year will look even better.

Cheers, Arpad