JIVE Wiki

TE045

Date: 26 Oct. 2006 Time: 12:00-16:36 UT Participating: Cm, Jb, Mc, Tr, Wb14 Operator: Arpad

Tests soon before this run were done to prepare the new control computer for e-VLBI. A short run was done on 25 October to verify the system, that worked well. However there were network problems with Medicina and Torun. These were eventually sorted out. Cambridge reported a MkIV rack power supply unit problem -this could not be replaced in time. The main goal of TE045 was to have long jobs at 256 Mbps to see whether we can sustain this data rate without problems. Since this was a success, the following science experiments (RP003&4) were run at 256 Mbps as well. Unfortunately, Onsala could not take part in the observations because of a storm.

RP003 & 4

Date: 26/27 Oct. 2006 Time: 17:00-08:38 UT; 19:00-10:38 MET Participating: Cm, Jb, Mc, Tr, Wb14. Operator: Martin (17:00-24:00 MET), Zsolt (0:00-8:30 MET), Hans (8:30-10:38 MET)

Correlation setup information:

setup = prod_256 vex = rp003_256.vix obsdef = lags = 64 crosspol = 1 bocf = 16 speedup = unset maxchan = unset head = unset int = 1 telescopes = WbJbTrMcCm validity = Local firstjob = unset jccs = base

Below is detailed information about the jobs, with runtime comments. The job number, start and end scans, production, job status, lag number, integration time, cross-hands, and correlation statistics: number of skipped integrations/number of integrations with mismatching header information/number of integrations with some missing interferometers/ total number of interferometers/total number of integrations are listed; the percentage at the end shows what portion of all the interferometers were being correlated. Less than 100% means loss of one or more telescopes. Time is in MET.

Operator: Martin/(Zsolt) Telescopes/SU relations: <><><><>Mc<>Tr<>WbJb<><>Cm<><><>

610261902 No0001 No0001 PROD ABORT 64 1.000000 Xpol 0/0/0/220/102 80.0% First start did not go well, must restart the job; Cm problems.

610261908 No0002 No0002 PROD ABORT 64 1.000000 Xpol 0/42/0/220/43 80.0% Red LEDs for Cm - trying again. Agreed to restart Cambridge Mark5A.

610261922 No0003 No0004 PROD GOOD 64 1.000000 Xpol 0/0/0/220/560 100.0% Cm does not work still. Stopped the job and rebooted the local Mk5 for Cm.

610261940 No0006 No0007 PROD GOOD 64 1.000000 Xpol 0/0/0/220/297 100.0% Trouble with Cm and MC as well. Rebooted some local Mark5As. Correlator processes also died.

610262002 No0010 No0012 PROD GOOD 64 1.000000 Xpol 0/0/0/220/426 100.0% Tr, Jb, Cm fringes -Wb does not see the source yet. Local Mark5A for Mc crashed.

Medicina moved from SU4 to SU11 Telescopes/SU relations: <><><><><><>Tr<>WbJb<>McCm<><><>

610262017 No0014 No0014 PROD ABORT 64 1.000000 Xpol 0/13/0/220/14 100.0% It turned out that the TSP unit also crashed.

610262024 No0016 No0030 PROD GOOD 64 1.000000 Xpol 0/196/603/220/2829 100.0% Wb sees the targets - fringes to all.

610262115 No0031 No0067 PROD GOOD 64 1.000000 Xpol 0/0/0/220/6899 100.0% Lost Jb -started a new job.

610262314 PROD ABORT 64 1.000000 Xpol 610262321 No0070 No0070 PROD ABORT 64 1.000000 Xpol 0/30/0/220/31 80.0% 610262331 PROD ABORT 64 1.000000 Xpol 0/0/0/220/130 0.0% Things crashed during a job start; restarted correlator and JCCS.

610262343 PROD ABORT 64 1.000000 Xpol 0/0/0/220/115 0.0% Trouble with Mc - asked station for a reboot.

610262355 PROD ABORT 64 1.000000 Xpol 0/0/0/144/101 0.0% Correlator trouble.

Operator: Zsolt only

610270016 No0081 No0081 PROD GOOD 64 1.000000 Xpol 0/0/0/220/60 100.0% Mc problem solved - the correlator not yet. Eventually correlator fixed. Job started but Tr Mk5 seems to be dead.

610270027 No0083 No0096 PROD GOOD 64 1.000000 Xpol 0/0/0/220/2708 100.0% Lost Cm, reconnection did not work; will stop job and restart lokal Mk5 after pref scan This was a nice long job without Cm. Cm Mark5A restarted by Paul from home. Restarted local Mark5A for Cm.

610270142 No0104 No0118 PROD GOOD 64 1.000000 Xpol 0/0/0/220/3387 100.0% Before this job ~1/2 hour was lost due to an SU12 or Cm Mk5 problem. Same happened at the end of job - diagw? (It was not a diagonal weight.) Cm Mk5 or SU trouble again - try to reset only the SU first.

610270246 No0120 No0120 PROD ABORT 64 1.000000 Xpol 0/0/0/220/57 100.0% … did not help - restart Mk5 software … reset Mk5 …

610270309 No0126 No0132 PROD GOOD 64 1.000000 Xpol 0/0/0/220/1275 100.0% Wasted ~20 minutes on this, Cm still not OK; must restart there, but I cannot… Tr has a bit more dropouts in this job. Jb controller woke up Paul again. Cm Mark5A restarted.

610270339 No0133 No0134 PROD FAIL 64 1.000000 Xpol 0/0/0/220/82 80.0%

Cambridge moved from SU12 to SU4. Telescopes/SU relations: <><><><>Cm<>Tr<>WbJb<>Mc<><><><>

610270347 No0136 No0141 PROD GOOD 64 1.000000 Xpol 0/0/0/220/1009 100.0% Cm restarted their Mk, I connected it to SU4 - green LEDs! -however I see no fringe for Cm… Tr fell out, then Cm soon after…

610270416 No0143 No0165 PROD GOOD 64 1.000000 Xpol 0/0/0/220/4686 80.0% Cm completely red - but since there are no Cm fringes, I let this job run!!! Jb controller restarted Cm - however controlEVLBI says Cm is not online (data ip).

Cambridge moved from SU4 to SU12. Telescopes/SU relations: <><><><><><>Tr<>WbJb<>McCm<><><>

610270542 No0167 No0170 PROD GOOD 64 1.000000 Xpol 0/0/0/220/760 100.0% Trying to get Cm back - now on SU12. Works! SU6 Mark5 (Tr) crashed. Wow! -controlEvlbi crashed.

610270608 No0173 No0175 PROD GOOD 64 1.000000 Xpol 0/0/0/220/484 100.0% Tr still has problems. Rebooting remote Mk5?

610270634 PROD FAIL 64 1.000000 Xpol 610270646 No0181 No0209 PROD GOOD 64 1.000000 Xpol 0/0/5/220/6049 100.0% After calling Arpad, I could start this job. Green LEDs and some 3C84 fringes. But not to Cm! -Jb claims there are fringes for Cm in MERLIN. Now they realise that they have had a communication failure… back ~5:40UT?

Operator: Hans

610270831 No0211 No0219 PROD GOOD 64 1.000000 Xpol 0/54/401/220/1971 100.0% 610270913 No0221 No0226 PROD GOOD 64 1.000000 Xpol 0/0/0/220/2699 100.0% Mc Diagonal weight

610271005 No0227 No0228 PROD GOOD 64 1.000000 Xpol 0/0/5/220/1348 100.0%

The experiment was a success, the first 256 Mbps science e-VLBI run, and the first with joint MERLIN observations. But the two projects suffered from a significant data loss (RP003 - 30%; RP004 - 20%). This was due to restarts because of various connection problems, first with Cm and Mc Mark5a, and later again with Cm. The data rate itself did not seem to be a problem. Torun suffered from data rate problems from time-to-time, but this always fixed itself in a minute or so automatically. During the whole night, we had only one diagonal weight. The TSP failure in the beginning of the run was not discovered early enough, partly because the operators did not think of this problem, and partly because the TSP unit cannot be seen from behind the control computer.

General notes on data quality: both target sources were nicely detected in the pipeline. The increase of data rate from 128 to 256 Mbps compensated somewhat for the losses. Medicina used a new 5 GHz receiver, it worked well. The Mc D-terms became significantly lower. The amplitude calibration of the run was awful. Cm has no Tsys, Mc receiver new, Jb Tcal seemed to be completely wrong (they use a different receiver every time). This ampcal problem should be somehow solved for e-VLBI.