Performance and Reliability of the EVN ====================================== EVN Session 3/2009 ------------------------ The session consisted of five frequency blocks (6cm, 1.3cm, 5cm, 18cm, 7mm). In each block, ftp-fringe tests were carried out before user experiments. Furthermore, separate NME experiments at 6cm, 5cm, and 7mm were also performed.The automatically data extraction was affected by the problem of "scan_set" at Nt, Mc and Ur. Due to no "tpicd" messages and no "preob" in the FS log files of Mc, Tr, and Nt in N09M3, Tsys data were missing during the "midob" and the JIVE software log2vex cound not parse it properly. These stations have been required to update their FS, MK5 SDK, firmware, and OS in the last TOG at Bonn on 4 Dec. 2009. There were 20 user experiments (5 at 6cm, 2 at 1.3cm, 6 at 5cm, 6 at 18cm, and 1 at 7mm) in the session. There were 5 1-Gbps expriments (EM077A, EP066, GV020A, EC029A & B) scheduled and observed. Four Global VLBI experiments (GV020A, GB065B, GF015A & B) were performed. There were 3 experiments (EM077A, EP066, & GB065B) correlated at Bonn. A disk-recording ToO experient (RE001) at 18cm was carried out on 1 Feb 2010 (out of session). To test binning and gating correlation with the JIVE SFXC software correlator, a test experiment (FP003) was carried out. The post test correlations shows that the binning correlations works well for pulsar B0329+54. Station and correlator feedback for individual stations: Ef - Hard drive failure of the antenna control computer caused a delay of 3 hours in EB042. Missed about an hour (~0440-0540) because of a Mark5 hang during a pack change in EC029A. Wb - Single dish was used at 5 cm. As the absence of 0.5 MHz filters in the backend, 0.625 MHz filters were used in ES062A. On - The noise source did not work stably in the session. A new one has been installed after the session. It lost several hours due to communication problems with the antenna in ES062B. Due to strong wind, it started observations about 6.5 hours late in ES063A and 5.3 hours late in EB039E. Tr - No fringes in ES062B and N09M3 due to 1 MHz LO offset.The reason was reported by K. Borkowski on 23 Mar 2010. No fringes in the ToO experiment RE001 probably due to the LO offset too. Lost signal for half hour (9h10m -- 9h40m UT) in ES063B due to a problem with a spontaneous switch in the distribution system on the antenna. Nt - Recording failure prevented recording of the data for more than 4 hours in EB042. BBC 7 (IF 7-8, RCP) was dead in GF015A. No fringes in BBC 7 & 8 in EB044A & B, in BBC 8 in EB039D. Period of non-detections ~10h30m -- 12h30m and ~13h20m -- 13h50m in EC029A, which might be related to the FS system. Formatter was not synchronized from 04:30 to 05:30 UT in EM078. Mc - Maser problem (12 -- 14 UT) reported in the feedback, no loss of coherence of coherence seen in the data, though. Missed the first couple hours because of Mark5 pack problems (0400-0646) in EC029A. Due to a problem with the Mark5 recorder, MC lost scans from 05:00 UT to 07:16 UT in GF015B. No fringes were detected in the ToO experiment RE001. Ur - There was a clock jump in scan 77 in EB044B and two formatter jumps at ~12h45m UT (falls out at scan 202) and at~ 15h49m UT in EP067. Due to no internet connection, Ur did not participate in any ftp-fringe tests. Sh - The 2nd head connection between formatter and MK5A was on VLBA indead of MK4 connection. The problem was solved after F09C2. Ys - Because of technical problems, it started 8.5 hours later in EP066. Mh - Participated in N09K3, and EA041 at 1.3cm. Jb - Poor sampler statitics in some BBCs were reported in the NME experiments. The Lovel telescope ran properly in all the required experiments. Cm - Replaced by Darnhall and Knockin at 6 and 18 cm. Dn had no fringes in IF3 in N09C3, EB044A, EB044B. Dn had a power failure at 13:44 UT in EP067 and was still off in the later experiment EM078. Ro - Participated in EC029B for first 4.25 hours, EM078 ang GB065B. Ar - Participated in only one experiment: GV020A. e-VLBI Sessions ---------------------- 19-20 November 2009 Ef: OK in first part. Ef had to be re-routed during the night, and (as a consequence) it was limited to 512 Mbps in the second part. Jb: The UK telescopes were parked for most of the time because of high winds. Ar: Started at 512 Mbps, which had to be dropped to 256 Mbps later on due to bandwidth restrictions after 10 UT. Near the end of the Arecibo part connection was completely lost to the telescope. 10-11 December 2009 Wb: Serious power failure at the station caused loss of fringes sometime after the start of EL039A. The problem was fixed during the e-VLBI run. Jb: No data because of a network problem in the UK. 27-28 January 2010 Ef: Antennas was stowed for a long time because of snow. There was network maintanance at Ef for a half an hour, during which the data rate was limited to 512 Mbps (very little loss, but still unfortunate timing). 10-11 February 2010 Ef: Due to a network problem, 1024 Mbps could not be maintained. The data rate had to be decreased to 512 Mbps. It was snowing in most of Europe and Ef had to stop for extended periods during the run. Tr: showed amplitude/phase instability in some parts of the run (reason not known at the time). Jb: Jodrell MkII sent data but there were no fringes during the run. There were synthesizer and receiver problems reported. At 14:55 UT there was a serious power failure affecting all the UK telescopes (Jb2, Cm, Kn). Wb: Westerbork had a serious cooling machine problem and started late in the run. Mc: The Medicina gps-formatter clock-offset was variable before the run. This was fixed in the test part, but the clock jumped again. Formatter had to be resynced again during the run. The problem was fixed just before RY001, a ToO project in which Mc was one of the key telescopes (target had very low declination). Ar: the telescope had a structural problem, could not participate. Snow in the US prevented the "tiger team" to travel to Porto Rico and fix the problem before the run. 30-31 March 2010 Tr: Could not participate because of a serious control computer crash. Ys: Could participate only during the daytime (no operator available). Mc: AIPS IFs R3-R4 apparently had LCP data instead of RCP. Note this was not a polarization swap. The other channels were not affected. Ar: the telescope had a structural problem but participated in the 30-31 March run (albeit with limited ZA coverage). JIVE Correlator --------------------------- 1) for some reason, there were considerably more cases of mismatches between internal & external VSNs in incoming packs this past session: by my count 6 coming here & 1 going to Socorro (at least they only complained [loudly] about one). The usual culprit now is the "+"/"-" in the VSN. Tr made 6 new 8TB packs coming here, and at least one going to Socorro -- they all should have been "+" in their system, but 3 for us & 1 for Socorro were "-" inside. Jb had a case (also a new 8TB pack) of "-" outside & "+" inside (they had 3 others going to Socorro, for which the FS log (reflecting the internal VSN) had "-"). Mc had the opposite: like Tr a "+" outside and a "-" inside. Nt managed to put inconsistent bar-code labels front/back on a pack (the wrong one was a valid VSN for another pack incoming this session). This is annoying for us (& Socorro, apparently) since the expectation of incoming packs & their association with experiments is set by the internal VSN via the FS logs, while the entry into the physical storage (& TRACK) based on the external VSN. Unless there's something obviously wrong with one or the other, it's easy for packs to slip through the cracks (expecting something that's already arrived, but hard to find without knowing the external VSN). At least these "+"/"-" transpositions are relatively easy to check for if one knows to look. The previous cases in which a VSN was tranferred from one pack to another while conditioning two at once was more pernicious. As far as I know, there were no instances of that this past session (but we certainly haven't tried to start any more than a small fraction of experiments yet). I'll harangue the stations about this closer to the session. 2) One of the new Tr 8TB packs was giving really bad playback; scan_checks associated with in from the FS logs show a large number of missed bytes. Unfortunately, it was the 24hr Gbps experiment that we tried to start with for which this pack participated -- jumping around to other Gbps experiments using other of the new packs gives much better results. 3) The problem from sessions 2-3/2009 in which there were no fringes to Tr in only some 5cm experiments seems to have been tracked down by Kaz to a situation in which they run some single-dish observation in the EVN gaps (the 5cm sub-sessions can have meaningful gaps, since almost everybody is looks in the Galactic plane at around the same l). It turns out that the LO set in sched for 5cm is 5900; in their single-dish observations it's typically 5901 -- and the first LO setting apparently isn't under field-system control. We went back and correlated some scans from N09M3 (originally no Tr fringes) with a 1MHz LO offset, and fringes were fine. Of course, the usual 5cm observation is using 2 or 0.5 MHz sub-bands, so even being able to correct for the 1MHz LO offset wouldn't really help salvage the observations of the maser lines, which sched would try to center in the band. Still working with Kaz to try to avoid this kind of thing in the future -- my current pet scheme would be to change the LO in the skd files (i.e., freq.dat) for Tr 5cm to 5901 to match -- they have 500MHz IFs, so this should be entirely benign and hopefully remove the risk of them not handling the EVN/single-dish transitions properly every time. 4) An uncomfortable amount for formatter/maser jumps or re-sets in ongoing experiments in the past sessions/e-VLBI days (Ur, Mc, Nt). Nt had a H-maser causualty in the L-band part of session 1/2010, and went to Rb, and fixed the H-maser in the 5cm part. Grep'ing/plotting the gps-fmout from the FS log gives a reasonably good overview of what was happening. (as an aside, still no gps-fmout at Tr, which would make diagnosing formatter/maser behavior on <1d timescales hard) 5) Tr seems to have really low amps in RCP channels from a couple C-band experiments in session 1/2010, but not all. We haven't looked at enough to get a firm idea of what all the correlations are (isn't just the worse playing-back pack; isn't explicitly correlated with Gbps; etc.). Some problems in Session 3/2009 were reported in the last TOG at Bonn, on 4 Dec. 2009. Since Session 1/2010, Urumqi has started to use MK5B recorder. Through ftp-fringe tests of Session 1/2010, we found strong fringes to all the participated stations except for CM. Ur did not participate in any ftp-fringe tests again due to problem of internet. The Russian QUASAR stations Bd and Zc have also participated in the FTP-fringe tests and NME experiments at 18cm and 6cm. The first ftp fringes to Bd and Zc were successfully detected during the observations. We detected the first EVN fringes on the longest baseline to the Sheshan stations at 5 cm in N10M1. Jun Yang, Zsolt Paragi, and Bob Campbell Science Operations and Support Group (JIVE)