TOG Meeting, 2006 March 24, Dwingeloo. Performance and Reliability of the EVN ====================================== Sesion 2/2005 ------------- Since the last TOG meeting all the NMEs have been produced and pipelined from session 2/2005: N05L2 (ERI=0.90), N05L3 (ERI=0.56), N05L4 (ERI=0.67), and N05C2 (ERI=0.59). The drop in ERI after N05L2 was due to the Noto azimuth engine problem. Session 3/2005 -------------- Noto still could not join session 3/2005, because the wrong azimuth engine was shipped. They used its encounter with the old engine. After this fix they had a "communication problem", schedules could not be run. The Lovell Telescope was not ready by the start of the session. The MarkII telescope was used in the C-band experiments instead. The K-band session did not go very well. Urumqi and Shanghai had a sensitivity problem. The weather was also not good and this caused some trouble, especially at Metsahovi. In the L-band session there were a number of 1 Gbps experiments. Arecibo performed well (joined GL028A-E), but Westerbork and Robledo (in EP051) had serious problems recording at this rate. Some of these problems were probably related to RFI in the ribbon cable connecting the Mark5A and the formatter. Also, in Robledo fringes appeared only in the lower half of the channels (we had 4 out of the expected 8 channels per polarization). Several disk packs needed recover=2 to fix the recorder pointer position. Some of the disk packs that could not be recovered this way were sent to Conduant. This is why Jodrell Bank was left out from the first correlation of N05K2; later this NME was produced again with the corrected disk pack. There were two "trouble experiments". In EP053 the schedule was replaced at a late time by the PI, without changing the schedule version number. Wb and Sh observed the wrong version of the schedule. ES055 was scheduled as a single, long scan. There were no Tsys measurements taken during the experiment, and disk pack change did not go smoothly. All the NMEs have been pipelined from this session: N05C3 (ERI=0.74), N05K2 (ERI=0.36; ERI*=0.47), N05L5 (ERI=0.79). As in the previous session, the Noto problem lowered the ERIs for all the NMEs very significantly. Otherwise the C- and L-band NMEs went very well. Cm - Mk5 kept dropping communication throughout N05L5. VLBA rack power supply was nearing failure. N05L5 still had fringes, but most of the channels failed in EP051. EA035A was schedule was not run because of the power supply problem. Ef - Had some problems with the formatter, and sometimes lost soem data due to trouble with Mark5A. Reported severe formatter communication problems in EP051, but the data were in fact very good. In EA035A had pointing problems for a very short time. Wb - Had trouble with Mark5A before the session, Borrowed one from JIVE. Later it turned out that this had serious recording problems at 1 Gbps, probably due to RFI in the ribbon cable between formatter and Mark5A. In the L-band session Iridium RFI messed up calibration in some of the channels (around 1620-1630 MHz - affected the 1 Gbps experiments). Removed RT9 from two experiments because of delay hardware problems. Swapped polarizations in the 90cm session. Jb - The Lovell Telescope was not ready for the 6cm session, only started in the L-band. The MarkII telescope had azimuth drive trips in some of the experiments, each time a little data being lost. In GB055A stopped for an hour to check reason for low sensitivity. Had vacuum loss repeatedly for about an hour in GM057A. The Lovell Telescope azimuth drive failed completely in GL028A, a 1 Gbps experiment with only four telescopes. Lost the beginning of EK022B (the following experiment), and 5 hours from GB055B for the same reason. The telescope was parked for about two hours in ES055 because of high winds. On - Had tracking problems due to high winds in many experiments, and had to stow the telescope a number of times (EP051, GV017B, EA035A). Some RFI reported. Mc - Produced first ever ftp fringes in the K-band (after worked around a software correlator bug). Missed several hours in EP051 because antenna was not tracking due to an error in the antenna control unit configuration. Some scans were lost in GV017B due to "errors in positioning in the schedule". Had a power outage in GM057B, after which the tape footage counter was no longer synchronized with the schedule. GB052 feedback: "the pass length did not match the tape length. As a result, stations recording on tape lost the last scan of each pass." -? Nt - could not join the session. Tr - In GB052 lost half an hour due to accidental duplication of a Mark5A process. In EP051 BBC#5 lost 1pps - this problem had been fixed before the next experiment started. Was off-source for a short time in ES055 due to a power outage. Did not observe GB055B because there were not enough disks. GV017B was successful but the data were lost together with GM057B, in an attempt to fix a Mark5A problem by running sstest. Ur - Reported high Tsys in the K-band session - had no fringes in the ftp test and in most of the time in the NME, probably due to the low sensitivity (SEFD ~10000 Jy). In EK022A started recording late because of a power supply problem. Sh - Had no fringes in half of the channels in F06C1, due to a sampler problem. This had been fixed before the session started. There were problems with the cryogenic system in the C- and K-bands that resulted in high system temperatures. Had no fringes in the K-band experiments N05K2 and EM059A. In the L-band used a new, cryogenically cooled receiver. The polarizations were swapped in the first two experiments, otherwise everything went very well. Hh - Started late N05C3 because of a secondary reflector controller failure. RCP receiver was unstable as well, for about 2.5 hours. Ar - The lower sideband of BBC#8 had a fluctuating level and bad sampler statistics. Mh - Had massive power outages before N05K2, because of sudden rain, snow, and high winds. The receiver was warm, could not observe the NME, and was late from EM059A. Ro - Robledo produced first ever ftp fringes in N05L5. In EP051 observed first at 1 Gbps data rate. Produced good data in the first half of the experiment, but the upper half of the channels had no fringes. All data was lost in the second half of the experiment. May have had the same ribbon cable interference problem as Westerbork. Session 1/2006 -------------- Noto was finally back for session 1/2006. There were four band sessions, each with a pre-session ftp test, and an NME including an ftp test. The data were processed in a fully automated way, with minimal support scientist interaction. Started reporting on clock offsets from the ftp results. Webpages now include autocorrelation spectra for each station, and the correlation vexfile. The 5cm band ftp tests failed because the software correlator could not handle the recording mode (should have tested this in advance). The limit of the maximum number of users sending data to jop13 was extended to 50. The ftp tests solved some problems quickly: Jb had no fringes from VC#6 in F06C1 (fixed during N06C1), and Mc had no fringes from VCs#5-6 in F06X1 (fixed before N06X1). Torun had no fringes at all in F06C1. In N06C1 they had fringes in most channels, except for BBC#8. The weather was not good at all during the session. There was rain, snow, and sometimes high winds. In Torun the telescope was frozen for several days, they missed the 5cm experiments EB031A, N06M1, EB031B, and part of EF012. Effelsberg was down for several hours in EB031B, EF012, EM058A, EM061B and EM061C. Still in the 5cm session Jodrell indicated a failure in the feedback for most of the experiments. In F06M1, N06M1, EF012 and EM058A "offsets were incorrectly set hance sources probably out of telescope beam". In EB031B, EF012, EP054, EM061A-C, EM058A and EA035B they also had a disk pack problem, data were apparently lost (but might be recoverable). Note that the disk pack (JOD-0023) was sent from JIVE to Jodrell for a repair, not observing, but this was probably not communicated well. The C-band NME is being pipelined at the time of writing this report, the other NMEs have not been clock-searched yet. The pipeline has been replaced with a new, easier-to-use version written in ParselTongue (Python interface to AIPS). Cm - OK Ef - Many problems due to snow, 5cm was the worst affected (see above). In EF012 also had a receiver problem - stopped experiment to fix that. Wb - There were some tied array problems in EL033B and RO001B. In GC025A observed with a wrong polarization setup (half the pols were wrong). Formatter out of sync for half an hour in EB031B. Jb - Had a short telescope communication problem in EL033B, and a Mark5A problem for an hour in GM060. A disk failed in one disk pack during RO001B, but recording seemed to go well. Apparently had many failures in the 5cm session (see above). On - It was frequenctly snowing in Onsala. Had a short antenna problem in N06X1, and some trouble with the FS in EF012. Reported some RFI. Mc - Had antenna tracking problems due to high winds in ED026 and RO001A-B. Telescope was late by 1.5 hours from EP054 because of an antenna problem. Nt - Lost most of EL033B and the beginning of GM060 because the formatter was not synchronized. Stopped in RO001B because of high winds. Formatter gave false sync and slewing error messages in ED026, but things were apparently OK. Had pointing problems in EB031B (no details in feedback). In EM061A the pointing computer was several seconds late. Did not record for 5 hours in EX006 due to a disk pack problem. Tr - Lost an hour in EL033B due to a Mark5A problem. The telescope froze during the 5cm session (see above). Probably had no BBC#8 fringes during most of the experiments. Ur - Had weaker fringes from VC#7 in N06C1. In EL033B started late due to a terminal problem, then had trouble with a disk pack for half an hour. In N06L1 had a formatter problem for about an hour. In EX006 had some more trouble with the formatter, because the serial number was accidentally changed. Sh - Failed EL033B because of a formatter problem (exact reason not known yet, it was not a power supply problem). Lost the first ten scans in EX006 because of a disk problem. Hh - Very strong RFI in N06C1. The whole L-band session was also affected by RFI. Had some rain at the end of the 5cm session that resulted high Tsys. Had to stow telescope for some time in EX006 and RO001A due to high winds. Ar - There were ftp fringes in all channels and polarizations in F06X1, but the residual clocks compared to the GPS data were very large (-28.2 us). ED026 and GK034 apparently went well. Ro - No problems reported from GK034. Zsolt Paragi for the Science Operations and Support Group (JIVE)