Performance and Reliability of the EVN; November 2007 ====================================== Session 3/2006 -------------- The session consisted of two frequency-band blocks (C,X), each with an NME including an ftp test. The ftp test data were processed in a semi-automated way. Several stations reported brief data losses due to problems with either Mk5 or FS control computers which required rebooting. Station and correlator feedback for individual stations: Ef - Subreflector drive failed during N06C3. This severely reduced the gain and produced large pointing errors. Time was lost due to work to fix the subreflector during experiment EL033D. Wb - The station reported: "Total power readout on VC08 was very bad during the whole session." However, this did not have effect on fringes detection. Last scan of N06C3 was lost due to operator error. Mc - GC028: VC08 was wongly connected on the patch panel for a few hours. No fringes detected on VC08 during that time. Nt - EL033D: The subreflector was reported in a wrong position several times in the observation, probably with effects on the pointing. EF017: From the beginning of the experiment until about 00:00 UT (4.5/7 hr), data lost due to problems with the active surface. Tr - N06C3 was observed with about 1.5 minutes offset of telescope computer time (this was discovered after the first ftp report). BBC5 was unlocked throughout this experiment. Hh - During EF017 and N06C4, RR pol was flagged in BBC#8 because of data corruption. Cm - Sampler analog inputs to the formatter had been transposed, most likely during a PSU repair, leading to an apparent re-ordering of tracks as though two head-stacks had been "swapped". Ar - GC028 sched was re-run on a key file, leading to a slightly different observing schedule than used by other stations. Ar had a one-version older stations.dat that had a different Az slew-rate for Ef. This introduced slightly longer gaps bewteen scans in a few places, eventually working up to a ~40s difference in the time of source-changes. Session 1/2007 -------------- The session consisted of four frequency-band blocks (C,K,L,5cm), each starting with an ftp fringe test experiment and an NME including an ftp test. The ftp test data were processed in a semi-automated way. Bad weather and strong wind affected many stations during the whole session. Some stations reported lots of RFI at L-band. Several reports were made about data losses due to problems with either Mk5 or FS control computers which required rebooting. Several scan_check errors reported from many stations. Problems also reported about disks and Mark5 disk modules. In several occasions, the problems occurred when switching disks. Effelsberg and Torun lost some time due to cable wrapping during some experiments. The cable wrappings were not predicted by SCHED. Station and correlator feedback for individual stations: JB - The second of 3 disk-packs didn't record properly during EF018C. EB032B seems to have no fringes in LCP in IF 3-4 (corresponding to BBC3) in the geodetic blocks only. No fringes detected in all the EB032* experiments on IF 3-4. EB032A was in session 2/2006 and EB032C in 2/2007 (see below). Ef - Experiment EK024A was not observed because the mechanism to change between secondary and prime focus receivers failed. No fringes detected for the experiment EF018C. Wb - In EA037B, one of the 14 telescopes (RT3) was mispointing. On - Power failure reported during EF018B. During EO005, the telescope was unattended for a few hours. The Mark5 system hung during that time and 2 hours of data were lost. The second of three disk packs of EL034 were unable to play back. It was also discovered that there were occasional 180degree phase jumps between scans in L-band experiments. The cause was traced to resetting the first LO before each scan. Since the first LO would be in the range 1-2GHz, any such 180deg phase jumps arising during the reset would feed directly into the data. At higher frequencies, the first LO would typically be doubled some number of times (i.e., C-band 4800 = 1200*4), thus such a phase jump in the first LO would correspond to full cycle(s) of phase, and not be visible in the phase(t) data. On took steps to prevent this, and the problem did not recur in session 2/2007. Mc - The LO was not locked during EF016: all data flagged. Wrong LO at the beginning (0.5 hour) of EA037A. Corrupted disk caused the loss of one hour at the beginning of the experiment EF020B. Nt - EA037A and EF018C: BBC#8 unlocked. BBC#8 had problems during the whole session. Tr - In EK024B, the cryogenic system failed so that Tsys began to grow gradually. The first hour of EA037B was lost due to a mistake of the operator. GT007 failed due to a disk pack failure and the L-band cryogenic system failure. At 5cm, only one receiver (one channel: LCP) worked. The recording Mark5 unit failed in the latter part of the experiment EL034 (3 hours lost). Session 2/2007 -------------- The session consisted of four frequency-band blocks (C,L,5cm,S/X), each starting with an ftp fringe test experiment and an NME including an ftp test. Due to the breakdown of the cluster computer running the NICT software correlator, only the first four subbands could be checked and the cross-hand correlations were not produced. In parallel, the JIVE Software correlator has been successfully used. Some stations reported significant RFI, in particular Wb. In some occasions, Wb had to restart the array. Several reports were made about data losses due to problems with either Mk5 or FS control computers which required rebooting. Several scan_check errors at many stations. Problems also reported about disks and Mark5 disk modules, in particular at Sh and Tr. Station and correlator feedback for individual stations: No feedback received from Cm and Jb. Wb - During EG037A, there was a severe problem with the Westerbork file server which compromised most of the data. The file server had to be replaced on Friday evening 1st June. The file server had also given errors on Thursday. We recorded only from 15:30 to 16:44 due to an operator error who had not enabled 2nd bank while checking the local system). Observations restarted at 20:49UT, but most of the data from 20:49 to 00:30 gave large numbers of data_check errors. Ur - The compressor of the C-band receiver did not work during EG037A, therefore a room temperature receiver was used. Sh - One of the disks in the mark5 module was 'slow' in EG037A (a Gbps experiment). This corrupted all the data. Problem with the LCP receiver caused extreme variations in the receiver temperature during EG037B. In EG037B, strong interference produced by a poorly shielded flat-cable was detected. This problem is a known issue for Gbps recordings: Stations need to remain alert to the environment in which their data cables lie. Together with a problem with the MK5 disk pack (as in EG037A), this interference made the correlation very difficult. Ef - VC05 not working for most of the session. In the experiment EG037A, no valid data before the system was restarted: more than 6 hours lost. Tr - At the beginning of EL035A, there was an error in writing to the disk-pack. The station reported that: "Since it was the second similar case, when problems appear while trying to continue recording on a pack used earlier by another experiment, Greg has inferred it may have something to do with the newest version of Mk5 system (version 2007y120d00h) installed before one of May e-VLBI experiments. Therefore, immediately after this experiment completion he has reinstalled the older version (2005y147d11h)." The experiment EH020 was observed with swapped polarizations. The experiment EB034C was not observed because of local scheduling error. At L-band, the BBC#7 had very low sensitivity. On - One hour lost at the beginning of EL035B because of missing operator. The data of EB034B were corrupted due to an error of the operator. The station could not take part to EB036 and EB032C because of antenna hardware problems. Mc - The antenna did not observe the experiment GI004 because of failure of the antenna motor drivers. Hh - Focusing mechanism failed during GI004, therefore the first hour was lost. Cm - No fringes detected in EK025. The FS log indicated formatter problems. Jb - No fringes in BBC#3 for ES057A, EI009A and EI009B. Also no fringes in the geodetic mode in EB032C. Giuseppe Cimo' for the Science Operations and Support Group (JIVE)