Report of the Global 3mm-VLBI Session May 11-15 (day 131-135), 2007 =================================================================== General: PdB observed only the first 12 hrs on day 131, then stopped VLBI observing by IRAM directors decision. After fringe verification on next Monday morning PdB resumed observing on May 14/15 (day 134/135) for SgrA*. EB, PV, ON, MH, VLBA observed without major problems. The whole experiment was scheduled in dual polarisation. EB, PV, PdB and VLBA support dual pol. Onsala supports only LCP. MH uncertain polarisation quality, LCP should be better than Oct. 2006 More details: ------------- Effelsberg: day 131 BLLac Rain, wind, 6-14h30 wind stop OJ287 ok, but still windy and cloudy day 132 SgrA* lost, heavy rain 3C454.3 weather not good, 3hrs lost due to wrong focus day 133 0716+71 good, clear sky, 5 scans lost to to RX problems 1803+78 ok, weather moderate, last 2 hrs lost due to thunderstorm day 134 SgrA* clear, but all lost due to receiver problems 3C84 receiver repaired, weather moderate, partly clear day 135 SgrA* rain, no usefull data Pico: day 131 - day 133, 17h00 good weather, only few scans lost day 133, 17h00 - day 134 03h30 no observing, high wind (affects last part of 1803+78 and SgrA*) day 134, 03h30 - end all ok Onsala: basically ok. variable weather, times of high Tsys Metsahovi: Metsahovi completed according to schedule UT 11:07. We had surprisingly good weather and no major apparent failures except two freeze-ups of the Mark5A software. The first one caused the loss of just one scan No0257, but the second one went unnoticed for a longer time: 2007.132.20:15:00.03/disk_pos/1052816846032,1024725916856 - last successful recorded position, end of scan No0313 2007.133.08:30:00.00/disk_pos/0,0 - resumed recording with scan No0359 So Saturday UT 20:15 -- Sunday 08:30 was lost. Initially, it seemed like the second freeze also erased the 4TB 8-pack which was being recorded when the freeze occurred (and already had 1TB of recorded data), but that was fixed with 'mk5=recover=2' as Martin advised on evntech. VLBA: bassically ok, day 133: NL light clouds; BR overcast; others good day 134: most stations have good weather, PT light clouds during end: FD rain, PT overcast (affects 3C84) in detail: VERY LONG BASELINE ARRAY OBSERVING LOG -------------------------------------- Project: C071A Observer: Krichbaum, T. Project type: VLBA Obs filename: c071acrd.* Date/Day: 2007MAY11/131 Ants Scheduled: NL FD LA PT KP OV BR MK =UT-Time===Comment===============================================MF#===%AD==AMD= Vclock, gps delta, UT1-UTC all check OK. Operator is Betty Ragan 0456 Begin at all stations except MK. 0505 %LA tsys look a little high at 3mm. Weather is cloudy. 0530 Begin at MK. 0537-0538 *MK focus drive fault. FRM 100 1 0545-0546 *MK focus drive fault. FRM 100 1 0555 Operator is Anthony Sowinski. 1236 FD reports clear, dry, no precip. 1345 New operator: T. Perreault 1425-1431 *PT autostowed. Power glitch. Repointed. COM 100 6 1514 OV reports clear skies. 1532-1538 *KP not observing. Site techs have taken antenna 20278 100 6 to work on 1cm. 1609 %NL 3mm receiver slightly warm (15K stage about 30K). 1735-1746 *FD Mark5 pack MPI-0503 in bank A claims to be full OTHER 100 11 with 0.00 Gb used out of 4000 Gb available. Pack in bank B (JOD-0020) is full; the error happened when recording switched from bank B to bank A. Email sent to Mark5 support. Called site, no tech answered the phone. 1743 FD sent an erase command to the pack MPI-0503 and this cleared the problem. Started recording data on the pack at 17:45:05 UT 1749-1751 *OV Mark5 pack MPI-0510 exhibited the same OTHER 100 2 problem as FD MPI-0503 did when switching recording from MPI-0150 (bank A) to MPI-0510 (bank B). Sent an erase command and this cleared the problem. 1830 %NL 3mm receiver temperature has stabilized, still running a little warm. 1846-1852 *PT autostowed, commercial power glitch. Repointed. COM 100 6 1918-1920 *KP autostowed. Power glitch. Repointed. COM 100 2 1947 MK wea: scattered cirrus, no precipitation 2058 KP wea: broken low clouds, no precipitation 2100-2102 *PT autostowed. Power glitch. Repointed. COM 100 2 2151 New operator is Betty Ragan. 2200-2208 *MK had some similar MK5 problems to other sites. OTHER 100 8 I switched over to 4000 GB pack MPI-0511 in bank B so they could switch out the module in bank A, but when they loaded MPI-0402 in bank A, the nearly empty module in B declared itself full and we began writing on the module in A... which apparently had not been properly erased, and was showing as nearly full. Sending an erase command to both modules seems to have set things right, but we lost several minutes of data. 2248-2251 *KP got AZ#1 & EL#1 armature/electronics CB COM 100 3 errors and autostowed. Probably a power glitch. Repointing. 2322-2325 *KP autostowed again. Power glitch. COM 100 3 2336-2339 *KP autostowed. Another power glitch. COM 100 3 2352-2353 *KP rotation drive fault. FRM 100 1 0000 Date is 2007MAY12/132. 0604 Operator is Anthony Sowinski. 0645-0651 *FD Focus position error. FRM 100 6 0712-0714 *KP Formatter Header Control Module #2 error. FMTR 100 2 1134-1136 *KP Tried to switch to MPI-0505 in mark5 bank 'B' REC 100 2 but module was write protected. Fixed by sending erase command. 1238-1239 *OV Formatter Header Control Module #2 error. FMTR 100 1 1349 New operator is Jim Ogle. 1613 KP reports clear skies. 2024 %High wind gusts at OV. 2033-2035 *PT autostowed, commercial power glitch. Repointed. COM 100 2 2049-2050 *KP autostowed, commercial power glitch. Repointed. COM 100 1 2123-2125 *PT autostowed, power glitch. Repointed. COM 100 2 2159-2200 *OV AZ#1 armature overcurrent fault; briefly AELEC 100 1 disabled the AZ axis to clear error then resumed pointing. 2248-2249 *PT autostowed, power glitch. Repointed. COM 100 1 0000 Date is 2007MAY13/133. 0130 New Operator: Paul Padilla 0620 *BR has precipitation 0905-0911 *KP ACU oscillation noted, cleared when 19993 100 6 put into standby momentarily 1242-1244 *KP ACU oscillation 19993 100 2 1300-1301 *OV autostowed, power glitch? COM 100 1 1322-1324 *PT autostowed in power glitch COM 100 2 1335 New operator is Jim Ogle. 1337-1339 *FD focus brake fault. FRM 100 2 1530 %High wind gusts at OV. 1655 LA reports scattered high clouds. 1803 OV reports clear skies. 1843 %KP FIFO errors on formatter? Checker messages for them popped up but it looks like they cleared on their own, they aren't on the FMTERR screen. 1924 %High wind gusts at LA. 2010-2011 *KP oscillating in azimuth again. Cleared after being 19993 100 1 briefly put in Standby mode. 2020-2022 *PT autostowed, power glitch. Repointed. COM 100 2 2026-2028 *PT autostowed again. Repointed. COM 100 2 2034-2035 *PT autostowed. Repointed. COM 100 1 2158 %High wind gusts at MK. 2303-2304 *PT autostowed, power glitch. Repointed. COM 100 1 0000 Date is 2007MAY14/134. 0135 New Operator: Paul Padilla 0135-1520 *NL 3mm cryo is warm. 20377 50 412 0215 %OV high winds 0337-0342 *KP oscillating again. Cleared after being put 19993 100 5 into Standby 0924 %KP 35+ mph caution 1258 %OV FIFO errors on formatter? Same situation as with KP earlier at 1843. Error seems to have cleared up on its own. 1315-1316 *KP oscillating again. 19993 100 1 1353 %LA reports high scattered clouds 1358 New Operator is Alan Kerr 1637-1642 *LA rot error. cleared. FRM 100 5 1750-1809 *PT not in point. repointed. COM 100 19 1822-1826 *MK rot error. 5 resets needed. kept stopping. FRM 100 4 1837-1842 *LA rot error. cleared. FRM 100 5 1837-1842 *FD rot error. cleared. FRM 100 5 1907-1908 *PT autostowed, commercial power glitch. Repointed. COM 100 1 1947-1949 *PT power glitch. repointed. Tech reports lightning COM 100 2 storm and to expect more power glitches. 1952-1956 *NL rot error. cleared FRM 100 4 1952-1956 *FD rot error. cleared. FRM 100 4 1952-1956 *BR rot error. cleared. FRM 100 4 1959 *LA precipitation starting. 1958-2000 *NL rot error. cleared. FRM 100 2 2002-2004 *PT power glitch. repointed. COM 100 2 2003 *LA tech reports a thunderstorm. 2009-2011 *PT power glitch. repointed. COM 100 2 2020 FD Wea: overcast. 2022-2024 *PT power glitch. repointed. COM 100 2 2020-2022 *FD rebooted itself. COM 100 2 2022-2032 *FD mark5 offline, not recording data COM 100 10 2024 FD precipitation starting. thunderstorm reported. 2024 LA precipitation stopped. 2043-2046 *PT power glitch. repointed. COM 100 3 2052-2055 *KP rot error. cleared. FRM 100 3 2105-2107 *PT power glitch. repointed. COM 100 2 2109-2111 *PT power glitch. repointed. COM 100 2 2120 New operator: T. Perreault 2128 KP wea: clear, no precipitation 2141-2143 *OV formatter problem. FMTR 100 2 Header control module 2 error HC2 FIFO error FIFO 0 1PPS detected Reset formatter 2142-2145 *FD power glitch at site caused station VME COM 100 3 computer to reboot. 2210-2214 *PT autostowed. Power glitch. Repointed. COM 100 4 2226-2230 *PT autostowed. Power glitch. Repointed. COM 100 4 2237 OV wea: clear, no precipitation 2316-2324 *FD power glitch at site caused station VME COM 100 8 computer to reboot. When rebooted, Mark5 unit came up OFFLINE, had to ONLINE before recording would restart. FD update: non-critical power is off, generator has 20381 not turned on. Contacted tech that is on the oncall list, he said to call the other tech (he forgot to update the oncall list). Contacted other tech, he is on his way out. 2324-2353 &FD non-critical power is off, generator did not 20381 100 29 come on following loss of commercial power to the site. Oncall site tech enroute. 2327-2331 *PT autostowed. Power glitch. Repointed. COM 100 4 2335 FD another loss of commercial power to site. Will remove site from observing for these last few minutes of the file - with non-critical power off, there is no sense of continuing to observe. 2353 End. Downtime Summary: Total downtime : 648 min Percentage downtime of observing: 1.5% Average downtime per hour : 0.9 min Total scheduled observing time (# Antennas): 43656 min (8) Notes: * = Entries where data was affected. % = Entries where data may have been affected. & = Entries where the site tech was called out. WEA = Weather entries. MF# = Maintenance form or major downtime category associated with a problem. %AD = The percentage of an antenna affected by a problem. AMD = Total antenna-minutes downtime for a problem. Tsys = System Temperature (TP/SP x Tcal/2) ACU = Antenna Control Unit FRM = Focus/Rotation Mount RFI = Radio Frequency Interference VME = Site control computer CB = Circuit Breaker vclock = Program that compares site clock time to a standard.