EVN Technical & Operations Group (TOG) Meeting Monday 22nd November 2004 Jodrell Bank Observatory, UK Introduction ------------ The EVN TOG held a meeting on 22nd November 2004 at Jodrell Bank Observatory, UK. Approximately 30 people attended the meeting. The chairman was Alef. Gunn welcomed the visitors to Jodrell Bank. 1. Approval and additions to agenda ----------------------------------- No additions to the agenda were requested from the participants, although Alef added an item under Calibration concerning the period of Tcal measurements during observations. 2. Review of action items from last meeting ------------------------------------------- The TOG reviewed action items from the previous meeting. 1. Action: Friends to contact Gunn if there is a problem with a schedule - Gunn reported that, in general, people are doing this but reiterated the action 2. Action: Wunderlich make a new list of spares - this had not been done and the TOG asked Burgess to continue this action, adding a section for Mk5 3. Action: Pogrebenko to distribute the 'lost stack' software to the Bonn correlator - done 4. Action: All: try to get a patching system to avoid these problems and get it working before May - mostly done. Reynolds was asked to get modifications into SCHED - done 5. Action: Graham, Himwich and Gunn: produce improved K-band corrections (particularly opacity) - will return to this issue - K-band is still a big problem 6. Action: All: Check BBC linearity up to 25000 units and zero-level offsets - several stations not done this yet (contact D. Graham) 7. Action: Noto improve flagging data - this will be implemented by the February session 8. Action: Alef to contact EVN chairman to express the need for SATA connection capability in Mk5B - done 9. Action: Smythe to ask Conduant to send shipping boxes with each disk pack kit - added boxes and foam to Conduant price list, but this is probably not cost effective 10. Action: Alef take the Mk5B recommendations of the pre-meeting to board of directors - done. Stations go to Mk5B when NRAO correlator gets Mk5B. 11. Action: All: Check station SCHED catalog entries and send updates to Reynolds - some stations doing this but slew times need looking at because the accelerarion may be elevation dependent. Stations asked to provide, if possible, a better model of slewing (Ef, Jb particularly) 12. Action: Wunderlich to request the decoder firmware code - done 3. Reliability/Performance -------------------------- Paragi gave a report on the results of recent Fringe Tests and NMEs. Session 2/2004 went very well in general. The only outright telescope failure was the loss of Urumqi in the UHF-session. Urumqi also could not participate in most of the experiments because of the installation of their new 6cm receiver. There were 3 NMEs (N04C2, N04U1, N04L2) which have been all correlated and pipelined. In the C-band there was a successful Gbit test, and a special test for Mk2/Lovell at Jodrell Bank. Altogether there were 5 ftp fringe tests that went very well. Apart from the old-style NME reports we introduced formal fringe reports that are being sent out after these tests. The response time signifiantly improved in 2004. In general session 3/2004 went very well. In most cases, the ftp fringe tests were successful. In the K-band session Medicina, Noto and Urumqi did not produce ftp fringes and the disks have not been checked yet. Bad weather at the stations may be partly responsible for this. In Shanghai the 6cm ftp fringe test failed (file transfer problems). The Sh new 6 cm receiver worked well, but the polarizations were swapped. Jb also had swapped polarizations (this was not realized during the ftp test since Jb produced fringes, and the RL plots were not made for the C-band test). Action Item: Paragi to include cross-hand fringe checks as the default for software correlation. The Noto BBC#1 problem was discovered early in the ftp test and fixed by the station. Another major problem was the failure of the H-maser in Noto during the C-band session, but there were still fringes with the Rubidium. Darnhall joined the EVN instead of Jb2 in the 5cm session, and produced the first fringes since the MkII era. Two 1 Gbps user experiments were observed at 6cm. Alef asked if any stations have problems with 1Gbit modes. None were so far identified. It was considered necessary to get 1 Gbps support in the software correlator. Action Item: Paragi and Reynolds to investigate 1Gbps support in the software correlator. There were some concerns about the phase-referencing time limitation at the Lovell Telescope which is not sufficient for 6cm observations. The station reported that, unfortunately, this limitation would not be relaxed. However, a recent experiment demonstrated the successful use of the Mk2 to track phase changes throughout Lovell observations. How this will be used in practice is yet to be resolved. Action Item: Garrington was asked to discuss Mk2/Lovell phase referencing with the EVN PC and form a working process for users. Gunn asked that more information be given from software correlation, e.g. results for all 8 channels. Action Item: Paragi to provide more comprehensive feedback from software correlation but to give immediate 'fringe' feedback to station. Action Item: All stations should look at data regularly with chchk program, use it to locate significant RFI and report the results to Polatidis, and use it to check pcal throughout the session. Action Item: All stations should upgrade the decoder and use it to monitor pcal. Action Item: Urumqi was asked to provide automatic control of pcal as soon as possible. There was a discussion of the smaller ftp file sizes from Noto. It was thought this may be due to the preob procedure taking too long. Himwich asked that the preob procedure is sent to him for checking. Action Item: Noto station should check reasons for smaller ftp file sizes. Mk5 operations in the last session were discussed. No major problems were reported by stations. Gunn pointed out the discrepancy of the disk size of experiments in drudg output. Action Item: Himwich to locate total file space error in drudg output and fix. Smythe reported that the problem with wrong VSN from SSErase is now fixed (details on Haystack website). Stations should not now SSErase incoming packs, just write some data and check statistics. Do this towards the end of the disks since that is where recording is slower. Make a note of any slow disks since these may be marginal for 1Gbps. 4. Amplitude Calibration ------------------------ Reynolds gave a report on network calibration. At C band and X band the overall calibration quality is quite good. However Tr, On, Cm, Ur and Hh have significant errors. The Tr problem appears to be related to an error in the estimation of the gain curve, which was discovered in the summer of 2004. This has now been corrected and significant improvement is expected for experiments in session 3 of 2004. The Hh problem at C-band is at least partially the result of the fact that the two observations were performed at low elevation where accurate calibration is significantly more difficult to obtain. Results from the other wavelengths indicate that there is no fundamental problem with the Hh calibration scheme. The poor calibration at Cm is due to the fact that the microwave link prevents the measurement of system temperatures, and nominal values of the SEFD are used to calibrate the telescope. This problem will not be resolved until e-MERLIN comes online. The reason for the poor Ur and On calibration is not yet understood, but will be investigated in session 3 of 2004. L band calibration is less accurate than C and X band, and this appears to be due to RFI. This is supported by the fact that there is a large scatter in the amplitude correction factors between experiments, which seems to be related to the observing frequency. Experiments in the cleanest part of the 18 cm band have significantly smaller errors than the median values. Again, the Tr values at L band are affected by the (now corrected) error in the gain curve. K band is clearly a problem for all stations. This is probably related to opacity effects and to the low sensitivity of many of the antennas at high frequency. Several people suggested that pointing accuracy may be a problem at K-band. It was agreed that we should come back to opacity problems when we have investigated sensitivity at K-band. It was suggested that the K-band NME be used as a test of calibration procedures since K-band is such a problem. Action Item: Alef to suggest to scheduler to provide 6-8 hours in the next K-band NME for calibration tests. Action Item: Graham to provide advice on K-band calibration targets. Action Item: All stations were encouraged to check amplitude corrections on pipelined experiments available from the JIVE webpage. Action Item: Stations were asked to provide detailed Wx and RFI comments in experiment logs. Calibration in the UHF band is very poor. This is because of the extremely difficult RFI environment at these frequencies. It was felt no immediate action could be taken to alleviate this. Comparison of Tcal tables from February 2003 and May 2004 sessions indicates that the Tcal versus frequency dependence can vary over the timescale of a few months. Gunn pointed out that this is not unexpected since the same diodes/receiver may not be in use from session to session. Continued monitoring of Tcal during every session is therefore required. Alef asked how often the cal diode should be fired for good calibration. This would require some investigation but a warning can be implemented in SCHED (currently this is 15 minutes). Action Item: All stations to continue monitoring Tcal throughout sessions. Action Item: Reynolds to look at tpidiff for a range of experiments and recommend a suitable time between diode firings. Update SCHED accordingly. Action Item: Colomer was asked to report on implementation of continuous tpi on VLBA systems. Reynolds again emphasised the timely need for amplitude calibration. Gunn suggested that stations need not analyse the whole session at the end and can produce calibration files after each frequency ends. 5. Automatic Flagging --------------------- Reynolds reported that stations continue to provide off-source monitoring data which allow data taken when the antenna was off-source to be flagged. This system continues to work well at most stations. Exceptions in session 3 of 2004 were Westerbork and Noto. Torun have been successfully using the off-source flagging since May 2004. Urumqi station has also introduced off-source flagging in session 3 of 2004, but the accuracy of this flagging has yet to be tested. Noto are reported to be investigating possible solutions and will solve this by February. This system is unlikely to be useful for the tied-array at Westerbork. A new version of the ANTAB scripts has been distributed to stations for the November 2004 session. Stations will attempt to make their own ANTAB files for this session (to be checked afterwards at JIVE). Action Item: Noto to resolve problems with automatic flagging of data. Action item: Stations to produce ANTAB and RXG files within 2 weeks of the end of a session. 6. Mk5 ------ Smythe reported on Mk5 developments. Conduant is proceeding to build a SATA prototype upgrade kit. This will involve a new backplane but SATA and PATA modules will be interchangeable in the Mk5. The SATA connectors are only rated for 250 cycles and so will need replacing more often. The new Mk5 card should cost around 2k per unit and the full upgrade to Mk5B should cost around 4k. Nearly all planned features in the Mk5 software are now supported with the exception of automatic hardware bank switching, which will be available very soon. All software/firmware is available online on the Mk5 website. Action Item: Stations must regularly check for Mk5 software updates and ensure the software matches the appropriate FS release. Some issues about use of disks in the field and at the correlator were discussed. Stations should try to fix broken disks with new ones if possible, and broken disks should be returned to the originating institute (because they may still have a warranty). The comment label on the side of units can be used to record replaced disks. It was pointed out that TRACK does not accept extended bar codes and Walker was asked to look into this. It was stated that the bar code writer software has two versions and one doesn't produce the correct check code. Gunn complained about disk pack labels which were inconvenient, messy, and unable to hold labels for more than a few experiments. Alef/Campbell commented that the correlators could feasibly run without any experiment labels, but would have to be sure that VSN's are believable. The TOG agreed to leave this to a later decision. In the meantime, Alan Whitney is investigating other solutions. The TOG recommends that this should not be a 'sticky' solution, possibly a plastic envelope/metal holder on the front panel. Alef talked about problems with acquiring foam for shipping. Medicina said they had found a local supplier. Action Item: All stations are to provide information to Alef about how much disk space they have purchased to date, what is currently avalable (for February session) and an indication of future plans for buying disk packs. Action Item: Walker to check out the status and report on the use of extended bar codes in TRACK. Action Item: Medicina will send a sample of their supplier's packing foam to Alef. Walker gave a brief presentation concerning the plans for Mk5 at NRAO. Should be at 128 Mbps at all VLBA stations by the end of 2005. Alef asked if for the February session, Noto could send disk packs to Socorro since they don't have a tape recorder. This would have to be by a session by session basis. Action Item: Walker to enquire of NRAO whether Noto could send disk packs to Socorro in the February session since they no longer have a tape recorder. 7. Field System --------------- Himwich gave a report on recent Field System (FS) developments. e-VLBI support had been improved and the SCHED-DRUDG-FS path tested. Also, some changes to in2net and disk2file had been implemented. The VEX parser now supports SCHED Mk5 schedules, although DRUDG does not (but may do by February 2005). Other improvements included integration of NTP into fmset/setcl, mk5 remaining capacity reported in monit, inclusion of a change_pack procedure and the provision of data_transfer overide options in DRUDG. Himwich explained that the FS now produces error files on crashing and asked that these are passed to him for investigation. Some discussion was made concerning 'chooser' for selecting equipment/ telescope on FS startup and the LO switching scheme. JIVE reported that they are adopting the standard file name convention for VLBI data that was suggested by Alan Whitney. Action Item: All stations to forward FS error log files to Himwich and include details of what the FS was doing when the crash occurred. Action Item: All stations to provide to Himwich detailed descriptions of all bugs encountered in the FS. 8. SCHED Developments --------------------- Walker gave a presentation on recent SCHED developments. A new release would be available very soon. This would be able to schedule Mk5 up to 1 Gbps (awaiting FS upgrade). A straightforward FTP fringe-finder and e-VLBI scheduling option would also be offered. Some bug fixes in the VEX writer were also being made. In the future, SCHED should be able to use VLBA-style 'autoallocate' for disk stations to remove scans outside of telescope limits. Also, a 'super-simplified' continuum setup selection will be offered where users select only 'band' and 'total bitrate'. Some method of indicating band-switching (frequency agility) times should be included in SCHED. Action item: Stations to provide frequency agility information to Reynolds. Action Item: Alef to ask Polatidis to update the EVN status table with new/additional frequency agility information. Walker suggested that slewing models need to be investigated. This resulted in some discussion about inadequate slew times at some stations. It was suggested that a a slew to Az 180 command in schedule_end procedure would aleviate some of the wrapping problems. Action Item: Stations to investigate whether SCHED slewing model and drive rates in station catalogues are adequate. Action Item: Jb (Lovell) and Ef should look at wrap limits to see if they can be changed. It was felt important to have target codes to identify the fringe-finders, p cals, bandpass cals etc. in a future release of SCHED. Schedule checking at JIVE seemed to work very well, with plenty of time to sort out problems. The TOG recommended that the 'three week rule' for schedules is retained. 9. If Switchbox --------------- Graham reported that building of the IF switchboxes, software and firmware are completed. The unit has been tested and should be available before the February session. Cost will be 1300 - 1500 Euros per station. 10. Decoder ----------- Smythe gave a synopsis of the MkIV decoder status. A brief discussion was made. Action Item: All stations to install new firmware for decoder. Update formatter firmware to version 41. 11. Spare Parts --------------- Burgess reported that the budget had been spent on remaining parts from (defunked) Metrum UK. It was suggested that a new "section" for Mk5/disks be established. Action Item: Burgess to establish Mk5/disk spare parts list. 12. Digital BBCs ---------------- Tucarri presented the prototype digital BBC system, which was also on display in the JBO VLBI observing room. The TOG discussed the system and thanked Tucarri for his hard work. 13. e-VLBI Status ----------------- Szomoru presented the cuurent status of e-VLBI from the JIVE perspective. Currently, 1Gbps connections were made to Westerbork, Torun and Onsala, and 155 Mbps connections to Jodrell Bank/Cambridge and Arecibo. An upgrade of Jodrell/Cambridge to 2.5 Gbps and to Medicina at 1 Gbps are planned for later this year. Currently, existing transfer protocols are being used but new ones (VSI-E?) will be available shortly. Szomoru reported that dramatic improvements in TCP performance were possible by adjusting the default settings. Various tests had been performed using iperf, in2net and disk2net. The first real-time e-VLBI EVN image had been produced in September. In the future a high bandwidth test with jumbo frames was planned for November, in December a transatlantic test involving Westerbork and Westford (with correlation both at Haystack and JIVE) will be attempted. Identification of network bottlenecks with the help of VSI-E software will also be performed. In 2005, a further increase in data rates involving more telescopes, both in Europe (Cm, Jb, Mc), and across the Atlantic will be achieved. Smythe gave an overview of e-VLBI developments at Haystack. 512 Mbps real-time e-VLBI demonstrations between Westford and GGAO had been progressing. Kashima K5 data had been successfully transfered by FTP to Haystack. e-VLBI connections were being established to Kokee Park and Wettzell for near-real-time Intensive UT1 experiments. A remote real-time fringe display had been presented at a Super-Computing conference in Pittsburg. 14. Station and Correlator Reports ---------------------------------- The TOG did not discuss, but noted, reports from stations and correlators. 15. AOB ------- None 16. Time and Place of Next Meeting ---------------------------------- Next meeting will probably be at Onsala Space Observatory after mid-summer. 17. Summary of Action Items --------------------------- Action Item 01: Paragi to include cross-hand fringe checks as the default for software correlation. Action Item 02: Paragi and Reynolds to investigate 1Gbps support in the software correlator. Action Item 03: Garrington was asked to discuss Mk2/Lovell phase referencing with the EVN PC and form a working process for users. Action Item 04: Paragi to provide more comprehensive feedback from software correlation but to give immediate 'fringe' feedback to station. Action Item 05: All stations should look at data regularly with chchk program, use it to locate significant RFI and report the results to Polatidis, and use it to check pcal throughout the session. Action Item 06: All stations should upgrade the decoder and use it to monitor pcal. Action Item 07: Urumqi was asked to provide automatic control of pcal as soon as possible. Action Item 08: Noto station should check reasons for smaller ftp file sizes. Action Item 09: Himwich to locate total file space error in drudg output and fix. Action Item 10: Alef to suggest to scheduler to provide 6-8 hours in the next K-band NME for calibration tests. Action Item 11: Graham to provide advice on K-band calibration targets. Action Item 12: All stations were encouraged to check amplitude corrections on pipelined experiments available from the JIVE webpage. Action Item 13: Stations were asked to provide detailed Wx and RFI comments in experiment logs. Action Item 14: All stations to continue monitoring Tcal throughout sessions. Action Item 15: Reynolds to look at tpidiff for a range of experiments and recommend a suitable time between diode firings. Update SCHED accordingly. Action Item 16: Colomer was asked to report on implementation of continuous tpi on VLBA systems. Action Item 17: Noto to resolve problems with automatic flagging of data. Action item 18: Stations to produce ANTAB and RXG files within 2 weeks of the end of a session. Action Item 19: Stations must regularly check for Mk5 software updates and ensure the software matches the appropriate FS release Action Item 20: All stations are to provide information to Alef about how much disk space they have purchased to date, what is currently avalable (for February session) and an indication of future plans for buying disk packs. Action Item 21: Walker to check out the status and report on the use of extended bar codes in TRACK. Action Item 22: Medicina will send a sample of their supplier's packing foam to Alef. Action Item 23: Walker to enquire of NRAO whether Noto could send disk packs to Socorro in the February session since they no longer have a tape recorder. Action Item 24: All stations to forward FS error log files to Himwich and include details of what the FS was doing when the crash occurred. Action Item 25: All stations to provide to Himwich detailed descriptions of all bugs encountered in the FS. Action item 26: Stations to provide frequency agility information to Reynolds. Action Item 27: Alef to ask Polatidis to update the EVN status table with new/additional frequency agility information. Action Item 28: Stations to investigate whether SCHED slewing model and drive rates in station catalogues are adequate. Action Item 29: Jb (Lovell) and Ef should look at wrap limits to see if they can be changed. Action Item 30: All stations to install new firmware for decoder. Update formatter firmware to version 41. Action Item 31: Burgess to establish Mk5/disk spare parts list.