CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN TOG MEETING, September 2008, Bologna 12 September 2008 (statistics cover 20 Oct 2007 - 12 Sep 2008) Bob Campbell Arpad Szomoru Mark Kettenis SCIENCE OPERATIONS The table below summarizes projects correlated, distributed, and released from 20 October to 18 September. The table lists the number of experiments as well as the network hours and correlator hours for both user and test/NME experiments. Here, correlator hours are the network hours multiplied by any multiple correlation passes required. User Experiments Test & Network Monitoring N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr Correlated 53 506 903 43 176 184 Distributed 48 441 822 42 171 179 Released 61 602 1082 46 194 202 The following table summarizes by session the user experiments from sessions since the previous TOG, with an additional column for experiments not yet distributed (entries = remaining to do / total). N_to.corr Corr.hrs N_to.dist session 3/2007 0/6 0/281 0/6 Feb ToO 0/1 0/8 0/1 session 1/2008 0/18 0/288 0/18 (excl. 3 wx-abandoned) Mar ToO 0/1 0/8.5 0/1 Apr e-VLBI 0/5 0/54 0/5 (incl. 3 ToO) May e-VLBI 0/1 0/3 0/1 session 2/2008 2.5/21 97/362 7/21 Jun e-VLBI 0/3 0/7 0/3 session 3/2008 15/15 200/200 (anticipated Corr_hr) Since the previous TOG, we have had a surfeit of personnel changes. Nico Schonewille retired at the end of February 2008. We then gained a new operator, Bert Harms, who "transferred" over from Wb, and have reverted to the original 80-hr weeks (moving away from the recent 80-80-40 hr/wk rotation). Hans Tenkink has assumed the mantle of senior operator, and Martin Leeuwinga has moved to a position as Hardware Support Engineer, taking over most of the responsibilities of the retired Jan Buiter, with a focus now on developments in the Mark5 family. Martin also continues to function as an operator. Cormac Reynolds and Hayley Bignall departed for Perth, and Rebeca Soria-Ruiz returned to Madrid. We have gained Antonis Polatidis and will soon add Yang Jun as support scientists. Zsolt Paragi has moved into the senior support scientist position, as well as remaining the EXPReS post-doc. Landmarks from recent sessions: Tactics: Since session 1/2008, we have begun to correlate Gbps experiments first, to recycle as many disks as possible back to the stations for the next session (as opposed to the conventional next+1 session). In the past two sessions, this has accelerated the circulation of about half the sessions' packs by size. Logistics: We seem to have lost our first disk-pack in transit (Mc's 2nd pack in GC031A after recording en route to the correlator, Jun'08). Schedule issues: GC029 (3/2007)- different versions for EVN & VLBA stations were observed, leading to a scan-by-scan correlation of the regions of overlap (VLBA schadule = base; EVN added in partial scans where appropriate). EP062 (3/2007) - PI used non-sched software to make the skd file; some difficulties ensued (phase-cal tone specification, cable-wrap timing). Network firsts: GB060A (3/2007) - 1st 7mm experiment correlated at JIVE GB063A (3/2007) - 1st participation by EVLA station (5cm); 48MHz front-end range "misplaced", better in GB063B (1/2008) GB064 (2/2008) - high-z H20 at 6120MHz; 1st participation of GBT at 5cm N08K3 (2/2008) - higher K-band (23.5GHz); attempt to see NH3 for first time on EVN (GST range precluded looking at known NH3 maser) Correlator firsts/highlights: GC029 (3/2007) - 17 correlator passes; 153 correlator hours; 483 production hours; 1028.7 GB of output FITS files (all records) EP062 (3/2007) - first 1/8s integration experiment (2 passes: SB0-7, 8-15) non-standard post-correlation processing to re-join all 16 SBs and to enable the PI's non-AIPS analysis EP067 (3/2007) - copied data (7sta * 2hr @ Gbps) from Mark5 packs to Lacie disks, sent to Swindburne GW019A/GS029 (2/2008) - 3 pass correlation for too-many-stations (GW019A = 22 stations, also a record) EA038 (2/2008) - 3 additional out-stations recorded onto Cm pack (1-pol) separate schedule needed for Cm (at 512 Mbps) in line pass - largest sub-job in terms of correlator output (43.8 GB) RT007 (Apr e-VLBI) - longest uninterrupted job (& sub-job) having good data = 12h48m21s Development of Astronomical Features: Work continues on recirculation (up to a factor of 8: 16MHz/BW_sb). New recirculation-ready versions of real-time processes run exclusively at 64 correlator frames per second. The minimum integration time would increase by the recirculation factor. The global validity mode has essentially been shelved, as the new bug noticed last fall persisted in defying solution on an affordable time-scale (Albert has already officially retired). This would have doubled the effective spectral capacity for all-VLBA-format data, such as would result from 5B played back via 5A+. The possibility of including a second MERLIN out-station in the EVN correlation continues for dual-pol recording modes that leave "unused" subbands in the Cm recording (i.e, >=256Mbps with BW_sb>=8MHz). A single-pol experiment recorded 4 out-stations on the Cm recording in session 2/2008. In this case, a separate schedule was needed for Cm, since it's recordings did not have any "unused" subbands in the default 128 Mbps schedule. Making use of this 'additional out-station' capability would entail more disk-space (perhaps at recording, and certainly at correlation, where the N-station Cm pack would need to be copied to N-1 other packs, so that each additional out-station could be correlated as separate stations). There may also be correlator-capacity consequences if the 8- (or 16-) station boundary is crossed. We began applying a better post-correlation fractional bit shift correction to the phase across the band for each (baseline) visibility for experiments starting in session 1/2007. With this in place, we were able to determine that there was also a remaining station-based fractional-bit delay problem remaining in correlation of over-sampled data, which is being looked into. Stations are all putting weather info into the FS logs (/wx/ lines). We are working to take advantage of this by incorporating these values into the apriori model (tropo dry, wet zenith delay estimates). TECHNICAL DEVELOPMENTS [e-VLBI developments are now treated in a separate report to the TOG] Mark5 All Mark5 units have been equipped with high-end motherboards and CPUs, and several were modified to B units and hooked up to the correlator via Correlator Interface Boards and optical serial links. Limited amounts of suitable 5B data has limited the number of tests. Initial tests with modified LBA data showed a difference between correlation using A+ and B playback. Extensive consultations with Haystack identified a 1-second discrepancy. New tests with a converted Mark5A-recorded experiment showed no significant differences in the results from A, A+ and B playback. Another consideration in the change-over to 5B at the stations is ensuring e-VLBI with Mark5B functions properly. The locally developed control code on the sending side (identical to the code used for Mark5A e-VLBI) works well, and first e-VLBI fringes were obtained on the 15th of April between an A and a B unit at Effelsberg, where both units were fed the same noise signal. The only EVN stations with Mark5B units in place are Ef and Wb (via the new TADUmax backend, which has been undergoing separate testing), so there has been little opportunity to test B-transfers. Unfortunately our last test, involving both Wb and Ef, produced some puzzling results, and further tests are planned. We are sure these problems will be resolved soon, and we do not think stations need to hold off converting their units from A to B. On the receiving side, an e-VLBI enabled version of DOMINO is being developed at Haystack and has been partly debugged at JIVE. As mentioned previously, normal B playback has been tested and is now fully integrated in the correlator control software. Software correlation Initial comparisons between the software correlator and the MkIV hardware correlator identified phase-drift and model issues, which have been fixed in the delay module of the software correlator. The existing software to convert correlator output into an aips++ MeasurementSet was adapted to accept the output of the software correlator, allowing us to create FITS files in the standard way. Comparison in AIPS now shows good agreement between the results of the hardware correlator and the software correlator. A slightly updated version of the software correlator was used for the FTP fringe test in session 1/2008. Meanwhile, much work had been put into instrumenting and benchmarking the software correlator. Decoding the Mark5A data was identified as a major bottleneck. After optimization of the relevant code, we are now able to decode a 1Gbps datastream in real time on a single cluster node. Further bottlenecks have been identified and are being addressed. After some initial optimizations of the delay module, we are now able to correlate a 4-station 256 Mbps experiment at 30%-40% of real-time on a 30-node cluster of quad-cure AMD Opteron machines (part of the DAS-3 supercomputing facility in the Netherlands). This optimized version was used for the fringe tests in session 2/2008. Web services for translating VEX files into correlator-control files (needed for the FABRIC Workflow Manager developed at PSNC), and distributing data to GridFTP servers (needed for the VLBI Grid Broker, also developed at PSNC) have been implemented. With these components in place, we should be able to start testing fully-distributed correlation soon. Huseyin Ozdemir and Nico Kruithof have left JIVE, resulting in slower progress on the software correlator over the last couple of months. Des Small has taken over from Huseyin on the webservices aspects of the FABRIC project and Aard Keimpema started on Septembert 1st, to continue the work of Nico in SCARIe. A joint demo between SCARIe and the DANTE AutoBahn team to demonstrate the use of their Bandwidth-on-Demand facility is scheduled for the GLIF workshop in Seattle and the big SC08 conference in Austin in November. Miscellaneous A new Solaris server equipped with a high-capacity raid array has replaced the aging data acquisition machine after a series of correlation tests. This machine is interchangeable with the two correlator control machines. With the installation of this machine, a re-circulation enabled version of the correlator code was installed and as a consequence, the whole system now runs permanently at 64 BOCFs. Terminal servers were installed to interface to the PCInt nodes and to the Mark5 units. Stephen Bourke has taken over on wide field imaging from James Anderson. He has started on generalizing the ParselTongue code written to support his PhD-thesis research for imaging wide field spectral line data. The new parallelization framework for ParselTongue that has been developed has received its first in-house testing. A public "beta" test version will be released soon. It seems the number of ParselTongue users is still increasing given the fact that periodically we see questions from new people. Since the last release only a small number of bugs have been fixed, which suggests that the existing functionality has reached maturity. NETWORK SUPPORT The automatic-ftp feature continues to be exercised in all network monitoring experiments, sending the specified portion of a scan directly to the software correlator computer at JIVE. The correlation is now carried out by the software correlator being developed under FABRIC/SCARIe. Correlation results go to a web page available to all the stations within a couple hours, and Skype chat sessions during the NME provides the station friends with even more immediate initial feedback. We continue to process all experiments, including NMEs, via the pipeline (now run via ParselTongue), with results being posted to the EVN web pages. The pipeline provides stations with feedback on their general performance and in particular on their gain corrections, and identifies stations/frequency bands with particular problems. Timely delivery of ANTAB amplitude calibration results from the telescopes seems to be improving, but remains an issue in e-VLBI experiments due to the shorter time-scales involved. Giuseppe Cimo has taken over the ANTAB-related responsibilities from Cormac Reynolds. USER SUPPORT The EVN Archive at JIVE continues to provide web access to the station feedback, standard plots, pipeline results, and FITS files. Access and public-release policy remains the same. Disk space available for the EVN Archive has been increased from 4.5 TB to about 16 TB. We are nearing 6 TB of FITS files on the archive. The NorthStar web-based proposal tool is now mature. It is the sole route for submitting EVN (including e-VLBI) and Global VLBI proposals (ToO proposals and requests for short observations remain outside). PC members access the proposals through the NorthStar repository, and distribution to non-EVN bodies (NRAO PC; MERLIN, DSN schedulers) is handled off-line as necessary. The NorthStar review tool has been implemented for the EVN, but has not yet been approved by the PC. We continue to contact all PI's once the block schedule is made public, and to check over schedules posted to VLBEER prior to stations downloading them. There have been a couple glitches in the system since the previous TOG, which have led to beefing up the redundancy in cross-checking the schedules deposited on VLBEER and ftp.aoc in Socorro. Antonis Polatidis has taken over the sched-related EVN responsibilities from Cormac Reynolds. Antonis also maintains his responsibilities for the EVN status table. Stations are reminded to inform him of any changes in a timely fashion. Information in the EVN status table also makes its way into the EVN calculator, Proposal Tool, and sched catalogues.