CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN TOG MEETING, November 2006, Yebes 4 November 2007 (statistics cover 22 Nov 2006 - 19 Oct 2007) Bob Campbell Arpad Szomoru Cormac Reynolds SCIENCE OPERATIONS The table below summarizes projects correlated, distributed, and released from 22 November to 18 October. The table lists the number of experiments as well as the network hours and correlator hours for both user and test/NME experiments. Here, correlator hours are the network hours multiplied by any multiple correlation passes required (e.g., continuum/line, >16 station, different phase centers, etc.) User Experiments Test & Network Monitoring N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr Correlated 55 528 707 43 229 253 Distributed 56 552 751 46 233 258 Released 51 515 728 44 222 251 The following table summarizes by session the user experiments from sessions since the previous TOG, with an additional column for experiments not yet distributed (entries = remaining to do / total). The processing factor (the ratio of the actual correlation time required divided by the Corr_hr) is typically between 1.5-2.5 times. N_to.corr Corr.hrs N_to.dist session 3/2006 0/5 0/47 0/5 Dec-Feb e-VLBI 0/5 0/52 0/5 session 1/2007 0/23 0/254 0/23 March e-VLBI 0/1 0/8 0/1 session 2/2007 0/17 0/295 0/17 Jun-Sep e-VLBI 0/4 0/43 0/4 session 3/2007 6/6 250/250 (anticipated Corr_hr) To review some landmarks from recent sessions: 3/2006: very abbreviated largely owing to engineering work at Effelsberg (sub-reflector) and Shanghai. 2/2007: New records for longest uninterrupted sub-job (10h14m30s) and job (10h28m45s) with good data established (in different experiments). general: All experiments observed prior to session 3/2007 correlated and distributed. All user e-VLBI experiments so far have run at 256 Mbps, although tests using UDP have shown great promise at 512 Mbps. The automatic-ftp feature added to the Field System last year has been exercised since session 3/2006. This automatically ftp's the specified portion of a scan directly to the software correlator computer at JIVE. A script running there detects the arrival of new data and automatically performs the correlation and posts the results to a web page which is available to all the stations (initial results are reported on the same day as the experiment is observed). The ftp fringe tests continue to be very successful in identifying station problems early in the session in time for the stations to take action to safeguard subsequent user experiments. We began applying a better post-correlation fractional bit-shift correction to the phase across the band for each (baseline) visibility for experiments starting in session 1/2007. Work continues on recirculation and a new bug noticed in the global validity mode (that would be useful for doubling the effective spectral resolution for all-VLBA-format data). The possibility of including a 3rd MERLIN station in the EVN correlation continues for recording modes that leave "unused" subbands in the Cm recording. TECHNICAL DEVELOPMENTS We currently have 16 Mark 5A units, all housed inside temperature-controlled cabinets. Twelve of these have been converted into Mark 5A+ units to allow playback of Mark 5B recordings; we have successfully exercised such playback in some modes. Two working DPU's remain in place; others could be used but would have to be moved to accommodate maximum data-cable length limits. Reconnecting a DPU would require swapping 2 cables. All non-e-VLBI experiments starting from session 2/2006 have been recorded entirely on disk. Developments in e-VLBI are now treated in a separate report to the TOG. New motherboard/CPU/power-supply combinations for the Mark5 units at JIVE have been purchased and (mostly) installed, with the last few units being upgraded now. An HP switching router with the capacity to handle multiple 10Gbps data streams was purchased as well. Although this router has performed very well with TCP-based e-VLBI, recent tests with UDP data streams brought a number of firmware bugs to the light, which are being investigated by HP. Until a real fix is provided, a temporary, but adequate, work-around has been designed. Two Mark5A units were modified to Mark5B; one of these was sent to Westerbork to help debug the new TADU hardware. The other was connected to the correlator via a Correlator Interface Board, and necessary changes to the correlator control software are being made. A new Solaris server, identical to the servers which replaced the old HP correlator control computer, was purchased to replace the data acquisition machine. For this purpose it is equipped with a large-capacity raid array. The system will be configured in such a way that any of the three servers will be able to take over the task of either correlator control or data acquisition machine. The ParselTongue software, which has been developed in the context of ALBUS, is being used by an increasing number of outside users (currently approximately 40). The improved installation procedure and extensive new user documentation wiki (www.jive.nl/jivewiki) seem to have been successful in simplifying the process of getting new users started with ParselTongue. Recent developments in ParselTongue have focused on establishing a simple framework for distributing data and tasks to remote hosts, e.g., for parallel execution in a computer cluster. A test release of this functionality should be available by the end of the year. The research part of the ionospheric calibration work package is essentially complete and documentation will be delivered in the coming weeks. A method was developed to apply differential ionospheric calibration, but it has been shown that the current algorithm only works well in areas where there is a dense GPS network. More data from GPS networks should become available in the coming years as the GPS community develop a centralized database for sharing data. Work has also begun on wide field imaging and I/O has been identified as the main bottleneck. This work will benefit from data distribution functionality described in the previous paragraph, but will probably also require some changes to the AIPS code to make processing steps such as gridding more atomic so they be carried out in parallel. James Anderson has recently left JIVE to take up a position in Bonn, and the wide field imaging task will be taken up by Stephen Bourke who is expected to arrive at JIVE in December. Ruud Oerlemans, who has been part of the FABRIC project from the start, found another job and left JIVE. Fortunately a new scientific programmer was found in the person of Huseyin Ozdemir, who started in August 2007. For the (closely related) SCARIe project, the UvA/SARA group finally found a suitable post-doc (Damien Marchal). In spite of the disruption and the problems finding suitable candidates, work progressed well. The JIVE software correlator was used in parallel with the NICT software correlator on the ftp fringe tests in session 2/2007, and processed the ftp fringe-tests from the (current) session 3/2007 alone. A bug in the fringe-rotation of LSB data was discovered and fixed. USER SUPPORT The EVN Archive at JIVE continues to provide web access to the station feedback, standard plots, pipeline results, and FITS files. Public access to source-specific information is governed by the EVN Archive Policy: the complete raw FITS files and pipeline results for sources identified by the PI as "private" have a one-year proprietary period, starting from distribution of the last experiment resulting from a proposal. PI's can access proprietary data via a password they arrange with JIVE, and receive a one-month warning prior to the expiration of their proprietary period. In addition to the existing tools for searching the Archive via the FITS-finder utility and the catalogue of EVN observations via the Bologna database, new functionality to connect to the Aladin Sky Atlas and the Sloan Digital Sky Survey have been developed. Disk space available for the EVN Archive has been increased from 4.5 TB to about 16 TB. The NorthStar web-based proposal tool is now the only means for submitting EVN and Global VLBI proposals. The NorthStar review tool has been implemented for the EVN, but has not yet been approved by the PC. A new release of SCHED with (limited) support for geodetic patching is expected soon. We continue to contact all PI's once the block schedule is made public to ensure they know how to obtain help with their scheduling, and to check over schedules posted to VLBEER prior to stations downloading them. Since the new safety features have been incorporated into the pre-observation logistics to avoid the risk that stations may observe an older version of a schedule, this situation has not recurred. The period prior to session 3/2007 proved unusually difficult, largely because many new things were being tried for the first time (e.g., 5cm at EVLA, 7mm at VLA in a global, non-sched/sked scheduling software). This resulted in some schedules being significantly late to the stations, but so far there has been no adverse consequence.