CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN TOG MEETING, December 2009, Effelsberg 28 October 2009 (statistics cover 19 May 2009 - 14 Oct 2009) Bob Campbell Arpad Szomoru Mark Kettenis SCIENCE OPERATIONS Personnel activity Since the previous CBD meeting, the Science Operations and Support group has gained two support scientists, bringing it to its full complement for the first time since the end of 2008. Mahreen Mahmud arrived on 3 August, after completing her thesis work at the University of Cork. Yurii Pidopryhora took up a full-time support scientist position on 1 September, following the end of the EXPReS/FABRIC project. Sessions and their Experiments The table below summarizes projects correlated, distributed, and released from 19 May to 14 October. The table lists the number of experiments as well as the network hours and correlator hours for both user and test/NME experiments. Here, correlator hours are the network hours multiplied by any multiple correlation passes required. User Experiments Test & Network Monitoring N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr Correlated 43 381 526 7 36 41 Distributed 43 392 499 10 47 52 Released 42 383 481 14 64 69 The following table summarizes by session the user experiments with activity since the previous CBD meeting , with an additional column for experiments not yet distributed (entries = remaining to do / total). N_to.corr Corr.hrs N_to.dist session 1/2009 0/20 0/273 0/20 Mar-May e-VLBI 0/12 0/91 0/12 session 2/2009 1/25 42/350 3/25 Jun-Sep e-VLBI 0/7 0/52 3/7 session 3/2009 17/17 317/317 (anticipated correlator hours) Some landmarks since the previous CBD report: First fringes in EVN experiments to Kunming (N09SX1, S-band; EY008A, X-band) and Miyun (EY008A, X-band), both in session 2/2009. First fringes in EVN experiments to Yamaguchi32 in session 1/2009 (EB039B; K5 to Mark5B translation resulted in a bit-stream/channel mapping that differs from "normal" Mark5B stations, which delayed finding the fringes until after the previous CBD meeting). First successful correlation using recirculation in two experiments in session 1/2009; in routine use for session 2/2009. Kashima and up to four Australian stations participated in a three-epoch e-VLBI target-of-opportunity experiment from May-July. Kashima had problems with the K5 to Mark5B translation in the first two epochs, but had good fringes in the third, marking the first Kashima fringes in an e-EVN user experiment at JIVE. Session 2/2009 saw the first incoming disk-pack recording load of over 60 TB per station, and over 500 TB total. The outgoing disk-pack distribution for both session 2 and 3/2009 was over 500 TB. Astronomical Features: We have begun to post two new types of FITS data on the EVN archive: (i) FITS files from the Westerbork-array data obtained during during the course of EVN observations, and (ii) the pipeline-calibrated UV-FITS files for individual sources. The Wb FITS files aren't directly available to users as such from the station; we take care of the data transformation from the Wb-archive Measurement Sets, and place the resulting FITS files on the archive along with the FITS files from the EVN correlation. We are working on expanding the pipeline to calibrate these Wb FITS data, to help with amplitude and polarization calibration of the full EVN data-set. In some cases (e.g., small-field continuum mapping), the Wb FITS data can be several times the size of the EVN FITS data, so this extra processing is currently driven by PI request. The pipeline-calibrated UV-FITS data for individual sources contain the cumulative effects of all steps of calibration within the EVN pipeline. These are now available via the main pipeline page for each experiment in the EVN archive. These pipeline-calibrated FITS files associated with "private" sources are protected by the same one-year proprietary period as are the plots/images of these sources and the full set of raw IDI FITS files. The possibilities for including additional MERLIN out-stations in the EVN correlation continue to expand. We have already incorporated one additional dual-pol and three additional single-pol stations into the Cambridge recording in user experiments (one station/pol per each of the 4 Cm recorder IFs; lower data-rate experiments may require a separate schedule for Cm to create enough "unused" subbands to hold these additional out-stations' data). We can use multi-casting techniques developed for e-VLBI to access all stations' data from the single Cm disk-pack directly (at the cost of 2 of the 16 Mark5 units to simulate the e-VLBI transfer into the switch/router). In recent e-VLBI testing, Jodrell was able to combine more than one dual-pol out-station into the Cambridge signal, placing the 16MHz signal from two station/pols into a single 500MHz IF by up-converting one of them when mixing into the IF and down-converting again mixing into the BBCs. This mode will be exercised in a user experiment in session 3/2009 if there prove to be enough out-stations available. Recirculation is a means of time-sharing correlator resources for experiments that don't use the maximum sampling rate (32Mb/s; 16MHz Nyquist-sampled subbands), in order to increase the apparent spectral capacity in experiments that would otherwise have had spectral resolution limited by the number of stations or polarizations. The recirculation factor, R, is 32 over the sampling rate, up to a maximum of 8, and the spectral capacity scales as R, subject to the maximum number of frequency points per baseline/subband/polarization that remains at 2048. The downside of recirculation is that the minimum integration time also scales with R, which may affect narrow-band spectral-line observations wanting a wide field of view. Recirculation has been used by 7 user experiments through sessions 1-2/2009, and will be needed by another 8 in session 3/2009. Of course, no improved capability remains sufficient for long... five of these eight experiments will need to combine recirculation with multiple passes to attain their desired spectral resolution. Also in session 3/2009 will come the first oversampled observation requiring recirculation. TECHNICAL DEVELOPMENTS e-VLBI The past half year started off with a 3-epoch ToO observation, the first e-VLBI science operations at K-band ever. For these observations, a truly global array was created that was only ever used before during the demonstration at the opening of the International Year of Astronomy. It involved the European EVN telescopes, Shanghai, Kashima, and up to four telescopes in Australia. In spite of some technical problems, and the difficulties of clock searching at high frequency on such long baselines, the observations went well and two papers are in progress. After the vacation period, several e-VLBI runs took place, most of which were remarkably unremarkable, with many hours of uneventful correlation -- a clear demonstration of the long way e-VLBI has come in the past years. During one of the runs Merlincast was successfully used to include the Lovell telescope and no less than 3 additional MERLIN stations. Unfortunately the azimuth bearing of the Shanghai telescope broke in the beginning of September, and as all accepted e-VLBI projects needed long baselines, only one triggered proposal was scheduled for the run at the end of September. However, with no trigger forthcoming, only tests were done. These included tests with simultaneous recording/transmitting at Onsala, and Mark5B-Mark5B e-VLBI. During the past month, several data transfer tests were done with Badary, showing that 32 Mbps e-VLBI should be possible (albeit at restricted times). A full fringe test was conducted on 21 October. Likewise, plans are underway to include Robledo in an e-VLBI session, at low data rates. Other potential future e-partners are Yamaguchi and the new telescopes of the Korean VLBI network. Mark5 Mark5B development is more or less completed, apart from some VEX and SCHED-related issues. When these are solved and the operational all-clear is given we will start retiring SUs as more stations move to Mark5B (or possibly Mark5C, depending on the availability of dBBCs). FPGA-based correlator development In the beginning of 2009 the UniBoard project kicked off, which is a Joint Research Activity in RadioNet FP7 led by JIVE. Its aim is to develop a generic computing platform, based on FPGA technology, to be used for a number of astronomical applications. In this project ASTRON is responsible for the hardware development, JIVE and ASTRON for the EVN/Apertif correlators/beam-former, the Universities of Manchester and Orleans for the pulsar binning machine, and the University of Bordeaux and Arcetri Observatory for the digital receiver application. This project was reinforced by the approval of a JIVE/ASTRON proposal to NWO (the Netherlands science council) to develop an FPGA-based correlator based on LOFAR-type boards combined on a backplane. The hardware development is right on track, with most of the design finished and the first prototype expected in the beginning of 2010. The actual correlator firmware development got underway when Jonathan Hargreaves started as digital engineer at JIVE, on the 1 June 2009. He was joined in October 2009 by Ying Xiang, a senior digital engineer from Shanghai Observatory, who will spend a year working on the UniBoard project as a visiting researcher at JIVE. Through yet another NWO grant, aimed at further developing the collaboration between ShAO and JIVE, we hope to hire an additional digital engineer soon, and to host a digital engineering student from ShAO for an extended period. In parallel to this, the development of a correlator control system took off when Des Small switched from the EXPReS to to ExBox project. Together with Harro Verkouter he is designing and developing a new EVN correlator control system, which will be capable of controlling both UniBoard and software correlators, and could be used as the basis of an Apertif control system. Software correlation Since the EXPReS project ended in August 2009, a lot of effort was spent on wrapping things up and doing the final tests/demos that were promised as deliverables to the EC. The distributed correlation scheme that we designed and implemented together with PSNC was tested. Some bugs in the system were found and either fixed or worked around. The system was demonstrated with some success. Unfortunately the approach chosen to comply with the Grid computing paradigm involves a lot of overhead transporting the data, so correlation doesn't seem to run at the desired speed. We hope to address this in a follow-on project to EXPReS. The SCARIe project is still going ahead at full steam. Aard Keimpema implemented pulsar gating/binning functionality that includes incoherent de-dispersion across each subband. Test observations have been scheduled for session 3 2009 in order to obtain data that can be used to test/verify this new functionality. Also work is being done to remove a restriction on the maximum number of spectral channels. At the University of Amsterdam, our partner institute within SCARIe, Mihai Cristea has started on the 1st of July in the position left by Damien Marchal at the end of 2008. His work will focus on the networking side of the correlator. Currently he is working out some ideas to guarantee the availability of bandwidth inside a shared compute cluster running the software correlator. NETWORK SUPPORT The automatic-ftp feature continues to be exercised in all network monitoring experiments. Stations send the specified portion of a scan directly to the software correlator computer at JIVE. The correlation is now carried out by the SXFC software correlator being developed under FABRIC/SCARIe. Correlation results go to a web page available to all the stations within a couple hours, and Skype chat sessions during the NME provides the station friends with even more immediate initial feedback. Irbene continues to participate in some C-band NMEs, but currently seems limited by the older version of their dBBC, which is being addressed separately apart from JIVE. We continue to process all experiments, including NMEs, via the pipeline (now run via ParselTongue), with results being posted to the EVN web pages. The pipeline provides stations with feedback on their general performance and in particular on their gain corrections, and identifies stations/frequency bands with particular problems. Timely delivery of ANTAB amplitude calibration results from the telescopes seems to be improving, but can remain an issue in e-VLBI experiments due to the shorter time-scales involved. Since the previous CBD meeting, Jun Yang determined improved SEFD's for Jodrell Bank Mark2 and Cambridge from analysis of a series of NME's. USER SUPPORT The EVN Archive at JIVE continues to provide web access to the station feedback, standard plots, pipeline results, and FITS files. Access and public-release policy remain the same. During this period we improved the redundancy of the RAID, prompted by the failure of two disks that required us to manually restore some FITS files from backup tapes. Following this reconfiguration, we have 12.8 TB of dedicated disk space, with a buffer of another 1.8 TB that also houses the pipeline work area. At the end of this reporting period, we have 7.0 TB of FITS files on the archive, a gain of 0.8 TB over the five months. Three users visited JIVE for EVN data reduction in this period, not including graduate students specifically associated with JIVE staff members or projects. There were a further two visitors who came to collaborate on development or non-EVN astronomical issues. Additionally, there were three EVN data-reduction visits to other EVN institutes (Jodrell Bank, MPIfR, IRA) supported by the RadioNet FP7 EVN trans-national access. We continue to contact all PI's once the block schedule is made public, and to check over schedules posted to VLBEER prior to stations downloading them. This occupies occupies a great deal of time in the fourth-second weeks before the start of the session, but helps to prevent avoidable errors in the observations themselves and thus makes the available observing time more productive.