CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN TOG MEETING, Onsala, 1 JULY 2005 16 June 2005 (statistics cover 11 Nov 2004 - 15 Jun 2005) Huib van Langevelde Bob Campbell SCIENCE OPERATIONS Sessions and their Experiments The table below summarizes projects correlated, distributed, and released from 11 November to 15 June. The table lists the number of experiments as well as the network hours and correlator hours for both user and test/NME experiments. Here, correlator hours are the network hours multiplied by any multiple correlation passes required (e.g., continuum/line, >16 station, 2 head stacks, different phase centers, etc.) User Experiments Test & Network Monitoring N Ntwk_hr Corr_hr N Ntwk_hr Corr_hr Correlated 26 366 533 18 78 84 Distributed 24 334 486 19 79 85 Released 31 381 488 18 69 69 The following table summarizes by session the user experiments still in the queue, with an additional column for experiments not yet distributed (entries = remaining to do / total). The actual correlator time is typically between 1.5-2.5 times these estimates, depending on the number of redos or other problems. N_to.corr Corr.hrs N_to.dist May/Jun'03 1/25 11/447 hr 1/25 ad-hocs 0/2 0/28 hr 0/2 Nov'03 0/9 0/127 hr 0/9 Feb'04 3/13 132/294 hr 3/13 May/Jun'04 0/14 0/237 hr 0/14 ad-hocs 0/6 0/40 hr 0/6 Oct/Nov'04 1/14 24/280 hr 2/14 ad-hocs 0/2 0/32 hr 0/2 Feb'05 0/10 0/239 hr 3/10 The remaining experiment from the Oct/Nov'04 session is the Gb/s experiment for which the Jb disks were unplayable as recorded (only 0's recorded on even tracks from "head-stack" 2). It proved a more complicated evolution than anticipated to take the data off the original disks, process them in a normal computer, and re-write new disks. We can recover the "head-stack" 1 tracks, but the "head-stack" 2 odd tracks show playback problems. The three remaining experiments from the Feb'04 session each require 4 passes to provide the requested 1/8s integration times, and each of the three would produce ~575GB of correlator output. The next stage of PCInt development (full correlator read-out at 1/8s) would cut the correlation time in half (such that the entry in the above table would be 66/228 hr). One user experiment from the May/June'03 session remains in the queue to finish its second correlation run, awaiting PIs to provide revised coordinates for their targets based on a first correlation run we did using only short baselines and a short integration time. To review some landmarks from recent sessions: Feb'04: 1st sub-second integration-time user experiments Regular Mk5 recording by 3-4 stations per experiment May'04: Regular Mk5 recording by up to 8 stations per experiment Oct'04: 1st all-disk experiments (the 5cm sub-session [8station] was recorded entirely on disk) 1st overwhelmingly large user datasets (260GB of FITS files from one experiment) ad hocs: Huygens dress-rehearsal and actual observations: 1st Mk5 recordings from VLBA stations 1st fringes from Australian, Japanese stations most disk-stations being correlated at once (15) Feb'05: 1st 1Gb/s user experiment correlated and distributed Infrastructure We currently have 13 working DPUs and 15 Mk5A units attached to station units (SU) for operations. Ten of the Mk5A units are housed inside temperature- controlled cabinets, with the others sitting on benches behind the row of DPUs. Three Mk5A units are fully connected to their SU (all 64 tracks); the rest share their SU with a tape playback unit. Through a simple re-connection of 2 cables, we can fully connect all Mk5As to their SU. We have a medium-term plan of a mix of 12 DPUs and 12 Mk5As in cabinets, wherein four of the Mk5s would be fully connected to their SU, and the other eight would share an SU. This is on hold until we complete the remaining experiments from Feb'04 that have 13 tapes. The Reduction in the number of tape players has provided the opportunity to cannibalize some of the key components (capstan motors, heads) to use as replacement parts. Our disk-shipping requirements are derived from the recording capacity needed by a session (from the EVN scheduler) and the supply on-hand at the stations (from the TOG chairman). Using the policy that stations should buy two sessions' worth of disks, our disk flux should balance over the same two-session time frame. The following table charts our net disk flux since Mk5A recording began; each row corresponds to a two-session cycle (the syntax for all entries: N_packs "for" N_TB). Ad-hoc experiments and out-of-session tests are not included in these numbers. IN OUT NET OVER-DISTRIBUTION Oct'03 23 for 22.000 -> May'04 28 for 30.520 8 for 8.520 Feb'04 53 for 63.689 -> Oct'04 68 for 86.312 15 for 22.623 May'04 78 for 111.516 -> Feb'05 47 for 69.890 -31 for -41.626 Oct'04 87 for 122.529 -> Jun'05 72 for 101.882 -15 for -20.547 Feb'05 88 for 140.583 -> to recycle in Oct'05 We began by distributing more disks than conventionally required, which allowed stations that didn't have enough disk-packs on hand to participate as Mk5 stations. The size of the Feb'05 session was cut back to match disk availability. Recycling prior to the Jun'05 session fulfilled all the distribution requirements for the scheduled experiments. After distributing disks to the stations for the Jun'05 session, but before receipt of any recordings from Jun'05, we had 242.38TB of disks in house, with ~52TB freeable. Disks from Jun'05 have begun to arrive, but we do not yet have all of them. From the FS logs, we expect 98 packs for 173.675TB from EVN stations. TECHNICAL DEVELOPMENTS The special correlator mode for eVLBI has been expanded and it now allows the Mk5 configurations at the stations to be controlled from the correlator setup. In addition it has become much more straightforward to start new integrations during ongoing eVLBI observations. Most importantly the correlator robustness has been improved considerably, allowing longer unattended operations, and improving the success rate of eVLBI. Recently this has resulted in sucessful eVLBI observations with 5 stations at 128 Mb/s. In addition a web-based fringe display is now operational, allowing anybody to inspect the fringes during eVLBI events. Our new focus point is the start-up of disk-based recording, where there is still a annoying delay of a few seconds before the playback is fully servoed. This has now been traced back to the specific way the timing of the correlator at JIVE interacts with the Mk5 units. The progress on the PCInt project is still slow, but an implementation of the data handling was made that allows the usage of the PCInt backend cluster for storage of extremely large projects (several 100 GB). This also required the purchase and installation of a new LTO-3 tape device, in order to make back-ups of these large projects. The archive machine was also expanded, as it was rapidly getting too small. Provisions have been made to store a copy of all the user data outside the main Dwingeloo building. Within the scope of the ALBUS project we have been working on the phase-cal detection hard- and firmware. For a moment it seemed it would be impossible to obtain phase-cal detections in the normal JIVE operational mode, which employs short correlator frames. Recently this seems to have been cured. Phase-cal calibration data will become part of the standard data product, similarly to system temperature and flagging data, which is currently being tested. For several other projects initial work has been done on the requirements documentation, prompted by the RadioNet software forum in Jodrell Bank. In particular several lines of investigation were initiated for the ionospheric calibration project; a series of test observations are planned in the upcoming session. In the context of the FP6 project ALBUS we have started developing a Python interface to classic AIPS in collaboration with Bill Cotton (NRAO). The aim is to produce and distribute the software that needs to be made available to the user community in this package. It focuses on new uv calibration tasks and a scripting environment for distributed computing. The current test version allows AIPS scripting with increased functionality over the traditional POPS interface. It has been successfully tried by in-house astronomers for batch processing and seems to be a very promising tool for a wide range of users. USER SUPPORT The EVN archive at JIVE is up and running. This provides web access to the station feedback, standard plots, pipeline results, and FITS files. Public access to source-specific information is governed by the EVN Archive Policy -- sources identified by the PI as "private" have a one-year proprietary period, starting from distribution of the last experiment resulting from a proposal, prior to their FITS data and pipeline results being made public. Password protection in enabled for proprietary data. The one-month public-release warning has been sent to PIs of experiments prior to 1 June 2004 (release date 1 June 2005), and the process of sending out targeted one-month warnings to PIs of individual subsequently-distributed experiments, based on actual distribution date, has begun (current through 14 June 2005). There are two independent ways to search the Archive other than by direct entry via a specific experiment. The EVN catalogue of observations (Bologna) can be used to search for observations of particular sources, and provides a link to the relevant experiments on the EVN data archive for experiments correlated at JIVE. In addition, a FITS-finder utility for archived data is operational. A database contains all the meta-data for the projects that are on-line. Searches can be key to source names or coordinates, observing frequency, and/or participating telescopes, among others characteristics.