CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN TOG MEETING, Wettzell, 1 APRIL 2004 19 March 2004 (statistics cover 24 Jun 2003 - 18 Mar 2004) Huib van Langevelde Bob Campbell SCIENCE OPERATIONS Sessions and their Experiments Since the previous TOG report (24jun03), 30 user projects and 23 NMEs/tests have been completed. Three user experiments from the May/June'03 session are still in the queue to correlate: EB022C - Jb had an 800kHz LO offset, which we can now accommodate; currently running. GP036B - a bug in the correlator control software was found in the case of 2-bit, fan-out=4 data that was 'reassigning' LSB channels in the upper half of the observed subbands to be USB. This is now repaired and correlation will occur after EB022C. EM048 - awaiting PIs to provide revised coordinates for their targets based on a preliminary correlation pass we did for them using only short baselines and a short integration time. One user experiment remains from the November session: GI001A - uses oversampling x8 (0.5MHz filters at 8Ms/s), which has never been advertised, nor fully tested. An initial attempt did not succeed, but it is too early to know how serious this problem is. One old user experiment remains on hold because parts of it use 40ips recording, which would require speed-up to correlate. During this time, we have also distributed 27 user experiments to the PIs, including 5 from the November session. 25 user experiments were both done and distributed in this period. To review some landmarks from the sessions that are still 'active': May/Jun: 1st 512Mb/s user experiment (more in Nov, Feb) 1st user experiments with >16sta (more in Nov) 1st user experiment with a sub-netted schedule (not yet advertised, required some manual hacks to make a VEX file for correlating) Nov: 1st station recording all experiments regularly onto Mk5 disks 1st fully successful 512Mb/s user experiment 1st oversampling x8 experiment (see above) Feb: 1st sub-second integration-time user experiments (directly coupled to this will be our first >60GB FITS files) The following table summarizes by session the user experiments still in the queue (entries = remaining to do / total): N_exp Est.Corr.Time May/Jun 3/25 68/542 hr (one of these running now) Nov 1/10 75/236 hr Feb 13/13 364/364 hr The actual correlator time is typically between 1.5-2.5 times these estimates, depending on the number of redos. Within the February session, three experiments account for 168 of the total hours; these require 4 passes each (each subband/polarization uses the full correlator), and will each produce ~575GB of FITS files. Infrastructure We seem to have encountered fewer problems arising from PIs not following the Bologna rules compared to our experience in the May/June session. We made more of an effort to be pro-active in offering help to PIs prior to the schedule due-date; hopefully there is a causal relationship between these statements. It's hard to see how to avoid the remaining sort of problems (e.g., experiment with oversampling x8) in our current scheme where PIs deposit schedules directly to where stations retrieve them. Were we to take the time to check the schedules & possibly confer with the PI leading to a new one, there's no safety against stations already having downloaded the original one, leading back to the familiar multiple-schedule syndrome. In the current world of multiple experiments per disk-pack, the station logs become even more indispensable to being able to associate physical media with experiment: the FS logs tell us which disks we need, but the disks are stored via scanning the paper label. Therefore mismatches between the internal VSN and external paper labels can cause lengthly manual searches of the field-labels for experiment name and time-range. We've encountered some growing pains in this area, but I'm sure things will improve with experience. (disk2net experiments, where the FS log mentions the disk-pack used at the station but which may or may not be sent to us, is a pernicious case for our current software.) Another consequence of multiple experiment per disk-pack is that releasing disk-packs is no longer a one-to-one function of completing experiments. Currently, we prioritize correlation for network tests, experiments with PhD-student authors, experiments for which people want to visit at a specific time, etc. And of course, some experiments have to be kept longer because of capability issues. Only after such experiment-specific factors does observing chronology enter in. An illustrative example: in Feb there were 9 test experiments, all of which have been completed. Three stations were disk-only throughout the whole session. The table below summarizes disk usage and releasability based on having completed all the test experiments for these stations: Ef On Wb tot Total number of disk-packs in session 12 13 10 35 Number of disk-packs involved in tests 6 7 5 18 Releasable disk-packs after finishing tests 1 1 0 2 From a different viewpoint, by not having correlated EB027A we can't release 6 disk-packs, but if we correlate it next, we could release only 2. For EB025, the corresponding numbers would be 5 & 0. We may have to put higher weight to chronology in order to release disk-packs at a reasonably constant rate. TECHNICAL DEVELOPMENTS Mixed disk/tape operations are now the norm. Progress continues incrementally on optimizing various aspects of the station-control and media-database software as increasingly subtle differences between the disks and tapes become apparent operationally. The biggest advance from an efficiency standpoint has been creating a messaging system between the SUs and the Mk5 units that allows multiple start/stop times to be specified at the job initialization. Prior to this, a new job had to be started at every gap in the schedule, incurring a ~10min penalty for the tape start-up operations. Now, from the operator's viewpoint the mix of tapes and disks requires no different treatment. In the worst case from the November session, not having this new fix carried a ~35% penalty in correlator time. Of course, the full efficiency gains won't come until we have only-disk experiments, since every job has to wait for the slowest station. We currently have 14 working DPUs and six Mk5A units attached to the SUs for operations, with another three in house. Enough EVN stations have shifted over to all-disk operations that this mix of DPU/Mk5 units is sufficient for 1-pass correlation of any reasonable experiment (that wouldn't need multi-pass correlation anyway because it had >16 stations). We have a medium-term plan to shift to a 12-12 DPU/Mk5 mix, wherein four of the Mk5s would be fully connected to their SU, and the other eight would share an SU with a DPU. Reduction in the number of DPUs provides the opportunity to cannibalize some of the key components (capstan motors, heads) as the need may arise. Progress continues on the PCInt project, which will eventually allow read out of the whole correlator with integration times as low as 1/64s. A principal milestone has been passed with the ability to read the entire correlator off with integration times of 0.25s, or half the correlator with integration times of 0.125s. Work towards purchasing the necessary hardware to further reduce the integration times is ongoing. We've created a document going over the field-of-view improvements resulting from these shorter integration times and the existing spectral resolution, and have put it on the Users_Guide portion of the EVN web site. We've also added some field-of-view capabilities to the "EVN calculator" there. These are currently parameterized by the field of view having less than a 10% reduction in the response to a point source; we aim to add more flexibility by allowing computations of percent loss at a specified radius as well as considering image broadening. As mentioned in the previous TOG, we now apply a post-correlation 2-bit van Vleck correction to compensate for the sampler statistics in each subband. In the February session, the On sampler statistics are much better than they were in May/June and November (when they had too low a fraction of high bits). A new version of the correlator software with the latest c++ compiler is almost complete. Better maintainability and more robust performance is expected from this. A long standing bug with midnight crossing scans has been fixed. The new EVN archive at JIVE is up and running. This provides web access to the standard-plots, pipeline results, and following a 1-yr proprietary period, the FITS data themselves. There are capabilities to search by experiment or by source (via the Bologna web page). The EVN Feedback facility is also linked via the archive. JIVE is preparing for the FP6 project ALBUS and is involved in a collaboration with the WSRT in NL-GRID. These projects concern the use of high performance (cluster) computers for radio-astronomy data processing.