CORRELATOR REPORT, EVN MkIV DATA PROCESSOR AT JIVE EVN PC MEETING, JUNE 2003 Bob Campbell June 18 2003 (statistics cover Mar 01 2003 - Jun 18 2003) INTRODUCTION We cleared the backlog of experiments awaiting correlation in early May. This provided some time for more dedicated testing prior to the arrival of tapes from the May/June session. Because of the many rescheduled experiments from previous cancelled sessions, the load from the current session appears rather heavy, and includes some "firsts" for us in terms of some specific experiment characteristics. Even though correlation of all May/June experiments may not be complete by the next session, tape availability for the next two sessions, assuming a demand more in line with past trends (~200-250 per session), appears to pose no problem at the current time. SCIENCE OPERATIONS Since 1 March, 9 user projects and 9 NMEs/tests have been completed. We finished the backlog of existing observations in early May (8 of the 9 completed user experiments were part of the backlog). The time from then until early June, when tests and user experiments from the May/June session began to arrive, was spent developing and testing various new features -- speed-up, making a new Mk5-savvy release, PCI groundwork, etc. One user experiment remains on hold because parts of it use 40ips recording, which would require speed-up to correlate. During this time, we have also distributed 9 user experiments to the PIs, including all of the backlogged experiments above. Thus, by early June there was no remaining backlog in experiments either to be correlated or to be distributed. We released 7 user and 5 test experiments, comprising 62 thin tapes, in the period. The released user experiments had all been completed and distributed prior to this reporting period. The load from the May/June session seems large: 23 user experiments containing 389 tapes and requiring an estimated 449-500 correlation hours, not counting network-test experiments or any potential recorrelations that may be necessary for the user experiments (the upper limit assumes that no station fails in GJ010A; this has 17-18 stations for a large fraction of the time, which would require three passes). (There is a further 61 hr implied by PI requests for separate line and continuum passes for spectral-line observations that have two identical sub-bands -- something whose value I find hard to discern, since the 2 subbands that would be correlated in the continuum pass don't have significantly different independent data with which to boost the SNR of the continuum subtraction -- but I haven't resolved this issue with the PIs yet). Included in this session are: our first 512Mb/s user experiment, our largest network to date (21 stations participating), and our first experiment that uses a sub-netted schedule (although this capability has never been advertized as being available). We currently have 225-246 thin tapes which should be able to ship for the next session without much risk (i.e., either already released, an automatic release-by date in July, a already-passed by-PI release date with the PI in our building, etc.) Of the incoming experiments, NMEs/tests together with just the 5 user experiments with the highest tape-completion/correlation-hour ratio will free up an additional 160. So even though the size of the current session will quite possibly give rise to a backlog at the time of the November session, having 200-250 tapes for each of the next two sessions doesn't seem problematic at current correlation efficiencies. (A linear fit to the 8-week-smoothed Correlator Yield efficiency over the past two years shows that it is 24.5%, with an upwards trend of 0.13%/week.) We have also recently sent 51 tapes to Socorro to help balance the net trans-Atlantic tape flux for global experiments over the last few sessions. These tapes are not included in the above numbers. TECHNICAL DEVELOPMENTS Construction of the new ASTRON computer room in the basement is complete. Production correlation wasn't disrupted (thanks to having cleared the backlog in time prior to the start of the construction work). The project to incorporate disk systems fully into the correlator software has been completed. This implementation takes full advantage of the operational characteristics of disks. The correlation control programs can now handle tape-only, disk-only, or mixed disk-tape experiments, but there remains somewhat more manual VEX-file preparation necessary for disks; fuller automation is under development and should be testable soon. Changes have been made to the logistics procedures to allow multiple experiments per disk and storage of the disks in the paternoster. Overall, we are ready to connect Mk5 units for production correlation. We currently have 4 Mk5 systems operational for use. It has turned out to be very difficult to start using speed-up correlation for production. Although we have achieved fringes months ago, firmware changes are still required to resolve some timing issues. The introduction of disk systems has actually allowed faster testing of these features. It has been known for quite a while that the EVN MkIV correlator is particularly sensitive to poor sampler statistics at the telescopes. The van Vleck correction for 2-bit data depends on the fraction of high- and low-magnitude bits (i.e., the sampler statistics), but this dependence wasn't being taken into account. If a BBC at a station had sampler statistics different from the expected value, a non-unity autocorrelation peak would result. The amplitudes on baselines containing it would be affected to a lesser degree. If the sampler statistics were especially bad on both stations in a baseline, even the closure amplitudes to which the baseline contributes could be affected at a ~1% level. The original on-line van Vleck correction has been switched off, and we now compensate for the observed sampler statistics per station/BBC/integration, as derived from the autocorrelation (lag) spectra peak amplitudes, in a post-correlation program in aips++ (similar to the AIPS task ACCOR, which is VLBA specific). Progress continues on the PCInt project, which will eventually allow read out of the whole correlator with integration times as low as 1/64s. New hardware has been added to the correlator and testing continues on its robustness in operational situations. At the same time, recirculation, is undergoing its initial testing. New computers have been acquired to facilitate the collection of high data rates and large projects. The new EVN archive at JIVE is up and running. This provides web-access to the standard-plots, pipeline results, and FITS data (the latter subject to an 18-month delay for public release) for experiments that have been correlated at JIVE. A further link to the EVN Feedback facility is planned soon.