EVN TOG - March 2006 Procedures for Pre-Session Distribution Bob Campbell, JIVE 1. The first step in deterimining how many disk-packs to distribute to stations prior to a network session is receipt of the block schedule. From this, the total recording load per station can be calculated by summing, over all experiments in which a station participates, the product of (recording-rate * duration) for each experiment. a. There is some question whether a net 90% duty-cycle should be assumed in PI scheduling -- that would obviously allow ~10% more recorded data to be scheduled, but if PIs exceed this (on net), then the distributed disk-packs may prove insufficient. Even though the 90% rule has empirically been honored on average in the past couple sessions, disk recording does have fewer intrinsic gaps, and I'm probably leaning towards using a 100% duty-cycle rule in calculating pack needs, to be sure enough would be distributed. b. Another factor to keep in mind is the destination correlator per experiment. Recordings destined for different correlators should not be put onto the same disk-pack, which may lead to some inefficient use of available space given the quantized sizes of packs. 2. The next step in determinig how much to distribute is to determine the inventory available at the stations & correlators. To minimize shipping costs and maintain flexibility to respond to unanticipated experiments, we aim to distribute just enough to bring individual stations up to their required load for the session. Therefore, it's important to know how much is already on hand at the station, either from excess from previous sessions or from new purchases. If there are multiple destination-correlators for a station's recordings, then it would also be useful to know the number/size distribution of the available packs. The number and sizes available for release at the correlators (Bonn, JIVE) would form the final piece of the puzzle. If the session is expected to be disk-space limited, then it's quite possible the creation of the block schedule may be an iterative process, taking into account various scenarios to see whether the resulting loads could be supported by the available packs. 3. At some point, a set of packs to distribute to each station is determined. This takes into account: a. the load per destination correlator -- we currently try to provide, as separate sub-distributions, the load + ~0.5TB for each correlator. The rationale for this safety margin is to allow ~2 individual disks to fail. Ideally, an entire pack safety margin would be more robust, but this could limit the total number of experiments that could be supported, especially when there are experiments going to JIVE, Socorro, and Haystack from several stations. b. I try to provide specific (sub)-shipment break-downs (N_packs, sizes) that should meet the above requirements for each destination correlator. c. There may be some individual stations having particular needs. Usually, this has included returning specific disk packs to the purchasing station to effect repairs. In the future, with the newly agreed tactic of JIVE repairing packs and sending individual disks to the purchasing station for warranty service/replacement, this category would become obsolete. 4. There are some logistical points to keep in mind; some agreed upon at the CBD level, and some that evolve as we learn more about optimizing the distribution: a. EVN full-member institutes pay for the shipping from JIVE to the station. Hence, we try to coordinate with the stations to use a shipper who provides reliable and economical service. Because we don't get charged for these shipments, we don't get feedback about these aspects unless the receiving station brings things to our attention. b. JIVE pays for shipments to non-full-member institutes (typically, Ro, Ar, VLBA). From these shipments over the past couple years, we have become aware of some undesirable characteristics of the shipper we had been using (expensive, often encountered difficulties clearing customs to the US, etc.), and have switched companies. c. We try to ship packs out to arrive at least 2 weeks prior to the session (shipments to China/South Africa leave 4-5 weeks prior; continental Europe typically the week after). Of course, a constraint on the earliest possible shipping date is the existence of the block schedule and an inventory of disk-packs already on hand at the stations, so at least in terms of the latter, you can see a direct benefit from providing this inventory in a timely fashion. In order to minimize shipping costs to the distant stations, we generally try to meet the load for non-continental stations with the fewest possible packs. 5. Tables of disk receipt/distribution: Total net disk flux (following 2-session rule): IN OUT Nov'03 -> May'04 22.00 30.52 Feb'04 -> Oct'04 63.69 86.31 May'04 -> Feb'05 111.52 69.89 Oct'04 -> Jun'05 122.53 100.67 Feb'05 -> Oct'05 140.58 170.15 Jun'05 -> Feb'06 176.99 184.53 Oct'05 -> Jun'06 245.41 Feb'06 -> Oct'06 221.27 EVN-only receipts/distributions: through Oct'04->Jun'05, same as above Feb'05 -> Oct'05 140.58 142.31 Jun'05 -> Feb'06 176.99 152.93 Oct'05 -> Jun'06 211.85 Feb'06 -> Oct'06 191.05 VLBA-only distributions/receipts (these are currently arranged on a pre-positioned basis): OUT IN Oct'05 27.84 33.56 Feb'06 31.60 30.22 (these numbers reflect only the packs distributed by JIVE for observations of global experiments that will be correlated at JIVE. There is an additional westward flux of disk-packs in globals observed at EVN stations to be correlated in Socorro, but I do not have a firm set of numbers for this -- I do know what we have sent EVN stations with the intent of use in experiments to be correlated in Socorro, but I don't know if those packs were so used -- and some stations may have sent packs to Socorro from out of their on-hand supply, which would not register in our bookkeeping at all.)