Download this report as a PDF.

This is the report for the end of the second qurter of the project (months 4-6).

Link to the Deliverables Table

Deliverables due during the second reporting quarter:

The following deliverables were submitted to and their receipt acknowledged by the Commission during this quarter:

NEXPReS and project officer Siren have exchanged emails and decided to move away from monthly updates to a quarterly format. From this month, we will produce a report every three months. The quarter in which the period (annual) review is held, no update will be provided as the materials should (had better) be in the annual report. I expect that this announcement will be received with minor celebration among the other work package leaders.

Astron is leading a call and has been in contact with JIVE to participate. Astron has also asked for input on the construction of the proposal as it is felt that the NEXPReS was well executed.

This is also an internal reminder indicating that we will try to be better about cc'ing the NEXPReS project's email address provided by the EC for all official communications. To this end, we have set up an internal mailing list called NEXPReS-deliverables that will be used for official communications. The mailing list members are our project officer, his support secretary, the project manager and the NEXPReS Secretary email accounts. Only the project manager and the project secretary account will be able to send to the list.

Note that the deliverables table has been edited slightly to remove some redundant information and to make it easier to read. It continues to be updated regularly as deliverables are completed.

The mailing lists for the project are being used and have seen the expected bursts of traffic at the start of the project along with some periods of silence.

WP1 has one outstanding deliverable, the Consortium Agreement, which should be confirmed by the board early in 2011.

No Meetings were hosted by NA1 this quarter.

The Project Manager attended the 8th e-Infrastructure Concertation event in Geneva, Switzerland. The meeting was held 4-5 November 2010 at the CERN facility. NEXPReS was given a chance to present a quick overview of the project along with other projects from the funded cohort. Additional information about the meeting is available from the meeting website: http://www.e-sciencetalk.org/e-concertation/. Note, the agenda is made available via the CERN website. A PDF copy is uploaded here in case the CERN site removes information about old meetings.

At the meeting, the Project Manager gave an informal video interview with the Grid Talk Project. The interview is available online on their website: http://gridtalk-project.blogspot.com/.

The EVN-NREN work package is scheduled to have a meeting in the new year. Discussions are currently underway with several other interested groups (non-NEXPReS organizations) to have the EVN-NREN meeting immediately before/after their meeting. The EVN-NREN meeting is seen as a good way to draw both researchers and networking professionals together. We should know by the end of January if we will partner with another group for the meeting and be able to start advertising information about the location.

No deliverables were due this quarter.

The workpackage hosted no meetings this quarter.

There were no presentations on behalf of the EVN-NREN this quarter.

Contacts with the EVN PC Chairperson (Tiziana Venturi) to coordinate a possible face-to-face eVSAG meeting to be organized at the time of an EVN PC meeting.

Collected information on EVN vs. e-EVN proposals submitted in the last years.

Collected information from users regarding their experiences and expectations on e-VLBI.

No deliverables were due this quarter.

The workpackage hosted no meetings this quarter.

There were no presentations on behalf of the eVSAG this quarter.

The WP4 leader collaborated with professional artists in months 2-5 to create a visual representation of the project that could be incorporated into all outreach materials, giving them a unified look and feel. This illustration was used in the large display (D4.4) and will also be used for slide templates, print materials and an updated header for the project web site.

The display made its debut at the 9th International e-VLBI Workshop in Perth, Australia, in month 4. It was therefore unavailable for the ASTRON/JIVE Open Day, and NEXPReS was represented instead with an A0 poster version.

Other WP4 activities this quarter included preparation for the ASTRON/JIVE Open Day (24/10/2010) and work on the first set of print materials (D4.6), due in the next quarter.

D4.5 Visual Guide/Elements was completed in month 5 (due month 6). Graphic resources including project logos, visual illustrations, colors and fonts have been posted to WP4 section of the wiki. An email about availability and use of these resources was also sent to all project partners in month 5.

No WP4 meetings are necessary.

WP4 leader Kristine Yun presented outreach activities of NEXPReS (plus EXPReS and JIVE) at a RadioNet Outreach Workshop in month 5. The idea was to share experiences and activities among all European radio astronomy institutes and to find ways to share resources. NEXPReS will likely be able to contribute to and benefit from shared work on brochures and videos, in particular.

At TUM, good progress was made on task 3, with updates, fixes and essential add-ons of existing code:

• Help pages forward
• Remote system call handling
• Update of MathPlot for monitoring plots
• BugFixes of internal modules like timecalc
• Fixes of compiler warnings and static code analysis, tests on different 32-bi/64-bit operating system platforms
• First attempts to use the communication generator under Windows
• Tests for an installation of a NASA Field System PC on Ubuntu Linux

Measuring functionality was added:

• System information, network-utility (traffic load etc.)
• Cyclic redundancy check (CRC32) for internal functionality cross-check
• Software quality code test coverage, automatic test suite

A Web server will be installed end of December/mid January. Several tele-conferences with colleagues at MIT Haystack Observatory and NASA Washington were held. Preparations for the standardization for System Monitoring within the MCI (Monitoring and Control Infrastructure) Collaboration Group are ongoing.

After an initial slow start, work at JIVE picked up considerably. One staff member is now working full-time on WP5, with several others contributing to the project part-time. A new staff position was advertised, but unfortunately no suitable candidates were found so far. A new advertisement for scientific software engineering positions in WP5 and WP7 will go out in January 2011.

Functionality was added to the locally developed Mark5 control code jive5ab. VDIF-Mark5B on-the-fly conversion software was developed and awaits testing. A first step was made towards a generic correlator user interface, which will be capable of controlling both hardware and software correlators and which will be essential for automating operations. 4Gbps-capable Mark5C recording systems were ordered from the Conduant Corporation and were delivered in November 2010. Several of these have been integrated into the correlator system and tests are ongoing.

The draft of D5.3, Definitions of specifications monitoring and remote control, was revised and the final version was submitted in October 2010. D5.2, Metrics, selection criteria and user feedback, was completed on December 21 2010.

None

Presentations:

• EXPReS, NEXPReS, UniBoard: Developments in European VLBI A. Szomoru, 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 19 November, 2010

• SysMon, a robust and flexible remote monitoring system for VLBI and more, Ettl M., Neidhardt A., Mühlbauer M., Riedere M., Kronschnabel G., Himwich, E., Beaudoin C., Plötz C., Dassing R., 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 19 November, 2010

• EXPReS, NEXPReS, UniBoard: Developments in European VLBI A. Szomoru, Working visit to Shanghai Astronomical Observatory, Shanghai, China, 25 November, 2010

• Network Communications, Schreiber, U., Neidhardt A. et al., GGOS Bureau for Networks and Communications at the AGU, San Francisco, USA, 15 December, 2010

• Monitoring and Control Infrastructure, Beaudoin C. et al., GGOS Bureau for Networks and Communications at the AGU, San Francisco, USA, 15 December, 2010

Posters:

• First proof of concept of remote attendance for future observation strategies between Wettzell (Germany) and Concepcion (Chile), Ettl M., Neidhardt A., Mühlbauer M., Plötz C., Hase H.; Sobarzo S., Herrera C., Oñate E., Zaror P., Pedreros F., Zapato O., 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Western Australia, 18 October, 2010

Several participants in WP6 came together for a face-to-face meeting just prior to the official NEXPReS kick-off meeting in Manchester.

Work on the architecture document (milestone MS601) is progressing well and should be finished shortly. It is somewhat delayed due to the late start of the project itself, and the fact that most participants could only start the hiring process once project was started. There have been three WP6 telecoms to coordinate the work on MS601: the minutes of all three are linked below.

Minutes of WP6 Kickoff Telecon on 2010-08-27

Minutes of WP6 MS601 telecon 2010-11-02

Minutes of WP6 MS601 telecon 2010-11-30

Work has also started on designing and configuring the first international BoD path, both as a test and as a first step towards deliverable D6.1.

(none in this quarter)

See above for the telecons held in WP6 and their minutes

The activity leader (Paul Boven, JIVE) gave several presentations:

High speed continuous recording and playback for VLBI, presentation for the Terena 7th TF-Storage meeting, Poznan, Poland, 2010-09-09 (11MB)

NEXPReS: Dynamische Lichtpaden voor Radio Astronomie, presentation for the SURFnet Relatiedagen 2010, Noordwijkerhout, NL, 2010-12-13 (10MB, in Dutch)

Dynamic Networking for NEXPReS, presentation at the 9th e-VLBI Workshop, Perth, Australia, 2010-10-20 (7MB)

Mark Kettenis attended the 9th International e-VLBI workshop in Perth, Australia, and gave a presentation on WP7.

Since several key people to doing e-VLBI with software correlators were present, a lot of useful exchange of ideas happened. These include discussions about the implementation of e-VLBI functionality in the DiFX software correlator with Chris Phillips (CSIRO). Also the VDIF/VTP taskforce met, and made some good progress in defining the VLBI Transport Protocol (VTP). There is still some discussion on reply packets. A proposal for the information within those packets has been made.

Progress is also being made to incorporate VDIF support into the international VEX standard. Within the VEX2 committee, of which Mark Kettenis is a member, consensus has been reached about most aspects, and an updated standard is expected to be finalized within the next few months. This will help us with scheduling experiments for distributed correlation.

After the workshop Mark also visited CURTIN for two days, and fleshed out some details of the collaboration. Cormac Reynolds, who is head of the VLBI group at CURTIN, will be our primary contact there.

Demonstration of the SFXC software correlator at the Dutch Research exhibit at SC10, The International Conference for High Performance Computing Networking Storage and Analysis in New Orleans, United States. During this demonstration, pre-recorded data of a pulsar observation sitting on diskpacks at JIVE was streamed into a cluster in Amsterdam where it was correlated. The result of the correlation, a pulse profile for the pulsar, was displayed at the exhibit in New Orleans and updated while the correlation was running. This shows that, with the right setup, streaming data into a software correlator is feasible and that it i possible to provide real-time feedback on the correlation. Some discussions with people form SARA, investigating the potential to make use of the Dutch National Grid Infrastructure (BIG Grid) in NEXPReS.

Unfortunately deliverable D7.2 wasn't finished before all this travel started, and very little time was found afterwards to finish the document. There is very little work that still needs to be done, so the report is expected to be finished early January.

Support for correlating multiple phase centers was added to SFXC and is currently being tested. This feature allows the scientists to observe a larger part of the sky and generate several smaller datasets instead of a single big dataset that is difficult to handle. This improves the survey capabilities of our instrument and infrastructure, which will help us achieve some of the science goals mentioned in the NEXPReS proposal.

Support for reading data directly from Mark5 diskpacks was added to SFXC. This allows us the bypass the fuseMk5A filesystem. This will greatly help deliverable D7.4 as it will simplify operations at the observatories since it eliminates an action that was done manually.

Also, the first EVN user experiments were correlated using SFXC. As a result a lot of fixes to the codebase were made that improve the stability. This is crucial for running automated e-VLBI correlations.

Some effort was put into optimizing the corner turning done by the channel extractor in the software correlator. Hand-written assembly language implementations that use SSE instructions were written for the most common setups that are in use. Compared to the (already optimized) C versions of the code these are twice as fast. The same code can be used to convert the legacy Mark4, VLBA and Mark5B data formats into VDIF such that separate data streams can be created that are needed to distribute the correlation over several clusters.

At OSO work continued on gathering telescope operator requirements for the work flow manager. These have now been compiled into a report which was distributed to relevant partners at the beginning of November, satisfying deliverable D7.1. Investigations into semi-automated telescope schedule creation have begun; existing tools used for working with schedules have been examined to get a feel for what is required as well as locate possibilities for re-using or taking inspiration from existing code. Programming languages appropriate to this task have also been looked at.

To provide better performance of Workflow Framework a proper workflow description language has to be chosen. In previous version of EXPReS a self-developed solution basing on JSON was harnessed, however the effort to maintain this solution is increasing - especially when adding functionalities not foreseen at the beginning of implementation. Another aspect is to support standards or de facto standards. Regarding those arguments we decided to use an already existing workflow language. Since there are many solutions available a wide research was made in this area. The first step was to narrow the area of interest. We have identified couple of groups of workflow languages and then found the most popular or of wider use among those groups. The groups are: BPEL family (BPEL itself and other BPEL-related languages), WebServices family (workflows basing on WS idea, using WS mechanisms or protocols), Grid/Scientific family(languages developed for Grid environments or to support scientist of different domains).

The first step was to narrow the area of interest. We have identified couple of groups of workflow languages and then found the most popular or of wider use among those groups. The groups are: BPEL family (BPEL itself and other BPEL-related languages), WebServices family (workflows basing on WS idea, using WS mechanisms or protocols), Grid/Scientific family(languages developed for Grid environments or to support scientist of different domains). See PDF for details.

Criteria for selecting a workflow language were formulated and each language was evaluated. Again see PDF for details. This allowed us to choose MoML as a new language for KIWI Workflow Platform. The Kiwi Workflow Platform will be used within NEXPReS to support telescope and correlator operators. The main highlights of MoML are its extensibility, action-centric design, de-facto standard in many scientific domains and support provided by groups of interest gathered around projects such as Kepler. Another point of great significance is the availability of both – MoML engine and graphical libraries as well as some work in field of web applications interfaces.

We have also started an initial work on MoML language, as well as MoML engine which we would like to use in the NEXPReS.

At Ventspils, Normunds Jekabsons has started experimenting with the SFXC correlator on their cluster.

D7.1 was completed on November 10 2010.

D7.2 is still in progress. It is almost completely done but still needs some small touch-ups the appendix on invoking the SFXC software correlator needs to be integrated. This is expected to be completed in early January 2011.

A telecon was held on Telecon October 12, 2010. The minutes can be found on the project WiKi.

Automated distribute correlation on a shared computing infrastructure, M.M. Kettenis, 9th International e-VLBI Workshop: e-VLBI and the Path to the Square Kilometre Array, Perth, Australia, 18 October 2010

Smart Infrastructure for VLBI Software Correlation (poster and demonstration), M.L. Cristea, R.J. Strijkers, R.J. Meijer, V. Korkhov, A. Belloum, M.M. Kettenis, K. Keimpema, C. de Laat, R.J. Meijer, New Orleans, US, 15-18 November 2010

The activity has continued as planned. WP8 has looked further into the subjects designated in the presentation of September board meeting.

It has been diffiuclt to hire people to some of the tasks requiring simultaneous expertise in several fields (Linux kernel, advanced communications, hardware and FPGAs, networking security etc.). We finally managed to hire Esa Turtiainen beginning of December 2010 but we are still looking for at least one additional resource. WP8 has also been able to extract in-kind contributions (not paid by NEXPReS) from Metsähovi Linux experts Timo Lindfors and Juha Aatrokoski in disk and 10GbE Linux performance related issues.

WP8 is on-track in getting explanations and solutions to most of the open questions of EXPReS 20-disk 4Gbps performance issues, CPU consumption, eliminating expensive memcpy()s, and further understanding how to deal with close-to-10Gbps UDP networking efficiently. Additionally, ideas how to do simultaneous reads and writes to the same set of disks and how to make best use of large RAM buffers are evolving.

Deliverable D8.1 was released mid-December (due month 6).

None.

There is a plan for WP8 leader Ari Mujunen to give a colloqium talk at ASTRON/JIVE in February 2011 on the subject of evolving storage in astronomy computing.