CERN/FOCUS 2000-007

Minutes 19

31^st October 2000

Present: T.Cass, M.Cattaneo (Secretary), M.Delfino, M.Ernst, B.Gobbo, R.Gokieli, A.Grant, F.Hemmer, H.-F.Hoffmann^*), V.Innocente, P.Jeffreys (Chairperson), M.Kienzle, J.Knobloch, M.Marquina, N.McCubbin, H.Meinhard, S.O’Neale, A.Norton, S.O'Neale, M.Pimiä, F.Ranjard, H.Renshall, L.Robertson, A.Sandoval, J.Shiers, A.Silverman, E.Valente^*), P.Vande Vyvre, W.von Rüden

Invited: J.-P.Baud^*), M.Dimou^*), J.Gordon, D.Heagerty^*), B.Segal^*), J.van Eldik^*)

Apologies: S.Jarp, J.May, K.Safarik, R.Voss (represented by H.Meinhard)

Absent: J.Altaber, R.Cashmore, F.Etienne, F.Gagliardi, D.Jacobs, W.Lerche, L.Mapelli, M.Mazzucato

*) part time

1. AGENDA
1. Consideration of Agenda
2. Minutes of last meeting and matters arising
3. Chairman's comments
4. Proposed dates for meetings in 2001 (M.Cattaneo)
2. Web servers and security (D.Heagerty)
3. CLASP project (D.Heagerty)
4. Proposal for improving service change announcements (M.Dimou)
5. Mass Storage Issues
   (Ref: FOCUS-99-11, minutes of 14^th FOCUS meeting, 1^st July 1999)
1. Review of current tape technologies and policy (Harry Renshall)
2. Overview of Storage Area Networks (Ben Segal)
3. Plans for HPSS, Castor etc. (Jean-Philippe Baud)
4. DataGrid and implications for CERN
1. WP2 (Ben Segal)
2. WP5 (John Gordon)
6. Update on ongoing IT activities
1. Termination of X terminal repair service (T.Cass)
2. Consolidation of SUN physics services (L.Robertson)
7. Actions outstanding
8. A.O.B. (includes raising of user issues not covered by agenda)
1. New CERN computing rules
2. LEP Higgs search MC productions (DELPHI: J.van Eldik, ALEPH: F.Ranjard)

The chairman opened the meeting by pointing out that the operation of FOCUS is still being tuned to ensure meetings finish on time. Changes introduced with this meeting are that IT has been asked to predict the time needed for items presented under "Update on ongoing IT activities", and that FOCUS members have been asked to request items under A.O.B. at the latest two days before the meeting.

The chairman thanked Helge Meinhard for distributing his excellent paper on requirements for Linux on laptops.

The minutes had already been approved by E-mail. There were no further comments.

The chairman welcomed new members to FOCUS and thanked departing members:

Wolfgang von Rüden replaces Eric McIntosh as IT/PDP group leader.

Maria Kienzle replaces Bob Clare as L3 representative.

Vincenzo Innocente replaces Lucas Taylor as second CMS representative.

In addition, on suggestion by Jürgen May, the Desktop Forum (DTF) secretary (Alan Silverman) has been invited to attend FOCUS meetings, and the FOCUS secretary has been invited to attend DTF meetings.

The chairman then pointed out that FOCUS should be more explicit when approving proposals presented to it. Some misunderstandings have arisen in the past when proposals made at FOCUS have been misinterpreted as decisions.

Finally he congratulated Denise Heagerty on her appointment as site-wide Computer Security Officer.

1.4 DATES FOR FOCUS MEETINGS IN 2001 (M.Cattaneo) (see slides)

The following dates for FOCUS meetings in 2001 were agreed:

2. WEB SERVERS AND SECURITY (D.Heagerty) (see slides)

Denise's talk highlighted the dangers of insecure web servers. There are hundreds of known security holes for web servers, and automatic tools publicly available to exploit these holes. Risks range from defaced web pages (unwanted publicity) to access to system scripts and deletion of files from inside the CERN firewall. A security check has been made of the web servers registered for external

access, but this needs to be extended to unregistered web servers, some of which bypass the firewall on non-standard port numbers..

The Security Team proposes the following actions:

• Run a site-wide scan of all IP addresses to detect all web servers. The scan will cover all ports numbers (including high numbered). Since there is a small risk of side-effects, the schedule of dates and IP addresses will be published well in advance. A change of date can be negotiated for critical systems.
• Web servers will only run on port numbers agreed by the security team, based on discussion with users and the new firewall capabilities.
• Web servers must be kept secure. This requires a named responsible: correctly entered in the CERN network database, staying informed of known security holes and pro-actively updating the system, available in case of emergencies.
• Insecure web servers will be closed, either by the relevant management hierarchy or on request of the security team
• Web sites will be moved to central servers where this is feasible, in order to minimise the overall resources required to maintain secure web servers at CERN.

There was some discussion on whether any script accepting the http protocol would be considered a web server. The goal is not to forbid such use but to find vulnerability. It was suggested to authorise certain users to look for their own vulnerabilities.

3. CLASP PROJECT (D.Heagerty) (see slides)

Denise presented the CLASP project, whose goal is to reduce the number of login/passwords required by users to access services. It addresses at least the services provided by IT and AS divisions, targeting Linux and W2000 use for web, mail, interactive use and file access, both from inside and outside CERN. Details can be found on the project web site http://cern.ch/proj-clasp. This is NOT a security project, but the definition of security levels and password policy are within its scope.

Under phase 1 of the project, the mandate has been defined and a service survey and feasibility study have been completed. The survey lists more than 30 services across divisions, using more than 12 different passwords. Most IT services use a single login ID through CCDB, AS division services are being integrated. Some password harmonisation exists wherever possible. The explosion of different login/passwords is mainly driven by Web authors.

The feasibility study concludes that Kerberos v5 provides a good basis for common authentication and Single Sign On, but some Public Key Infrastructure (PKI) based on certificates has to be integrated, for GRID applications. Non-Kerberos solutions have to be considered for the Web, since Netscape does not support Kerberos v5. Enhanced security is essential to overcome the vulnerability of the initial sign on. Common Access Rights, covering distribution lists, web page and file protections through "e-groups" looks useful - electronic grouping of people and/or accounts can be defined centrally and made available to applications through LDAP and Active Directory.

Phase 2 of the project has just started. The first milestone, this month, is to make available to services a test authentication environment, serving Kerberos v5, AFS and Grid certificates - services included in phase 2 are mail, web (IT+AS services), file access, interactive, batch (LSF), Oracle and future Grid services. By February 2001 implementation plans will be available for a production authentication service and for most IT and AS services. Phase 2 will conclude in May 2001with a final proposal, including a security review, password change and check policy, recommendations for offsite access and a proposal for common access control for web pages, files and e-mail lists. The final proposal will be presented to C5, FOCUS and Desktop Forum.

The audience made several suggestions about the scope of the project. External access to Windows files is requested, worries were expressed about the scheduling of acrontab tasks, about tapes ownership, and about the maturity of the PIE database. It was also suggested to collaborate with other laboratories (through HTASC and HEPCCC). In answer to a question from M.Ernst, Denise said that she is picking up as much as possible from the FNAL "strong authentication project".

4. PROPOSAL FOR IMPROVING SERVICE CHANGE ANNOUNCEMENTS (M.Dimou) (see slides)

Maria summarised her proposal, details are available at http://cern.ch/ref/cern/it/us/2000/040/. She addresses the problem of users complaining that they don't know the details of planned changes, or that insufficient explanation is given when problems occur, even though service managers are already using agreed announcement procedures. Several actions are proposed to improve the communication between managers of services (which also allows building up a knowledge base of side effects), and to better target announcements to affected groups of users. When announcing, the reasons and side effects of a change/problem should be explained. Several announcement channels should be used, to tailor to individual tastes: announcements should be replicated in e-mail distribution lists, news, web pages etc., letting people switch off those channels in which they are not interested.

5.1 Review of current tape technologies and policy (Harry Renshall) (see slides)

Harry began by reviewing current devices and media. CERN strategy is to use data centre quality tape technology for raw and processed physics data, and to expect to use commodity tape technology for import/export. The placing of Linear Tape Open (LTO), currently under field test at CERN, is not yet clear but it is expected to be the successor to DLT for small labs. End of service has been announced for the STK Redwood drive (end 2002), IBM 3590E (2CHF/GB) has replaced IBM 3590.

Since the March FOCUS meeting, a third STK silo has been added in the basement of bat 513. Eight IBM 3590E drives have been added (for adsm+HPSS). A new LHC testbed silo in bat 513 is currently equipped with 9 STK 99xx drives. 25 STK 99xx drives could be installed later but we have to go for tender before. The STK 99xx drive, currently on beta-test, is expected to be the next drive for bulk physics data. One IBM 3494 robot has been stopped, the second one and the TL820 DLT robot will be stopped later this year.

New objectives for 2001 are:

• Move second STK silo complex from bat 513 basement to new bat 613 (three existing silos, plus one or two more for 2002 data). The testbed silo will be left standalone.
• Migrate HPSS data from Redwood and 3590 to 3590E (transparent to users)
• Leave old 3590 physics data read only, or copy transparently to Castor
• Plan to get out of Redwoods, following STK announcement of end of service for end 2002.
• All new data to be written into Castor managed storage on IBM 3590E or STK 99xx. Media cost issues to be discussed in COCOTIME.
• Migrate existing 'active' STK Redwood data to Castor to stop or reduce to negligible Redwood use by end 2001
• Redwood migration transparent to end users, through use of VID mapping, but migration to Castor namespace is encouraged - this could be a way to get out of FATMEN for those who want to.
• New data stored in Castor and HPSS managed storage during 2001 will be charged at 2CHF/GB as an interim solution. The price is equivalent to buying tapes and guarantees storage for 5 years, after which there will be a new charge, or the tapes will be ejected and returned to their owners. There will be no charge for data migrated out of STK Redwood.
• Experiments/users will no longer buy tapes except for import/export. A charging algorithm based on volume and activity will be worked out.

The following points were raised during the discussion:

• A small number of Redwood drives will be kept after 2001
• It is not thought safe to move out of data-centre quality into commodity tape drives (as we have done for CPUs) because the known road-map for data centre tapes addresses our future needs.
• The issue of migrating existing FATMEN users and applications to Castor is being investigated, those concerned believe this will not be as straightforward as suggested by Harry. However, M.Delfino believes that it must be possible to provide a FATMEN API to Castor.

5.2 Overview of Storage Area Networks (Ben Segal) (see slides)

Ben introduced the SAN principles. SCSI commands are sent over a switched LAN (usually FibreChannel), offloading the general purpose LAN. Disk and Tape devices are attached directly to the network (as opposed to servers as for the conventional "Network Attached Storage" (NAS)). They are physically visible to the whole network but are usually logically partitioned among clients without sharing. Shared storage is possible but requires a cluster file system.

The Pros of SAN compared to NAS are extra network bandwidth, storage switchability without recabling, better storage redundancy and manageability, and higher speed due to the elimination of the IP protocol stack and intermediate servers. Cons are the cost per GB of disk storage, which tends to be higher than basic SCSI or EIDE, extra network management, and interoperability problems with FibreChannel. However, work is ongoing, also at CERN, to look for alternatives to FibreChannel, especially Gigabit Ethernet (encapsulating SCSI commands in the very lightweight ST protocol).

Thus SANs make sense for high quality, high availability, high performance, high reliability storage, but not for stageing, scratch or for direct desktop connections.

A small pilot SAN will be set up by IT/PDP with RAID disks and a few tape drives, to investigate high performance, highly reliable storage for ORACLE Parallel Server, and for CDR first level storage (i.e. the high performance buffer at the input of CDR). An investigation of Gigabit Ethernet client attachment will be made when ready.

5.3 Plans for HPSS, CASTOR etc. (Jean-Philippe Baud) (see slides)

This year HPSS has been used for CDR (NA57 and LHC test beams), "User tapes", DELPHI simulation data, all CMS data, and the ALICE MDC. CASTOR was used by the Tape movers, the Delphi stager, the ALICE MDC, COMPASS and L3C production, and as an interface to HPSS.

Next year it is proposed to limit HPSS to "user tapes" and existing datasets, and to use CASTOR for all CDR, LHC Data Challenges and high volume physics data.

In the longer term, CASTOR is viewed as the candidate solution for LHC, with HPSS as a possible fallback. There is no realistic hope for a common HEP solution, but Data Grid WP5 will provide a common API (with RFIO as proposed solution) and the definition of an exchange format for data and metadata.

As a possible migration path, two possibilities are suggested for importing existing SHIFT tapes into the CASTOR name space. Experiments can ask to move data from HPSS into CASTOR, but no systematic transfer is currently planned; the copy must be done while CERN has an active HPSS license.

S.O'Neale pointed out that the proposed "transparent" mechanism for migrating data to CASTOR loses the knowledge which allows staging of many files in one mount, and so may require changing the applications. This again led to a broader discussion on the difficulty of migrating out of FATMEN.

It was felt important to give to Tier 1 centres the message that CASTOR is the current direction of CERN, and that CERN should invest sufficient manpower into this project.

5.3.1 WP2 (Ben Segal) (see slides)

Ben gave a breakdown of the tasks composing DataGrid WP2, whose goal is to provide a universal name space, efficient data transfer between sites, secure WAN data access with caching/replication, and interfacing to mass storage management systems, and reviewed the resources available to the project. CERN manages the work package and leads tasks 2.2 (Data Access/Migration i.e. the interface to WP5) and 2.3 (Data Replication i.e. how to move data close to processing) but has also budgeted some manpower in the other tasks.

The schedule is similar for all work packages. A report on the survey of current technology should be produced by month 4, and a report on requirements and architecture by month 6. A first prototype release is scheduled for month 9, with second and third prototypes one and two years later.

5.3.1 WP5 (John Gordon) (see slides)

Data Grid WP5 addresses the problem of different regional centres having different mass storage hardware and software. The three main tasks of WP5 are to provide:

• A common API to Mass Storage Systems (for which RFIO is a candidate)
• A method of exchanging tapes and metadata between MSS (i.e. a clean way to move data by tape if network bandwidth is insufficient)
• Publishing of metadata associated with the data held in MSS (to interface with other Work Packages) - this includes information to allow estimates of how long it will take to access the data.

If RFIO is adopted, CERN will have to package it for use by others (e.g. documentation). Work will be needed to instrument services to provide resource metadata to the Datagrid Information Service.

Unsolved problems include data flow and grid architecture, and databases: so far only physical files have been considered - the implications of using e.g. Objectivity have still to be worked out.

As a conclusion, the chairman asked whether people were worried about grid activities diverting effort from IT division. It was felt too early to conclude anything, but if the model is correct this will actually bring in effort for something that anyway has to be solved.

6.1 Termination of X terminal repair service (T.Cass) (see slides)

The service handles requests for X terminal repairs. Usage of the service has been declining (15 repairs in 1999) and repairs take 5-10 weeks to complete at an average cost of 500CHF. No loan is available during the repair. Tony suggests stopping the service, since X-terminals have no long-term future as they are not much cheaper than PCs. Broken terminals could be replaced by old PCs running Linux.

The proposed termination was AGREED.

6.2 Consolidation of SUN physics services (L.Robertson) (slides)

(This item was in fact taken just before coffee)

Les presented one slide summarising the IT proposal. This is to concentrate physics SUN services on a single shared interactive service "SUNDEV". The platform would provide the environment for physics software development, including support for tools not available on Linux. It would NOT be a generic "desktop support service", i.e. not for mail, browsing, text processing etc.. Batch would be available only in background to the basic interactive service. The proposal is a consolidation proposal, not implying any additional investment in SUN support. Specialised servers (e.g. database servers) are not concerned by this proposal.

The Atlas and CMS representatives were unwilling to agree to this proposal without consulting their collaborations, and without more details of the scope and partitioning of the service. Worries were also expressed by NA48 about the impact of this policy on preserving backward compatibility for data processed on the CS2, and on preserving specialised interactive applications such as the NA48 news system. Given the urgency of a decision, N.McCubbin and M.Pimiä were asked to form an ad hoc working group together with IT/PDP (and NA48), to agree on a policy in time for the COCOTIME meeting on 6^th November. H.Hoffman stressed that IT division has defined a direction to follow, and that too many deviations taking manpower away from this direction are not welcome.

New Computing Rules, covering all users of CERN computing facilities, were recently issued. Full information is available at http://www.cern.ch/ComputingRules. Paper copies are available at divisional secretariats in both English and French.

8.2 LEP Higgs search MC productions (DELPHI: J.van Eldik, ALEPH: F.Ranjard (see slides)

Jan described his efforts to produce 3M Monte Carlo events in 3 days, to make them available for the Higgs analyses. This compares with 50k events/day normally produced by the biggest Delphi MC farm. 650 Linux and 45 OSF servers were made available by many experiments with the help of IT division. This was made possible by using the standard configuration (AFS, LSF, rfcp) and the LSF multicluster facility. There were few problems, except for book-keeping which was rather painful due to the short time available to develop it. Jan concluded by thanking IT/PDP and all the experiments.

Florence added that Aleph benefited from 150 CPUs from the EFF (originally allocated to CMS) for a similar production; she believes that this was only possible thanks to the standardised configuration of the Linux clusters managed by IT Division. It would never have been possible in the days of experiment specific "shift" environments.