DISTRIBUTED SYSTEMS FOR DATA HANDLING|
Joint Institute for Nuclear Research
Creation and use of distributed systems for computerized accumulation, transmissions and information processing in interests of scientific research started less than 50 years ago : apparently, the first such system in Soviet Union was realized in 1961 for radio-channel transmission of Atlantic ocean objects information to first Soviet semiconductor universal computer DNEPR in the Institute "Cybernetics" (Ukranian Academy of Science) , in Kiev. Basic realizations of multicomputers complexes for distributed jobs and data handling started in 60th/70th years of 20th centure practically in all prominent research organizations in Soviet Union and abroad. As the " top of success " of their development, we will have Grid systems in nearest future, that is complexes, which correspond to three criteria according to definition of one of Grid technology ideologists, Ian Foster  :
" Grid is a system that :
- coodinates resourses that are not subject to centralized control…
- using standard, open, general purpose protocols and interfaces…
- to deliver nontrivial qualities of service, relating for example to response time, throughput, availability, security and/or co-allocation of multiple resource types to meet complex user demands, so that the utility of the combined system is significantly greater than that of the sum of its parts."
As an ideal, such Grid system permits to ordinary user to input his/her job for batch or interactive processing without necessity to define in job`s "passport" (JDL-file) via attribute Requirements the address of certain computer (CE, Computing Element as part of computing Grid resourses), if system`s middleware has high level services like RB (Resourse Brokers) and a lot of CEs, as it is now in large-scaled realized projects such as EGEE (Enabling Grids for E-sciencE), see http://lcg.web.cern.ch/LCG/ and
http://egee-jra1.web.cern.ch; though, for example, there are no such services in American project OSG (Open Science Grid). We talk about distributed systems of 20th centure, but necessary to say, that many ideas, realized in contemporary Grid structured, were not quite unknown for specialists, which created systems more than 30 years ago, as it will be shown below. In other words, possible to say, that Grid - this is, in a sense, realization of "old" ideas on the base of new technology.
At such physical scientific centre in Russian territory, as Joint Institute for Nuclear Research (JINR), which was organized in 1956 and closely collaborated with Western physical centers, especially with European Centre for Nuclear Research ( CERN, Geneva), the problems arose, which demanded the common approach for applications realization and matched software environment in computing servers : this would make easier the data and programs exchange at least with use of magnetic tapes in time of direct computer links absence. As to main type of processed information, then at JINR, CERN and other centers ( in Brookhaven, Berkeley, Daresbury, etc.) this was a big amount of experimental data from accelerators detectors. At JINR, which also carried out theoretical and experimental research in fields of low and intermediate energy physics, it was necessary , in particular, to registrate and process data, taken from spectrometers in experiments with use of fast neutron pulsed reactor… According to these application problems, as one of main tasks of 60th years and later became the technical and software support of the whole sequence of measuring, preliminary and final data processing for different fields of experimental physics and for theoretical physics also. The chosen method to resolve this task was essential : to start to build distributed multileveled computing complexes with use of perspective and standard measuring equipment and standardized programming tools for applications ( such as compilers for high-level languages like FORTRAN ; compatible, with Western versions, libraries of of programs for universal and special destinations ), programming tools for algorithms parallelisation, tools for user access to computing resources of different types. Main works at JINR, mentioned below, were initiated and guided ( till July 1989) by corresponding member of Academy of Science (USSR) N.N.Govorun : chief of mathematical division in JINR Computing Centre (1963-1966), vice-director (1966-1988) and director (1988-1989) of Laboratory of Computing Technique and Automation (LCTA).
The first stage of distributed systems creation started at JINR in 1962, when the simple structure of two connected by cable computers M-20 and KIEV in JINR Computing Centre could collect ( via cable also) and process information from spectrum analyser, located in measuring centre of Neutron Physics Laboratory. They also processed the punched paper tapes with information, prepared with use of special measuring devices, where the events, which happened in accelerator detectors and automatically photographed ( in High Energy Physics Laboratory), were preliminary processed by operators. In 1965, this structure was modernized : additional M-20 was mounted and the KIEV, which was not reliable, replaced by the MINSK-22. This structure was also used for jobs batch processing in other fields of researches. Its schema looked like it is in graphycal presentation, acccessed by http://lit.jinr.ru/LCTA/lcta30/img/schema1.gif
1966-1974, the first stage of three-leveled complexes realization accordingly general principles and conditions ( also typical for modern, up-to-date distributed systems for physics), which provide:
1. Meeting the requirements of on-line experiments;
2. Remote batch and interactive processing for jobs;
3. Interactive mode for users and their terminals;
4. Algorithm parallelism with use of different computing elements in complex;
5. Maintenance of archives for programs and data.
The reality of realization for this set of services was defined by appearance at JINR of sizeble set of new computers:
"small" computers (first level) for on-line experiments (equipment control and data registration) or remote input/output stations (RIOS), used for batch processing support;
"middle" computers (second level) for preliminary processing of data, taken from first level, and also for RIOS regime support;
at last, "big" computers (third level) as basic computing servers.
This new set included some TPA computers (Hungarin production), SM-3/4, M-6000 (analog HP-2116B), original HP-2116Bs, BESM-3M and BESM-4, MINSK-2/22,, CDC-1604A, BESM-6… BESM-6 started to be used as basic computing server not only at JINR, but also in all remarkable scientific Soviet organizations, though it was necessary to develop preliminary its software system: this is why appeared the" DUBNA" multilanguage system with FORTRAN etc. compilers and programming library, compatible with CERN library [2, 3, 4]. Another problem concerned the necessity to develop technical and corresponding software tools in BESM-6 operating system (OS) for connectivity with various local external devices and other computers of first and second levels. As a result, the modernized at JINR BESM-6`s peripheral control unit and new version of operating system started to serve eight fast cabled lines : up to 500 Kbyte/sec per line and up to 4 abonents, connected to each line, in particular remote data processing computers and RIOSes (named also as "FORTRAN Stations", FS), equiped by own peripheral devices such as magnetic tape units, card readers, printers and terminals. So far as the "middle" computers in Laboratories measuring centers had to participate interactively in information processing together with BESM-6, then there was a need to realize peculiar "algorithm parallelism" tools: this is why the special "extracode" facility to serve external links was embaded in OS BESM-6 ; also in DUBNA monitoring system were added subroutines, which used this extracode and could be called in FORTRAN-written programs. Use of external interruption apparatus in BESM-6 and peripheral computers ( such as BESM-4, where also an analog of FORTRAN-oriented monitoring system was realized) helped to realize an analog of modern MPI (Message Passing Interface) service. Schema of complex in 1974 looks like it is in http://lit.jinr.ru/LCTA/lcta30/img/schema2.gif, where CDC-6200 was added in 1972 as BESM-6 partner on level 3 of the whole complex.
JINR was not unique in Russia, who created multicomputer complexes already in 60th years. It would be necessary to mention, at least, system AIST in Computing Centre (Sibiria), complexes in IAM RAS (Institute of Applied Mathematics in Russian Academy of Science) , Computing Centre RAS, ITEP ( Institute of Theoretical and Experimental Physics) and IPMaCT (Institute of Precise Mechanics and Computing Technique) in Moscow ; IHEP ( Institute of High Energy Physics ) in Protvino…
In principle, complex at JINR and mentioned systems had some features, similar to realized, for example, at CERN in system FOCUS, where existed concentrator (CDC-3200) for lines to servers CDC-6500/6400 ( 4 cabled lines, up to 1Mbyte/sec and 4 peripheral computing or measuring abonents per line), RIOS stations, terminal concentrator…But life was easier for Western scientific centers, because there was no need for them to create (themselves) the software environment like mentioned above…As to type of computing servers, physisists in West always prefered to use 60-bit processor technique for high-precision complicated calculations, this is why they used IBM computers for data registration and CDC seria 6000/7000 for processing and other calculations. Such computers were under embargo in 60th, but even 48-bit BESM-6 was good enough. Necessary to mention, that Grid started with use of 32-bit technique in clusters, but appearance of 64-bit AMD and Itanium 2 processors immediately initiated the use of new PC-clusters in scientific centers. So, already in 2003, in frames of CERN project "Openlab" , several major physics software packages have been successfully ported on the 64-bit environment : for example, PROOF ( Parallel ROOT Facility as a version of the popular CERN developed software ROOT for interactive analysis of large data files) . In Russia, approximately at the same time, Institute of Computing Mathematics in Moscow started to use similar cluster with Myrinet 2000 bus for simulations, which earlier were based on use of access to Supercomputer Centre RAS…
1975- beginning of 80th at JINR. Installation of new servers of ES-seria, compatible with IBM-360/370 ; development of tools for terminal access to level 3 in complex ( see picture in http://lit.jinr.ru/LCTA/lcta30/img/schema3.gif ).
This development included creation of special controllers (multiplexers) for "big" ES-computers with use of microprocessors, also creation of terminal concentrator on base of ES-1010 and programming subsystem INTERCOM, which permitted to have:
- unified language (INTERCOM, which was originally used only for terminal access to CDC seria 6000) and tools for access to different computing servers via terminal concentrator ES-1010 (equiped by own external memory on disks and tapes, also by special operating system, developed in JINR Laboratory of Computing Technique and Automation, former JINR Computing Centre ) or via ES-1060/1061/1066 microprogramed multiplexers : these ES-servers started to understand INTERCOM language by means of developed in LCTA software subsystem TERM;
- independent (of BESM-6, CDC-6000) service for terminals in process of information receiving, editing and archiving, when users prepared their job texts;
- execution of user directives (commands) to send prepared job to certain server (after automatical JDL-file edition ), to show the status of jobs queues, to stop temporary or kill job in server, to print OUTPUT-file on printer or terminal screen…
In principle, concentrator could, in essence, realize functions, similar to functions of program services UI, NS and RB ( Resource Broker) in modern Grid systems, especially if user used FORTRAN subset, understood by all computing servers ( BESM-6, CDC, ES ) . Possibility to choose a server ( this or that) simplified by existence of compatible versions of library CERNLIB in servers software.
1983-1985. Starting point for introduction of standard means for creation of local networks in scientific organizations and their external channels.
As to local network structure and technology, possible choice concerned Token Ring/Token Passing or "young" Ethernet technologies. At JINR, a local net was devised in 1985. This was the JINET (Joint Institute NETwork) with a marker (token) method of access to communication line of 12 km of 75 Ohm coaxial cable . Standard adapter nodes were connected to it in various JINR buildings, each node served 8 to 12 serial (standard RS-232C) ports for terminals or computer multiplexer`s ports. About 30 adapters were bought from Swiss firm Furrer-Gloor, they had built-in Z-80 microprocessors and built-in memories; afterwards, a set of their modernized ( by our LCTA engineers) analogs was produced at JINR and used in frames of JINET. The supporting software for all adapters was produced completely by LCTA programmers at JINR, because the firm`s software was too expensive; moreover, we foresaw the possible modernizations and wished to have not only built-in binary software, but also source texts. This software supported virtual communication channels ( communication sessions) according to users` applications through a common cable with a throughput of about 1Mbit/sec. Each abonent (terminal user or computer) could give various commands to his connection node, e.g. calling the text editor in his node, checking status of different nodes and their ports, setting the suitable transmission speed at his port or defining the terminating symbols for messages (end of text line, of package,etc.), if the defaults were inadequate. If no specifications were necessary, the abonent simply commanded to establish a virtual channel for communication with the given address (e.g. node and port number or the other computer`s symbolic name), started to use this channel and disconnected at the end of session. Abonents like computers could exchange files using the standard KERMIT program. Special node-administrator or his "hot reserve" partner automatically controlled the marker`s (token) movements, collected statistics of nodes connections and faults, recreated the lost markers, disconnected every other node in case of noticed frequent malfunctions, etc..
In 1988, JINET became the abonent of international X.25 net IASNET, being connected to its communication centre in Moscow . This permitted to directly exchange files with physical centers, to use telnet regime for access to certain servers ( many JINR physisists had passwords for such access, being members of different international collaborations ).
Till end of 1989, the standard JINR Ethernet was devised and connected to JINET , how it is pictured in http://lit.jinr.ru/LCTA/lcta30/img/schema4.gif. This was a time, when BESM-6 was dismounted and new servers were connected to Ethernet : cluster of two VAX-8350 with shared disk memory (about 12 Gbyte), cartridge units (200 Mbyte) and high density (up to 6250 bit/inch) magnetic tape units ; also powerfull MEGATEK/WIZZARD 7555 station with 3-dimensional graphics..
At the very beginning of 90th years, the throughput of Russian terrestrial external channels was inadequite for territorialy distributed information exchange and processing in high energy physics: practically in limits of 32Kbit/sec. This is why two direct satellite channels were organized at JINR and started to be exploited in 1994 for access to networks DFN (Germany) and HEPNET (High Energy Physics NETwork with entry point in Italy and terrestrial channel to CERN). The throughput of each channel was up to 64Kbit/sec; we used own antenna and Soviet "RADUGA" satellite for first channel , terrestrial link to near-by Cosmic Communication Station "DUBNA" and international satellite INTELSAT for second channel. We had a plan to give some other Soviet scientific institutes ( such as IHEP in Protvino) possibility to share this second channel with JINR, how it was described in  or shown in http://lit.jinr.ru/LCTA/lcta30/img/schema5.gif
Of course, terrestrial channels, with the same throughput as satellite channels, have an advantage in many cases, and when new facilities to increase their throughput appeared ( up to 1Gbit/sec nowadays) - we took this opportunity and finished to use satellite channels.
Our nearest perspectives in use of distributed systems - in final realization of the project RDIG (Russian Data Intensive Grid) as part of international project WLCG (Worldwide LHC Computing Grid  ) ; 8 institutes in Russian Federation (in Moscow, Moscow district, Petersburg) became the main participants in this project : JINR; SRINPofMSU (Scientific Research Institute for Nuclear Physics of Moscow State University; ITEP, IHEP, IAM, RSC KI (Research Scientific Centre of Kurchatov`s Institute ), PINP ( Peterburg`s Institute of Nuclear Physics) and IMPB (Institute of Mathematical Problems in Biology) as institutes of Russian Academy of Science )…
I am grateful to people ( especially to Georg Trogeman, Alexander N.Nitussov and Wolfgang Ernst), who organized publication of book, mentioned in [3,4] and containing a lot of information, concerning computing and networking in Soviet Union. Of course, thanks to Ministry for Scools, Education, Science and Research of the German state North-Rhine Westfalia and the Academy of Media Arts Cologne for this publication support.
- Ian Foster. What is the Grid? A Three Point Checklist. In:
- V.P.Shirikov. Matematicheskoye obespechenie vychislitel`nyh kompleksov i setei.[ The Mathematical Support of Computation Complexes and Networks]. Moscow: Nauka. In: Programmirovaniye, Number 3, 1991.
- Igor A.Apokin. The Development of Electronic Computers in the USSR. Germany: Vieweg. In: Computing in Russia. The History of Computer Devices and Information Technology revealed, 2001.
- V.P.Shirikov. Scientific Computer Networks in the Soviet Union. Germany: Vieweg. In: Computing in Russia. The History of Computer Devices and Information Technology revealed, 2001.
- The CERN openlab: a novel testbed for the Grid. In: CERN COURIER, October 2003.
- Memorandum of Understanding for Collaboration in the Development and Exploitation of the Worldwide LHC Computing Grid. In: http://lcg.web.cern.ch/LCG/C-RRB/MoU/WLCGMoU.pdf