Computing in the Soviet Space Program

Introduction

Bibliography

Computers

Discussion

Documents

Essays

Institutions

Interviews

Links

Sitemap

• When do you first start working with onboard computers?

• Who was writing programs - engineers who understood control problems or specialists in computing?

• Did the first Salyut computers ever perform actual control functions?

• When did onboard computer begin performing essential functions?

• In the early 1960s, was the decision to place a computer on board an obvious solution or was it controversial?

• What problems arose in connection with software documentation?

• To what extent did the computer control cosmonauts' actions? Did the machine set limits on what a cosmonaut could do?

• Did you rely on machine more than on human being?

• What were the arguments during the debate over which machine to install - the Argon-16 or the Salyut-4?

• How was the reliability problem solved?

• What methods were used to test and debug the software?

• Did you use American experience in any way?

• In software development it often happens that the programmer keeps the details of the program in his head...

• Did the huge experience you accumulated and the new methods you developed have any circulation outside your department?

Gerovitch: When do you first start working with onboard computers?

Zybin: I graduated from the Moscow Institute of Physics and Technology in 1961, and on April 4 of that year I came to Korolev's firm. In just a few days there was Gagarin's flight. From the very beginning, I began working on navigation of space vehicles. A year or two later a lunar program started. Since nobody was very confident that technology was completely reliable, we planned to control the lunar flight from the Earth, but as a backup we placed a sextant and other navigational equipment on board, so that cosmonauts could, for example, carry out lunar landing. This required data processing on board. This was approximately in 1964. By that time one firm in Zelenograd developed the Salyut-1 computer. It was a small-size machine weighing about 5 kg. Its memory had only 64 location cells. The problem was that trajectory calculations required a lot of memory. The division of responsibilities was as follows: Timur Yeneev's group at the Institute of Applied Mathematics worked on software for the flight to and from the Moon - the determination of trajectory and correcting propulsion impulse; and our department, headed by Ernst Gaushus, was responsible for calculations related to the circumvention of the Moon and the lunar landing. The problem was how to carry out all these calculations, using only 64 location cells. Yet everything was done and tested, the documentation was ready, but the Americans flew first, and our program was terminated.

After that, our firm turned to orbital stations. We placed the same computer on board the Salyut-1 station. The name of the station here coincided with the name of the machine. A major drawback was that the machine was not very powerful, and it required a lot of manual work. Cosmonauts had to take measurements and enter the data; even during computation they had to help the machine: the memory was so small that they had to enter intermediate results of calculations. On board the Salyut-2 and the Salyut-3 stations, the Salyut-2M computers were installed. Those were also made in Zelenograd. The Salyut-2M was a relatively advanced computing machine. It had a tape drive, made at a firm located at Novaia street (now it is called Kosmopribor). Magnetic tape for this drive was taken from American balloons. The Americans sent spy balloons over the Soviet territory; some of them fell down, and tapes were taken from those balloons.

We faced the following problems. First, other onboard equipment was poorly suited for working alongside a computer. The concept of electromagnetic compatibility had not been developed yet. Second, since everything was done very quickly, the software was not tested well. There were many malfunction reports which were difficult to sort out - whether this was a hardware or a software error. For the Salyut-4 station, we rewrote the Salyut-2M operating system completely. This station had an advanced navigational system, solar sensors (which timed the moments of entering and leaving a shadow), a Doppler velocity meter (on the Soviet territory there were transmitters, which generated a precise, stable base frequency, and there was a receiver on board, which measured the velocity based on the Doppler effect). On the Salyut-4 station, the trajectory was calculated automatically on board. As it turned out, however, there was no big need for that, since the ground control calculated everything precisely enough. The onboard system, in fact, served as a backup.

The main purpose for which this machine was actually used was displaying information for the crew: when will be the next communication session, how long will it be, when the spacecraft will enter a shadow, and so on. As for various measurement and calculations, those were not used but served as a backup, for information purposes only. This machine was not turned off for months in a row, and it operated normally.

On the Salyut-7, the last Salyut station, despite many breakdowns on board, the computer worked until the last minute, until power was shut off. There was also many computer malfunction reports - not clear whether software or hardware errors. Diagnostics at that time was not developed well enough. If there was a suspicion that it was a hardware error and pressure was applied on computer designers, they could not firmly reject this hypothesis. Especially since this machine had no redundancy. There was only a "cold" backup on board: normally the first machine operated, and only if it broke down, the second one would be turned on.

In the 1960s, in the Control System Division, headed by Boris Chertok, there were two departments, one led by Vladimir Branets, and the other by Ernst Gaushus. Initially it was planned that Branets would be working on cargo ships, and Gaushus on orbital stations. Branets' department preferred Argon onboard computers, and Gaushus' department Salyut computers. Based on the experience with the Salyut-2M, a blueprint for the Salyut-4 computer was created. In the early 1970s, in Chertok's division there were constant debates which machine to choose. We insisted that the Salyut-4 must be installed on all space vehicles, and Branets pushed the Argon. The Zelenograd firm built the Salyut-4, but then a "pause" occurred, and its production was delayed. More funds were needed. In the end, funds were not allocated, and the Zelenograd firm began the development of the Salyut-4 on its own.

In parallel with the Salyut stations, work on the Mir station began. Since Gaushus' department was overloaded, this project was assigned to Branets. Branets' department had more or less finished the cargo ship, and people in his department had time to do a new project. It was planned that the Mir would fly in orbital orientation, and in order to point equipment toward the Earth and the stars, a rotating platform was made in Czechoslovakia (this work was supervised by the Institute of Space Research). Wherever the station was, it could monitor a specific spot on Earth. Gaushus' department received an assignment to design a control system for this platform. This was in the mid-1980s. At the same time the Central Specialized Design Bureau in Kuibyshev, headed by Dmitrii Kozlov, was developing a new series of satellites, and together with the Zelenograd Science Center they decided to build a large computer complex: a high-speed computing machine with a single Manchester cable, which connected to individual pieces of equipment through peripheral exchange adapters (PEAs). The Zelenograd firm built this PEA. It was, in fact, a real computer weighing about 15-16 kg. It was planned that each system, for example, the engine control system, would have its own PEA, which would transfer data into the main computer. We decided to use this PEA as a control computer for the rotating platform on the Mir station, and this was done. As it turned out, I have never seen a better computer. It worked the whole time when the Mir was on the orbit. The Mir station itself was equipped with Argon-16 computers, but since those were not powerful enough, Branets designed the onboard network in such a way that the Argons could later be replaced with Salyut-4 computers, which were under development at that time. Branets was responsible for the Mir control system; they used both the Argon-16 and the Salyut-4 computers, and they had to swap the computers all the time. One day they shipped one computer to the station, another day another computer; there were constant malfunctions. And the first PEA that we installed worked on the Mir all the time without a single malfunction. Later other PEAs were added to it, as the functions expanded and other pieces of equipment were added, but this first PEA remained the central part of this system.

I am flabbergasted to see how ungrateful sometimes people are. Nobody thanked the chief designer of this PEA, Anatolii Shchagin. They could have written a newspaper article about him or given him a medal. In my opinion, there has never been such a computer. It had triple redundancy and was built with 16-layer ceramic boards. It did not have a single malfunction during the entire operation period.

Later on, in parallel with the Mir station, we started working on the Buran. There was a large onboard computer installed on the Buran. Whereas in all previous projects all software was written by a separate department, on the Buran everybody wrote their own piece and tested it as they could. The job of Gaushus' department was to put everything together and test it. In reality, the tests were somewhat superficial. It was a miracle that the Buran's flight was successful. They were right that they did not launch it the second time. It is well known that most problems occur at junction points: one did not say it right, another did not understand it right, and so forth. Even though there were special instructions given on how to avoid such situations, nobody could give assurances.

Gerovitch: After the Zelenograd firm handed the Salyut-1 over to you, did they do any programming for this machine? Did they do any modifications of hardware?

Zybin: No, we did all the programming. There were no hardware modifications. Later they developed the Salyut-2M model, which had a tape drive and larger memory (256 location cells, I think), and we moved to this new model.

Gerovitch: Who was writing programs - engineers who understood control problems or specialists in computing?

Zybin: Software for the first Salyut computers was written by the group led by Viktor Aleksandrovich Sharov. They wrote programs based on algorithms developed by us and by the Institute of Applied Mathematics. For the Salyut-2M computer, Sharov's group wrote both he operating system and the applications. Since on the Salyut-6 station there were many malfunctions, we decided to rewrite the operating system of the Salyut-2M. This project was assigned to me, and we carried it out in Sharov's group and in other groups under my supervision.

The situation with PEAs on the Mir station was different. PEAs already had a built-in operating system written by the Zelenograd firm. It had a debugging system, and programming was done in a machine-oriented language. One could use high-level languages, for example, Basic. In this case, the person who wrote the algorithm also wrote the program. The programmers also helped, but they largely worked on data exchange problems.

The Mir had a distributed complex. One PEA was installed on board, and there was a digital communication channel with the ground control. On the ground, there was a PEA and a personal computer next to it. When an operator on board saw something interesting on Earth, he could turn the rotating platform and point it in the desired direction, and the platform would then track this spot. The same could be done from the ground. Within the visibility zone, the ground and the onboard systems worked in parallel and in coordination with each other. Once the ground control was performed from Australia, later they took it to Europe, but this was at the time when I already left. This was a very good computer, but expensive: it cost about 100,000 rubles.

In Zelenograd, they had a large computer design and manufacturing complex. One organization was at the top, and it had many divisions. Different Salyut models was made by different people, but they all bear the same name.

Gerovitch: There were both Argon-16 and Salyut-4 computer on board the Mir station. Which of them kept breaking down?

Zybin: Branets would answer this question better. We did not go into this. We had certain competition [with Branets' department], so there was no information exchange. This was not a friendly competition. Gaushus and Branets competed for primacy with varied success. Sometimes Gaushus would take the upper hand, sometimes Branets. When Branets prevailed the last time, Aleksei Yeliseev offered Gaushus to move to the Bauman Moscow Higher Technical School (MVTU), and Gaushus did. When Yeliseev became the rector of MVTU, he decided to reform its structure and created seven scientific research institutes, corresponding to the seven faculties at MVTU. In those institutes research is conducted separately from teaching. He invited leading engineers from industry to be the heads of these institutes. At MVTU it used to be like that: a student graduates, goes to graduate school, becomes a professor, and then a department head. He teaches textbook knowledge; he never designed anything himself, and consequently he cannot teach any practical knowledge. But Yeliseev's term proved too short, only five years. He did not expect that the old cadres would not vote for him. Of course, they felt pinched. And here teaching is primary. So all department heads voted against him during the elections. He left, and for the last 10 years they have tried to roll everything back as it used to be, but they have not got enough power to do that.

Gerovitch: You mentioned that the first Salyut computers performed largely a backup function in case ground control fails.

Zybin: They performed not only the backup function, but also a monitoring function. The ground control makes calculations, and the onboard computer makes calculations, and if they match, then everything is correct.

Gerovitch: Were those computers actually used?

Zybin: The lunar program did not take off, so in this case it was done in vain. On the Salyut station, the crew used the computer for their own needs. The machine calculated the time of the next communication session, it assisted with planning, and so on. When the ground complex is activated, all ground tracking stations are working.

Gerovitch: Did ever a situation occur when control had to be switched from the ground to the onboard computer?

Zybin: No, there was no such case. If a problem occurs on an orbital station, it can usually be put off for a week or a month. By contrast, during a flight towards the Moon or during lunar landing, you cannot put off the problems. Processes here are irreversible, and there are specific time intervals when certain things have to be done, and here you absolutely need a backup.

Gerovitch: When did onboard computer begin performing essential functions?

Zybin: It was planned that the first Salyut stations would fly in orbital orientation, and they had an analog control system. In essence, a station is just flying and is not doing much. Later on new tasks were added: to perform various experiments, the station had to be pointed this way or that way. Those stations did not have a rotating platform, so the entire station had to maneuver. On the Salyut-7, we linked the Salyut-2M computer to the analog control loop, and with the help of this computer the station could do everything that was needed. The Salyut-2M manipulated with signals from various sensors and was able to "deceive" the analog control system and to make it perform what was needed. On the Salyut-7, the computer was part of the control system. It was used for many experiments and for trajectory corrections. On the Mir, of course, the machine was also actively used. The Soyuz-TK spacecraft had a digital control system based on Argon-16 computers.

Gerovitch: On the Soyuz-TK, the onboard computer was an essential part of the control system. What would happen if it broke down?

Zybin: There was a backup analog control system on board.

Gerovitch: In the early 1960s, was the decision to place a computer on board an obvious solution or was it controversial?

Zybin: For the lunar program, this was an obvious solution, since everyone realized that no risks could be taken. In this case, this was absolutely necessary. On the first Salyut stations, these machines and perhaps the entire navigation system were installed largely because there was a department engaged in these issues, and everything depended on personal contacts. They agreed to install this equipment, and that was it. There was no huge need for that, but everybody considered this important.

Gerovitch: Were there any doubts whether this technology proves reliable in flight?

Zybin: No, there were no such doubts. The main problem was the lack of equipment. We made the first Salyut and thought that it would be the last. It was launched, and then an order came down to build the next. Every two years we built a new Salyut, and every time we thought it was the last. The slogan was: "In order to do everything fast, don't change anything; launch it as is." Yet every time we changed something. Such a policy - "this is the last station; there is no need for updates" - slowed down everything. The policy was that computer equipment also needed no updating. So it dragged on with all of the Salyut stations.

Gerovitch: What problems arose in connection with software documentation?

Zybin: For the Salyut-1 there was practically no documentation. The description of instructions was no good. For example, the addition was described like that: this flag is set here, these digits are changed in the registers, and so on. There were many nuances, and the programmer had to figure it all out. The machine was unfinished; there were hidden problems everywhere. Even addition was carried out differently at different times. It all depended on history: if something was left after the previous operation, then the buffer had to be cleared or something of this kind. Our programmers had practically no documentation. They received hardware and some notes that the designers had scribbled for their own use. Our programmers had to figure things out on their own. They finally did it and produced their own documentation, and after that everything was okay.

Documentation for the Salyut-2M was improved, and the PEAs had complete documentation, which had only one error related to temporal functions. This error was noticed during ground tests. Since these computers worked in real time, several times per second they received exact time signals and also clock frequency (1,000 Hz) signals. The operating system had certain tasks to turn something on at specific moments. We noticed that some of these tasks were being simply skipped and were not carried out. The machine was malfunctioning! We called the chief designer in. It turned out, the operating system had a bug. The exact time signals and the clock frequency signals arrived in the system asynchronously from different generators. The machine, however, expected them to be synchronous, and for this reason it skipped. When they tested it on their simulator, both signals came from a single generator, and this glitch did not show. The documentation did not specify that the signals had to be synchronous, and we used different generators, which resulted in the malfunction. This was noticed during the ground tests, and we spent about a month on that. We had hard time figuring out why was it happening. Eventually they fixed it; they made a patch in the operating system. After that there was not a single problem.

Gerovitch: To what extent did the computer control cosmonauts' actions? Did the machine set limits on what a cosmonaut could do?

Zybin: In the beginning we tried to limit everything. On the Salyut-1 the cosmonaut was required to enter a code (for example, data from the first column of sextant measurements). Then the machine checked the data for consistency. There was no guarantee, of course, but rough testing was done. If he missed a digit in a high-order position, he would get an error message. However, if a mistake occurred in a low-order position, it was impossible to detect it. We handled it this way: we required the cosmonaut to enter the data twice. If it was the same both times, then it was assumed correct.

Gerovitch: Could the cosmonaut ignore the limits set by the machine? Did the machine only warn the cosmonaut of a potential mistake, or did it firmly tell him that he must redo the task?

Zybin: If he made a mistake in a high-order position, the machine would display the message "Enter the first code" again and again.

Gerovitch: That is, he could not do anything until the machine accepts his data?

Zybin: That's right.

Gerovitch: And if, say, there was an error in the algorithm, and correct data were rejected because of that, could he do anything about it?

Zybin: Nothing. We could not count on the cosmonaut in this regard. For example, one gave a lecture to cosmonauts and explained how to enter "sine equals 0.8." Gagarin asked: "But here it already shows zero and the decimal point. What if sine equals one and a half, how would I enter it then?" He should have learned this in high school.

Gerovitch: Did you rely on machine more than on human being?

Zybin: That's right. On the Salyut-7, where the station itself had to be guided, the cosmonaut had an opportunity to make corrections using a hand controller. The machine received signals from this column and made corrections. Here is a dilemma. On the one hand, you may want to limit the use of the machine just in case it has an error or it uses the data incorrectly. On the other, if you set such limits, you often cannot use the machine. The latter happened in the automatic docking system developed by Branets. They set strict limits on the velocity range, but it so happened that a sensor was giving incorrect data [as a result, velocity was calculated incorrectly], and the docking control system was simply turned off. They had to perform manual docking.

Gerovitch: Could the cosmonauts make any adjustments in the machine so that this velocity range, for example, could be corrected?

Zybin: Initially they could not. Later on, when the Salyut-2M with a tape drive was installed, they could make corrections using the tape. After that, for the Soyuz spacecraft Branets provided for the possibility of entering new programs. Nevertheless, corrections were made only if something did not work the first time.

Gerovitch: Have there been any cases when a cosmonaut on board made program corrections based on instructions from the ground?

Zybin: I think they have, but I cannot say for sure. Such possibility existed on the Salyut-2M. The Salyut did not have a digital communication line; they used the Stroka teletype system. Since we provided for the possibility of correcting programs, we linked the computer with the Stroka, and we entered new programs and corrected old ones through this system. The station stays on the orbit for a long time; new tasks arise; and we provided for the possibility to carry out operations which had not been anticipated before, and this was widely used. Just in case, there was also a possibility for the cosmonauts to receive programs from the ground by voice and then enter them in the machine. We never used that path; we sent everything over the teletype.

Gerovitch: That is, even though it was possible to correct programs for the Salyut-2M on board, this possibility was never used, right?

Zybin: Right, we never used it.

Gerovitch: Were the cosmonauts taught the fundamentals of computer programming, or just the operator functions?

Zybin: They were taught how to enter information.

Gerovitch: On the later models of space stations and spacecraft, did the cosmonauts have functions that required a deeper understanding of the functioning of computer?

Zybin: No, quite the reverse. The cosmonauts that used the PEA set only the mode and did not even touch the keyboard. Everything was controlled from the ground, while the cosmonauts used the hand controller only for corrections.

Gerovitch: What were the arguments during the debate over which machine to install - the Argon-16 or the Salyut-4? What were the advantages and the disadvantages of one machine compared to the other?

Zybin: The debates were mostly about the instruction set. Nobody thought about reliability at this point. In fact, the comparison was very difficult. The Salyut-4 had the Unified Series instruction set, which was considered more advanced. It was indeed more advanced, but this factor proved insignificant. Take a personal computer: it has a miserable instruction set, and yet... In fact, one can do anything with any instruction set. If the processor is fast, then the instruction set does not matter. Our debates centered on insignificant factors - as we know now - but back then it seemed crucially important.

Gerovitch: How did these machines compare in terms of weight and speed?

Zybin: The Salyuts were always lighter and were considered more advanced. Branets argued that the Argons were widely used, that they were installed on artillery, that they were better tested, but these were insignificant factors.

Gerovitch: How was the reliability problem solved?

Zybin: Reliability was ensured by redundancy. The PEA has triple redundancy; the Argon double redundancy, I believe.

Gerovitch: In case of the Argon, was it "hot" redundancy? That is, did both computers work in parallel?

Zybin: To tell the truth, I do not know. I was immersed in algorithms and programs and did not care much about hardware. Gaushus and Sharov took active part in those debates.

Gerovitch: What methods were used to test and debug the software?

Zybin: Testing of software for the Mir space station was extremely thorough. Before a programmer wrote a program, he had to write a clear description of what the program would do. After that, he would write the program, while another programmer received a separate assignment to write tests for this program. Each program was tested in this way, and when they were integrated, different programmers were again assigned to test this integrated product. For this reason, the software for the PEA was done so well.

For the Buran, we also planned many tests, but we lacked time. The organization of work was complex: Chertok was responsible for the entire control system, while the core control system was developed by Pilyugin's firm. The latter designed a dispatcher machine. The integration of all software was assigned to us, that is, to Chertok. It was difficult to sort out who is responsible for what. On the one hand, Chertok was in charge; on the other, Pilyugin's firm was in charge. We developed a testing system, but there was not enough time, and we just muddled through. The deadline was approaching, and they pressured us. To test everything thoroughly, one needed probably two more years. It was impossible to tests everything [in the given time]. We managed to complete the tests of the regular sequence of operations, but nothing was tested for irregular sequences. We all had fears. Thankfully, everything ran on the regular sequence. There were no questions afterwards.

Gerovitch: Where there any attempts to write two parallel programs for the same task to create software redundancy?

Zybin: That would not make any sense. Which one would you choose then? It is much better to assign another person to write tests for all program functions.

Gerovitch: Did you use American experience in any way?

Zybin: No, we did not. There was no literature to speak of. It appeared only later. We had the feeling that there was nothing to borrow from them. At least from the literature that was available it appeared that they either made a mistake or tried to mislead. For example, they wrote that they used certain methods in their algorithms, while those methods sometimes would not converge and would lead to failure. Our top priority always was reliability. Our methods were, perhaps, somewhat primitive, but we believed they were more reliable.

For example, the Americans at that time widely used the Kalman method for measurement data processing. Both we and the Institute of Applied Mathematics thoroughly checked it, and it turned out that if measurement errors were large then it produced nonsense. We rejected it right away; we used the mean-square method and cut off everything extra. They did not do it; they acted directly. All the time we had the feeling that they were wrong or maybe they just published the wrong approach and we did not know what their real approach was.

Gerovitch: In software development it often happens that the programmer keeps the details of the program in his head, and if he is not available, then it is very difficult to understand the program.

Zybin: Yes, Sharov is exactly this kind of person. For this reason at some point he was transferred to other tasks and his software was rewritten in such a way that anyone could understand how it was done and to criticize the program. But in the early days that problem was widely spread.

Gerovitch: When was this programming discipline introduced?

Zybin: About the time we worked on the software for the Salyut-4 station. For the Salyut-2M machine we introduced the rule: before you start programming, you must write down a verbal description of what the program would be doing and how. You give the description, and only then start programming. The program must have comments. This was not the case with the first machines. For example, on the Salyut-2 station, they turned the system on, and a problem occurred. They turned it off right away. Sharov then spent a week in Evpatoriia [flight control center] trying to find the error. Nobody could help him; he sat all alone and browsed through telemetry data. Gaushus told him: "The flight will soon be over, and will you still be sitting here looking for the error?" This led to the question of providing proper documentation in order to make it possible to get help or a substitute if the programmer is sick. By the way, in Branets's departments, I think, the things are still the old way: everyone knows only his own segment and there is no proper documentation.

Gerovitch: In his memoirs, Chertok describes cases when the program author was absent and nobody could understand the program.

Zybin: Much of it depends on specific people. Branets did not get around to this, and it all stayed as before.

Gerovitch: Is Branets still a department head?

Zybin: No, he is now in charge of the entire control system complex, in place of Chertok. At first Chertok was replaced by Viktor Legostaev, but the latter was soon assigned different tasks. He is in charge of contacts with the Americans, of the production of wheelchairs, and so on.

Gerovitch: Did the huge experience you accumulated and the new methods you developed have any circulation outside your department?

Zybin: Gaushus left in 1986 and I left in 1991. After that Sharov was appointed head of the department, and this whole business gradually withered away. Sharov is on good terms with us, but he tends to pay too close attention to tiny details. He has finished the Sigma system for the Mir space station (with the PEAs). I do not know what he is doing now.

Gerovitch: Thank you very much for the interview.