Computing in the Soviet Space Program

Introduction

Bibliography

Computers

Discussion

Documents

Essays

Institutions

Interviews

Links

Sitemap

• When did you first start working with onboard computers?

• Who was writing programs - engineers who understood control problems or specialists in computing?

• Did the first Salyut computers ever perform actual control functions?

• When did onboard computer begin performing essential functions?

• Back in the early 1960s, how controversial was the decision to place a computer on board?

• What problems arose in connection with software documentation?

• To what extent did the computer control cosmonauts' actions? Did the machine set limits on what a cosmonaut could do?

• Did you rely on machine more than on human being?

• What were the arguments during the debate over which machine to install - the Argon-16 or the Salyut-4?

• How was the reliability problem solved?

• What methods were used to test and debug the software?

• Did you use American experience in any way?

• In software development it often happens that the programmer keeps the details of the program in his head...

• Did the huge experience you accumulated and the new methods you developed have any circulation outside your department?

Gerovitch: When did you first start working with onboard computers?

Zybin: I graduated from the Moscow Institute of Physics and Technology in 1961, and on April 4 of that year I came to Korolev's firm. In just a few days there was Gagarin's flight. From the very beginning, I began working on navigation of space vehicles. A year or two later a lunar program started. Since nobody was very confident that technology was completely reliable, we planned to control the lunar flight from the Earth, but as a backup we placed a sextant and other navigational equipment on board, so that cosmonauts could, for example, carry out lunar landing. This required data processing on board. This was approximately in 1964. By that time one firm in Zelenograd developed the Salyut-1 computer. It was a small-size machine weighing about 5 kg. Its memory had only 64 words. The problem was that trajectory calculations required a lot of memory. The division of responsibilities was as follows: Timur Eneev's group at the Institute of Applied Mathematics worked on the software for the flight to and from the Moon (the trajectory and correcting burn); and our department, headed by Ernst Gaushus, was responsible for the calculations related to the orbiting of the Moon and the lunar landing. The problem was how to carry out all these calculations, using only 64 words. Yet everything was done and tested, the documentation was ready, but the Americans flew first, and our program was terminated.

After that, our firm turned to orbital stations. We placed the same computer on board the Salyut-1 station. The name of the station here coincided with the name of the machine. A major drawback was that the machine was not very powerful, and it required a lot of manual work. Cosmonauts had to take measurements and enter the data; even during computation they had to help the machine: the memory was so small that they had to enter intermediate results of calculations. The Salyut-2 and the Salyut-3 space stations carried the Salyut-2M computers. Those were also made in Zelenograd. The Salyut-2M was a relatively advanced computing machine. It had a tape drive, made at a firm located on Novaia street (now it is called Kosmopribor). Magnetic tape for this drive was taken from American balloons. The Americans sent spy balloons over the Soviet territory; some of them fell down, and tapes were taken from those balloons.

We faced the following problems. First, other onboard equipment was poorly suited for working alongside a computer. The concept of electromagnetic non-interference had not been developed yet. Second, since everything was done very quickly, the software was not tested well. There were many malfunction reports which were difficult to sort out - whether those were hardware or software errors. For the Salyut-4 station, we rewrote the Salyut-2M operating system completely. This station had an advanced navigation system, solar sensors (which timed the moments of entering and leaving the shadow zone), and a Doppler velocity meter (on the Soviet territory there were transmitters, which generated a precise, stable base frequency, and there was a receiver on board, which measured the velocity based on the Doppler effect). On the Salyut-4 station, the trajectory was calculated automatically on board. As it turned out, however, there was no big need for that, since the ground control calculated everything precisely enough. The onboard system, in fact, served as a backup.

The main purpose for which this machine was actually used was displaying information for the crew: when will be the next communication session, how long will it be, when the spacecraft will enter the shadow zone, and so on. As for various measurements and calculations, those were not used but served as a backup, for information purposes only. This machine kept working for months in a row, and it operated normally.

On Salyut-7, the last Salyut station, despite many breakdowns on board, the computer worked until the last minute, until the power was shut off. There was also many computer malfunction reports - not clear whether software or hardware errors. Diagnostic techniques were not developed well enough at the time. If there was a suspicion that it was a hardware error, and computer designers were hard pressed, they could not firmly reject this hypothesis. Especially since this machine had no redundancy. There was only a "cold" backup on board: normally the first machine operated, and only if it broke down, the second one would be turned on.

In the 1960s, in the Control System Division, headed by Boris Chertok, there were two departments, one led by Vladimir Branets, and the other by Ernst Gaushus. Initially it was planned that Branets would be working on cargo ships, and Gaushus on orbital stations. Branets' department preferred Argon onboard computers, and Gaushus' department Salyut computers. Based on the experience with the Salyut-2M, a blueprint for the Salyut-4 computer was created. In the early 1970s, in Chertok's division there were constant debates which machine to choose. We insisted that the Salyut-4 must be installed on all space vehicles, and Branets pushed the Argon. The Zelenograd firm built the Salyut-4, but then a "pause" occurred, and its production was delayed. More funds were needed. In the end, funds were not allocated, and the Zelenograd firm began the development of the Salyut-4 on its own.

In parallel with the Salyut stations, work on the Mir station began. Since Gaushus' department was overloaded, this project was assigned to Branets. Branets' department had more or less finished the cargo ship, and people in his department had time to do a new project. It was planned that the Mir would fly in orbital orientation, and in order to point equipment toward the Earth and the stars, a rotating platform was made in Czechoslovakia (this work was supervised by the Institute of Space Research). Wherever the station was, it could monitor a specific spot on Earth. Gaushus' department received an assignment to design a control system for this platform. This was in the mid-1980s. At the same time the Central Specialized Design Bureau in Kuibyshev, headed by Dmitrii Kozlov, was developing a new series of satellites, and together with the Zelenograd Science Center they decided to build a large computer complex: a high-speed computing machine with a single Manchester cable, which connected to individual pieces of equipment through peripheral exchange adapters (PEAs). The Zelenograd firm built this PEA. It was, in fact, a real computer weighing about 15-16 kg. It was planned that each system, for example, the engine control system, would have its own PEA, which would transfer data into the main computer. We decided to use this PEA as a control computer for the rotating platform on the Mir station, and this was done. As it turned out, I have never seen a better computer. It worked the whole time when Mir was on the orbit. The Mir station itself was equipped with Argon-16 computers, but since those were not powerful enough, Branets designed an onboard network in such a way that the Argons could later be replaced with Salyut-4 computers, which were under development at that time. Branets was responsible for the Mir control system; they used both the Argon-16 and the Salyut-4 computers, and they had to swap the computers all the time. One day they shipped one computer to the station, another day another computer; there were constant malfunctions. And the first PEA that we installed worked on Mir all the time without a single malfunction. Later other PEAs were added to it, as the functions expanded and other pieces of equipment were added, but this first PEA remained the central part of this system.

I am flabbergasted to see how ungrateful sometimes people are. Nobody thanked the chief designer of this PEA, Anatolii Shchagin. They could have written a newspaper article about him or given him a medal. In my opinion, there has never been another computer as good as that. It had triple redundancy and was built with 16-layer ceramic boards. It did not have a single malfunction during the entire operation period.

Later on, in parallel with the Mir station, we started working on the Buran. There was a large onboard computer installed on the Buran. Whereas in all previous projects all software was written by a separate department, on the Buran everybody wrote their own piece and tested it as they could. The job of Gaushus' department was to put everything together and test it. In reality, the tests were somewhat superficial. It was a miracle that the Buran's flight was successful. They were right that they did not launch it the second time. It is well known that most problems occur at junction points: one did not say it right, another did not understand it right, and so forth. Even though they issued special instructions on how to avoid such situations, nobody could give assurances.

Gerovitch: After the Zelenograd firm handed the Salyut-1 over to you, did they do any programming for this machine? Did they do any modifications of hardware?

Zybin: No, we did all the programming. There were no hardware modifications. Later they developed the Salyut-2M model, which had a tape drive and larger memory (256 words, I think), and we switched to this new model.

Gerovitch: Who was writing programs - engineers who understood control problems or specialists in computing?

Zybin: Software for the first Salyut computers was written by the group led by Viktor Aleksandrovich Sharov. They wrote programs based on algorithms developed by us and by the Institute of Applied Mathematics. For the Salyut-2M computer, Sharov's group wrote both the operating system and the applications. Since on the Salyut-6 station there were many malfunctions, we decided to rewrite the operating system of the Salyut-2M. This project was assigned to me, and we carried it out in Sharov's group and in other groups under my supervision.

The situation with PEAs on the Mir station was different. PEAs already had a built-in operating system written by the Zelenograd firm. It had a debugging system, and programming was done in autocode. One could also use high-level languages, for example, BASIC. In this case, the person who wrote the algorithm also wrote the program. The programmers also helped, but they largely worked on data exchange problems.

Mir had a distributed complex. One PEA was installed on board, and there was a digital communication channel with the ground control. On the ground, there was a PEA and a personal computer next to it. When an operator on board saw something interesting on Earth, he could turn the rotating platform and point it in the desired direction, and the platform would then track this spot. The same could be done from the ground. Within a direct visibility zone, the ground systems and the onboard systems worked in parallel and in coordination with each other. Once ground control was performed from Australia; later they took it to Europe, but this was at the time when I had already left. This was a very good computer, but expensive: it cost about 100,000 rubles.

In Zelenograd, they had a large computer design and manufacturing complex. One organization was at the top, and it had many divisions. Different Salyut models were made by different people, but they all bear the same name.

Gerovitch: There were both Argon-16 and Salyut-4 computer on board the Mir station. Which of them kept breaking down?

Zybin: Branets would answer this question better. We did not go into this. We had certain competition [with Branets' department], so there was no information exchange. This was not a friendly competition. Gaushus and Branets competed for primacy with varied success. Sometimes Gaushus would take the upper hand, sometimes Branets. When Branets prevailed the last time, Aleksei Yeliseev offered Gaushus to move to the Bauman Moscow Higher Technical School (MVTU), and Gaushus did. When Yeliseev became the rector of MVTU, he decided to reform its structure and created seven scientific research institutes, corresponding to the seven faculties at MVTU. In those institutes research is conducted separately from teaching. He invited leading engineers from industry to be the heads of these institutes. At MVTU a typical scenario was that a student graduates, goes to graduate school, becomes a professor, and then a department head. He teaches textbook knowledge; he never designed anything himself, and consequently he cannot teach any practical knowledge. But Yeliseev's tenure proved too short, only five years. He did not expect that the faculty would not vote for him. Of course, the old cadres felt that the new policies infringed on them. And here teaching is primary. So all department heads voted against him during the elections. He left, and for the last 10 years they have tried to roll everything back as it used to be, but they have not got enough power to do that.

Gerovitch: You mentioned that the first Salyut computers performed largely a backup function in case ground control fails.

Zybin: They performed not only the backup function, but also a monitoring function. The ground control makes calculations, and the onboard computer makes calculations, and if they match, then everything is correct.

Gerovitch: Were those computers actually used?

Zybin: The lunar program did not take off, so in this case it was done in vain. On the Salyut stations, the crews used the computer for their own needs. The machine calculated the time of the next communication session, it assisted with planning, and so on.

Gerovitch: Did a situation ever occur when control had to be switched from the ground to the onboard computer?

Zybin: No, there was no such case. If a problem occurs on an orbital station, it can usually be put off for a week or a month. By contrast, during a flight towards the Moon or during lunar landing, you cannot put off the problems. Processes here are irreversible, and there are specific time intervals when certain things have to be done, and here you absolutely need a backup.

Gerovitch: When did an onboard computer begin performing essential functions?

Zybin: It was planned that the first Salyut stations would fly in orbital orientation, and they had an analog control system. In essence, a station is just staying in orbit and is not doing much. Later on new tasks were added: to perform various experiments, the station had to be pointed this way or that way. Those stations did not have a rotating platform, so the entire station had to maneuver. On Salyut-7, we linked the Salyut-2M computer to the analog control loop, and with the help of this computer the station could do everything that was needed. The Salyut-2M computer manipulated signals from various sensors and was able to "deceive" the analog control system and to make it perform what was needed. On Salyut-7, the computer was part of the control system. It was used for many experiments and for trajectory corrections. On Mir, of course, the machine was also actively used. The Soyuz-T spacecraft had a digital control system based on Argon-16 computers.

Gerovitch: On Soyuz-T, the onboard computer was an essential part of the control system. What would happen if it broke down?

Zybin: There was a backup analog control system on board.

Gerovitch: Back in the early 1960s, how controversial was the decision to place a computer on board?

Zybin: For the lunar program, this was an obvious solution, since everyone realized that no risks could be taken. In this case, this was absolutely necessary. On the first Salyut stations, these machines and perhaps the entire navigation system were installed largely because there was a department engaged in these issues, and everything depended on personal contacts. They agreed to install this equipment, and that was it. There was no huge need for that, but everybody considered this important.

Gerovitch: Were there any doubts whether this technology would prove reliable in flight?

Zybin: No, there were no such doubts. The main problem was the lack of hardware. We made the first Salyut and thought that it would be the last one. It was launched, and then an order arrived to build the next one. Every two years we built a new Salyut, and every time we thought it would be the last one. The slogan was: "In order to do everything fast, don't change anything; launch it as is." Yet every time we changed something. This policy - "this is the last station; there is no need for updates" - slowed down everything. The policy was that computer hardware needed no updating either. So it dragged on with all of the Salyut stations.

Gerovitch: What problems arose in connection with software documentation?

Zybin: For the Salyut-1 computer there was practically no documentation. The description of instructions was no good. For example, the addition was described like that: this flag is set up here, these digits are changed in the processor registers, and so on. There were many nuances, and the programmer had to figure it all out. The machine was unfinished; there were hidden bugs everywhere. Even addition was carried out differently at different times. It all depended on history: if something was left over after the previous operation, then the buffer had to be cleared or something of this kind. Our programmers had practically no documentation. They received hardware and some notes that the designers had scribbled for their own use. Our programmers had to figure things out on their own. They finally did it and produced their own documentation, and after that everything was okay.

The documentation for the Salyut-2M was better, and the PEAs had complete documentation, which had only one error related to temporal functions. This error was noticed during ground tests. Since these computers worked in real time, several times per second they received precise time signals and also clock frequency (1,000 Hz) signals. The operating system had certain tasks to turn something on at specific moments. We noticed that some of these tasks were simply skipped and not carried out. The machine was malfunctioning! We called the chief designer in. As it turned out, the operating system had a bug. The precise time signals and the clock frequency signals arrived in the system asynchronously from different generators. The machine, however, expected them to be synchronous, and for this reason it skipped. When they tested it on their simulator, both signals came from a single generator, and this glitch did not show. The documentation did not specify that the signals had to be synchronous, and we used different generators, which resulted in the malfunction. We noticed this problem during ground tests and spent about a month working on it. We had hard time figuring it out. Eventually they fixed it; they made a patch in the operating system. After that there was not a single problem.

Gerovitch: To what extent did the computer control cosmonauts' actions? Did the machine set limits on what a cosmonaut could do?

Zybin: In the beginning we tried to limit everything. On the Salyut-1 the cosmonaut was required to enter a code (for example, data from the first column of sextant measurements). Then the machine checked the data for consistency. There was no guarantee, of course, but rough testing was done. If he missed a digit in a high-order position, he would get an error message. However, if a mistake occurred in a low-order position, it was impossible to detect it. We handled it this way: we required the cosmonaut to enter the data twice. If it was the same both times, then it was assumed correct.

Gerovitch: Could the cosmonaut ignore the limits set by the machine? Did the machine only warn the cosmonaut of a potential mistake, or did it firmly tell him that he must redo the task?

Zybin: If he made a mistake in a high-order position, the machine would display the message "Enter the first code" again and again.

Gerovitch: That is, the cosmonaut could not do anything until the machine accepted the input?

Zybin: That's right.

Gerovitch: And if, say, there was an error in the algorithm, and the machine rejected correct data, could the cosmonaut do anything about it?

Zybin: Nothing. We could not count on the cosmonaut in this regard. For example, someone gave a lecture to cosmonauts and explained how to enter "the sine equals 0.8." Gagarin asked: "But here it already shows zero and the decimal point. What if the sine equals one and a half, how would I enter it then?" He should have learned this in high school.

Gerovitch: Did you rely on machine more than on human being?

Zybin: That's right. On Salyut-7, in order to adjust the station’s attitude, the cosmonaut could make corrections using a hand controller. The machine received signals from this controller and made corrections. Here is a dilemma. On the one hand, you may want to limit the use of the machine just in case it has an error or it uses data incorrectly. On the other, if you set such limits, you often cannot use the machine. The latter happened in the automatic docking system developed by Branets. They set strict limits on the velocity range, but it so happened that a sensor was giving incorrect data [as a result, the velocity was calculated incorrectly], and the automatic docking control system was simply turned off. The crew had to perform manual docking.

Gerovitch: Could the cosmonauts make any adjustments in the machine so that this velocity range, for example, could be corrected?

Zybin: Initially they could not. Later on, when the Salyut-2M computer with a tape drive was installed, they could make corrections using the tape. After that, for the Soyuz spacecraft Branets provided for the possibility of entering new programs. Nevertheless, corrections were made only if something did not work the first time.

Gerovitch: Did the cosmonauts on board ever make program corrections based on instructions from the ground?

Zybin: I think they did, but I cannot say for sure. This possibility existed on Salyut-2M. Salyut did not have a digital telemetry link; they used the Stroka teletype system. Since we provided for the possibility of correcting programs, we linked the computer with the Stroka, and we entered new programs and corrected old ones through this system. The station stays on its orbit for a long time; new tasks arise; and we provided for the possibility to carry out operations which had not been anticipated before, and this was widely used. Just in case, there was also the possibility for the cosmonauts to receive programs from the ground by voice and then enter them into the machine. We never used that option; we sent everything over the teletype.

Gerovitch: That is, even though it was possible to correct programs for Salyut-2M on board, this possibility was never used, right?

Zybin: Right, we never used it.

Gerovitch: Were the cosmonauts taught the fundamentals of computer programming, or just the operator functions?

Zybin: They were taught how to enter information.

Gerovitch: On the later models of space stations and spacecraft, did the cosmonauts have functions that required a deeper understanding of the functioning of computer?

Zybin: No, quite the opposite. The cosmonauts that used the PEA set only the mode and did not even touch the keyboard. Everything was controlled from the ground, while the cosmonauts used the hand controller only for corrections.

Gerovitch: What were the arguments during the debate over which machine to install - Argon-16 or Salyut-4? What were the advantages and the disadvantages of one machine compared to the other?

Zybin: The debates were mostly about the instruction set. Nobody thought about reliability at that point. In fact, the choice was very difficult. The Salyut-4 had the Unified Series instruction set, which was considered more advanced. It was indeed more advanced, but this factor proved insignificant. Take a personal computer: it has a miserable instruction set, and yet... In fact, one can do anything with any instruction set. If the processor is fast, then the instruction set does not matter. Our debates centered on insignificant factors - as we know now - but back then it seemed crucially important.

Gerovitch: How did these machines compare in terms of weight and speed?

Zybin: The Salyuts were always lighter and were considered more advanced. Branets argued that the Argons were widely used, that they were installed on artillery, that they were better tested, but these were insignificant factors.

Gerovitch: How was the reliability problem solved?

Zybin: Reliability was achieved through redundancy. The PEAs had triple redundancy; the Argons double redundancy, I believe.

Gerovitch: In case of Argon, was it a "hot" backup? That is, did both computers work in parallel?

Zybin: To tell the truth, I do not know. I was immersed in algorithms and programs and did not care much about hardware. Gaushus and Sharov took active part in those debates.

Gerovitch: What methods were used to test and debug the software?

Zybin: The testing of software for the Mir space station was extremely thorough. Before a programmer wrote a program, he had to write a clear description of what the program would do. After that, he would write the program, while another programmer received a separate assignment to write tests for that program. Each program was tested in this way, and when they were integrated, different programmers were again assigned to test this integrated product. For this reason, the software for the PEA worked so well.

For Buran, we also planned many tests, but we lacked time. The division of functions was complicated: Chertok was responsible for the entire control system, while the core control system was developed by Pilyugin's firm. The latter designed a dispatcher machine. The integration of all software was assigned to us, that is, to Chertok. It was difficult to sort out who is responsible for what. On the one hand, Chertok was in charge; on the other, Pilyugin's firm was in charge. We developed a testing system, but there was not enough time, so we just muddled through. The deadline was approaching, and they pressured us. To test everything thoroughly, one needed probably two more years. It was impossible to tests everything [in the given time]. We managed to complete the tests of the nominal regime, but nothing was tested for any deviations from the nominal mode. Everybody feared the worst. Thankfully, everything ran in the nominal regime. Then nobody asked any questions.

Gerovitch: Where there any attempts to write two parallel programs for the same task to create software redundancy?

Zybin: That would not make any sense. Which one would you choose then? It is much better to assign another person to write tests for all program functions.

Gerovitch: Did you draw on the American experience in any way?

Zybin: No, we did not. There was no literature to speak of. It appeared only later. We had the feeling that there was nothing to borrow from them. At least from the literature that was available it appeared that they either made mistakes or tried to mislead us. For example, they wrote that they used certain methods in their algorithms, while those methods sometimes would not converge and would lead to failure. Our top priority always was reliability. Our methods were, perhaps, somewhat primitive, but we believed they were more reliable.

For example, the Americans at that time widely used the Kalman method for sensor data processing. Both we and the Institute of Applied Mathematics thoroughly checked it, and it turned out that if measurement errors were large then it produced nonsense. We rejected it right away; we used the mean-square method and cut off everything extra. They did not do it; they calculated directly. All the time we had the feeling that they were wrong or maybe they just published the wrong approach and we did not know what their real approach was.

Gerovitch: In software development it sometimes happens that the programmer keeps the details of the program in the head, and if the author of the program is not available, then it is very difficult to understand the program.

Zybin: Yes, Sharov is exactly that kind of person. For this reason at some point he was transferred to other tasks and his software was rewritten in such a way that anyone could understand how it was done and to criticize the program. But in the early days that problem was widely spread.

Gerovitch: When was this programming discipline introduced?

Zybin: This happened around the time we worked on the software for the Salyut-4 station. For the Salyut-2M machine we introduced the rule: before you start programming, you must write down a verbal description of what the program would be doing and how. You give the description, and only then start programming. The program must have comments. This was not the case with the first machines. For example, on the Salyut-2 station, they turned the system on, and a problem popped up. They turned it off right away. Sharov then spent a week in Evpatoriia [flight control center] trying to find the error. Nobody could help him; he sat all alone and browsed telemetry data. Gaushus told him: "The flight will soon be over, and will you still be sitting here looking for an error?" This led to the question of providing proper documentation in order to make it possible to get help or a substitute if the programmer is sick. By the way, in Branets's departments, I think, the things are still done in the old way: everyone knows only his own segment of the program and there is no proper documentation.

Gerovitch: Is Branets still a department head?

Zybin: No, he is now in charge of the entire control system complex, in place of Chertok. At first Chertok was replaced by Viktor Legostaev, but the latter was soon assigned different tasks. He is in charge of contacts with the Americans.

Gerovitch: Did the huge experience you accumulated and the new methods you developed have any circulation outside your department?

Zybin: Gaushus left [the Energiya Association] in 1986 and I left in 1991. After that Sharov was appointed the head of our department, and this whole business gradually withered away. Sharov is on the good terms with us, but he likes to immerse himself in internal intricacies of programming. He has finished the Sigma system for the Mir space station (with the PEAs). I do not know what he is working on now.

Gerovitch: Thank you very much for the interview.