Image credits: NASA
The film ‘First Man’ has its UK release this month, celebrating the life of Neil Armstrong, the first man to set foot on the Moon. There can’t be many people who haven’t seen the dramatic footage of Armstrong and Aldrin steering their spider-like LEM down to a safe landing with only seconds of fuel to spare. As far as the public was concerned, Apollo hardware consisted of the Command Module and the LEM. Engineers marvelled at the technology of the Apollo Guidance Computers fitted in both. Everybody forgot about the rest of the vehicle that launched the astronauts on their way: that engineering marvel, the Saturn V rocket. Very impressive take-off, but its mission only lasted a few minutes – nothing sophisticated: just a very big firework, right?
Apollo 6: The last unmanned test flight
Wrong. Apollo 6 was meant to be the final unmanned test before the first manned flight of Apollo 7. Trouble started with the first stage booster (S-IC) just before staging when severe longitudinal oscillations (Pogo effect) started. This could have caused structural failure, particularly as chunks of the rocket further up were breaking off. Fortunately, everything held together and the second stage (S-II) ignited after first stage separation. Sighs of relief in mission control were short lived however: almost immediately one of the five engines began to lose power. It was shutdown automatically, but then a second engine on the same side of the rocket failed completely. Apollo 6 was in real trouble now: it wasn’t designed to survive the failure of two engines causing a serious asymmetric thrust on the second stage. And yet somehow the crippled rocket kept going on the correct trajectory, albeit rather erratically, and achieved orbit. It ought to have a been disastrous setback. Instead, Apollo 7 went ahead on schedule using the smaller Saturn 1B and so did Apollo 8 with a Saturn V vehicle. The faults in Apollo 6 had been quickly identified and fixed. The pogo effect was traced to an instability in the fuel system, frictional heating caused trapped moisture and air to expand and force off the outer skin - cured with a few ventilation holes, the first engine failure was traced to a fuel line design fault and a wiring error caused the second engine to be shut down. Saturn’s ability to complete its mission despite all these problems gave NASA the confidence to sign it off for manned flight. How could a mere firework be so fault-tolerant?
Saturn Instrument Unit (IU)
The IU consisted of a 1m high ring about 6.5m in diameter sitting atop the third stage (S-IVB), containing the Saturn’s electronics distributed around its inner surface.
The Saturn V rocket never failed to complete its mission to deliver three astronauts safely into Earth orbit. It was in modern terminology, an intelligent rocket, controlled by an ‘embedded’ fault-tolerant digital computer called the Launch Vehicle Digital Computer (LVDC). Saturn was truly autonomous: in principle when the rocket was fully fuelled and on the launch pad, all that had to be done was to feed the destination 3D coordinates into the LVDC and press the launch button. The astronauts and mission control just monitored the telemetry until the S-IVB engine shut down at the desired point in orbit. A launch into orbit required three stages:
- From lift-off the S-1C booster heaves 3000 tons into the upper atmosphere, running for just two minutes and accelerating the ‘stack’ to about 8500 km/hour. The LVDC steers the rocket by ‘gimballing’ the engine nozzles according to a pre-set time sequence. There is no feedback from the on-board gyro platform for fear that large corrections might be applied, overstressing the structure. It’s during this period that ‘Max-Q’ is achieved, the point of maximum stress due to atmospheric resistance. Working in parallel with the LVDC is the Emergency Detection System (EDS) which monitors critical systems including its own independent set of accelerometers and gyros. The EDS can sense if something potentially catastrophic is happening and is able to command an abort, firing the escape rocket which pulls the capsule clear. The EDS is on automatic until Max-Q is passed simply because the astronauts may not be able to react to a warning in time. After Max-Q the EDS merely lights an indicator lamp in front of the Commander and he makes the decision whether or not to twist the abort handle he’s holding.
- The LVDC carries out ‘staging’ and starts the S-II engines. So far, the flight trajectory has been close to vertical under open-loop control. Now the control loop is closed by engaging the Flight Control System with its gyro platform to steer the rocket on a pre-determined trajectory. Effectively, an autopilot is engaged which compensates for wind forces and varying engine performance.
- Finally, the second stage is ejected and the S-IVB takes over, also on autopilot, until the requested orbital position is reached at a speed of about 28000 km/hour.
All the above actions take just over 11 minutes. The IU still has two more tasks though: setting course for the Moon and after the LEM has been extracted, sending the S-IVB to a separate crash landing elsewhere on the Lunar surface.
The Ordeal of Apollo 12
The rest of the Moon flights, at least as far as the IU was concerned, were largely uneventful apart from the big scare of Apollo 12. This event alone justified the design decision to give the rocket its own control system (the IU) independent of that in the command module. Soon after take-off the cockpit was lit-up by a blinding flash – lightning had struck the spacecraft. Immediately, control panels went dead or were awash with warning lights. Back at mission control, all telemetry from the rocket was garbled. The Saturn EDS was on automatic – but it didn’t trigger an abort because, as it dawned on all concerned, the S-IC was still running normally, still on course. The Commander, Pete Conrad was gripping the abort handle, hesitating when the obscure, but now famous words came through on the radio from CapCom: “Try SCE to Aux”. Astronaut Alan Bean flipped the switch, all the cockpit displays and telemetry went back to normal, and EECOM Flight Controller John Aaron became a national hero. The other ‘heroes’, the IU and LVDC went largely unrecognised.
One lesson that was nearly learned the hard way: don’t launch a rocket into a storm cloud. The ionised gas of the exhaust plume had formed an excellent electrical connection between the rocket and the gantry, so encouraging the lightning strike!
Saturn Launch Vehicle Digital Computer (LVDC)
The basic functional sections of the LVDC would be familiar to any computer engineer today: clock generator and timing logic, arithmetic logic unit, program counter, program RAM, and so on. Nowadays, all these functional sections would be contained on a single chip of encapsulated silicon called a microprocessor. For comparison with a modern micro, here are some LVDC vital statistics:
- Clock Frequency: 2.048 MHz
- Word Length: 13bits + 1 parity bit
- Bus system: serial
- Clock cycles per instruction: 168 (More for multiply/divide)
- Processing Speed: 12190 instructions/sec
- Memory: magnetic cores, up to 32K words, duplex with single-bit error correction
Take a look at Fig.2 which shows an LVDC with the front cover removed. There are many small plug-in modules inside, each composed of two printed circuit boards mounted back to back. Each PCB contains up to 35, what look like, modern integrated circuits. In fact, they themselves are tiny PCBs containing a few diode or transistor chips and resistors under a ceramic cover, connected to provide say, a four-input logic NOR gate. They look like modern 14-pin Surface-Mount (SMT) ICs, except that the ‘chip’ end of the pin is formed into a clamp which grips a pad on the edge of the tiny PCB. This means that they can be removed just by sliding them sideways, leaving the pins behind. I suspect this quick replacement system was used because many of these tiny hand-soldered modules either didn’t work on initial test, or failed subsequent stress testing. A working flight board would then have been covered in epoxy to fix everything in place. Very little detailed documentation still exists on the LVDC and this information comes from a recent ‘teardown’ here.
This is where things get a lot more complicated. The LVDC actually consists of three identical processors running in parallel, giving it Triple Modular Redundancy (TMR). There’s not much point in triplicated circuitry unless there is some means of identifying the faulty component and mitigating its effects. The LVDC uses voting logic (Fig.3) to monitor the serial busses between the functional sections of each processor.
Each data bit passes through the voter logic and on to the next section. Normally, the three inputs to each voter will be the same, either 000 or 111, so the three outputs will be 000 or 111. But, if a single bit is reversed, so that the input becomes say, 100 or 110, you can see from the truth table that the outputs will still be 000 or 111 by a majority decision. The error, permanent or transient is detected and corrected automatically. One unwanted side-effect is that defects could be masked when carrying out pre-flight checks. This is avoided by adding the 3-input Exclusive-OR gate as shown. The EXOR gate output is 0 when all its inputs are the same: 000 or 111. Anything else will lead to a logic 1 output, signalling an error. The Error Detected signal is only monitored when carrying out checks just before launch, avoiding the mission starting with already faulty hardware!
The memory could be split into two redundant blocks (duplex mode). Each block held the same data or program code. A word would be read from one block and the parity bit checked. If an error was indicated the corresponding word was read from the other block and then written back into the first to correct the error.
No Small Task
The LVDC program code controlled the operation of systems that were not just complex, but extremely powerful and dangerous. The five giant first-stage engines developed 60 gigawatts of power, burning fuel fed from five 53,000 horse power pumps at a rate of 2.5 metric tons per second. One small programming error and Neil Armstrong would not have survived the launch, let alone land on the Moon. Think about that the next time you’re debugging some program code…..
High-reliability computing using redundant components has been an important part of military and commercial aircraft design ever since the 1960’s. Nowadays it is crucial to the safe operation of automobiles with electronic driver assistance (ADAS) and of course those capable of fully autonomous driving. Designers are increasingly programming FPGAs with multiple ‘soft’ processor cores. Tools are becoming available to scan the FPGA layout and work out where to insert voting logic to achieve a desired level of reliability. The legacy of the LVDC may be to prevent a fault in your car’s systems from causing an accident one day. The Saturn’s mission was short and the TMR gave the LVDC a 99.6% probability of successful operation for 250 hours. Tomorrow’s cars will need to do a lot better than that.
If you're stuck for something to do, follow my posts on Twitter. I link to interesting articles on new electronics and related technologies, retweeting posts I spot about robots, space exploration and other issues.