How the 'freeze-the-field' rule led to NASCAR's Mobile Technology Center

We want to race the truck

Inside NASCAR's mobile timing and scoring Technology Center vehicle.

You almost can't enter the NASCAR Mobile Technology Center (MTC) vehicle without your mind replaying the annoying little buzzy theme from "Knight Rider." It's a 52-foot Freightliner that serves as a complete office complex for the timing and scoring team. After you step through the main entrance of the vehicle - which is at the back end, through the kitchen - you find yourself in Mission Control. There are banks of LCD monitors along the walls, and rows of notebook computers along both sides, mostly toward the rear and left side of the vehicle.

But you don't see the servers - at least, not right away. They're literally tucked away in a closet...a specially designed closet resting atop a two-ton air conditioning unit: a pair of 32U racks, one for the main system, one for the backup. The main system houses an HP BL25p blade server configuration with five blades. Three of these blades are single-core, 2.8 GHz AMD Opterons with 2 GB of RAM apiece - they're used for the communications network with the transceiver devices. The two other operational units are dual-core, 2.4 GHz AMD Opterons with 4 GB of RAM apiece, and are used for managing the SQL Server database of car positions, which is updated for every split second of the race.

These things shouldn't be here in a closet - at least that's what everyone told NASCAR's Steve Worling in September 2005, when he was first charged with this mission. "I sat down with a specialist on HP and blades," he told us, "and the first thing he said is, 'How you gonna cool 'em'" I said, 'Well, here's my parameters. I'm building a 52' hauler, I've got x by x by x space, which is very small...[and] basically we have two 32 U racks, and I'm going to stick a 2,000 pound air conditioner under them. And they were like, 'No, you can't do that.' I said, 'Trust me, we can make it work.'"

NASCAR national IT manager Steve Worling explains the MTC, with AMD marketing official Travis Bullard looking on.

With the help of Featherlite, the mobile systems construction firm that had built dozens of crew trailers for NASCAR in the past, Worling and his team constructed a 3D model of his dream truck. It took ten iterations for Featherlite to meet Worling's exacting specifications. "I did the calculations on the BTUs, on what the servers could do if they were fully populated with all the equipment," he said, "[and] two tons of air was going to be the maximum we could do."

By the time HP was coming up with specifications for the system, Worling and the Featherlite team had already constructed the truck. "We ordered two 32 U racks with smoked glass doors front and back, and the first thing [HP] said was, 'Why are you doing smoked glass doors?' Well, when you stash these into this room, seal them and then close the doors, now that is your server room environment. It's completely closed off, [and] right underneath this room is a two-ton air unit. The air is fed only to it, nothing else."

The base of the server closet is about eight inches higher than the floor of the truck. That space is empty - on purpose. Fans from Delphi Automotive mounted on the bottom of the racks draw air from the unit into pocket of the open space. When the smoked glass doors are closed, the fans gather cool air from the pocket and expel it out the back of the unit, through two vent holes that Featherlite cut in the top of the closet. Another space, in-between the wall and the work area, acts as a plenum that vents the exhausted air back into the room. The air conditioning unit then recirculates this air, because it's easier to cool cleaner air from the inside than dirtier air from the outside. Besides, we all know how dirty the air can get on a racecourse.

NASCAR's Steve Worling shows off the two 32 U HP ProLiant server racks to analyst Rob Enderle.

Worling's goal throughout the design process was to eliminate all possible single points of failure. There are redundant server racks, and among the timing and scoring server blades, the third is a backup. The HP StorageWorks component provides redundant storage arrays to all workstations in the MTC. And for every connection between the truck and the outside world of fiberoptic connections rigged all around the racecourse, there are two switches. If one connection is cut or one switch is misappropriated, system software can detect the breach and move to the bypass switch.

Even the power system is redundant. On the front of the truck is mounted a 40 KW generator capable of producing 150 amps, even though the servers, workstations, microwave oven, and coffee maker only require about 90 amps maximum. But that's not even the truck's main power source - it's the backup. Main power is supplied by a separate truck, provided by Caterpillar, which provides 998 KW of power for both the MTC and the mobile crews for US television networks.

So suppose the air conditioner unit goes out, as could conceivably happen during an electrical event such as a lightning strike. Would the race be red-flagged? NASCAR engineers have a "never-say-die" attitude. In the event of a cooling system failure, the team is ready to actually cannibalize a part of the truck itself, to serve as a standby cooling device.

"In the front, we have this 8 x 12 office," Worling told us. "I knew this office was [going to be there, so I said], 'You know what? Drop me in as much air as you can. I don't care if it snows in here while we're working; I want as much air as we can get in here.'" The wall on one side of the eight-inch air pocket butts up to the front office. "So I said, 'Cut me as big a hole as you can that fits inside of our plenum, out of this office.' So [if] this AC breaks, I go pull this little piece of wood out of the way, it becomes an inlet for cold air. We close the door to the office, it becomes a meat locker."