Toyota Shut Down 14 Factories Due to 'Insufficient Disk Space'
You can never have too much storage
Even major car manufacturers can never have too much storage space. In late August, Toyota had to shut down 28 assembly lines at 14 auto plants in Japan due to computer issues. Today, Toyota provided more details on the shutdown, including that "an error occurred due to insufficient disk space."
According to a press release from the car maker, the shutdown of these plants was a result of system maintenance that occurred a few days prior. During this maintenance, "data that had accumulated in the database was deleted and organized, and an error occurred due to insufficient disk space, causing the system to stop." The company further wrote that the servers were running on the same system as its backup, causing the same issue there, so the company couldn't make a switch.
The fix occurred when Toyota's engineers moved to a server with more storage space, with the plants restarting the next day. The company claims that "[c]ountermeasures have also been put in place by replicating and verifying the situation."
At the time, there was speculation that Toyota had fallen prey to a cyberattack. But the company's release is meant to "reaffirm" that security wasn't the issue.
"Going forward, we will review our maintenance procedures and strengthen our efforts to prevent a recurrence, so that we can deliver as many vehicles to our customers as soon as possible," the unsigned note reads.
News reports of the initial shutdown suggest that the issue was with a system that deals with orders for incoming vehicle components. The Guardian suggests this could have caused an issue with Toyota's "just-in-time" system of managing parts, keeping only what is needed to build cars at their exact points of assembly. This is meant to reduce costs (Apple is known for its mastery of the practice), but — clearly — everything falls apart when you can't order the parts you need.
Toyota's statement also included an apology to "customers, suppliers, and related parties" for "any inconvenience caused by the suspension of our domestic plants as a result of the malfunction in our production order system at the end of last month."
Reuters estimates that these Japan-based plants account for roughly one third of Toyota's global car production. In Feb. 2022, Toyota shutdown the same 14 plants due to a cyberattack.
Moral of the story: Always keep some extra storage space on hand for anything mission-critical, especially if you're a massive company that requires software that can be timed to exact specifications in order to build complicated machines.
Stay On the Cutting Edge: Get the Tom's Hardware Newsletter
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
Andrew E. Freedman is a senior editor at Tom's Hardware focusing on laptops, desktops and gaming. He also keeps up with the latest news. A lover of all things gaming and tech, his previous work has shown up in Tom's Guide, Laptop Mag, Kotaku, PCMag and Complex, among others. Follow him on Threads @FreedmanAE and Mastodon @FreedmanAE.mastodon.social.
-
washmc The company further wrote that the servers were running on the same system as its backup, causing the same issue there, so the company couldn't make a switch.
This is the most hilarious part. Who knew I could do better sys admin work than whoever is in charge of all these Toyota plants? -
Elusive Ruse You'd think at some point they'd use the profits from their overpriced cars and invest it on their servers.Reply -
PEnns What the ever-loving f......???Reply
Toyota doesn't have one competent sys. admin?? Who runs out of disk space in this pathetic manner!! -
derekullo Yo dog we heard you liked virtualization!Reply
So we virtualized all your servers so you can backup while you virtualize and virtualize while you backup! -
Sippincider I could forgive an early startup, with money tight and "Bring Your Own IT".Reply
But a company with roots to 1924, and doing $286+ billion in annual revenue today?? Get the freaking storage you need. -
umeng2002_2 Is this one of those cases where they need to install Windows 11 from thousands of floppy discs?Reply -
neojack believe it or not, but it happens a lot really.Reply
2 classic cases :
- logs on an SQL serveur. to clear the logs you have to merge the logs with the main DB file.
- snapshots on a Virtual Machine. It creates two files than needs to be merged eventually.
Those merging operations needs twice the space as the separate files. The process can start but if there is not enough space, it would fail and all the files would be corrupt. You need space to restore the backup but at this point there is any. it's a catch 22.
Restoring a backup needs space wich is a problem if the backup is on the same disk as it was the case here.
Also, by default, Microsoft SQL server stores it's files on the system partition. So if the company maintaning it didn't payed attention, the problem would be even worse, the OS would crash.