Musk Says Biggest Model 3 Bottleneck Was Software Related To Battery Module Production

Tesla Model 3


Writing 20 to 30 man-years of software code in just weeks is no easy task.

That’s apparently what had to be done at Tesla to correct a bottleneck that was holding up Model 3 production.

Tesla Model 3

Tesla Model 3 Production

Following the recent Q3 report, Musk explained on the earnings call that one of the major production constraints for the Model 3 was that software for two of the four Model 3 battery modules/production zones had to be recreated. Quoting Musk:

“To date, our primary production constraint has been in the battery module assembly line at Gigafactory 1, where cells are packaged into modules.”

“We had to rewrite all of the software, from scratch. We managed to write 20 to 30 man-years of software in 4 weeks.” 

Tesla outs the issues on suppliers, saying that the initial programming was “done by manufacturing systems suppliers.”  Musk further noted that Tesla itself “significantly redesigned” the system/software to correct the issues.

The Tesla CEO also noted specific issues to the zones (both software and hardware related):

 “Zone two (of four) in particular, we had a subcontractor, a systems integration subcontractor, that unfortunately really dropped the ball, and we did not realize the degree to which the ball was dropped until quite recently, and this is a very complex manufacturing area. We had to rewrite all of the software from scratch, and redo many of the mechanical and electrical elements of zone two of module production.”

This bottleneck was hinted at awhile back, but now we get a clearer idea of what the actual issue was. The rewritten software should allow full automation of the battery module assembly process, which in turn should increase Model 3 production. However, the hiccup was enough to force Tesla to push back production targets for the Model 3 and to issue emails to reservation holders noting delivery delays.

Tesla now says it’ll produce 5,000 Model 3s per week by the end of Q1 2018, basically a three-month setback from initial projections.

Categories: Tesla


Leave a Reply

97 Comments on "Musk Says Biggest Model 3 Bottleneck Was Software Related To Battery Module Production"

newest oldest most voted
(⌐■_■) Trollnonymous

I think it was caused by the over-engineered door handles.


These are the New Cells so it’s possibly believable . Hopefully this glitch is rectified and Tesla can get on with building cars again. I trust that all they say is true, Being the gullible optimist that I am…… lol

Seems like the totally new Pack Modules, as a build process for the Robotics, was not well tested, prior to the July 1st Deadline!

It was My Impression, some time ago, that last year, they ordered enough parts for 300 Model 3’s! It would have been great if they had built the 1st 100, and sent them out to Test Centers, then the next 100 and sent them out into the wild for Testing, and the 3rd Hundred, using the processes they want to build them: IE the Robotics, in this case, for at least that Hundred (if not all of those 300)!

I’m not quite sure from your comments, Robert, but you seem to be under the impression that the problem was with the battery packs themselves, and that the cars should have received more testing.

No, the problem was with the “modules” of the battery pack production line, and — at least according to Tesla’s side of the story — with the software for running the machines/robots in that assembly line module.

If the cars worked 100% perfectly, that still wouldn’t alter the problem with the assembly line producing the battery pack for those cars not working properly …or from the sound of it, part of the line not working at all.

…. And there is no way to know weather issues Tesla encountered would be visible with smaller batch of packs.

It’s also possible Tesla used manual build process for first power packs. And lets be frank 100 powerpacks done by hand is entirely doable, so “bigger initial” batch is not a guarantee that such issue would not occur.

Project Management says get critical path issues on line and tested in advance. There are NO excuses.

Except in production of a car EVERYTHING is in the critical path.

Also: it is anybodys dues when we would see the first cars when everything possible would have been outsourced, i.e. no way to fix something like this by yourself?

Not everything, but the issues that are there better be resolved in advance.

Rings hollow.

Yes, rings are.

.ah.ah..ah just like my head.

(⌐■_■) Trollnonymous

Like my wallet!

You Still use a Wallet? I Thought Apple ‘Pie’ (Apple Pay) was the thing these days!

From scratch? That seems like a fundamental error, as usually you can just fix your mistakes, that’s the advantage of software.
Oh well, hopefully now it will work properly.

Right! So the two questions are:
1) why was the first architecture so flawed that it had to be scrapped entirely and
2) given that what makes them think the second one would not be flawed?

Scrapping code is sometimes much easier and faster than fixing faulty code, especially if the faulty code is poorly documented.

Probably – Poorly Documented, and also written by people who just write code differently!

Imagine – Elon sees the process stumbling, crashing, sluggish, or Jerky, in the Robot Motions for the Pack Builds, and then has a chance to see the Code: “Hmmmmm, Why did they put that THERE?” And What is THAT? Wow! – They really Messed This Up! Guys – Guys, we need to get this fixed, Like – NOW!”

Maybe the comments were in Japanese.

(⌐■_■) Trollnonymous


Yup, I’ve had to deal with that myself. When faced with an undocumented program that’s not designed with a clear structure, that seems to be a patchwork of fixes that jump from one part of the program to another, then it’s best to just scrap it and start from scratch, so you can understand what the software is doing.

Production hell indeed!

Of course, the unanswered question here is why nobody at Tesla was exercising enough oversight of the battery pack assembly line installers to realize there was a major failure of performance in that “module”, if that’s really the reason for this major delay.

I think I read the Wall Street Journal reported that the problem was disagreements between the installers and Tesla engineers over how the assembly line should function. Sounds like the same situation, but reported from a different viewpoint. It’s a glittering generality to say “Every story has two sides”, but that’s usually true; and we haven’t heard the installer’s side of this story.

I’m not a programmer, but I worked with few. Documenting their works was always the last task they wanted to do. So, the next team of programmers always had the same opinion: we have to redo the whole thing or we cannot be responsible for that mess.

Or if it was written in a poorly chosen language for the task.

“You wrote the entire thing in bash???”


First programmer used Windows. It’s being re-written on Linux. 🙂

The complete quote:
“To date, our primary production constraint has been in the battery module assembly line at Gigafactory 1, where cells are packaged into modules. Four modules are packaged into an aluminum case to form a Model 3 battery pack. The combined complexity of module design and its automated manufacturing process has taken this line longer to ramp than expected. The biggest challenge is that the first two zones of a four zone process, key elements of which were done by manufacturing systems suppliers, had to be taken over and significantly redesigned by Tesla. We have redirected our best engineering talent to fine-tune the automated processes and related robotic programming, and we are confident that throughput will increase substantially in upcoming weeks and ultimately be capable of production rates significantly greater than the original specification.”

Seems like a lot of words to say SNAFU.

In this case, it morph in FUBAR!

No, they could not. It was NOT THEIR software.
They did not own it.
Also, id SW is POS it really IS better to start new.

Their contract with the sub might not have included Tesla having ownership of the underlying source code. Those rights may have been retained under the ownership of the sub as their intellectual property.

Let’s say you contract with a company to provide something like a database. If you don’t like their database, you don’t get to take their source code and rewrite it to work the way you want it to work. You don’t get access to their underlying code when you contract for them to provide the database.

Why would they not use Tesla Grohmann Automation in the first place? Wasn’t this the reason for purchasing them?

My Model 3 delivery date slipped from October-December 2017 to November-January.

$1 that’s not the last slip.

IIRC they did inverters or drive train part

He wouldn’t ‘out’ who the company was in the conference call. Just took the blame himself for picking the wrong supplier and is now working to get it resolved.

Yes, he is a very humble person.

Unlike all the Other ARROGANT EX and Current Big Auto CEO’s , Musk Is indeed the Most Knowledgeable, Humble,& selfless who is Not full of Himself. BY FAR ! It’s Impossible To Not Like & admire Elon Musk ..No matter what any ICE Head Tesla Hater may Think!

(⌐■_■) Trollnonymous

Don’t forget he get’s paid minimum wage by law.

Which is worse? If the story is true or if the story is false? If it’s true then that means the battery supplier is patently incompetent even though it’s been doing it for years. If it’s false then well that’s not very comforting either. Dude….everyone has to write control software. Quit acting like this is something special.

(⌐■_■) Trollnonymous

2 of the 4 modules….

Why would they design each module different?

Two of the four manufacturing zones, not 2 of 4 battery modules.

(⌐■_■) Trollnonymous

“one of the major production constraints for the Model 3 was that software for two of the four Model 3 battery modules had to be recreated.”

Where did you read “Zones”?
Of course if you consider each module a ‘Zone’ then sure?

Or, I’m reading it wrong, he11 it’s Friday after all.

It was described that way in a previous story.

It certainly is confusing, the way that Elon is describing it. But as I understand it, the “modules” in question are not modules that are assembled to make the battery pack; the modules are sections of the battery pack assembly line.

The Packs contain 4 Modules: power, Coolant, etc, connect into first one and go through it to the 2nd one, it goes around out to the 3rd one, in a particular way, and enters into the 3rd one in a particular way, and then out the 4th one, to the final connection, as a story I saw on the pack design showed. Each aspect of the pact, likely is programmed or designed in a particular way, and each Module like this also. So – at Stage 1, Cells might be Grouped, and at stage 2, cells could be having Coolant Elements inserted, and then cells joined in Stage 3, Finally, in Stage 4, they might be connecting main feed wires! (All for the sake of some idea of how this might be separated into work stages, my own interpretation only!) If a Couple of those steps got messed up, in small things, like the robot missing the alignment, horizontally, or Vertically, Cell Damage could occur, scrapping that whole Module! Pressures on Contact could be elements of programming, Weld Duration might not be done right, Positional (X/Y & Z) errors could be different at different parts of the Stage Process, requiring… Read more »

A vender dropped the ball in relation original event,but Tesla is ultimately responsible for a lack of managerial oversight on both their venders and at the GF1. With proper oversight the problems wouldn’t have had time to compound themselves. Because Tesla was out of the loop it will now cost Tesla millions in lost revenue to fix it. Plus the actual additional cost of reprogramming the automated machines.

A reasonable assessment.

Yes, I think you have given a good summary of the situation. Thank you.

You seem to be vastly underestimating the capability of people to hide problems from their bosses/customers/family/etc.

Until the s*** hits the fan.

It always goes down to Tesla being at fault. Elon shouldn’t even had said the word supplier. Either the machine is in spec or out of spec. If it was out of spec Tesla shouldn’t have taking delivery of the machine. More than likely with Tesla the spec changed at the last minute (pencils down).

Either way you don’t determine a machine is out of spec after job one is down the line. They would have known about this problem for awhile. Also most manufactures give themselves enough time for issues to arise. Tesla’s timing was questioned from day one.

If you’re curious: that’d be 41,600 to 62,400 hours of coding… wow.

How long will it take BMW or Mercedes to just get these programmers to write that much code? And that is just for the battery packs fellows.

Actually – it seems like 2 of 4 operational Stages needed Re-Coding only! (Bottle Neck = a Squeezed in middle or Choke Point)

“The biggest challenge is that the first two zones of a four zone process, key elements of which were done by manufacturing systems suppliers, had to be taken over and significantly redesigned by Tesla.” – So – at least 50% of the Battery Module and Pack Building Process – was AWOL as to not being Right On The Money!

I doubt a traditional auto maker would have sufficient depth in its data processing / software design department to diagnose problems with assembly line machines/robots and write software specifically for them.

In general, I would expect legacy auto makers to outsource the engineering and/or writing of any software they need. I see from the article linked below that BMW has “acquired Nokia’s software team in Chicago”, but as far as I can see from the article, that team is just working on the infotainment system for BMW cars. I see no reason to believe that even BMW would try to develop its own software for assembly line machines; that would almost certainly be outsourced, or at worst performed by contractors hired specifically for the project.

A modern car has about 100 million lines of code. So not very long.

Also this isn’t software used in the car, but rather software used for a tool, or probably multiple tools, that build modules of cells. So basically Tesla took their Grohmann engineering guys and let them redo this thing.

This is in no way especially impressive and has been done a hundred times over for other parts of the production line.

You can ask them when/if they ever get to the point where they try to ramp up battery pack production for 5K to 10K cars a week.

Tesla is way far out in front of them on the volume they are trying to ramp up to. Being first has unique challenges.

Didn’t Elon also say that now they had to recalibrate the robots and would take some time?

That’s just another way of describing the same problem.

It is a poor workman who blames his tools.

It seems to me his response was more like ‘The Axe was Dull, we did not realize it, since we figured our Axe Supplier shipped us Sharp Axes, but when we did, we got our Best Axe Sharpeners on it, so as to get out cutting speed back up to Snuff!’

Good analogy.


He didn’t, he blamed himself for getting a poor toolmaker.

I wonder if the LG Model 3 would have had this problem…

All joking aside, this just shows that there’s a reason why GM doesn’t manufacture their own batteries and why Nissan stopped doing so. Not every decision they make is a dastardly attempt to undermine the future of EVs.

Fisker didn’t build their own Karma packs and that didn’t work out very well for them.

Fisker “choose poorly”… =)

The Model s and X have the best battery pack of any existing EV. And there is no reason to believe that LG can do anything better than Panasonic.

“…this just shows that there’s a reason why GM doesn’t manufacture their own batteries and why Nissan stopped doing so.” Wut? This has nothing to do with that subject. This is a matter of Tesla not exercising proper oversight of the company installing part of the production line for the Model 3. The fact that the particular part of the production line in question happens to be for producing the battery pack, and is physically located inside Gigafactory One rather than Tesla’s Fremont assembly plant, is rather irrelevant. The reason GM has not yet paid to build battery factories to supply its own EVs is because they aren’t yet planning to make EVs in large numbers. If and when GM finally decides to get serious about making PEVs in large numbers, then they will do what Tesla has done: They will partner with a battery maker (likely LG Chem, since they have an existing relationship with that company) to build large-scale battery factories so they can control their own battery supply. In the future, auto makers will compete on the basis of battery tech and battery pack tech, the way gasmobile makers currently compete on the basis of their ICEngines.… Read more »

I agreed with you until you said in the future manufactures will compete on battery production and cell technology. Batteries are and will be commodities No manufacture wants to be in that business. Even Tesla isn’t in that business. Panasonic is housed in the Gigafactory. Tesla assemblies packs from Panasonic cells.

GM went one step further and said why do I even care about the pack. There is really no IP in it. It’s just a bunch of cells gathered together. It’s the vehicle integration that’s important when hitting cost targets.

Actually there is a ton of GM’s IP in the battery pack. GM’s battery cooling system is state of the art, and purely their own intellectual property.

I doubt LG will be able to scale to the level of production GM wants to do. They’re planning two new all-electric models in the next 18 months, and 20 more in the next 5-7 years.

On the upside, Tesla’s recent “bottlenecks” should only delay my Model 3 by a month.

I think we should wait for a single example, ever, of EV sales actually outpacing LG’s ability to build batteries before we continue talk about how LG can’t scale fast enough.

When LG started selling its so-called “200 mile battery”, it was taking orders two years in advance. Why do you think they were taking orders that far in advance, if it wasn’t because they can’t possibly supply enough batteries for current demand at any time, and can’t possibly ramp up supply fast enough to meet demand? Let us please try exercising a bit of logic here, Spider-Dan. Furthermore, it is fairly obvious that Tesla was at several stages production constrained because it could not get Panasonic to ramp up battery cell production fast enough for Tesla’s needs. That’s why Tesla was eventually forced to spend billions of dollars building its own battery “Gigafactory”. Also, note that the highest volume EV maker is BYD, which also has its own battery factories. Spider-Dan, do you honestly think that is a coincidence? And do you think that LG Chem has some magical ability to ramp up production instantly? No, it takes LG about two years (or at least a year and a half) to build new factories and get them fine-tuned for mass production, just like everybody else. This idea that GM or LG Chem can just turn some sort of magic knob… Read more »

GM doesn’t have to stick with LG as it’s sole supplier. In fact, they use hitachi automotive batteries in their malibu’s and I think the sierra’s as they use a different battery design. They will contract with whoever will deliver.

Tesla on the other hand is pretty much stuck with panasonic, as the gigafactory is partly owned by panasonic, and they are leasing equipment from panasonic, and using panasonic battery chemistries.

I think you’re right on the time it takes LG to build supply. But you don’t account for the fact LG has been building out to meet their demand. Tesla does everything big and in your face. LG has built quite a list of customers where as Panasonic has basically Tesla.

LG knows everyone’s delivery schedule and order quantity. They will build out to meet that demand – that’s good business.

I will reiterate my previous statement: can you name a single example of any EV ever being production constrained by LG’s ability to build batteries?

You say that LG takes orders two years out. All existing evidence indicates that two years is plenty of time to accurately forecast the market need for EV batteries. The only executive who thinks that isn’t enough time is the same guy who thought he was going to build 100k-200k units of his new BEV in the first six months of its release.

To continue with this point: the only EV I can think of that has had battery-constrained sales is the Mitsubishi Outlander PHEV. The manufacturer of the Outlander PHEV’s batteries is Lithium Energy Japan… a venture partially owned by Mitsubishi.

So it seems to me that the only major automaker that has been battery constrained was in that position because instead of simply buying batteries from existing suppliers on the market, they decided to become their own supplier and couldn’t make them fast enough. Which seems relevant to this article.

On my phone so harder to read all the comments…

Question, is Fremont then building cars without batteries at a rate of 5000/week?

No. The 5000/week goal is for sometime in 2018.

Currently, I think Elon said they have tested some parts of the TM3 production line with burst runs at a rate of 500/week. Five hundred, not five thousand; and obviously those are just short test runs at that speed. It’s not that the Fremont factory is cranking out 500 TM3 gliders a week and then letting them pile up in some gigantic parking lot somewhere!

Why didn’t Tesla just build the damn cars without a battery and delivered them? I would have gladly received a car sans battery and bought a whole bunch of 12v lead acid batteries and used them till Gigafactory caught up.
Thanks Tesla!

1. They only discovered this problem when they pushed the GO button?

2. They put 240 programmers on the case to accomplish this? Because that’s how you accomplish 240 man-months of work in 1 month.

Another day, another thing to blame.

“To date, our primary production constraint has been in the battery module assembly line at Gigafactory 1, where cells are packaged into modules.”

“We had to rewrite all of the software, from scratch. We managed to write 20 to 30 man-years of software in 4 weeks.”

I’m not 100% sure how to interpret Musk’s statement (was the 20-30 man-years the amount of SW in the original robot control systems as delivered by a supplier? Was an estimate of how much would need to be written to correct the problems with the existing code? Rewrite it from scratch?)

However, under all interpretations, I’ve just lost a lot of the respect I had for the guy. If it was really 20-30 man years, they haven’t fixed or rewritten it in weeks — and if they tried to, they’ll continue having issues. Lots of them. If it wasn’t 20-30, either that’s a completely hyperbolic statement, or they had a completely incompetent supplier (which makes Tesla’s management also incompetent, in the sense of insufficient oversight when you’re betting the entire company on good execution of the Model 3).

I’m sure he’s heartbroken by your armchair interpretations and loss of respect. Hopefully he’ll get over it and continue on his way.


You appear to have a good grasp of the reality of the situation; a grasp which is sadly lacking by many posting comments here. Elon is putting a brave face on things, but — as you say — the problem as described isn’t one that’s behind them; it will be an ongoing problem going forward. And that is almost certainly why Tesla has scaled back its guidance for how fast it’s going to be able to ramp up TM3 production.

I hope the situation isn’t quite as dire as you describe it, but that glass is definitely at best 50% full, whether you describe it as half-empty or half-full.

PP, the glasse is just too big!

Should have filled the glass with wine instead of water. Where is Jesus when you need him?

What code? PLC Code? Which brand PLC? Or robot code?

I’d like more specifics.. but probably won’t get them.

Yeah, we have gotten a perfect blizzard of “explanations” from Tesla about the reason for this major delay, yet an amazing paucity of details. Unfortunately, that suggests to me that Tesla is too embarrassed about what’s actually going on to give us the nitty-gritty. 🙁

As someone who isn’t a “financial guy”, I found the latest investor conference call to be astonishingly content-free. Perhaps those who are “financial guys” will find it informative, but I got almost nothing from reading the transcript that had not already been excerpted in the InsideEVs news coverage by Domenick Yoney (link below). This is by far the least informative Tesla investors’ conference call for which I’ve ever read a transcript.

Heck, what Tesla is saying is so vague and content-free that we’re arguing over whether the software problem is in the modules of cells that go together to make up the battery pack, or the “modules” that are different sections of the battery pack assembly line!

I’ve just canceled my M3 reservation. I am going to get a SMART electric. At least they are not totally dependent on a battery.

With a fuzzy range-extender (only 1hp though).

It’s an old myth that a horse has only one horse power. A horse can top out at 15 horse powers.

hum… “20 to 30 man-year of software code in 4 weeks” means putting a team of 240 to 360 programmers to work for a one month project… I have doubts.

My thoughts exactly. Unless there were a high number of completely independent software tasks, this is hard to manage; especially if it was a last minute rush, as opposed to a long term project.

I can believe that there was something like a team of 6 who worked on the generic software for 5 years, deploying it in various manufacturing scenarios for different clients. Tesla would not have needed all the options, code paths, etc. that the original code had. They would have needed just a fraction of the functionality…but they would need that to run very fast and reliably. And if that original team couldn’t get it to ramp up to dreadnaught speeds, then getting a fresh team on it to purpose build the software is a likely outcome.

FYI, I know the TSLA cell/module/pack scenario over the years firsthand, and Do Not Forget if the firmware and software for Each stage AND the entire System is not Bulletproof, then you can have Imbalances and Runaway Race Conditions that If any cell is Damaged propagate like dominoes then You have a Disaster (Literally and PR) like Boeing 787 Dreamliner Fires and Samsung Note Explosions. So the Model 3 “guinea pigs” better watch out, Similar Matters happened with the initial model S and Garage Home chargers Etc.

Dear Dr. Musk, most of the world thanks you and congratulates your work as to EV development, production and sales. We cannot wait until the M3 is in full production and rolling along. Thank you and I wish you continued successes!

Unfortunately with the tax credit on the chopping block by the GOP combined with not putting out 20,000 M3s by the end of the year will have cost Tesla customers around $150,000,000.

I don’t believe a single word of his excuse.