The big question we asked Consumer Reports was: “How come?”
We have recently seen the latest results from the Consumer Reports (CR) Reliability Survey, and they made us scratch our heads. Among 30 brands, Tesla is in the 23rd position. It is in the bottom third, with a score of 39. Yet, the Model 3 and the Model S regained a Recommended rating. They represent ⅔ of Tesla’s current lineup – or around 67 percent of it. How is this Recommended rating possible with such a low overall score?
Gallery: Tesla Model S
You may ask what this has to do with Dennis Christopher Wang and his video above, but we will get to that in good time. First, you have to understand how CR performs its reliability survey.
It asks its members to answer an extensive online survey on multiple problems the cars may present in 17 trouble areas. These vehicles can be brand-new, but the oldest ones date back to the model year 2000.
In 2019, CR got answers on 420,000 vehicles, of which a little less than 4,000 cars were from Tesla.
Gallery: Tesla Model 3
Putting that in perspective, 0.95 percent of the total amount of cars is from the EV maker. When you realize CR evaluates 30 brands, each of them should represent at least 3.33… percent of the number of answers. So Tesla is clearly underrepresented in the survey.
CR explains how it compensates that in the FAQ of the reliability survey:
“How many samples do you have of each model?
A typical model has about 200 to 300 samples for each model year. When we have small sample sizes for models, we may use brand history and the reliability of similar models that may share major components in calculating our predictions.”
Gallery: Tesla Model X
Here comes the first issue: Tesla does not work with model years. Owners typically discuss problems with the VINs to know if their cars are recent or older.
Even if it did, there would be another problem, which CR reports:
“Some of Tesla’s inconsistent reliability may result from its unique approach to manufacturing, says CR’s Fisher (Jake Fisher, senior director of auto testing at CR). Though most other automakers tend to make improvements on the assembly line all at once, before a new model year begins, Tesla makes numerous running changes throughout the year. Tesla has changed parts, including major items such as motors and suspension components, only a few months after beginning production on a new model. It can take time for any manufacturer to work out reliability problems with new parts, and it’s even harder if they’re introduced frequently, Fisher says.”
As the own magazine puts it, it is a challenge to evaluate Tesla vehicles. They have no model year, the company has a “unique approach to manufacturing,” and it can perform over-the-air updates that change how the car performs. Even range can be altered, which already cost Tesla a lawsuit and an NHTSA defect investigation.
To make matters more difficult, Tesla is not a company that communicates freely with the press. Not even to CR, as you can see below:
“CR emailed multiple questions last week to a Tesla representative, seeking comment about the new CR recommendations and any changes Tesla might have made to improve reliability.
The company did not respond by press time.”
You may not remember this, but the British magazine What Car? had a similar problem. Related almost to the same cause.
What Car? tests EVs as if they are combustion-engined vehicles. It measures energy consumption for some miles and calculates the range as if EVs offered constant energy expenditure.
Well, they don’t: EVs have regenerative braking, which puts some energy back on the batteries. They also have hidden buffers that can make the car run for a lot more miles after reading on the dashboard you have nothing left. The only way to actually know their real ranges is to test them in a lab until they can no longer run.
In CR’s case, besides having a smaller sample than it should, it also evaluates all cars according to the same standards and aspects. These ones below, described by the own magazine:
“What do the trouble areas cover?
Our Reliability History charts cover problems in any of 17 trouble areas. Here's a look at what's covered in each of those areas, listed in order of mechanical and more serious problems first:
ENGINE MAJOR: Engine rebuild or replacement, cylinder head, head gasket, turbocharger or supercharger, timing chain or belt.
ENGINE MINOR: Accessory belts and pulleys, engine computer, engine mounts, engine knock or ping, fuel leaks, oil leaks.
ENGINE COOLING: Radiator, cooling fan, water pump, thermostat, antifreeze leaks, overheating.
TRANSMISSION MAJOR: Transmission rebuild or replacement, torque converter, premature clutch replacement.
TRANSMISSION MINOR: Gear selector and linkage, transmission computer, transmission sensor or solenoid, clutch adjustment, rough shifting, slipping transmission, leaks.
DRIVE SYSTEM: Driveshaft or axle, CV joint, differential, transfer case, four-wheel-drive/all-wheel-drive components, driveline vibration, electrical failure, traction control, electronic stability control (ESC).
FUEL SYSTEM/EMISSIONS: Sensors (O2 or oxygen sensor), emission-control devices (includes EGR), fuel-injection system, fuel cap, fuel gauge/sender, fuel pump.
ELECTRICAL SYSTEM: Alternator, starter, hybrid/electric battery replacement, hybrid/electric battery-related systems, regular battery, battery cables, engine harness, coil, ignition switch, electronic ignition, distributor or rotor failure, spark plugs and wires failure.
CLIMATE SYSTEM: A/C compressor, blower (fan) motor, condenser, evaporator, heater system, automatic climate system, electrical failure, refrigerant leakage.
SUSPENSION/STEERING: Shocks or struts, ball joints, tie rods, wheel bearings, alignment, steering linkage (includes rack and pinion), power steering (pumps and hoses, leaks), wheel balance, springs or torsion bars, bushings, electronic or air suspension.
BRAKES: Antilock system (ABS), parking brake, master cylinder, calipers, rotors, pulsation or vibration, squeaking, brake failure, premature wear.
EXHAUST: Muffler, pipes, catalytic converter, exhaust manifold, heat shields, leaks.
PAINT/TRIM: Paint (fading, chalking, peeling or cracking), loose interior or exterior trim or moldings, rust.
NOISES/LEAKS: Squeaks, rattles, wind noises, seals, and/or weather-stripping, air and water leaks.
BODY HARDWARE: Windows, locks and latches, doors or sliding doors, tailgate, trunk or hatch, mirrors, seat controls (movement and temperature), seat belts, sunroof, convertible top, glass defect.
POWER EQUIPMENT AND ACCESSORIES: Cruise control, clock, warning lights, body control module, keyless entry, wiper motor or washer, tire pressure monitor, interior or exterior lights, horn, gauges, 12V power plug, USB port, alarm or security system, remote engine start, heated or cooled seats.
IN-CAR ELECTRONICS: CD player, rear entertainment system (rear screen or DVD player), radio, speakers, in-dash GPS, display screen freezes or goes blank, phone pairing (e.g. Bluetooth), voice control commands, steering wheel controls, portable music device interface (e.g., iPod/MP3 player), backup or other camera/sensors, Android Auto/Apple CarPlay, infotainment hardware replacement and software fixes.”
As you can see, the survey focuses on ICE vehicles. After all, they are currently the majority of the market. We have searched for special procedures regarding EVs but found nothing on the Reliability Survey FAQ. If CR has a different approach towards EVs, that is not stated anywhere we could find. The fact is that it has to be different.
Electric cars have fewer parts to break down. Their main concern is the battery pack, which is their most expensive and most sensitive part. Electric motors are very robust. Most EVs do not need a transmission.
If the evaluation criteria are the same for ICE cars and EVs, the latter gets a considerable headstart. All the glitchy things on a Tesla are well below in CR's importance scale. If the EVs do not stop running – which they usually don’t – they may get the Recommended status in an easier way.
CR reinforces that a little below the list of evaluated issues:
“Are all problems considered equally serious?
Engine major, engine cooling, transmission major, and drive system problems are more likely to take a car out of service and to be more expensive to repair than the other problem areas. Consequently, we weight these areas more heavily in our calculations of model year Overall Reliability Verdict. Problems such as broken trim and in-car electronics have a much smaller weight. Problems in any area can be an expense and a bother, though, so we report them all in the Reliability History charts.”
Would that be the reason for the Model 3 and the Model S to have recovered their Recommended status this year? Check what the magazine credits the upgrade to below:
“For the Model 3, there were fewer reports of stuck latches or malfunctioning doors. For the Model S, owners reported fewer problems with paint and trim quality—issues that have plagued it in the past. Owners also reported fewer problems with power equipment, such as cruise control, cameras, and warning lights.”
The Model X, a third of Tesla’s current lineup, is among the ten least reliable cars in the US, according to CR. It has a score of 15. With that in mind, how does the company get its reliability rating? A sum of all its vehicle’s reliability scores divided by their number? Do models with higher production and sales numbers have more weight than niche vehicles?
We have asked this to CR and here is the official response:
“Sales are not factored into our brand reliability rankings and reliability scores in any way. Only the performance of the vehicles in each brand lineup (that we have sufficient data on). In Tesla's case, as you know, it's only three vehicles. Some other brands have over a dozen vehicles. The Model X has a reliability score of 15, among the worst of any car in our survey. So because it's only one of three Tesla vehicles, it does significantly drag down overall average for the brand.”
It is a pity the ratings of the Model 3 and the Model S are not publicly available, but that is part of CR's strategy to attract more subscribers. Only they have access to the scores. We’ll ask the magazine exactly which their grades were, but math will always come to our rescue.
CR did not inform how it calculates the brand reliability rating, but it seems it sums up all the scores obtained by cars in the lineup and divides that number by how many they are. If that is correct, the Model S and the Model 3 would have to have a number that – divided by the three models Tesla currently sells – allowed a score of 39.
Tesla’s rating times three is 117. The Model X scored 15. Take that number out of 117, and you'll get 102, or 51 for each of the other vehicles. When you consider the total amount of points is 100, both the Model S and the Model X would be slightly above half.
Is that enough for a car to have the Recommended rating? We have dug up recent CR texts and found this:
“To earn a CR recommendation, a car must have a high enough Overall Score within its individual category. The score is calculated using results from CR’s stringent testing, reliability, and owner satisfaction surveys, and safety tests.”
In other words, the Recommended rating does not depend solely on the reliability score, as some articles about the Reliability Survey may have led you to think. Dependability is just one of these status’ components, and the cars only need to “have average or better reliability,” according to CR.
We have no idea what that means. If it refers to a score above 50, that is like celebrating a C-. Especially when you see the ten most reliable vehicles had ratings above 87. The best one scored 97.
In the Teslas’ case, safety tests must have made the difference. They perform really well in these evaluations. It may also have gone well on owner satisfaction, but that is a tricky thing to consider because of its subjective nature.
More than one Tesla owner we interviewed already said that the company matters because of “the mission” it has. One of them literally told us this – off the record:
“That is the frustrating part. The cars are so compelling that you tolerate BS that – if it was from any other car manufacturer – you would tell it to go to hell a long time ago.”
There is no way to measure the devotion to a car brand on any customer satisfaction survey, but perhaps this is something psychologists should try to address to take passion out of the equation.
What about the 17 trouble areas?
After trying to explain Tesla’s low performance in reliability even with 67 percent of Recommended status, we were left with CR’s reliability assessment and the weight it gives each of the 17 trouble areas. With the publicly available info, it does not seem adequate to deal with EVs in general and with Tesla vehicles in particular.
We have to discover where battery pack problems are classified. In an electric car, it should be in an area equivalent to “Engine Major.” Any problem with it may lead the car to stop running. If it is damaged and needs to be replaced, that will demand many thousands of dollars to fix. If it catches fire, your car is lost in the best-case scenario. In the worst, you may see damages to much more than your ride.
A known serious issue with Tesla vehicles is the paint. That is where Wang’s story fits.
“The one thing that sparked all of this was a delivery checklist I found.”
It was not the one we published in October to help Tesla Model 3 buyers avoid the most common issues the car presents at delivery. Anyway, the checklist allowed Wang to find three paint defects on his Model 3 Performance, which Tesla took nine days to fix.
Sometime later, Wang discovered another issue that is proving to be recurrent in the Model 3: wrinkled paint on the front bumper corners. It is as if the paint did not adhere to the bumper and got wrinkled by rubbing on the body panels when it was put in place.
There is a thread on a Tesla Model 3 Facebook private group about this started by Joseph Hill. And many more owners said they had the same issue: Johnny La, Tommy Dutton, Jim Walther, Benito Nguyen, Michael Brown (who said Tesla Service Centers see this all the time), Brown’s wife, Thi Dinh, Christopher Le, Joerg Schuster, Tim Shinn, Edan Acosta, Sam Wickert, Stephen Yen, Ron Morales, and Alvin Lee. So far.
Wang discovered that issue in his car spontaneously.
“I was about to do PPF and was getting quotes to get it done.”
Gallery: Tesla's Performance On Consumer Reports Reliability Survey Made Us Wonder
PPF is the acronym for paint protection film. It is almost mandatory for all Model 3 drivers due to the fragile and thin paint these vehicles still have. Wang said he just thought about it because he uses the highway very often.
The paint in the bumper is just a nuisance since plastic does not rust. But the Model 3 has an issue at the left A-pillar that starts corrosion on that part. The upper extreme of the left front fender rubs the frame and extracts the paint from there, exposing the steel. Every single Model 3 we have had the chance to check in person has this issue, even in sales stands.
That is not the worst paint problem the Model 3 presents. Its rocker panels are peeling and chipping too quickly, something that is a severe concern in colder countries, such as Finland and Canada.
Tesla told most owners that asked for a warranty repair that this paint wear was related to using, hence not covered by warranty. It said that even to people that only drove on asphalt and for less than 12,000 km. Some are currently painting the whole car and suing Tesla to pay for their expenses.
Among the 17 trouble areas CR’s reliability survey analyzes, paint is in the 13th place in terms of importance. But if a car rusts prematurely, isn’t that a serious concern, even if it is not an immediate problem? Isn’t fixing rust and repainting the affected parts expensive and time-consuming enough for that to be in a higher position?
CR probably considered that not to be so relevant because paint issues are not so frequent as they used to be decades ago. Anyway, it involves more than seeing your vehicle rot. That may also compromise the structural integrity of the car.
Finally, in-car electronics are the last item of CR’s list. They have a much smaller weight in ICE automobiles, in which they do not play an important role. In-car electronics in a Tesla is basically the MCU, which handles critical elements in the EV, such as climate control or headlights.
In a “computer on wheels” – as Musk defines Tesla vehicles – it is a crucial part of the experience. And MCUv1 has a known problem that will make all of them fail. How should that be considered in terms of reliability?
CR is between a rock and a hard place here. We understand it has to submit all assessed vehicles to the same requirements and standards in order to have a fair evaluation of them. At the same time, it cannot use the same rule to measure very different machines – such as Tesla's – because that is also unfair to all automobiles involved.
Things in which Tesla stands out may not have a high weight in the reliability assessment. Criteria in which it goes badly can also be overlooked, as in the examples we gave. We have asked the magazine about all this and will write a new article when we have the answers.
Will CR do anything to correct these standards and their weight for electric vehicles? Will readers and members consider eventual changes adequate to Tesla and other cars? We would not like to be in CR’s shoes, but we sure hope it finds a way to treat the different differently, yet justly.