# Consumer Reports on Model 3 Owner Reliability



## LittleTess

I've had concerns about how Consumer Reports comes up with their Overall Reliability. Just sent the following to CR:
"You state in the April 2019 issue that the Owner Reliability is "calculated from the total number of problems reported by our members in the 17 trouble spots." For the Tesla Model 3, as an example, you have 14 much better than average, 2 better than average and one average measure but come up with a worse than average Owner Reliability. I hope that you realize that it is logically IMPOSSIBLE to come up with that Owner Reliability given the 17 measures. The only way is to factor in other biases that you overlook in your definition! Can you correct your definition please?"


----------



## Montblanc

Interesting! Please share their response.


----------



## JasonF

It will probably be something like they decided that the issues were so important and/or awful that they had to ignore the good responses. What that really translates to is "pressure from sponsors".


----------



## Enginerd

_Tesla Daily _podcast had a really good piece on this issue late last week. The host, Rob, actually scored a good long phone interview with the Consumer Reports guy who's in charge of the rating system. It's educational, and he has some reasonable explanations. Highly recommend.


----------



## John

Podcast interviewing CR's Jake Fisher explaining weighting system:
http://traffic.libsyn.com/techcastd...ctor_of_Auto_Testing_Jake_Fisher_02.27.19.mp3

This was discussed on a Reddit Ask Me Anything thread with Jake as well:

__
https://www.reddit.com/r/IAmA/comments/av1fzz


----------



## Midnit3

I listened and he sounds unsure himself... was that just me?


----------



## John

Midnit3 said:


> I listened and he sounds unsure himself... was that just me?


I like that he was friendly and was trying to be open, but the bottom line is that the way they present results is confusing. He pretty much admitted that, but I don't think they are rushing to change anything. They care about feedback, but only perhaps superficially. As CR would say if they were reviewing themselves: "We found these issues in the current rating system, so we have to mark them down for this. But if they fix this it should show up in future reviews and we'll give them a better rating on this then."

My opinion:

1. They should not mix quality and reliability together. A paint blemish won't strand you on the side of the road.

2. They should add rows called "Reliability issues" and "Quality issues" and mark them red when necessary, then explain that all of the other categories of that type can be green, but if they all have minor issues they can add up to a poor rating in that category. In other words, there can be only a few incidents in each category-meaning you'd be unlikely to have a particular issue-but there can be so many issues like that spread across similar categories that you'd be too likely to have at least ONE of those issues in any car.

3. He said they don't share weightings because they don't want the OEMs to game the ratings, but that's definitely not a valid argument. The weightings are there to reflect what's important, and the OEMs *SHOULD be encouraged to game them* (and thus place priorities as CR wants them to).


----------



## ChristianZ

I was at Barnes & Noble today, so I checked out the magazine. There were mostly positive statements in the narrative, with nothing glaring about why it wouldn't be recommended. The handling is "stiff" and something about challenges to charge the car. I am shocked it's not recommended, especially since it's the safest car to drive.


----------



## Dr. Prunesquallor

John said:


> Snip
> 
> 3. He said they don't share weightings because they don't want the OEMs to game the ratings, but that's definitely not a valid argument. The weightings are there to reflect what's important, and the OEMs *SHOULD be encouraged to game them* (and thus place priorities as CR wants them to).


This is known as Kepner-Tragoe analysis and it is infamous for its susceptibility to "gaming". It is extremely easy to decide what you want the answer to be and subtlety tweak the weightings to get it. It can be a lazy man's way to try to quantify qualitative inputs.


----------



## M3OC Rules

I think there is a little too much incentive for them to push the extremes on high profile products. Not recommending a Samsung washing machine is not going to get headlines but an iPhone or Tesla does. Then later they will change the recommendations and get headlines again. They do some good testing and have a lot of data. Props to them for finding the braking issue. It's worth getting their opinion/information but I also regret the vacuum cleaner and mattress purchases I made on their recommendation. And I can't imagine liking another car more than this Model 3.


----------



## Montblanc

I also wasn't impressed by the answers provided by Consumer Reports. Their rating system is confusing and not based on statistical principles, as he freely admitted. I don't have a problem with looking at the issue rates of a particular car compared to the average issue rates but the fact that the scores are not revealed and the methodology is not specified is very weak. I think a better approach would be to separate the quality and reliability indicators and score each one on a modified quartile (or quintile) basis that, by definition, would be based on the spread of the data points of the sample. Average score, median score and standard deviation measures would allow us to really understand how a specific car scores relative to others. Come on Consumer Reports, you can do so much better than what you are doing now.


----------



## ADK46

The article admits that customer satisfaction is extremely high. _They are telling people not to buy a car they will probably be delighted with. _This is a very weird thing about CR, and always has been. I don't recommend that anyone pay much attention to their "recommended" advice.

On the other hand, we do want Tesla to pay attention to it, and it seems they are.


----------



## John

I wonder if CR are seeing the same effect as Munro & Assoc (teardown, initially dismissive of Model 3 quality): at least two Munro guys bought Model 3s.


----------



## M3OC Rules

ADK46 said:


> The article admits that customer satisfaction is extremely high. _They are telling people not to buy a car they will probably be delighted with. _This is a very weird thing about CR, and always has been. I don't recommend that anyone pay much attention to their "recommended" advice.
> 
> On the other hand, we do want Tesla to pay attention to it, and it seems they are.


I agree. After listening to the reasoning they do make it sound like they are following their methodology. But the methodology ends up not recommending products that people will be delighted to have. Reliability isn't everything. My previous cars have been Japanese largely because of their reliability. But there is no way I'm going to recommend a Camry over a Model 3 because the Model 3 is twice as likely to have some issue requiring service. He says "they are just laying out the data." That's fine but it devalues their recommendations. Good info but bad recommendations. The final question about the unknowns if ICE data on reliability translates to EV's was a good one. There is a risk buying new technology but there is a risk buying old technology that only gets worse over time as well.


----------



## LittleTess

Good point about reliability versus quality. I've sent CR emails stating that, to me, reliability is the assuredness that the car will get you from point A to point B. Paint blemishes have nothing to do with that!


----------



## LittleTess

Mr. Fisher appears to admit that EVs have a much simpler drive train and that it should have less long term problems than an ICE vehicle. But he also states that the overall reliability factors in the long term prognosis (which they don't state in their definition for the overall reliability). I would think that the long term would push the reliability up. As stated before, almost all body issues except for the cracked glass don't have long term impacts - a bad body gap isn't going to get worse. His interview just left me more confused than I was before!

For those who asked, I've heard back from CR on prior queries but they always provide short, nebulous answers that don't really answer my questions. The Jake Fisher interview in the link above comes the closest to attempting to answer my concerns.


----------

