FW rollout process

  • Our merch store is back! Check out our line of quality apparel and accessories featuring the TOO logo. Let us know if you'd like something specific you don't see 👍https://teespring.com/stores/tesla-owners-online-store
  • It's OK to discuss software issues here but please report bugs to Tesla directly at servicehelpna@teslamotors.com if you want things fixed.

barjohn

Well-Known Member
Joined
Aug 31, 2017
Messages
215
Location
Riverside, CA
Country
Country
#1
I haven't received 12 or 12.1 yet and I enjoy getting the updates but it still bugs me that they frequently fail to fix existing bugs (they don't tell us what is supposed to have been fixed) and they seem to introduce new bugs we didn't have before. With all of the early access folks how does this happen? Don't they provide feedback and let Tesla know about these bugs? If so, why doesn't Tesla listen and fix them?
 

theblindtree

Active Member
Joined
Apr 30, 2018
Messages
41
Location
Knoxville Tennessee
Tesla Owner
Model 3
Country
Country
#2
I haven't received 12 or 12.1 yet and I enjoy getting the updates but it still bugs me that they frequently fail to fix existing bugs (they don't tell us what is supposed to have been fixed) and they seem to introduce new bugs we didn't have before. With all of the early access folks how does this happen? Don't they provide feedback and let Tesla know about these bugs? If so, why doesn't Tesla listen and fix them?
That's par for the course when it comes to software development, especially if you're constantly introducing new features. Knowing is only half the battle.

It's like I told a friend of mine that was interested in buying a Tesla: understand that you're purchasing a work-in-progress, and that is only a good thing depending upon your outlook. :)
 
Joined
Sep 25, 2017
Messages
23
Location
Crystal Lake
Country
Country
#3
The software issues we see seem like issues that are not 1 in a million corner cases. At some point it's no longer being fast to market and approaches incompetence. Who is in charge of QA? Or have they decided to have no quality control and simply let customers figure it out.
 

DocScott

Well-Known Member
TOO Supporting Member
Joined
Mar 6, 2019
Messages
269
Location
Westchester, NY
Tesla Owner
Model 3
Country
Country
#4
I haven't received 12 or 12.1 yet and I enjoy getting the updates but it still bugs me that they frequently fail to fix existing bugs (they don't tell us what is supposed to have been fixed) and they seem to introduce new bugs we didn't have before. With all of the early access folks how does this happen? Don't they provide feedback and let Tesla know about these bugs? If so, why doesn't Tesla listen and fix them?
I also think there are three kinds of problems that people are referring to as "bugs":

1. Problems with the firmware code. This is what people are thinking of when they call something a bug.

2. Problems with installing a new firmware update. There have been a number of reports of partial installs causing problems. This looks like a bug, because it appears with the new firmware and may go away with a subsequent firmware update. But it's not actually a bug, because there's nothing wrong with the code in the firmware itself. If users were able to reinstall the same version of the firmware, it might fix it if the install went right the second time, but that's not an option owners have.

3. Problems that have nothing to do with firmware installation. For example, maybe a sensor gets finicky. It may coincidentally happen around the time of a new firmware update, and the owner might think it was a firmware bug. But it won't ever go away with subsequent updates, because it's actually a hardware problem.

I think a lot of people tend to assume when there's a problem, it's of type 1. But sometimes it's actually type 2 or 3.

I agree that more detailed change logs for firmware updates would be very helpful, and would help clarify which kind of problem a given car was having.
 

Ed Woodrick

Top-Contributor
Joined
May 25, 2018
Messages
649
Location
Atlanta, GA
Tesla Owner
Model 3
Country
Country
#5
The software issues we see seem like issues that are not 1 in a million corner cases. At some point it's no longer being fast to market and approaches incompetence. Who is in charge of QA? Or have they decided to have no quality control and simply let customers figure it out.
Read what you agreed to, it's beta software and we'll known to be incomplete
 
Joined
Aug 31, 2018
Messages
12
Location
Virginia
Tesla Owner
Model 3
Country
Country
#6
Honest question: Is the entire display “beta”? Would think reported problems of screen blanking out on 8.5 wouldn’t exist in 12.1? 4 weeks of development/debugging for a pretty integral part of the car. Can’t speak for OP, but I think that’s an example of “how did this get out in the wild?”, or at least a “How is this not fixed yet?”
 

DocScott

Well-Known Member
TOO Supporting Member
Joined
Mar 6, 2019
Messages
269
Location
Westchester, NY
Tesla Owner
Model 3
Country
Country
#7
Honest question: Is the entire display “beta”? Would think reported problems of screen blanking out on 8.5 wouldn’t exist in 12.1? 4 weeks of development/debugging for a pretty integral part of the car. Can’t speak for OP, but I think that’s an example of “how did this get out in the wild?”, or at least a “How is this not fixed yet?”
I did a Google search for Tesla screen blackout. I find descriptions of that kind of problem from every year since 2013. Here is just one thread on that kind of issue.

So why do we think this is an "8.5" issue rather than, essentially, a Tesla issue? And here are reports from people running 2018.42.2. This has sometimes been a problem with Teslas. When it happens to someone, it often happens to the same person repeatedly.

So it's not necessarily a problem with insufficient testing of 8.5. It just seems to be a more fundamental problem with Teslas that eventually crops up for some people.

And yes, that's on Tesla. They should track down what causes that problem and fix it. Maybe they'll figure out that it can be fixed via a firmware update at some point. But I don't think it should be thought of as a recent problem that has started because they just introduced some new bugs in recent firmware.
 
Joined
Aug 31, 2018
Messages
12
Location
Virginia
Tesla Owner
Model 3
Country
Country
#8
So it's not necessarily a problem with insufficient testing of 8.5. It just seems to be a more fundamental problem with Teslas that eventually crops up for some people.

And yes, that's on Tesla. They should track down what causes that problem and fix it. Maybe they'll figure out that it can be fixed via a firmware update at some point. But I don't think it should be thought of as a recent problem that has started because they just introduced some new bugs in recent firmware.
Understood. I too experienced it, albeit rarely, on earlier builds, so I’m not saying that’s where it started. However, it seemed to be much more evident on 8.5 builds (check the 8.5 thread). Literally happened to me more in 8.5 than had occurred in all builds combined since taking delivery of my Model 3 in August. I don’t believe the primary display going out during driving is an acceptable bug, flaw, feature, whatever you want to call it. The fact similar issues have existed for so long, and seem to get worse/better with various software builds, makes me believe a root cause could be identified. I’d gladly trade Tetris for a stable display.
 

MelindaV

☰ > 3
Moderator
Joined
Apr 2, 2016
Messages
9,561
Location
Vancouver, WA
Tesla Owner
Model 3
Country
Country
#9
I did a Google search for Tesla screen blackout. I find descriptions of that kind of problem from every year since 2013. Here is just one thread on that kind of issue.

So why do we think this is an "8.5" issue rather than, essentially, a Tesla issue? And here are reports from people running 2018.42.2. This has sometimes been a problem with Teslas. When it happens to someone, it often happens to the same person repeatedly.

So it's not necessarily a problem with insufficient testing of 8.5. It just seems to be a more fundamental problem with Teslas that eventually crops up for some people.

And yes, that's on Tesla. They should track down what causes that problem and fix it. Maybe they'll figure out that it can be fixed via a firmware update at some point. But I don't think it should be thought of as a recent problem that has started because they just introduced some new bugs in recent firmware.
remember, people will not typically post publicly when everything is working as expected.
exception being the following quote ;)
My Tesla software worked flawlessly during my commute this morning, even with a light rain.
It unlocked with no delay when I operated the door handle to open it.
NOA performed correctly, and even suggested that I get out of the passing lane at one point.
No rebooting was required at any point, before or after.
The charge port door opened as expected with just a slight push on the charge port door.
And the car continues to charge at the full rate that the ChargePoint station is capable of providing.

I'll be sure to post this message after every single commute during which I don't have any problems with the software. :expressionless:
 

epmenard

Active Member
Joined
Mar 5, 2019
Messages
41
Location
Rosemère
Tesla Owner
Model 3
Country
Country
#11
I'm still having a hard time understanding why some software releases don't make it to everyone. I'm still on 2019.8.4 or 10 releases behind the latest official release. Almost 4% of teslaFi registered users appear to be stuck on even older firmware versions than mine.

I understand some of the minor updates but why would my car have skipped 2019.8.5, 2019.8.6 and could skip the latest 2019.12.1?

I am also aware of Tesla's Early Access Program, but that would surely not represent the bulk of users and would also not be able to justify so many version skips. Can anyone shed a light on the seemingly random deployment schedule?

1556548352496-png.25361
 

GDN

Moderator
Moderator
TOO Supporting Member
Joined
Oct 30, 2017
Messages
3,515
Location
Far North Dallas, TX
Tesla Owner
Model 3
Country
Country
#12
I'm still having a hard time understanding why some software releases don't make it to everyone. I'm still on 2019.8.4 or 10 releases behind the latest official release. Almost 4% of teslaFi registered users appear to be stuck on even older firmware versions than mine.

I understand some of the minor updates but why would my car have skipped 2019.8.5, 2019.8.6 and could skip the latest 2019.12.1?

I am also aware of Tesla's Early Access Program, but that would surely not represent the bulk of users and would also not be able to justify so many version skips. Can anyone shed a light on the seemingly random deployment schedule?

View attachment 25361
It just simply isn't meant to be. While it isn't officially part of the early access program, I believe they are likely to collect different measurements and feedback from different cars on different firmware. They don't want all of us on a single release. I figure it could easily be another 24 months before we see every car start to get the same firmware. We will all be disappointed at that time too because it likely means they've implemented all of the features they want to implement and things are very stable.

Each of us will leap frog others depending on options selected on the car, feedback they need and features being updated in the software. Every few months they do let it stabilize a bit and get most caught up and then we will see another round of roll outs.

We could be even more spread out now perhaps because of FSD HW. It seems those cars are running different SW than the rest of us. It could be a while before they again have a single version that supports all HW configs.

All I can say is don't get too hung up on it. I did at one time, but it just isn't meant to be. Only when you find yourself perhaps down in the very last percentage running a release and everyone else has moved on then you might have a problem or concern, but those cases are few and far between.
 

DocScott

Well-Known Member
TOO Supporting Member
Joined
Mar 6, 2019
Messages
269
Location
Westchester, NY
Tesla Owner
Model 3
Country
Country
#13
I'm still having a hard time understanding why some software releases don't make it to everyone. I'm still on 2019.8.4 or 10 releases behind the latest official release. Almost 4% of teslaFi registered users appear to be stuck on even older firmware versions than mine.

I understand some of the minor updates but why would my car have skipped 2019.8.5, 2019.8.6 and could skip the latest 2019.12.1?

I am also aware of Tesla's Early Access Program, but that would surely not represent the bulk of users and would also not be able to justify so many version skips. Can anyone shed a light on the seemingly random deployment schedule?

View attachment 25361
Do you have EAP or FSD?

My understanding is the big difference between 8.4 and 8.5 is no-confirmation lane change. If you're not either EAP or FSD, then that's not a feature you'd get anyway.

I'm still on 8.3, but that seems fine to me, because I only have vanilla AP. So I don't need the no-confirmation lane change from 8.5. 9 was an early-access version, and 12.x is still in limited roll-out. So I don't feel behind at all.
 

Ed Woodrick

Top-Contributor
Joined
May 25, 2018
Messages
649
Location
Atlanta, GA
Tesla Owner
Model 3
Country
Country
#14
Honest question: Is the entire display “beta”? Would think reported problems of screen blanking out on 8.5 wouldn’t exist in 12.1? 4 weeks of development/debugging for a pretty integral part of the car. Can’t speak for OP, but I think that’s an example of “how did this get out in the wild?”, or at least a “How is this not fixed yet?”
No, but I wasn't talking about it. But yes, there are portions of the screen code that are beta. As @DocScott mentioned, these have been occurring for a long time. There are some releases that have never reset for me, some releases that reset more. I've had a couple in 2019.8.5, but before that it was months. I had one release last year that was really bad last year. The main thing that you learn is that it is just a reboot, life goes on, the car goes on. Go find a safe road and turn autopilot on and let it drive. Then just reboot the display (hold both buttons in) and see what happens. Sure, it's not the best thing, but at the worse, you just drive the car.
 

Ed Woodrick

Top-Contributor
Joined
May 25, 2018
Messages
649
Location
Atlanta, GA
Tesla Owner
Model 3
Country
Country
#15
I'm still having a hard time understanding why some software releases don't make it to everyone. I'm still on 2019.8.4 or 10 releases behind the latest official release. Almost 4% of teslaFi registered users appear to be stuck on even older firmware versions than mine.

I understand some of the minor updates but why would my car have skipped 2019.8.5, 2019.8.6 and could skip the latest 2019.12.1?
So, the first reason would be, are all the cars identical? We've got 2 platforms (S/X and 3) which easily suggest possible difference and then within, various variants.

  • Next, let's say that you start rolling out software as any software producer does. As it hits the public, you start hearing or seeing issues. Do you let this continue or do you stop the rollout. That's what happened with the Enhanced Summons.

  • Now, let's say that you've got folks in northern climates that are having cold battery issues. You've got a fix that should go out pretty quickly. Do you start with Southern California as is their common practice or do you just start up North?

  • Now that the cold battery fix is rolling out, here comes another update for other features, do you complete rolling out the cold update or do you just start with the new update?

  • Oh, and the new update is a main development branch and the cold battery was a branch. So now the cold battery is rolled back into the main development branch and it now needs to be redeployed.
This is a pretty common development cycle and you are dealing with a product that is under development. I hope that you didn't buy the Tesla expecting it to never change, as if you did, you bought the wrong car.

It is kinda funny, we went through the first of the year with no updates, people complained. Now we are back into an active update cycle, people complaing.

Just stop reading the forums and none of these issues will occur.
 

MelindaV

☰ > 3
Moderator
Joined
Apr 2, 2016
Messages
9,561
Location
Vancouver, WA
Tesla Owner
Model 3
Country
Country
#16
@Ed Woodrick - nicely summarized. and beyond just the 3 vs S/X and regional variants... even within just the 3, where there are few models, there are still tons of variants within those models. Each time a part is redesigned, it potentially has a different impact on an unrelated part. IE a steering wheel/column is replaced with one 9 months newer and it interacts with other systems the slightest bit differently than the earlier part. multiply that by the number of parts in the car with a chip in it and you end up with more variants that could impact the FW than any casual observer could imagine.
 

epmenard

Active Member
Joined
Mar 5, 2019
Messages
41
Location
Rosemère
Tesla Owner
Model 3
Country
Country
#17
So, the first reason would be, are all the cars identical? We've got 2 platforms (S/X and 3) which easily suggest possible difference and then within, various variants.

  • Next, let's say that you start rolling out software as any software producer does. As it hits the public, you start hearing or seeing issues. Do you let this continue or do you stop the rollout. That's what happened with the Enhanced Summons.

  • Now, let's say that you've got folks in northern climates that are having cold battery issues. You've got a fix that should go out pretty quickly. Do you start with Southern California as is their common practice or do you just start up North?

  • Now that the cold battery fix is rolling out, here comes another update for other features, do you complete rolling out the cold update or do you just start with the new update?

  • Oh, and the new update is a main development branch and the cold battery was a branch. So now the cold battery is rolled back into the main development branch and it now needs to be redeployed.
This is a pretty common development cycle and you are dealing with a product that is under development. I hope that you didn't buy the Tesla expecting it to never change, as if you did, you bought the wrong car.

It is kinda funny, we went through the first of the year with no updates, people complained. Now we are back into an active update cycle, people complaing.

Just stop reading the forums and none of these issues will occur.
I'm not complaining about not getting every single release, I am simply interested in understanding how the deployment is organised. Thank you all for the great insight.

TOO is a great community and your comments are much appreciated!
 

Ed Woodrick

Top-Contributor
Joined
May 25, 2018
Messages
649
Location
Atlanta, GA
Tesla Owner
Model 3
Country
Country
#18
I'm not complaining about not getting every single release, I am simply interested in understanding how the deployment is organised. Thank you all for the great insight.

TOO is a great community and your comments are much appreciated!
No problems, oh, I forgot one, country specific localizations! Different laws, steering wheels in wrong places, Klingon languages
 

barjohn

Well-Known Member
Joined
Aug 31, 2017
Messages
215
Location
Riverside, CA
Country
Country
#19
I just tweeted Elon a suggestion that they reduce the number of versions of software they publish and support. He wants to reduce cost, this is an area he could significantly cut costs. I have managed large software development projects for the Navy and I kept the number of active versions we were supporting to no more than 3 and preferably just 2. When you have 6-10 versions it takes a lot of resources to just to track which versions have which bugs and as we see in the software releases, things that worked in one release are broken in the next. With that number of teams it is easy to have one fix broken by a different team fixing a different bug. The communications become a nightmare. They could narrow their focus, get bugs fixed more quickly and get new features rolled out faster. A win for everyone. They could put the extra resources it would free up to more thoroughly testing code for both bug fixes and whether they have introduced any new ones.
 
Joined
Apr 18, 2019
Messages
17
Location
McDonough, GA
Tesla Owner
Model 3
Country
Country
#20
I just tweeted Elon a suggestion that they reduce the number of versions of software they publish and support. He wants to reduce cost, this is an area he could significantly cut costs. I have managed large software development projects for the Navy and I kept the number of active versions we were supporting to no more than 3 and preferably just 2. When you have 6-10 versions it takes a lot of resources to just to track which versions have which bugs and as we see in the software releases, things that worked in one release are broken in the next. With that number of teams it is easy to have one fix broken by a different team fixing a different bug. The communications become a nightmare. They could narrow their focus, get bugs fixed more quickly and get new features rolled out faster. A win for everyone. They could put the extra resources it would free up to more thoroughly testing code for both bug fixes and whether they have introduced any new ones.
It's probably going to get worse before it gets better with the introduction of FSD. Since the architecture is brand-new (ie, not nVidia), they are probably going to be stuck in multi-version hades for the foreseeable future. I'd imagine they won't clear the hurdle until they label non-FSD "feature complete" and are able to focus all new development on *just* the FSD hardware.