Sentry Mode: Empty Files, Corrupt Video

  • SUPPORT THE SITE AND ENJOY A PREMIUM EXPERIENCE!
    Welcome to Tesla Owners Online, four years young! For a low subscription fee, you will receive access to an ad-free version of TOO. We now offer yearly memberships! You can subscribe via this direct link:
    https://teslaownersonline.com/account/upgrades

    SUBSCRIBE TO OUR YOUTUBE CHANNEL!
    Did you know we have a YouTube channel that's all about Tesla? Lots of Tesla information, fun, vlogs, product reviews, and a weekly Tesla Owners Online Podcast as well!

  • It's OK to discuss software issues here but please report bugs to Tesla directly at servicehelpna@teslamotors.com if you want things fixed.

simpsonhomer

Active member
Joined
Aug 28, 2018
Messages
27
Location
Washington, DC
Tesla Owner
Model 3
Country
Country
#1
Roughly 1/6 of my Sentry Mode recordings are blank (0-byte) files. Each video stream is only 4 Mbps (total 1.2 MB/s) so it's certainly not a speed issue (any flash drive on the market will easily handle many times that). Plus, the recent clips (from the past hour) are all okay.

Many of the non-empty files have corrupt video (bottom half is a giant green blob and the rest is extremely pixelated).

The drives were freshly formatted (GPT partition) and a new TeslaCam folder was created before being re-inserted. These problems occur on multiple different flash drives.

Is anyone else experiencing this? I'm on 2019.8.2 but I saw this on 2019.5.15, too.
 

Bokonon

Self-identified Teslaholic
Moderator
TOO Supporting Member
Joined
Apr 12, 2017
Messages
3,111
Location
Boston
Tesla Owner
Model 3
Country
Country
#3
Roughly 1/6 of my Sentry Mode recordings are blank (0-byte) files. Each video stream is only 4 Mbps (total 1.2 MB/s) so it's certainly not a speed issue (any flash drive on the market will easily handle many times that). Plus, the recent clips (from the past hour) are all okay.
I'm seeing something similar. After the first few minutes of Sentry recording, about 1/6 - 1/8 of the left-repeater camera files are zero bytes. And I've seen video clips that are missing keyframes so they end up discolored or pixelated.

I suspect sustained (rather than burst) write-speed is the issue here... after a while, the flash drive gets too hot to handle the I/O. I'm considering either a UHS SD card or some kind of SSD setup.
 

MelindaV

☰ > 3
Moderator
Joined
Apr 2, 2016
Messages
9,852
Location
Vancouver, WA
Tesla Owner
Model 3
Country
Country
#5
yesterday about half of my passenger side repeater camera videos had similar distortion as this one (on 2019.8.2). I've emailed it to Tesla to ask if this is a MCU processing issue, camera hardware, USB recording, etc issue. Videos start out mostly ok, then get a little wavy, then totally garbled before returning to mostly ok.
 

JWardell

Legendary Member
Joined
May 9, 2016
Messages
3,806
Location
Boston
Tesla Owner
Model 3
Country
Country
#7
yesterday about half of my passenger side repeater camera videos had similar distortion as this one (on 2019.8.2). I've emailed it to Tesla to ask if this is a MCU processing issue, camera hardware, USB recording, etc issue. Videos start out mostly ok, then get a little wavy, then totally garbled before returning to mostly ok.
It's just bad/missing blocks in the file saved to the stick. Nothing to worry over with the car.
I bet eventually Tesla will combine these in the computer and write one clean video that will work better with most usb sticks. Question is if HW3 will be required to do so.
 

garsh

Dis Member
Moderator
TOO Supporting Member
Joined
Apr 4, 2016
Messages
11,505
Location
Pittsburgh PA
Tesla Owner
Model 3
Country
Country
#8
Roughly 1/6 of my Sentry Mode recordings are blank (0-byte) files. Each video stream is only 4 Mbps (total 1.2 MB/s) so it's certainly not a speed issue (any flash drive on the market will easily handle many times that).
I could easily imagine that Tesla's early coding attempts try to create a new file, and "time out" too quickly, then give up on writing to that file. In the meantime, the file creation actually finishes shortly after the code gives up waiting on it, leaving you with a nice, fresh 0-length file.

As you say, almost any USB drive nowadays can handle the requested throughput. But 0-length files probably aren't a throughput issue. But it could still be speed-related if the USB drive isn't very quick at "seeking" for some reason.

The "missing file blocks" that @MelindaV and @SoFlaModel3 are seeing sound like a possible throughput issue.

In either case, the quick & easy thing to try is a different USB drive. So far I appear to be 2-for-3. The first one I tried actually appeared to fry from overheating and I had to pitch it. The other two actually seem to be holding up well, even though they're probably at least 5 years old.
 

SoFlaModel3

@Teslatunity
Moderator
TOO Supporting Member
Joined
Apr 15, 2017
Messages
9,855
Location
Florida
Tesla Owner
Model 3
Country
Country
#9
I could easily imagine that Tesla's early coding attempts try to create a new file, and "time out" too quickly, then give up on writing to that file. In the meantime, the file creation actually finishes shortly after the code gives up waiting on it, leaving you with a nice, fresh 0-length file.

As you say, almost any USB drive nowadays can handle the requested throughput. But 0-length files probably aren't a throughput issue. But it could still be speed-related if the USB drive isn't very quick at "seeking" for some reason.

The "missing file blocks" that @MelindaV and @SoFlaModel3 are seeing sound like a possible throughput issue.

In either case, the quick & easy thing to try is a different USB drive. So far I appear to be 2-for-3. The first one I tried actually appeared to fry from overheating and I had to pitch it. The other two actually seem to be holding up well, even though they're probably at least 5 years old.
I actually have all of the files (I should have elaborated). What happens in my case is that a large portion of the clip will be covered by solid green and/or complete pixelation.
 

MelindaV

☰ > 3
Moderator
Joined
Apr 2, 2016
Messages
9,852
Location
Vancouver, WA
Tesla Owner
Model 3
Country
Country
#10
I could easily imagine that Tesla's early coding attempts try to create a new file, and "time out" too quickly, then give up on writing to that file. In the meantime, the file creation actually finishes shortly after the code gives up waiting on it, leaving you with a nice, fresh 0-length file.

As you say, almost any USB drive nowadays can handle the requested throughput. But 0-length files probably aren't a throughput issue. But it could still be speed-related if the USB drive isn't very quick at "seeking" for some reason.

The "missing file blocks" that @MelindaV and @SoFlaModel3 are seeing sound like a possible throughput issue.

In either case, the quick & easy thing to try is a different USB drive. So far I appear to be 2-for-3. The first one I tried actually appeared to fry from overheating and I had to pitch it. The other two actually seem to be holding up well, even though they're probably at least 5 years old.
Mine is a USB3 scandisk ultra. scandisk gives read speeds (150MB/s), but lists write speeds as “varies by capacity”. Since the car’s ports are limited to USB2, the drive should be able to handle as much as the port and then some, in theory, right?
It also lists operation temps with a max of 118F. The only days I’ve noticed it like this has been the last few that were in the 70s, inside the car around 85F, so still well under the ambient temp but maybe temp was an issue. I’ve also not seen it do this when recording drives, just sentry.
The other high speed usb drive I have (can’t remember what model it is), I can tell from how it reacts on my computer isn’t as good as the scandisk ultra, but will try it today and see if it’s the same or different.
(The scandisk is the one I’ve been using since the beginning, have never had a grey x, hasn’t had any 0kb files since the very first TeslaCam release, so it has been pretty solid. Worked without issues for the 1 ½ day airport test on 15.5].
 

garsh

Dis Member
Moderator
TOO Supporting Member
Joined
Apr 4, 2016
Messages
11,505
Location
Pittsburgh PA
Tesla Owner
Model 3
Country
Country
#11
I actually have all of the files (I should have elaborated). What happens in my case is that a large portion of the clip will be covered by solid green and/or complete pixelation.
Right. Those are caused by missing MPEG frames.. Most frames of an MPEG file contain only delta (change) information from the previous frame. If the file is missing a frame or two, then your video remains corrupted until the next I-Frame is recorded (those contain complete frame information instead of a delta).
 
Last edited:

garsh

Dis Member
Moderator
TOO Supporting Member
Joined
Apr 4, 2016
Messages
11,505
Location
Pittsburgh PA
Tesla Owner
Model 3
Country
Country
#12
Mine is a USB3 scandisk ultra. scandisk gives read speeds (150MB/s), but lists write speeds as “varies by capacity”.
And at the other extreme, all of my USB drives are crappy cheap ones that I bought 5-10 years ago, and they're either working wonderfully or dying from overheating.

I bought the Kingston seven years ago, and the Toshiba five years ago, both from Best Buy. They're both working just fine so far:
 
Last edited:

JasonF

Top-Contributor
Joined
Oct 26, 2018
Messages
964
Location
Orlando FL
Tesla Owner
Model 3
Country
Country
#13
I have some familiarity with digital video recording in Linux-based OS's. The video above looks like it's being recorded at a variable bit rate (VBR), but the final output bit rate is too inconsistent for clean playback. You can see that effect if you watch a badly edited compilation video on youtube - as the bit rate changes abruptly from one to another, you get that pixellation/color shift/pause. If I'm correct about it, that's often done on low-power CPU devices (like dashcams, ironically enough!) to keep the video from freezing when CPU power runs out.

The good news is that might be fixable in software by increasing caching so the bit rate doesn't have to change so abruptly when CPU usage goes up. Down side to that is if the MCU crashes, you can lose a few minutes of footage.
 

Skione65

Top-Contributor
Joined
Apr 4, 2016
Messages
819
Location
Kentucky
Tesla Owner
Reservation
Country
Country
#14
yesterday about half of my passenger side repeater camera videos had similar distortion as this one (on 2019.8.2). I've emailed it to Tesla to ask if this is a MCU processing issue, camera hardware, USB recording, etc issue. Videos start out mostly ok, then get a little wavy, then totally garbled before returning to mostly ok.
@MelindaV,

Thanks for this Data Point. Let us know (a) if you ever hear back from Tesla regarding their analysis of this and (b) what their actual interpretation of this is.

Ski
 
Joined
Jun 9, 2018
Messages
15
Location
Chantilly
Tesla Owner
Model 3
Country
Country
#15
I love this sentry mode feature. I caught someone cleaning their car at work put a ding on my door. I know its a Nissan and have the person on video but haven't seen him since or the car. I guess he realized that I may be looking for him and started to park somewhere else. I wish the pillar cameras were also active as part of the sentry mode.
 

Skione65

Top-Contributor
Joined
Apr 4, 2016
Messages
819
Location
Kentucky
Tesla Owner
Reservation
Country
Country
#16
I love this sentry mode feature. I caught someone cleaning their car at work put a ding on my door. I know its a Nissan and have the person on video but haven't seen him since or the car. I guess he realized that I may be looking for him and started to park somewhere else. I wish the pillar cameras were also active as part of the sentry mode.
@Mehul,

Great! Have the vid? You can always cruise the lot and see if you recognize his car.

Ski
 

MelindaV

☰ > 3
Moderator
Joined
Apr 2, 2016
Messages
9,852
Location
Vancouver, WA
Tesla Owner
Model 3
Country
Country
#17
yesterday, I used my backup USB drive (32GB PNY USB3.0). It had similar results as my other. Always the passenger side repeater gets wonky, but only in Sentry mode.
 

Bokonon

Self-identified Teslaholic
Moderator
TOO Supporting Member
Joined
Apr 12, 2017
Messages
3,111
Location
Boston
Tesla Owner
Model 3
Country
Country
#19
Another data point that this may actually be a firmware bug...

Last night I was reviewing some fun dashcam footage (which I will share later :) ), and noticed that the corruption in my repeater cameras follows a very simple and consistent pattern. When the car is in motion, there are no issues with the side cam videos, but as soon as the car stops, BAM, corruption starts to seep in. When the car starts moving again, the corruption immediately vanishes. This happens sometimes with the front camera video as well, but I never saw this happen when only the front camera was recording in 50.6 and prior.

It's as though there's a firmware (over-)optimization in 2019.5 and later to minimize dashcam I/O by writing very few (if any) keyframes to the video files while the car is at rest. Or at the very least there seems to be some significant change/reduction in I/O activity that only happens while the car is at rest, and it can produce some buggy side effects.

Makes me wonder whether some of the zero-byte files have a similar cause, rather than being an issue with drive write speed that the firmware doesn't handle gracefully.