Logging software in my own server

  • SUPPORT THE SITE AND ENJOY A PREMIUM EXPERIENCE!
    Welcome to Tesla Owners Online, four years young! For a low subscription fee, you will receive access to an ad-free version of TOO. We now offer yearly memberships! You can subscribe via this direct link:
    https://teslaownersonline.com/account/upgrades

    SUBSCRIBE TO OUR YOUTUBE CHANNEL!
    Did you know we have a YouTube channel that's all about Tesla? Lots of Tesla information, fun, vlogs, product reviews, and a weekly Tesla Owners Online Podcast as well!

  • It's OK to discuss software issues here but please report bugs to Tesla directly at servicehelpna@teslamotors.com if you want things fixed.

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#1
I want to keep data logs for my Model 3 but I don't want to use a 3rd party service. Essentially I want something like TeslaFi but running on my own computer.

I've looked around a bit and found DoctorMcKay's Tesla Data Recorder which is the closest to what I need. I am going to test this and would like to enhance it.

Since I don't really like javascript I plan to write my own software. I would prefer to use C or PERL but I haven't found API bindings for these languages. So I will likely do it in Ruby, the same language as used in timdorr's API. (Will be my first program in Ruby, but it sure looks like a cool language).

If anyone has done something like this I would like to hear about it. If I am successful in developing this project (i.e. if I have enough stamina and time to dedicate to it), I will be very happy to open source it, as I've been doing with my scientific software (for 30 years).

For now I am going to install DoctorMcKay's software and see how it works, before I embark in writing code. At least the database is likely to keep the same structure so this will be good preparation.

So watch this space. I will use this thread to provide updates.
 

AmpHog

Active member
Joined
Oct 28, 2018
Messages
90
Location
NorCal
Tesla Owner
Model 3
Country
Country
#2
I'll be interested to follow your progress. I do some rudimentary logging, command scheduling, voice control and geolocation using the Tesla API, but but since my programming skills are now classified as 'really prehistoric, Dad' I did most of my intial work via SmartThings/webCoRE. I've thought about trying Ruby, but haven't made the push just yet.
 

FF35

Well-known member
TOO Supporting Member
Joined
Jul 13, 2018
Messages
358
Location
New Jersey
Tesla Owner
Model 3
Country
Country
#3
I’ve thought about this also and will be following this thread.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#4
So far so good: I've managed to get DoctorMcKay's Tesla Data Recorder to work. It is currently logging Niki's data to the SQL database in my server.

For the next step I will concentrate on creating useful reports from the database, since that is something DoctorMcKay's code doesn't do (it is simply a logger with no frontend). Adding reports will already expand the capability of that code.

Candidates for the reporting are a) R statistical software or b) PERL+gnuplot. I guess R is probably better for people using Windows/Mac as it may be easier to install and I know there are many packages that may help with visualization (eg maps).
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#5
The DoctorMcKay's script has been running for a few days now so I get to appreciate how it is organizing things. One thing that I don't like is that it stores four JSON structures verbatim in the database (ie JSON stored in text strings). This makes it harder to use SQL to make complex queries on the data inside those JSON objects. The first thing I would change in the logger script therefore would be to convert all the data inside the JSON object into table columns. That way we could query things like charge_current_request, charger_voltage or battery_heater_on (though this one may only be interesting to people with models X and S).

So if I am to touch the logger software the first thing will be to put everything in SQL as it will make more queries possible (without complex code).

Another thing I am starting to grapple with is that SQL is not great for time series data. But since I am writing the analysis in R, I think I am just going to pass large chunks of data to the R scripts and let these deal with the time series.

The first analysis script I am working on is a weekly charging summary (later can easily change to monthly, yearly, etc.). Basically the R script will do an SQL query to pull all the charging entries for the week and then do the rest of the work itself.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#6
The DoctorMcKay's script has been running for a few days now so I get to appreciate how it is organizing things. One thing that I don't like is that it stores four JSON structures verbatim in the database (ie JSON stored in text strings).
Another thing I don't like is that the logging daemon needs to be 'awaken' before starting to drive, otherwise it may miss a chunk of the ride (or even the whole thing). This is because when the car is in sleep mode the logger only queries it every 6 hours (to avoid vampire drain); or every 30 minutes when it is parked and not asleep, etc.

Currently I have set up a web page that I call from my cell phone or from the car's browser just before starting. But this is not practical, and I am sure I will forget many times. Ideally this should be triggered automatically by the car or the phone app. But I have not figured out how to do this yet (there's a method that could be used with Android phones but does not work on iPhones.)

It would be great if we could program the car to load some web page automatically as it started...
 
Joined
Jan 1, 2019
Messages
12
Location
Edison, NJ
Tesla Owner
Model 3
Country
Country
#7
I chose python and found an existing package that supported the Tesla API. I then extended it with a polling/logging program and a text parser. I also created an RPC interface to support integration with a CLI or web server or whatever.

However, with my program you don't need to explicitly tell it to start polling. The polling daemon I wrote checks the car up every (configurable) minute when asleep to see if something has changed (like, the car is awake because you started to drive). It also wakes up the car every (configurable) 2.8 hours to track temperature and vampire drain. It lets the car go back to sleep after ~10m after the last state change (stop driving/charging etc). My vampire drain is in the 4.6 to 5.3 mile per day range, which seems to be about par for Tesla 3s so this isn't a big burden.

It currently doesn't have any packaged analytics for producing graphs or whatever, but graphing from json isn't too challenging.

My version is at https://github.com/tesladdicts/teslajson (was https://github.com/SethRobertson/teslajson) while the original version is at https://github.com/gglockner/teslajson
 
Last edited:
Joined
May 5, 2018
Messages
21
Location
Ontario
Tesla Owner
Model 3
Country
Country
#8
I did most of my intial work via SmartThings/webCoRE.
Have you published your SmartThings/webCoRE handlers/pistons? I've written some python stuff to interface with the car but I haven't had success with a decent device for ST with the Tesla.
 

AmpHog

Active member
Joined
Oct 28, 2018
Messages
90
Location
NorCal
Tesla Owner
Model 3
Country
Country
#9
Have you published your SmartThings/webCoRE handlers/pistons? I've written some python stuff to interface with the car but I haven't had success with a decent device for ST with the Tesla.
I haven't published all of my pistons, as much of the work is still in its infancy and the code is not as clean or complete as I'd prefer, but the foundation for the work that I've done is published here.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#10
I chose python and found an existing package that supported the Tesla API. I then extended it with a polling/logging program and a text parser. I also created an RPC interface to support integration with a CLI or web server or whatever.

However, with my program you don't need to explicitly tell it to start polling. The polling daemon I wrote checks the car up every (configurable) minute when asleep to see if something has changed (like, the car is awake because you started to drive). It also wakes up the car every (configurable) 2.8 hours to track temperature and vampire drain. It lets the car go back to sleep after ~10m after the last state change (stop driving/charging etc). My vampire drain is in the 4.6 to 5.3 mile per day range, which seems to be about par for Tesla 3s so this isn't a big burden.

It currently doesn't have any packaged analytics for producing graphs or whatever, but graphing from json isn't too challenging.

My version is at https://github.com/SethRobertson/teslajson while the original version is at https://github.com/gglockner/teslajson
Thanks for that. From what you describe and a quick look at your code, it seems that your approach has some advantages to what I am using, though also some things I'd change. I will evaluate it some time soon to get a better feel.

These are the things I like in your approach:
  • no need to wake up the logger
  • python code (even though I don't write python, I prefer it to javascript)
but there are some negatives:
  • drain: 4.6 - 5.3 mile per day is a bit more than I'm getting (3.8-4.0)
  • writing output in json to files (I'd rather use a database)
I have a question about the difference between "checking the car" versus "awaking the car". I thought that checking the car would also awake it, no?
The current poller I am using waits 6 hours when it finds the car was asleep (which is why it drains so little); of course this comes at the cost of having to manually wake up the poller (which will then wake up the car) before starting to drive.

Another question: are you following the polling approach used by TeslaFi? (does anyone know what is the approach they follow? I guess it cannot be much different)

I'm thinking that I may end up adapting your code to write out the data to a relational database; that would allow me to recycle the SQL/R code I've already written. I think I can write out reports as good as those from TeslaFi that way, which is really my objective. That way I could get the same functionality as in TeslaFi but not have my data outside my control (except Tesla itself, of course).
 

Bokonon

Self-identified Teslaholic
Moderator
TOO Supporting Member
Joined
Apr 12, 2017
Messages
3,119
Location
Boston
Tesla Owner
Model 3
Country
Country
#11
I have a question about the difference between "checking the car" versus "awaking the car". I thought that checking the car would also awake it, no?
The "vehicles" API endpoint provides a way of checking whether a given car is awake or asleep without waking it up. (Note that while the car is asleep, a minimal amount of information relating to the car's identity is available through the API, such as its name, configuration, and VIN.)

TeslaFi basically calls this "vehicles" endpoint every [n] minutes to check your car's status, except when TeslaFi pauses polling to allow your car to sleep (per your configured sleep settings). Once TeslaFi confirms that your car is asleep, it resumes polling every [n] minutes so it will know when your car wakes up. Also, when TeslaFi determines that your car is driving, it increases the polling frequency to 2-3 times a minute until the drive ends, in order to improve the accuracy of its data.
 
Joined
Jan 1, 2019
Messages
12
Location
Edison, NJ
Tesla Owner
Model 3
Country
Country
#12
I'd like to know if you find a longer timeout gives you lower drain. I didn't notice a difference, but I was not doing data-driven analysis before I had the data but other people online were reporting more lower and higher drains without monitoring so I figured I was close enough. But that is why I make all of the timeouts configurable on the command line.

I thought about throwing it into PostgreSQL, but I thought that the ability to compress older files probably would be more valuable to save space. However, if you have a great analytics package, that would certainly make more sense.

I was looking at some of the posts by the teslafi people when they were first playing around, and it seems to be approximately the same. I'm sure there are specific details on query selection and timeouts which may vary (though my impression is the timeout are configurable on their system as well, on a per-account basis. (And this agrees with what Bokonon just said).

I could probably hack up a quick json to sql conversion if you want and provided the schema.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#13
I'd like to know if you find a longer timeout gives you lower drain. I didn't notice a difference, but I was not doing data-driven analysis before I had the data but other people online were reporting more lower and higher drains without monitoring so I figured I was close enough. But that is why I make all of the timeouts configurable on the command line.

I thought about throwing it into PostgreSQL, but I thought that the ability to compress older files probably would be more valuable to save space. However, if you have a great analytics package, that would certainly make more sense.

I was looking at some of the posts by the teslafi people when they were first playing around, and it seems to be approximately the same. I'm sure there are specific details on query selection and timeouts which may vary (though my impression is the timeout are configurable on their system as well, on a per-account basis. (And this agrees with what Bokonon just said).

I could probably hack up a quick json to sql conversion if you want and provided the schema.
I think I am going to fork your project and add the code for SQL logging (probably using MariaDB but if you strongly prefer PostgreSQL I can also use that, let me know). Then I will send you a pull request and you can add my mods if you like them.

I will base the schema on the one from DoctorMcKay's, except that I will flatten all of the json objects and put everything as columns.

I definitely don't want to concentrate on the logger, I would like to put most of my effort on reports. Python can also be a good option and I may end up using that for reports (would allow me to learn more python too). But reporting doesn't have to be done in the same language as the logger.

I am going to do this on a few evening hours per week and some weekend time, so don't expect it to happen very fast...
 
Joined
Jan 1, 2019
Messages
12
Location
Edison, NJ
Tesla Owner
Model 3
Country
Country
#14
I certainly prefer postgresql. I just looked at his sql and it is interesting how we have approximately the same number of "important data" elements (I have 14, he has 17) but we only have five things in common.

Probably I'd suggest a union of the two.
 

jat255

New Member
Joined
Sep 12, 2017
Messages
1
Location
Northern VA
Country
Country
#15
I think I am going to fork your project and add the code for SQL logging (probably using MariaDB but if you strongly prefer PostgreSQL I can also use that, let me know). Then I will send you a pull request and you can add my mods if you like them.

I am going to do this on a few evening hours per week and some weekend time, so don't expect it to happen very fast...
@Niki-and-I is your code public anywhere? I'm somewhat capable hacking around on SQL, R, and Python, and I'd love to contribute to the efforts.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#17
I certainly prefer postgresql. I just looked at his sql and it is interesting how we have approximately the same number of "important data" elements (I have 14, he has 17) but we only have five things in common.

Probably I'd suggest a union of the two.
I've run your logger this morning and I am now looking over the json file it created. I'm now going through the data to see what I would like to log into a relational table.

What I am not sure is whether a) I should just create a separate app that logs to SQL, independent from your app, or b) if I should extend yours to allow users the option to use json or SQL data logging. What do you think?
 
Joined
Jan 1, 2019
Messages
12
Location
Edison, NJ
Tesla Owner
Model 3
Country
Country
#18
What I am not sure is whether a) I should just create a separate app that logs to SQL, independent from your app, or b) if I should extend yours to allow users the option to use json or SQL data logging. What do you think?
I think it should be extended. I think ideally the tesla_parselib.py should have the SQL output since it already parses the json---it also would let people (me I guess :p) switch from json to SQL. Obviously the parser would have to be extended to support whatever new variables you wanted to directly put into the SQL schema. Then the poller can just import that class, parse the json it retrieves, and export it. Probably I would then convert the poller to use the parsed class data instead of directly accessing the json data like I'm doing now.
 

Niki-and-I

Active member
Joined
Nov 18, 2018
Messages
93
Location
Farmington, CT
Tesla Owner
Model 3
Country
Country
#19
I've forked @Climate Change Denier's package to https://github.com/pmendes/teslajson and I am now working on adding SQL to it. Feel free to check my progress if you want; when I am done I will then issue a Pull Request. Right now I have pushed a preliminary SQL schema https://github.com/pmendes/teslajson/blob/master/create_tables.sql please discuss whether there are important missing features.

I decided to create two tables, one for cars, the other for their statuses. This is because this code can log several cars simultaneously and I like to keep the data normalized. Currently the car table has only a few items of information. I'd like to see if other items are important to people, including those with Model X and S (the code should work for all Tesla models).

I'm going to use the Psycopg2 package to do the connection from Python to Postgresql. If anyone here has very strong opinions to use another package instead please come forward (I've been spending more time reading about why are there so many postgresql connectors for python than I wanted!)