Don’t Put It on the Internet: Tesla Backup Gateway Edition

Derek Abdine, formerly Director of Rapid7 Labs, now CTO at Censys, contributed this blog post.

This blog post aims to increase user awareness of the privacy and security risks of connecting devices to the internet. In this edition, we address Tesla Backup Gateways and identify some key areas where Tesla could improve security and privacy to help customers protect themselves. The security risks associated with these devices have been explored previously by other researchers, as detailed below, and we hope to help boost awareness of their findings and provide some additional insight into the degree of exposure.

Recent studies of the internet from Project Sonar indicated 97 active home/commercial solar/battery installations (“power plants”) connected to the internet, with plenty of information that can be collected from them without prior authentication. The risk here is not just a violation of privacy; there is also the potential for a malicious actor to cause some action to be taken. Given that we’re talking about powerful batteries, that action could cause significant harm.

Since January 2020, we have tracked 379 total unique Tesla Backup Gateway installations. Of these, a subset are commercial-grade Tesla Powerpacks (very large battery arrays). While these numbers may seem low, the potential for malicious activity leading to serious harm could be great, and is thus worth highlighting and mitigating as soon as possible.

Background

In February 2020, I decided to take a leap into Tesla. I’m a fan of the mission of both Tesla and SpaceX, but I won’t turn this into an endorsement for Tesla itself. Suffice to say, I bought into the idea of powerwalls and solar panels. Especially with an infant on the way, I wanted to ensure the family would be comfortable in the event of power outages, which happen every so often where we live.

There are various components to the energy install, including a subpanel that was installed for battery backed-breakers, some AC disconnect switches, the solar inverter, a main subpanel for power source breakers, and finally, the Tesla Backup Gateway. The Tesla Backup Gateway is one of the most interesting pieces of technology in this mix, since it is effectively managing a tri-power source (battery, grid, and solar), with a common software stack, and determining:

1. When to charge the battery
2. When to send energy to the grid
3. What source/mix of energy should be supplied to a house (“load”)

To aid its customers, Tesla includes monitoring of your energy installation. To do this, the Tesla Backup Gateway communicates with Tesla via three different networking schemes:

1. Wi-Fi
2. AT&T mobile (radio)
3. Ethernet

Accessing data

In my experience, Tesla Energy installers prefer the backup gateway is connected via Wi-Fi to your home network, and the AT&T radio is used only for backup communication. As an aside, this is a good time to use vlan segmentation or other network isolation models if you don’t already do so. The Tesla Backup Gateway exposes, among other things, a web server with a self-signed certificate running on the standard HTTPS port 443, that listens on your local network (unsurprisingly, this also makes these extremely easy to find on the internet). Tesla encourages interaction with the gateway through its documentation for the backup gateway. The web interface resembles the sample that they have on the documentation site, showing the flow of electricity from all your connected sinks and sources of power:

There’s a seemingly rich amount of information you can glean from the interface. You can tell how much power is being used by your home, what your grid draw is, and what your battery percentage is, along with some ownership information (“Doe’s Powerwall”). But, as it would turn out, there’s a much richer set of APIs that power this web UI that are unofficially documented (reverse engineered) by Vince Loschiavo on GitHub.

The API calls reveal that quite a bit of information can be gleaned from the gateway sans authentication. For example, you can determine your solar, battery, grid, and home energy statistics simply by querying the /api/meters/aggregates endpoint with cURL:

$ curl -s https://gateway/api/meters/aggregates --insecure | jq
{
  "site": {
    "last_communication_time": "2020-04-15T13:24:48.086747663-07:00",
    "instant_power": -4828.510009765625,
    "instant_reactive_power": -673.8599853515625,
    "instant_apparent_power": 4875.30472834928,
    "frequency": 60,
    "energy_exported": 65134.13694506048,
    "energy_imported": 1298075.5183339494,
    "instant_average_voltage": 121.01499938964844,
    "instant_total_current": 0,
    "i_a_current": 0,
    "i_b_current": 0,
    "i_c_current": 0,
    "timeout": 1500000000
  },
  "battery": {
    "last_communication_time": "2020-04-15T13:24:48.090099305-07:00",
    "instant_power": -10,
    "instant_reactive_power": 630,
    "instant_apparent_power": 630.0793600809345,
    "frequency": 60.021,
    "energy_exported": 53250,
    "energy_imported": 89000,
    "instant_average_voltage": 241.55,
    "instant_total_current": -0.4,
    "i_a_current": 0,
    "i_b_current": 0,
    "i_c_current": 0,
    "timeout": 1500000000
  },
  "load": {
    "last_communication_time": "2020-04-15T13:24:48.086747663-07:00",
    "instant_power": 540.5096700141057,
    "instant_reactive_power": -24.131958716463117,
    "instant_apparent_power": 541.0481076671931,
    "frequency": 60,
    "energy_exported": 0,
    "energy_imported": 1382705.7686111112,
    "instant_average_voltage": 121.01499938964844,
    "instant_total_current": 4.466468394333113,
    "i_a_current": 0,
    "i_b_current": 0,
    "i_c_current": 0,
    "timeout": 1500000000
  },
  "solar": {
    "last_communication_time": "2020-04-15T13:24:48.086860329-07:00",
    "instant_power": 5379.058881364439,
    "instant_reactive_power": 10.182297104210392,
    "instant_apparent_power": 5379.068518652646,
    "frequency": 60,
    "energy_exported": 185756.7500000004,
    "energy_imported": 242.36277777818367,
    "instant_average_voltage": 120.75,
    "instant_total_current": 0,
    "i_a_current": 0,
    "i_b_current": 0,
    "i_c_current": 0,
    "timeout": 1500000000
  }
}

These numbers alone tell you interesting things, such as:

1. Solar/instant_power: 5379 is 5379 watts, which implies that the solar array is producing 5.3kW of energy at the moment the command was run.
2. Load/instant_power: The “load” is the house, which in this case was using around 500 watts of energy (0.5 kW).
3. Site/instant_power: The “site” is the grid, which in this case was SENT 4.8 kW of energy due to overproduction of the panels, and the powerwall batteries being full.
4. Frequency: This is the frequency of the AC power being supplied. In this case, 60hz is common in North America. Elsewhere, such as the UK, you would see 50hz.

Going a tiny bit deeper, we can inspect some other fun endpoints:

$ curl -s https://gateway/api/site_info --insecure | jq
{
  "max_site_meter_power_kW": 1000000000,
  "min_site_meter_power_kW": -1000000000,
  "nominal_system_energy_kWh": 13.5,
  "nominal_system_power_kW": 10,
  "site_name": "Loschiavo",
  "timezone": "America/Los_Angeles",
  "grid_code": "60Hz_240V_s_UL1741SA:2016_California",
  "grid_voltage_setting": 240,
  "grid_freq_setting": 60,
  "grid_phase_setting": "Split",
  "country": "United States",
  "state": "California",
  "distributor": "*",
  "utility": "Pacific Gas and Electric Company",
  "retailer": "*",
  "region": "UL1741SA"
}

This can tell us the time zone we’re in, the name of the site that appears in the web UI, and the grid code, which seemingly controls things such as the frequency of the various components. Last, we can tell which utility we’re a part of as well.

Potential harm

I’m not going to completely reproduce Vince’s work here—I highly recommend you read his documentation on GitHub for more details. It’s a great read, with interesting results. Suffice it to say that there are fairly broad security and privacy implications of leaving a Tesla Backup Gateway exposed to the internet.

There are direct impact implications of leaving these devices connected to the internet as well, as detailed by The Hacker’s Choice. In theory, the voltage, cycle, and other settings of the energy managed by the Backup Gateway, and the batteries connected to it are configurable. It may be possible to do damage to a battery, or even the electrical grid, if these settings could be tampered with. However, this is not an area of research I would delve into for obvious ethical, legal, and monetary reasons.

Weak credentials

Even if the gateway were properly secured and all APIs required authentication, our next finding makes that irrelevant: weak default credentials. Home users and installers can log in to the Tesla Backup Gateway using weak credentials, which is outlined on the Tesla website.

For “first-time login,” which seems to be after the gateway has been rebooted (new install, or after a firmware update), a user can log in using any email address, and the last five characters of the gateway serial number. It’s unlikely that most users have logged in the first time, since logging in also stops powerwall/battery operation.

Once you’re authenticated, an API call can be used to grab the full serial number for subsequent access. Since mine has alphanumeric characters in it, the brute force worst case for the five-character first-time login is 10 digits (0–9) plus 26 alpha characters (A–Z, uppercase only), which gives us (26+10)^5 = 60.4 million password combinations. Although from testing, there does not appear to be any max login attempt lockout, on the surface, the sheer theoretical number of possible passwords makes it seem as if brute forcing against the web server is unlikely to succeed. However, there are a number of techniques to reduce that number low enough to be brute-forceable.

From a purely physical perspective, It’s important to note that the serial number is printed on the device, so simple recon outside a person’s house to find the gateway and take a picture of the tag containing the serial is a real possibility. At first thought, this may sound hard to do. However, many county websites publish household Tesla Solar and Powerwall install permits to the internet, such as what is done in El Dorado county in California. These permits can provide a wealth of information about which backup gateways to target (though some permits are solar-only), and the physical location of the address where the device was installed, making a drive-by easy.

However, you may not even need to leave your car, as the gateway also broadcasts a Wi-Fi access point, also noted in the Tesla documentation, that uses the same password (serial number). The Wi-Fi access point SSID naming scheme includes the last three characters of the serial number (TEG-XXX), further reducing the amount of work needed to brute force access of the first-time login password (you only have to guess the first two characters, which may be a worst case of 1,296 attempts). Using Wi-Fi OSINT databases like wigle.net, it’s trivial to find many more SSIDs (free account required) that give us a greater sense of the patterns backup gateways use, and how many of them exist (spoiler: wigle.net reports 4,294 Tesla Backup Gateways broadcasting Wi-Fi SSIDs, based on SSID and MAC–00:23:A7 or 88:DA:1A, associated with Redpine Signals, who manufacture the Wi-Fi chip in the gateway). I have not tried, but you can likely glue this with the latitude and longitude of the home addresses of physical installations to optimize setup time before brute forcing).

Moreover, when the backup gateway is connected to the LAN, it broadcasts its hostname to the network using its full serial number. Some home routers, such as Xfinity routers, have been known to leak entire MAC/IP/Hostname information sets from the LAN to the public internet. We have observed one instance where this leak has occurred. Last, the Tesla app also will show the serial number in full, and Tesla accounts do not support MFA (yet), which enables phishing as another viable attack vector.

It is worth pointing out that logging into a Gateway belonging to someone else, or otherwise without authorization, using these methods likely violates the Computer Fraud and Abuse Act (CFAA) and state law equivalents. This is true even if the credentials are easy to guess, so any testing is best performed on equipment you own, or to which you have authorized access. The purpose of detailing these methods for bypassing the login is to encourage public awareness of the risk.

Internet presence

As it would turn out, these devices are absolutely on the internet—strangers around the world can directly access these things and attempt to log in and monkey with the configuration. As with most equipment offering services on the internet, they’re fairly easily discoverable using a straightforward google dork. HTTP title tags and self-signed certificates are a sure, easy way to find them. We’ve tracked at least 379 of these since January 2020 around the world:

Gray tiles indicate no data for that region, while colored tiles indicate an approximate number of Tesla Backup Gateways for that region. The U.S., of course, would be expected to have a larger footprint of them, since Tesla is headquartered in California. Indeed, this is the case, with a large concentration in Tesla’s backyard/home state:

A view of Europe is also interesting, when looking at publicly exposed Tesla Backup Gateway devices. Most notably, there are a significant number of them in Italy:

Bonus: Commercial Tesla Powerpacks are on the internet, too!

While investigating exposed Tesla Backup Gateway devices, we also ran into an anomaly: Simply grabbing /api/ returned a 404 response from the web server for some of these devices. Digging further, we discovered that the reason for this is that the device was not actually a Gateway device, but rather it was a Tesla Powerpack.

Tesla Powerpacks appear to be (and I say appear because, though I’d love to have one, I don’t have the $200K minimum to dish out for one) large, commercial battery arrays.

Here’s an example of the user interface you’re presented with, just browsing to/on the server (in this case, over port 10443). Using simple algebra, it would appear this battery is around 220kWh in size (114kWh/.52). To give you a frame of reference, a Tesla Powerwall 2 battery is 14kWh in size. So, this publicly exposed Powerpack is about 15 times the size of a single residential powerwall battery.

On the same host, there tends to be another interface, which in one case, was running on the standard HTTPS port 443:

Slightly more concerning is that these devices can also speak modbus/dnp3. In this case, this specific server was also exposing modbus on the default port 502, though DNP/20000 appeared closed (which, in some additional Tesla documentation, may be due to DNP only being exposed over serial).

Tesla documents these protocols openly on its website:

Closing thoughts

I am fairly alarmed at the number of these devices on the internet. As mentioned above, the numbers may be relatively low, but given the devices are massive batteries that deal in high voltage and current, malicious manipulation could lead to potential physical harms. It is also possible to pinpoint these even if they aren’t internet connected through weakly configured home routers, and pivot from those routers into the LAN to control them that way.

Though placing a Tesla Backup Gateway or Tesla Powerpack on the internet may be tempting, we should remember that the internet is noisy by nature, with lots of unsolicited traffic being passed through various ports on a regular basis. It is never safe to assume that placing a service on a port that is non-standard will prevent it from being discovered. Projects such as zmap and masscan have demonstrated that the entire IPv4 internet can be scanned in under five minutes.

Moreover, the implications of a decentralized energy system and network connectivity will start to become more clear over the next decade. The solar market has heated up, and some states, such as California, have passed laws that are starting to require energy production come from onsite or offsite solar farms. How will energy standards such as NERC CIP be impacted in the future? Suddenly, the footprint of an electrical grid now includes decentralized “mini power generation plants” that have a network footprint. Dumping energy on the line when linemen are working, messing with the phase of energy sent back to the utility, and other oddities that these systems allow for aren’t really fully understood yet. NERC itself requires registration when a generating facility produces over 75 megawatt-hours of energy. Most home solar generation will be well under that. But in a world where there are many more home solar deployments, it’s not far-fetched to imagine the aggregate capacity easily exceeding those requirements. However, there is still real damage that can be done by individual houses on local infrastructure. There are also intense privacy concerns related to the publishing of energy usage statistics, that may allow physical break-ins to coincide with a lull in energy use, correlated with sleeping occupants, or an empty home.

To harden these devices a bit more, and help customers stay safe, Tesla could require passwords to be set on initial configuration by their installers, with no defaults. Tesla is also highly encouraged to support 2FA on user accounts (though Elon Musk hinted that this functionality is coming soon, as of April 22, 2020, even when it does come, this is assumed to be applied to mobile apps and Tesla SaaS, not necessarily the private web server which seems completely unintegrated with Tesla SSO).

Prior to publication, we reached out to Tesla’s Product Security about this, and they let us know that they are working on further mitigating accidental exposure in some upcoming security features. In the meantime, they let us know that “predictable installer passwords have been fixed for some time on newly commissioned Backup Gateway V1 devices, but some previously commissioned devices still had them, and all online Backup Gateway V1 devices have had their installer passwords randomized.” In addition, “all Backup Gateway V2 devices come from the factory with non-predictable random passwords,” so that all sounds like good news going forward.