AT&T Internet

Tuesday 18 August 2020 22:26 -05:00

AT&T Technician arrived at 08:40 on Monday 17 August 2020 to install Internet 1000 service. It took approximately 5 hours for him to complete the hookup, because he had to run fiber line on the utility poles in the back yard, presumably from the nearest fiber pedestal, trimming trees and shrubs in the process, and had to return to the office to fetch different equipment. Apparently, mine was the first house he installed with the capability for 10 gigabit/sec bandwidth, and he was unprepared. He completed his work and left around 13:45.

First Tests

I performed the first tests of the AT&T Internet connection by connecting Toad directly to the provided "modem." This device, an Arris BGW210-700, or one of other similar devices, seems to be mandatory for all AT&T Internet/U-Verse customers. As far as I can tell, it is a typical Ethernet router, with support for 802.1x/EAP wired port authentication.

With this setup, I got a pretty disappointing Speed Test score: http://www.dslreports.com/speedtest/65265252. Additionally, I am not sure the "IP Passthrough" functionality of the device is going to be sufficient for my needs. Although it makes it appear to the downstream device as though it is directly connected to the Internet, the intermediate router still seems to be managing the traffic. I have seen a few vague reports that certain services "don't work" when the router is set up in this mode.

Bypassing the Residential Gateway

With the poor network performance presumably caused by the AT&T-provided Residential Gateway, and my suspicions that it would cause me trouble with my self-hosted services (particularly the IPsec VPN), I decided to research a way to bypass it.

Initial Research: EAP Proxy

The first method I found for trying to bypass the AT&T-provided Residential Gateway (RG) was to use an "EAP Proxy" running on the USG: https://medium.com/@mrtcve/at-t-gigabit-fiber-modem-bypass-using-unifi-usg-updated-c628f7f458cf. The proxy would receive the EAPOL frames from the Optical Network Terminator (ONT) and forward them to the RG, which is connected to the LAN 2 port on the USG. It would then forward the responses from the RG to the ONT, thus the ONT would think the USG itself was performing the authentication.

I was initially turned off from this method, because it seemed like it required connecting the RG directly to the LAN 2 port, without a switch in between. Since I am already using the LAN 2 port for the auxiliary VLANs on my network (test, guest, Home Assistant, management), I decided to revisit it only if I could find no other option.

Attempt 1: Device Swapping

The next bypass mechanism I found was this "true bridge mode" post on DSLReports: https://www.dslreports.com/forum/r29903721-AT-T-Residential-Gateway-Bypass-True-bridge-mode. This method uses the RG temporarily to perform the 802.1x port authentication/authorization, and then swap it out for another router. The described procedure uses port-based VLANs on a NetGear ProSafe GS108E. By changing the VLAN assignment of the ports where the AT&T RG and the desired router are connected, the idea is to "trick" the Optical Network Terminator (ONT), such that it does not notice the device on the other end has changed, and thus does not require it to reauthenticate.

I attempted this method using my UniFi Switch 48, assigning 3 ports to a special "AT&T" VLAN. I planned to bring up the ONT and the RG together on the same VLAN, then once the authentication was complete, kill the RG and bring up the USG. Unfortunately, the RG never managed to authenticate. The "Broadband" light blinked red most of the time, and never acquired an IP address. I suspect the issue was that the switch was filtering the 802.1x traffic, since this is a layer 2 protocol that is intended to authenticate a device to a switch, rather than a device to another device.

Some musings in the DSLReports thread suggested that it should also be possible to use a "dumb switch" to work around this problem. If the switch is dumb enough (as is apparently the GS108E), it will forward EAPOL frames exactly the same as it would any other frame. I have an extremely cheap 5-port Tenda gigabit switch, so I tried using it instead of the UniFi switch. In this scenario, the RG did manage to authenticate to the ONT, and come online completely. Unfortunately, even with MAC address spoofing, I was not able to "swap out" the RG. When I connected Toad to the dumb switch and disconnected the RG, Toad never received a DHCP response, and using a static IP address did not work. (In hindsight, I realized that this was probably because I needed to use an 802.1Q tag to mark the outbound frames with VLAN ID 0.)

This method seemed really fragile to me, so even if I had gotten it to work, I probably would not have been happy with it. I considered using some trick like this "2 dumb switch relay" method (https://www.dslreports.com/forum/r32061385-) to automatically disconnect the USG and reconnect the RG whenever some issue caused the port to become deauthenticated. If this had been my only option, I probably would have tried harder, but instead, I decided to investigate some other options.

Attempt 2: wpa_supplicant

During my initial research for a bypass method, I noted some posts regarding using wpa_supplicant on a UniFi Dream Machine to perform the 802.1x authentication directly. Since I had had no luck with the first couple of options, I decided to research this option further and see if there was an equivalent option for the UniFi Security Gateway. Sure enough, a quick Duck Duck Go revealed a blog post specifically about that: https://wells.ee/journal/2020-03-01-bypassing-att-fiber-modem-unifi-usg/ (Aside: the title of this person's blog, It Kinda Works initially gave me pause, as I interpreted that to refer to this specific technique. Luckily, I decided to read it anyway :).

I knew wpa_supplicant could be used to perform 802.1x authentication for wired interfaces, the same as it does for WPA2-Enterprise networks on wireless interfaces, so I read the post hoping for some insight into how to set that up on the USG. Sadly, it seemed like this may be difficult, since it involves "obtaining" the X.509 certificates and private keys from the RG. The author of this post advocates using eBay to obtain them, but I was not willing to pursue that avenue. Instead, I decided to research a mechanism for extracting them from the RG itself.

A quick DDG search for bgw210 certificate led me to this blog post: https://www.dupuis.xyz/bgw210-700-root-and-certs/. The author does a magnificent job describing the vulnerability in the RG that allows this, how an exploit was developed, and how to use it to gain root access to the operating system. The process itself was absolutely trivial. In only a few minutes, I was able to exploit the vulnerability, gain root access to the OS, and extract the mfg.dat file.

The only tool I could find to decode themfg.dat file and extract the certificate and private key from it was a closed-source program called (naturally) mfg_dat_decode: https://www.devicelocksmith.com/2018/12/eap-tls-credentials-decoder-for-nvg-and.html?m=1. Fortunately, the author provided a native Linux executable, so although I would rather have inspected the code to see what it does and how it works, I could at least use it without any hassle. Again, the process was so trivial, I had access to everything I needed to set up my USG to perform the EAP-TLS authentication itself, completely and permanently eliminating the AT&T RG from my network.

I first tested this method using wpa_supplicant on Toad, and it worked flawlessly. Later, when I had some spare time, I went ahead and deployed it on the USG. With some minor variation (I put the certificates and private key in /config/auth, the wpa_supplicant binary in /config/scripts, the wpa_supplicant.conf file in /config, and wrote my own /config/scripts/post-config.d/wpa_supplicant.sh because the one suggested was written by someone who had extremely limited knowledge of shell scripting), I was able to set it up as described in the Bypassing the AT&T Fiber modem… blog post. The process was actually rather simple:

Install the wpa_supplicant binary
Copy the CA certificate, client certificate, and private key files
Configure wpa_supplicant to use these to authenticate the port using EAP-TLS
Ensure wpa_supplicant starts automatically using a script in /config/scripts/post-config.d
Change the MAC address of the WAN port on the USG to that of the RG
Set the WAN port to use tagged VLAN ID 0
Reboot the USG

The only thing that bothers me about this method is using the precompiled wpa_supplicant binary. I definitely want to come up with a process for building it myself, preferably automatically in Jenkins, so I can keep it up-to-date.

Fortunately, the certificate I extracted from the RG does not expire until 2040! Hopefully sometime in 20 years, something changes…

Without the RG in the path, the next speed test was much better: http://www.dslreports.com/speedtest/65268235.

IPv6 Support

During my initial test, Toad was assigned a couple of IPv6 addresses (presumably 1 via SLAAC and 1 by DHCPv6). Once I eliminated the RG and connected the USG directly, no IPv6 addresses were assigned either to the USG or to its clients. Based on various posts on the web, I believe this is a "known issue" with AT&T. Apparently, there is a 2-week period after changing devices during which IPv6 address assignment is unavailable. This seems to be because the DHCPv6 lease is assigned with a 14-day lifetime, and only a single lease is allowed per customer connection. I will again try to get IPv6 working in a couple of weeks, then.

Self-Hosting

So far, after switching to AT&T Internet, I have not noticed any issues with my self-hosted services. I have confirmed that all of my websites (including Nextcloud, Bitwarden, Gitea, etc.) are all working, as is my Wireguard VPN.

Before I signed up with AT&T, I read the Terms of Service and the Acceptable Use Policy. Obtuse legalese notwithstanding, I did not see anything that forbade self-hosted services, so I imagine there will not be any issues going forward, especially with the crappy RG out of the way.