AT&T Technician arrived at 08:40 on Monday 17 August 2020 to install Internet 1000 service. It took approximately 5 hours for him to complete the hookup, because he had to run fiber line on the utility poles in the back yard, presumably from the nearest fiber pedestal, trimming trees and shrubs in the process, and had to return to the office to fetch different equipment. Apparently, mine was the first house he installed with the capability for 10 gigabit/sec bandwidth, and he was unprepared. He completed his work and left around 13:45.
I performed the first tests of the AT&T Internet connection by connecting Toad directly to the provided "modem." This device, an Arris BGW210-700, or one of other similar devices, seems to be mandatory for all AT&T Internet/U-Verse customers. As far as I can tell, it is a typical Ethernet router, with support for 802.1x/EAP wired port authentication.
With this setup, I got a pretty disappointing Speed Test score: http://www.dslreports.com/speedtest/65265252. Additionally, I am not sure the "IP Passthrough" functionality of the device is going to be sufficient for my needs. Although it makes it appear to the downstream device as though it is directly connected to the Internet, the intermediate router still seems to be managing the traffic. I have seen a few vague reports that certain services "don't work" when the router is set up in this mode.
Bypassing the Residential Gateway
With the poor network performance presumably caused by the AT&T-provided Residential Gateway, and my suspicions that it would cause me trouble with my self-hosted services (particularly the IPsec VPN), I decided to research a way to bypass it.
Initial Research: EAP Proxy
The first method I found for trying to bypass the AT&T-provided Residential Gateway (RG) was to use an "EAP Proxy" running on the USG: https://medium.com/@mrtcve/at-t-gigabit-fiber-modem-bypass-using-unifi-usg-updated-c628f7f458cf. The proxy would receive the EAPOL frames from the Optical Network Terminator (ONT) and forward them to the RG, which is connected to the LAN 2 port on the USG. It would then forward the responses from the RG to the ONT, thus the ONT would think the USG itself was performing the authentication.
I was initially turned off from this method, because it seemed like it required connecting the RG directly to the LAN 2 port, without a switch in between. Since I am already using the LAN 2 port for the auxiliary VLANs on my network (test, guest, Home Assistant, management), I decided to revisit it only if I could find no other option.
Attempt 1: Device Swapping
The next bypass mechanism I found was this "true bridge mode" post on DSLReports: https://www.dslreports.com/forum/r29903721-AT-T-Residential-Gateway-Bypass-True-bridge-mode. This method uses the RG temporarily to perform the 802.1x port authentication/authorization, and then swap it out for another router. The described procedure uses port-based VLANs on a NetGear ProSafe GS108E. By changing the VLAN assignment of the ports where the AT&T RG and the desired router are connected, the idea is to "trick" the Optical Network Terminator (ONT), such that it does not notice the device on the other end has changed, and thus does not require it to reauthenticate.
I attempted this method using my UniFi Switch 48, assigning 3 ports to a special "AT&T" VLAN. I planned to bring up the ONT and the RG together on the same VLAN, then once the authentication was complete, kill the RG and bring up the USG. Unfortunately, the RG never managed to authenticate. The "Broadband" light blinked red most of the time, and never acquired an IP address. I suspect the issue was that the switch was filtering the 802.1x traffic, since this is a layer 2 protocol that is intended to authenticate a device to a switch, rather than a device to another device.
Some musings in the DSLReports thread suggested that it should also be possible to use a "dumb switch" to work around this problem. If the switch is dumb enough (as is apparently the GS108E), it will forward EAPOL frames exactly the same as it would any other frame. I have an extremely cheap 5-port Tenda gigabit switch, so I tried using it instead of the UniFi switch. In this scenario, the RG did manage to authenticate to the ONT, and come online completely. Unfortunately, even with MAC address spoofing, I was not able to "swap out" the RG. When I connected Toad to the dumb switch and disconnected the RG, Toad never received a DHCP response, and using a static IP address did not work. (In hindsight, I realized that this was probably because I needed to use an 802.1Q tag to mark the outbound frames with VLAN ID 0.)
This method seemed really fragile to me, so even if I had gotten it to work, I probably would not have been happy with it. I considered using some trick like this "2 dumb switch relay" method (https://www.dslreports.com/forum/r32061385-) to automatically disconnect the USG and reconnect the RG whenever some issue caused the port to become deauthenticated. If this had been my only option, I probably would have tried harder, but instead, I decided to investigate some other options.
Attempt 2: wpa_supplicant
During my initial research for a bypass method, I noted some posts regarding
wpa_supplicant on a UniFi Dream Machine to perform the 802.1x
authentication directly. Since I had had no luck with the first couple of
options, I decided to research this option further and see if there was an
equivalent option for the UniFi Security Gateway. Sure enough, a quick Duck
Duck Go revealed a blog post specifically about that:
(Aside: the title of this person's blog, It Kinda Works initially gave me
pause, as I interpreted that to refer to this specific technique. Luckily, I
decided to read it anyway :).
I knew wpa_supplicant could be used to perform 802.1x authentication for wired interfaces, the same as it does for WPA2-Enterprise networks on wireless interfaces, so I read the post hoping for some insight into how to set that up on the USG. Sadly, it seemed like this may be difficult, since it involves "obtaining" the X.509 certificates and private keys from the RG. The author of this post advocates using eBay to obtain them, but I was not willing to pursue that avenue. Instead, I decided to research a mechanism for extracting them from the RG itself.
A quick DDG search for bgw210 certificate led me to this blog post:
https://www.dupuis.xyz/bgw210-700-root-and-certs/. The author does a
magnificent job describing the vulnerability in the RG that allows this, how
an exploit was developed, and how to use it to gain root access to the
operating system. The process itself was absolutely trivial. In only a
few minutes, I was able to exploit the vulnerability, gain root access to the
OS, and extract the
The only tool I could find to decode the
mfg.dat file and extract the
certificate and private key from it was a closed-source program called
Fortunately, the author provided a native Linux executable, so although I
would rather have inspected the code to see what it does and how it works, I
could at least use it without any hassle. Again, the process was so trivial,
I had access to everything I needed to set up my USG to perform the EAP-TLS
authentication itself, completely and permanently eliminating the AT&T RG from
I first tested this method using
wpa_supplicant on Toad, and it worked
flawlessly. Later, when I had some spare time, I went ahead and deployed it
on the USG. With some minor variation (I put the certificates and private key
wpa_supplicant binary in
wpa_supplicant.conf file in
/config, and wrote my own
/config/scripts/post-config.d/wpa_supplicant.sh because the one suggested
was written by someone who had extremely limited knowledge of shell
scripting), I was able to set it up as described in the Bypassing the AT&T
Fiber modem… blog post. The process was actually rather simple:
- Install the
- Copy the CA certificate, client certificate, and private key files
wpa_supplicantto use these to authenticate the port using EAP-TLS
wpa_supplicantstarts automatically using a script in
- Change the MAC address of the WAN port on the USG to that of the RG
- Set the WAN port to use tagged VLAN ID 0
- Reboot the USG
The only thing that bothers me about this method is using the precompiled
wpa_supplicant binary. I definitely want to come up with a process for
building it myself, preferably automatically in Jenkins, so I can keep it
Fortunately, the certificate I extracted from the RG does not expire until 2040! Hopefully sometime in 20 years, something changes…
Without the RG in the path, the next speed test was much better: http://www.dslreports.com/speedtest/65268235.
During my initial test, Toad was assigned a couple of IPv6 addresses (presumably 1 via SLAAC and 1 by DHCPv6). Once I eliminated the RG and connected the USG directly, no IPv6 addresses were assigned either to the USG or to its clients. Based on various posts on the web, I believe this is a "known issue" with AT&T. Apparently, there is a 2-week period after changing devices during which IPv6 address assignment is unavailable. This seems to be because the DHCPv6 lease is assigned with a 14-day lifetime, and only a single lease is allowed per customer connection. I will again try to get IPv6 working in a couple of weeks, then.
So far, after switching to AT&T Internet, I have not noticed any issues with my self-hosted services. I have confirmed that all of my websites (including Nextcloud, Bitwarden, Gitea, etc.) are all working, as is my Wireguard VPN.
Before I signed up with AT&T, I read the Terms of Service and the Acceptable Use Policy. Obtuse legalese notwithstanding, I did not see anything that forbade self-hosted services, so I imagine there will not be any issues going forward, especially with the crappy RG out of the way.