a serious issue with pop!_os 22.04 and esp32 dev boards — FIXED

Remember when I spoke glowingly of the latest version of Pop!_OS, 22.04? Well the bloom is now definitely off the rose. I’ve been working with the ProS3 on my MBP which means I haven’t been working with any of the ESP IDF development boards on my updated Pop!_OS Linux system. Last night I decided to fire everything back up on that system and install the Rust crates and tools for ESP embedded development via Rust. The updated Python 3.10.4 tripped me up when I tried to enable the ESP-IDF environment. That forced me to re-run the ESP-IDF installation script to rebuild the Python virtual environment it uses. I’d expected that that would happen. The biggest issue is critical: I can’t see any of the ESP development boards as tty devices. Here’s what happens when I plug one of the ESP32 development boards in via USB.

[  872.282830] usb 1-4.3: new full-speed USB device number 9 using xhci_hcd
[  872.405203] usb 1-4.3: New USB device found, idVendor=10c4, idProduct=ea60, bcdDevice= 1.00
[  872.405220] usb 1-4.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  872.405224] usb 1-4.3: Product: CP2102N USB to UART Bridge Controller
[  872.405227] usb 1-4.3: Manufacturer: Silicon Labs
[  872.405229] usb 1-4.3: SerialNumber: 448454f7dffaeb11aa2038a4c6d924ec
[  872.407305] cp210x 1-4.3:1.0: cp210x converter detected
[  872.417691] usb 1-4.3: cp210x converter now attached to ttyUSB1
[  876.147068] usb 1-4.3: usbfs: interface 0 claimed by cp210x while 'brltty' sets config #1
[  876.150583] cp210x ttyUSB1: cp210x converter now disconnected from ttyUSB1
[  876.150819] cp210x 1-4.3:1.0: device disconnected

I checked with lsusb and sure enough the developer boards as USB devices are all there. It’s whatever the kernel driver usbsf is doing that’s seemingly forcing the device ttyUSB1 to immediately be dropped. It looks like an off-by-one bug but I can’t say that with certainty. The fact it’s happening right now is really annoying me. I have both Fedora 35 in a VM with kernel 5.17 and Ubuntu 22.04 in another VM with kernel 5.15. I’ll use them to see if they exhibit the same problem. If either Ubuntu or Fedora doesn’t exhibit this issue then Pop!_OS is off this machine to be replaced by one or the other. This is a one strike and you’re outa here issue. You don’t mess with my development environment. I’ve considered buying a System76 Linux machine in the past, but if I ever do in the future then I’ll buy it with Ubuntu installed. I’ll never recommend Pop!_OS again.

Update and Near Solution

Turns out it wasn’t completely Pop!_OS’ fault. The primary culprit is an application called brltty ( https://brltty.app ), a “background process (daemon) which provides access to the Linux/Unix console (when in text mode) for a blind person using a refreshable braille display.” It was automatically installed when Pop!_OS was upgraded, and is also part of the bog standard Ubuntu 22.04 installation, which is why it wound up in Pop!_OS.

Figure 1: Output portion of dmesg

I fired up my Ubuntu 22.04 VM and allowed it to access one of the ESP32-C3 developer boards. As Figure 1 shows it failed just like it had under Pop!_OS.

I ran dmesg to look at the tail of the log, and sure enough I saw the same failure mode. That’s when I went looking for brltty and why it was running on Linux. Folks, unless you’re a blind developer, you don’t need to have this running at all.

I also tried this on my Fedora 35 VM. Because Fedora didn’t have brltty running it mapped the developer board’s USB port to a tty port just fine. I checked to see if it was installed, and it wasn’t. I hope and pray that brltty doesn’t show up in Fedora 36 or any future Fedora release.

To remove brltty from Ubuntu execute in a shell sudo apt remove brltty. This worked for Ubuntu and allowed the USB port mapping to function properly.

Update for Pop!_OS

What works for Ubuntu won’t work for Pop!_OS. Try to remove brltty and you will remove all the many bits that support the desktop, thus disabling graphical login. Instead, I had to use systemctl to shut down and full disable the service. I found these instructions on a Reddit forum:

$ sudo systemctl stop brltty-udev.service
$ sudo systemctl mask brltty-udev.service
Created symlink /etc/systemd/system/brltty-udev.service → /dev/null.
$ sudo systemctl stop brltty.service
$ sudo systemctl disable brltty.service
Unit /etc/systemd/system/brltty.service is masked, ignoring.

This disabled both parts of the brltty service (!) and allowed my ESP32 development boards to properly connect. What I’m afraid of is that any future Pop!_OS updates will re-enable the service and I’m going to have to go through this process all over again.

I’ve made up my mind. I’m waiting for Fedora 36 to be released so I can install it in one of my VMs. If it looks solid then I’m switching over to Fedora from that point forward. As I wrote about above, this isn’t a part of Fedora. One other fact that makes my blood boil is that this is a known problem that goes back to at least 2011. Over ten years of this cropping up intermittently.

what i miss in retirement

Two former work colleagues.

Friendship. No matter how bad the work environment or crazy the problems I had to face, I could always count on a core group of workmates I could talk to that would make my dark days brighter. While I most certainly enjoy retirement in general (no commuting, no sitting on my butt in an office, no constantly broken build systems, no constantly failing tests, etc, etc, etc), I miss being with those people I worked with. Last week I drove the long commute back to their side of Orlando and had a wonderful lunch with two of them. There was very little shop talk; we talked the majority of the time about families and technology and families and TV shows and even more about families. And we did it without masks (this was after all a luncheon), but we made sure to eat outside in the marvelous weather. I appreciated their taking time out to drive over and spend the time with me, and especially the companionship. They’re bright, effusive, kind folks and I’m blessed to know them. Here’s to hoping I can still maintain contact with them (and others) in the years to come.