My girlfriends and I are big coffee enjoyers, and since getting an espresso machine in 2024, we've gotten deep into trying different types of coffee beans in lattes and other drinks. So when I stumbled upon terminal.shop last summer, I figured it would be a fun gimmick and a way to try out some different beans. The process is pretty simple: SSH into terminal.shop and use your keyboard to navigate the menus and order yourself some coffee.

Running a storefront over SSH works really well! Pages load almost instantly (since they're just an 80x24 grid of characters). While you don't have the full abilities of CSS, they've made extensive use of ANSI codes to create an aesthetically pleasing interface. Unlike HTTPS, SSH doesn't have public key infrastructure -- there's no certificate authority asserting that the terminal.shop server is legitimate. However, they put their SSH public key on their website so you can verify it yourself, and once you login the first time, your client will store the fingerprint to ensure the server is legitimate on future logins.

While we placed our first order because buying something over SSH sounded too fun to pass up, we ended up really liking the decaf and dark roast. However, once we become semi-regular customers, ordering via SSH lost its initial novelty, and I wanted to try something new.

None Coffee with Left Shipping

terminal.shop provides an HTTP API that allows you to place orders. They also have a bunch of client libraries, but I found the bare HTTP documentation easier to read, so I set out to write a Bash script so I could place orders from my terminal without the hassle of a TUI. My goal was to have one command I could run any time we needed to stock up.

Unfortunately, the development backend at api.dev.terminal.shop was down when I tried to test my code, so I sighed, took out my real credit card, crossed my fingers and hit enter. I immediately got a notification that I had been charged $8 and hadn't received a confirmation email. After a bit more digging, I realized where I had gone wrong -- I had supplied the product ID of each product I wanted to buy instead of the product variant ID, causing the order to be created with an empty cart, myself to be billed $8 for shipping and $0 for my nonexistent items, and then (presumably) the backend crashing later when trying to generate my order confirmation.

Immediately after I realized this, I sent an email to support@terminal.shop. I didn't get a response and, for a time, resigned myself to the fact that I had paid $8 for an interesting story. (I don't blame them for not getting back to me, Terminal Products seems to be a side project of a few content creators and I'm sure that job can be incredibly hectic.)

Bash hacking

I patched my code and ordered again. This time, it was a success! However, something was up with the packing slip...

All four items in the shop were present in both the email and the packing slip, but the two I did not order had a quantity set to zero. This was a byproduct of how my ordering script worked: instead of adding each product to the payload when a user ordered it, I always put all four available variant IDs into the payload with a quantity set to zero unless that argument was set to zero.

#!/bin/bash
# usage: ./order.sh --segfault 1 --404 1

OBJECT_OBJECT_QTY=0
SEGFAULT_QTY=0
DARK_MODE_QTY=0
_404_QTY=0


while [[ $# -gt 0 ]]; do
  case "$1" in
    --object-object)
      OBJECT_OBJECT_QTY=$2; shift 2 ;;
    --segfault)
      SEGFAULT_QTY=$2; shift 2 ;;
    --dark-mode)
      DARK_MODE_QTY=$2; shift 2 ;;
    --404)
      _404_QTY=$2; shift 2 ;;
    *)
      echo "Unknown option: $1"; exit 1 ;;
  esac
done

# ...


ORDER_PAYLOAD=$(jq -n '{
    addressID: "'$SAVED_ADDRESS_ID'",
    cardID: "'$SAVED_CARD_ID'",
    variants: {
        "'$ITEM_OBJECT_OBJECT'": '$OBJECT_OBJECT_QTY',
        "'$ITEM_SEGFAULT'": '$SEGFAULT_QTY',
        "'$ITEM_DARK_MODE'": '$DARK_MODE_QTY',
        "'$ITEM_404'": '$_404_QTY',
    }
}')

This was a purely arbitrary choice I made since I thought the ergonomics of doing it this way was easier in Bash versus trying to conditionally add things to the JSON payload. I figured that the backend would just filter out any items with a quantity of zero before the order processed. But it seemed like those "quantity zero" items were still there!

Just how much does this endpoint allow?

An idea dawned on us. It seemed like this endpoint had pretty poor validation overall. If we placed an order for the 2 types of coffee we wanted, then ordered -1 of a coffee we didn't care about, maybe we could receive these items for less money and thus get back the $8 we lost earlier!

We were almost certain that it wouldn't work, but I ran the script again:

./order.sh --404 1 --dark-mode 1 --segfault -1

Verified the payload it created:

{
  "addressID": "shp_XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "cardID": "crd_XXXXXXXXXXXXXXXXXXXXXXXXXX",
  "variants": {
    "var_01J1JFE53306NT180RC4HGPWH8": 0,
    "var_01J1JFDMNBXB5GJCQF6C3AEBCQ": -1,
    "var_01J1JFF4D5PBGT0W2RJ7FREHRR": 1,
    "var_01J1JFEP8WXK5MKXNBTR2FJ1YC": 1
  }
}

Then sent it off, and watched the response come in:

{ "data": "ord_XXXXXXXXXXXXXXXXXXXXXXXXXX" }

We successfully created an order! Just like that, we got a notification that the card had been charged $30! Then, checking my email, I saw the order confirmation also listed payment of $30: 1 bag of 404 at $22, 1 bag of Dark Mode at $22, 0 bags of [object Object] at $0, and -1 bags of Segfault for $-22.

We waited patiently for the tracking number. Once it arrived, we checked the metadata, and the shipping label listed the package as weighing 12 oz -- the weight of one bag. It would seem, somehow, our order got corrected and we were receiving only the one bag we had paid for.

Until... a box showed up outside our apartment! I picked it up and noticed it weighed far more than a single bag. Once I opened it, I realized we had been sent all four types of coffee!

It appears the order was able to make it all the way to the human fulfilling it without being filtered out, who then probably saw the packing list, thought "oh, I guess the quantity column is messed up on this one," and packed and shipped it off.

It quickly dawned on us what we had actually done: paid $30 for $88 worth of coffee and put Terminal Products in the position of shipping a 48 oz package with a label that said 12 oz. Not ideal!

Aftermath

As it turns out, my girlfriend Ava realized that she unexpectedly shares a mutual acquaintance with some of the terminal.shop team, so word reached them pretty quickly.

Thankfully, the folks at Terminal were super chill about this and quickly patched the issue. Not only do I get to keep four times as much coffee as I expected, but we were given a reward for reporting this in the form of even more coffee. (Friends in Boston, please hit me up if you want a latte sometime!)

From what I can tell, everything exploitable via bare HTTP requests would be similar to accomplish with the official client libraries. The only "trick" was using the API, calling the /order endpoint directly instead of adding items through /cart/item, and just putting weird stuff in the variants field.

It was fun to watch the hypotheses come in once this was announced online: was it an obscure SSH feature? Something with SendEnv? Terminal control characters?

These explanations were all unlikely. You might think terminal.shop is built on a traditional server like OpenSSH, with the login shell set to a TUI program, which would thus leave it open to bugs or misconfigurations in a wide range of obscure SSH features. However, the answer is much nicer -- the terminal.shop TUI is built with Wish, a framework by Charm that allows you to create apps accessible over SSH without ever creating an actual shell. It's the same framework I used for fissh.breq.dev, a tiny app I made about a year ago with Ava that presents you with an ASCII drawing of a fish every day at 11:11.

Discovering this vulnerability was a long adventure in the making! I want to thank my girlfriends Ava and Mia for encouraging me and offering advice, AJ Stuyvenberg for getting us connected to the team at Terminal, and of course all the folks at Terminal Products for having an open and positive attitude towards security research. Sometimes, the most powerful bugs are the ones that require the least complicated exploits!

Transit and Data

Transit systems and public data are a great match. In my daily life, I interact with so many devices and applications which pull from transit data, from the LED matrix in my living room, to the app on my phone, and to the countdown clocks within the station itself. It's also incredibly accessible, too, as the myriad of DIY projects pulling in transit data demonstrates.

There are, in general, two types of data about a transit system: what the system is in theory (schedules, routes, and stations determined months in advance), and what the system is in practice (vehicle locations, arrival predictions, and dropped/added trips updated in realtime).

Transit Systems in Theory

Describing what a transit system does in theory is the easy part. The General Transit Feed Specification defines a common format made up (in true 2006 fashion) of a set of TXT files contained within a ZIP file, each describing a different aspect of the system. For instance, stops.txt describes the vehicle stops and stations in the system, stop_times.txt describes when each stop is serviced by a vehicle trip, and trips.txt can specify the train number, whether bikes are aloud, and more.

The benefit of this approach is clear: sharing a common standard means that code written for one city can work seamlessly with others. Smaller transit operators can create these files by hand, and larger ones can build up the necessary automation to handle hundreds of routes and thousands of stops. Since these ZIP files usually only change when new schedules are determined, distributing them is straightforward and storing historical ones is easy to do.

Transit Systems in Practice

Realtime data is where things get messy. The requirements are more demanding. Data usually needs to be generated and distributed without a human in the loop, and clients need to pull updates every minute or faster.

GTFS does offer a solution to this in the form of GTFS-Realtime. However, being a Google initiative, they chose to build this using Protobuf as an interchange format, which requires specific language bindings to work with. My home system, the MBTA, chose to offer a JSON version of their feeds as well.

Even still, I tend to use their excellent, well-documented service API which makes it easier to ingest only the data relevant to the lines and stations I need. Interoperability with other systems is usually not a priority for my projects.

GTFS-Realtime, however, does not specify how historical data should be represented and stored, leaving transit systems to invent bespoke formats for this data -- that is, if they choose to make it available at all.

Historical Data at the MBTA

LAMP

The MBTA has a service called LAMP (Lightweight Application for Measuring Performance), which does three things:

That second point is what we'll focus on for parsing historical realtime data.

Open Data Portal

The MBTA Open Data Portal contains lots of additional reports generated by the MBTA covering ridership, predictions accuracy, and more across the various transit modes. One such dataset is the Bus Arrival Departure Times 2025 dataset, which nicely complements the subway data published by the LAMP team.

Let's get parsing

Subway data

Data format

The subway data we're interested in is distributed in the Parquet format, using URLs like this for each day of service:

https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/YYYY-MM-DD-subway-on-time-performance-v1.parquet

The Parquet file format is an Apache project specification for storing tabular data efficiently. They pack the column data efficiently but don't require the receiver to know the schema definition like Protobuf does, which means we can easily throw these files into Pandas, the popular Python data analysis library.

import pandas as pd

path = "https://performancedata.mbta.com/lamp/subway-on-time-performance-v1/2025-10-31-subway-on-time-performance-v1.parquet"
df = pd.read_parquet(path)

with open("data.json", "w") as out:
    out.write(df.to_json(orient="records"))

We can see that the data has 27 columns. While these aren't documented anywhere, here's how I assume the data is structured:

Simulating trips

Suppose I had been on the southbound platform at Sullivan Station at 5:15 PM. When would I have made it to North Station?

Let's start by finding all the trips which arrived at North Station coming southbound.

trips_to_target = df[(df["parent_station"] == "place-north") & (df["direction"] == "South")]

Now, let's find the earliest one of those trips which also stopped at Sullivan.

trip_ids = trips_to_target["trip_id"].unique()

trips_from_start = df[
    (df["parent_station"] == "place-sull")
    & (df["direction"] == "South")
    & (df["trip_id"].isin(trip_ids))
]

Right now, most of these are the same trip IDs, but now the timestamps match up with the train's stop at Sullivan. If we were dealing with different branches of the Green Line, for instance, this step would also filter out trips which don't run between our station pair.

Finally, let's find the first trip from the start which departed after we got to the station.

Times in this data are represented as Unix timestamps (i.e., seconds since 1970), seemingly in the local timezone (U.S. Eastern time). So, for instance, the first trip in our list arrived at the station at 1761902737 seconds, or at 5:25:37 AM on 2025-10-31.

import datetime

timestamp = datetime.datetime.fromisoformat("2025-10-31 17:15:00").timestamp()
trips_after_time = trips_from_start[trips_from_start["stop_timestamp"] > timestamp]

The first of those trips is trip ID 70526030, which arrived at Sullivan at 5:27:35 PM.

next_train = trips_after_time.loc[trips_after_time["stop_timestamp"].idxmin()]

train_arrival = df[
    (df["trip_id"] == next_train["trip_id"])
    & (df["parent_station"] == "place-north")
]

Looking up its arrival into North Station, we see:

It appears we would have arrived at 5:32:02 PM.

Bus data

The bus data is a little different. The MBTA provides it as a ZIP file for each year, with a CSV file for each month. At time of writing, the latest one is MBTA-Bus-Arrival-Departure-Times_2025-09.csv.

The data is pretty straightforward, providing the following columns:

Notably, not every stop is covered in this data, only the time points (places along the route with a scheduled arrival time). However, the time points are typically placed at high-traffic stops like subway stations or the start and end of the line, so they are probably useful anyway.

Basic parsing

Let's see if we can repeat a similar "simulated journey" to what we did with the subway, but with the bus data. Suppose I'm trying to get from Harvard to Hynes Convention Center station using the 1 bus on 2025-09-20 at 11:00 AM.

Ingesting the CSV file is quite easy:

import pandas as pd

path = "MBTA_Bus_Arrival_Departure_Times_2025/MBTA-Bus-Arrival-Departure-Times_2025-09.csv"
df = pd.read_csv(path)

The first part is quite similar to simulating a subway trip. Let's try selecting all of the trips that arrived at our destination stop ID, stop 79, then find the records for those trips departing from Harvard, stop 110.

trips_to_target = df[
    (df['stop_id'] == 79)
    & (df['direction_id'] == "Inbound")
]

trip_ids = trips_to_target["half_trip_id"].unique()

trips_from_start = df[
    (df["stop_id"] == 110)
    & (df["direction_id"] == "Inbound")
    & (df["half_trip_id"].isin(trip_ids))
]

And finally, we need to filter by trips happening after our chosen start time, so we need to handle the problem of dates.

Dates and times nonsense

How do we assemble the service_date and scheduled/actual fields into an actual timestamp like we got with the subway data?

Let's start by parsing the service date.

timestamp = datetime.datetime.fromisoformat("2025-09-01")
>>> datetime.datetime(2025, 9, 1, 0, 0)

Now, let's parse the timestamp offset. Note that it is in UTC, not local time.

offset = datetime.datetime.fromisoformat("1900-01-01T11:31:00Z")
>>> datetime.datetime(1900, 1, 1, 11, 31, tzinfo=datetime.timezone.utc)

Right now, the offset is an aware datetime, meaning it contains timezone info. Combining aware datetimes with their counterparts, naive datetimes, is usually not allowed. Let's make the service date an aware datetime as well.

You might be tempted to use the astimezone method, but this will convert the naive datetime to an aware datetime assuming the naive datetime is in local time, which is not what we want -- we want to keep the year/month/day values the same, but just attach timezone information to this instance. We can use the replace method and replace the tzinfo field.

timestamp = timestamp.replace(tzinfo=datetime.UTC)
>>> datetime.datetime(2025, 9, 1, 0, 0, tzinfo=datetime.timezone.utc)

Cool. Now the only thing we need to do with the offset is subtract the placeholder date (1900-01-01). Python implements this nicely, where subtracting one datetime from another gives a timedelta object.

offset -= datetime.datetime(1900, 1, 1, tzinfo=datetime.UTC)
>>> datetime.timedelta(seconds=41460)

Now, we can add this to our service date:

timestamp += offset
>>> datetime.datetime(2025, 9, 1, 11, 31, tzinfo=datetime.timezone.utc)

And finally, convert it from UTC to our local time, then strip the timezone info to match the behavior of our other code. If this were production code, we would want to only use aware datetimes... but we're just messing around so let's do what's easy.

import zoneinfo
timestamp = timestamp.astimezone(zoneinfo.ZoneInfo("America/New_York"))
timestamp = timestamp.replace(tzinfo=None)
>>> datetime.datetime(2025, 9, 1, 7, 31)

Cool! So that trip was at 2025-09-01 at 7:31 AM.

Pulling it together

Okay, back to the show. We were trying to filter by trips happening after a given time. Let's add a column to our dataframe with a proper timestamp to match our other data.

import zoneinfo

def convert_timestamp(row):
    if not isinstance(row["actual"], str):
        return pd.NA

    timestamp = datetime.datetime.fromisoformat(row["service_date"])
    offset = datetime.datetime.fromisoformat(row["actual"])

    timestamp = timestamp.replace(tzinfo=datetime.UTC)
    offset -= datetime.datetime(1900, 1, 1, tzinfo=datetime.UTC)

    timestamp += offset
    timestamp = timestamp.astimezone(zoneinfo.ZoneInfo("America/New_York"))
    timestamp = timestamp.replace(tzinfo=None)
    return timestamp

df['timestamp'] = df.apply(convert_timestamp, axis=1)

After rebuilding trips_from_start with this new column, we can select the trips after our chosen time, choose the one that departed earliest, and see when it got to Hynes in a similar way to our subway trip simulator.

timestamp = datetime.datetime.fromisoformat("2025-09-20 11:00:00")
trips_after_time = trips_from_start[trips_from_start['timestamp'] > timestamp]
next_bus = trips_after_time.loc[trips_after_time['timestamp'].idxmin()]
bus_arrival = df[(df['half_trip_id'] == next_bus['half_trip_id']) & (df['stop_id'] == 79)]

I would've made it to Hynes at 11:24 AM. Not bad!

Simulating longer journeys

Why do we care about simulating historical journeys, you might ask? It's useful for answering a few different types of questions:

And, my favorite:

Simulating an existing speedrun

Transit system speedrunning is a phenomenon in which competitors attempt to travel through all stations within a system as quickly as possible.

My friend Tris completed a speedrun about a year ago and documented her route in a Mastodon thread. Let's see if we can accurately simulate it!

I made a few improvements to the algorithm:

For parts of the run involving walking, I set very optimistic transfer times because I can vouch for the fact that Tris walks very fast.

The code for this is on GitHub if you want to play around with it.

Not too bad! Especially considering that the "actual time" column is based on Mastodon post timestamps so probably 1-2 minutes behind the actual timing.

Generating the optimal route

My eventual plan with this is to use it to learn more about what makes an ideal MBTA speedrun, including the route, timing, etc. It's been a bucket list item of mine for a while to do a speedrun, and I want to see if I can find something new in terms of route, timing, etc. I think it's unlikely I'll find anything that substantial that could make a difference, but maybe it's possible...

Since running this type of simulation becomes extremely fast, there are a lot of places you could take this other than just tracing manually entered routes:

Takeaways

I hadn't really done an in-depth data analysis project quite like this before. I definitely learned a ton!

Ingesting data into a data structure like this is something I usually consider "grunt work," but working on this showed me how wrong that assumption can be. I tried to throw a lot of this work to an LLM, only to watch it struggle against the date and timezone issues and completely miss the nuance between stop_timestamp and move_timestamp. While I got much further than it did (thankfully for the sake of my job security), it still required me to step away from the problem for a day before I could nail the accuracy.

I had also heard people constantly talk about how timezones are hard and date parsing is a mess, but had been spared the brunt of that struggle until now. I have discovered that there are a lot of bad ways to represent dates and times in software. I feel lucky that I do not need to deal with things like this more often.

The last thing I'll mention is that this project highlighted the difference between working with data in a domain where a dominant format exists (i.e., realtime GTFS), and working in a domain where implementors have no common format to use (i.e., historical transit data). I find it easy to get annoyed at format specifications not perfectly matching what I want to do... but when standardized formats work, they're pretty great!

I've built several projects based on LED matrices: my first one in 2021, and a redesign earlier in 2025 with a larger panel and slimmer enclosure. I've also gotten to spend lots of time fine-tuning the display UI as I learn more about icon design, balancing readability and "glanceability" with information density, and dealing with the unique constraints of a display with a 3mm pixel pitch.

Motivation

While the hardware was a dramatic step up from my first iteration, I still thought there was room for improvement: more robust mounting of the encoder and the DC power jack, a slimmer construction that could fit closer to the wall, and simpler internal wiring. The obvious next step was a custom PCB. This project came at a time when I was hoping to get back into PCB design, so this seemed like the perfect opportunity!

At the same time, I had relatively recently started full-time work and wanted to add some decorations to my desk area. Since my previous builds had proven extremely useful at indicating bus times and bike availability, I figured that one on my work desk would be similarly useful! Of course, this one had slightly different requirements, so I ended up building a single board and two very different enclosures for it.

PCB Design

The schematic of the board is very similar to that of the previous build, with a few changes:

The layout followed pretty naturally. I kept the board at 192mm wide to match either a 64-column 3mm-pitch display or a 32-column 6mm-pitch display, gave it rounded bottom corners to match the style of the previous build, and mounted the encoder at the center on one side.

Shameless plug: The renders in this section were made in my KiCAD SVG prettifier tool.

While I like the enclosed build in my apartment, I wanted to leave the option of doing a build with an exposed PCB. The components are all mounted towards the back, and the routing was mostly done only on the back side, leaving the front part of the board mostly empty on both the silkscreen and copper layers. I thought about putting some artwork or text on the silkscreen but couldn't think of a good way to make it look professional -- the usable space on the board is asymmetrical and interrupted by through-hole components.

In the end, I decided to put a repeating pattern in the copper layer of the board, then order it with OSHPark's After Dark colorway, which uses transparent soldermask to let the copper design shine through. I wanted an angular style to match the vibes of the rigid exposed pixel grid, but worried that an actual grid pattern could clash or appear misaligned. Thus, I chose a hexagonal pattern.

Building and importing this pattern turned into more of an adventure than I had predicted! The process I found is convoluted and by no means ideal. If this were something I was doing more often I would definitely design an automated tool for it, but as it stands, the best way I know to accomplish this is:

At my girlfriend Mia's suggestion, I added a photoresistor and ADC to the circuit to implement automatic brightness control. I haven't gotten around to using this yet, mainly because the build at my work desk does not require it and the enclosure design of the one at home blocks outside light. In the future I think I could add a small hole to the bottom of the design which would let light into the enclosure but not be too visible.

PCB Assembly

This was my first time doing surface-mount soldering! I kept the components manageable (SOIC chips and 0805 passives). Overall I found the experience much easier than I anticipated. Big thanks to Mia for giving very helpful advice throughout. Thankfully, the board worked on the first try!

Enclosure

For the build sitting at my work desk, I wanted to build something that showed off the internals while still appearing sturdy and thoughtfully designed. I went with two small 3D-printed parts on opposite sides of the assembly which attach to both the PCB and the panel, allowing the device to be freestanding on a desk or table.

For the one in my apartment, I wanted to continue the theme of building something as unobtrusive as possible. Since the PCB was now the same width of the panel, if I wanted to enclose the PCB fully, the enclosure would also need to be a few millimeters wider than the panel. I took this opportunity to recess the panel into the enclosure, giving the system a much more polished look from the side view.

I am not super happy with the large gaps in the design caused by the imprecision of my 3D printer. I do like that 3D printing can give a brightly colored end result, and the form of the parts seems to lend itself well to a 3D-printed design over something like CNC milling. Perhaps I just need to get access to a better 3D printer.

The knob was given to me by a coworker who happened to be getting rid of it. It fits my encoder shaft perfectly! It is a bit taller than I would like but that is mostly a function of how much the shaft protrudes. In the future I will get a shorter encoder shaft or find a way to mount the encoder farther back. I unfortunately can't move the entire PCB back without increasing the thickness of the system because of the height of the 2x8 connector going to the LED panel.

One thing I do love about the new design is that it sits dramatically closer to the wall, making it match the thickness of photos and artwork I have. Things have come a long way from the ~50mm thickness of the original version!

Software

The largest changes on the software standpoint were extending the previous version to support multiple display sizes and moving a lot of the hardcoded logic (ZIP codes, etc) into a configuration file so my display at work could show different information to my display at home.

I hacked together a barely-working implementation to test the panel which Ava kindly refactored into something much nicer.

Conclusion

I think time has shown that I'm unlikely to stop building this kind of display anytime soon. They serve a really interesting niche of a device that provides basic information at a glance without being obtrusive, and have a unique and interesting look.

I really enjoyed getting back into PCB design and finally conquering my fears regarding surface-mount work. I hadn't actually designed and gotten a board fabricated since I was in high school, more than 5 years ago! It was nice to exercise that skill again. SMT soldering, on the other hand, is something I had never attempted before, being scared off by my general clumsiness and the precision and tools often recommended for that kind of work. While I am very glad I bought flux at the recommendation of a friend, I found it to be smooth sailing even without new tips for my iron. Perhaps I'll invest in some better tools in the future.

I loved getting to design this iteration with a focus on making it easy to potentially build more in the future. The board came out quite nice, and while there are definitely some things I would change in the future (mostly relating to making the connectors protrude farther out of the case), I am overall quite happy with the results!

Throughout this process, several few people have told me that they would love to build something similar to this for their home. I've considered a few times trying to scale up production, but the parts cost (mostly the panel itself and the Raspberry Pi) makes the total cost of each unit pretty high. There are a few companies making something similar (Tidbyt and RideOnTime are the two I've seen), but they both use small panels in a thick enclosure style like I did in the first iteration of this project. I've grown to really appreciate the large 64x64 panel size as an option, a flat enclosure, and a design more focused on information density.

I don't want to start a business selling these or anything, but I could see myself creating better documentation of what hardware to buy, making the software easier to install and use, and organizing a group buy of boards and parts. Shoot me a message if that sounds interesting to you!

I've owned a 3D printer for about ten years now. I got my first 3D printer, a Monoprice MP Select Mini, when I was in middle school as the 3D printing craze was beginning to take off in "maker" culture. My dad likened it to buying his first computer as a kid. But while the home computer was on the verge of an explosion in popularity, the home 3D printer is still uncommon. The machines have undoubtedly gotten better, but FDM (fused deposition modeling) 3D printers still struggle to find broad appeal beyond the same group of tinkerers who were using them years ago.

Disclaimer: My employer works in the field of 3D printing, but that is neither FDM nor for personal use!

My journey with FDM

My MP Select Mini held up okay over quite a few years. The hotend was quite prone to clogs and other issues, so I replaced it with a generic upgrade kit one after printing an adapter I found online. Eventually the mainboard started to give up.

I now own an Ender 3 Pro printer that I impulse-bought at Micro Center a few years ago for $99. It works... okay. I've replaced the bed and upgraded to a metal extruder kit. The Z axis still misbehaves a lot, leading occasionally to layers that are too short (causing nozzle drag). I could try replacing the stepper motor, but I don't feel like paying $25 just to try something that might not be the actual fix.

Through my work on the Northeastern Mars Rover Team, I've gotten to work with slightly better printers from Bambu and Prusa. They're generally nicer to get working, but they still experience frequent issues and require consistent maintenance. They're much closer to a tool you'd find in a machine shop than an appliance you'd find in a home.

I've printed plenty of parts over the years, some more useful than others. Owning a 3D printer has enabled plenty of projects that would otherwise be impossible without a more substantial workshop, like my LED wall sign and its sequel, the custom mini-ITX case I made, and more. That said, it requires consistent maintenance to keep working, frequent retries slowing down the process, and produces lots of scrap.

While I'm happy to print models for my friends, few take advantage of this -- most people just do not reach for 3D printing when they encounter a problem.

STL websites

Broadly speaking, most personal 3D printing use cases can be divided into categories: printing existing STL models from sites like Thingiverse and Printables, and doing CAD work to produce unique parts.

The culture of sharing STLs within the hobbyist 3D printing community is huge. It's trivially easy to download a model from a website, drag and drop it into your slicer, and export G-Code to an SD card for your printer. However, in my experience, most of the models available on these sites are not particularly useful. The vast majority are either decorative parts, or utilitarian parts like headphone stands, shelving and organization products, and cable management parts.

In most cases, commercially bought parts are available at similar cost and with dramatically better build quality, surface finish, and durability. In some cases it is even faster to go to a physical store to buy a product than to run an FDM printer for hours.

I can't speak as much to printing artistic models or figurines, since that aspect never particularly appealed to me. Without postprocessing, FDM 3D prints just don't look that great.

There is undoubtedly a sweet spot here: items that are popular enough that you're likely to find an existing model, but not popular enough to warrant that product being commercially produced. Mounting brackets for specific network switches and similar parts sometimes fall into this category. But the bulk of the time I've searched for such a part, 3D models haven't been available at the usual places, and I've had to make it myself.

CAD software

A print I recently designed and made for organizing floppy disks for my Mavica.

In my experience, without an understanding of some form of CAD software, owning a 3D printer is of limited utility. Contrast this with the personal computer: most people who use computers on a daily basis do not know how to program! The typical user can meet most or all of their needs by downloading and utilizing existing software.

I was lucky to learn CAD software in middle school as part of "Technology and Engineering" class, in the golden age where SketchUp was the teaching software of choice instead of TinkerCAD. SketchUp undoubtedly has limitations (I still don't know how to draw a sphere), and the mechanical engineers I've worked with have generally laughed at me for using it. That said, I can reasonably efficiently build things.

Most people I know do not have the motivation or time to learn CAD software just to design and produce one or two parts every few months.

CAD takes a while, too! Designing good parts requires thinking about the direction of layer lines, need for supports (or lack thereof), balance between strength and filament usage/time, and more, on top of just constructing a part that fits the desired functionality and looks pleasant.

Can we make CAD more accessible somehow? The advent of tools like TinkerCAD have certainly helped, but building useful parts within its constraints can be difficult. Maybe generative AI can democratize building shitty 3D models in the same way it democratized building shitty software? The attempts I've seen at extending generative AI to CAD haven't been great.

The whole promise of 3D printing is customized parts built at much faster timescales and with less equipment than otherwise possible. But the typical home user only has the occasional need for a customized part. I use my printer about once a month, sometimes less, mostly because of the time investment of CAD.

Fitting into the home

My 3D printer sitting atop an IKEA KALLAX. It is quite big!

Will 3D printers ever be common home appliances?

One could argue that the limited utility of them precludes that being a possibility. But the success of Cricut machines proves that "maker"-oriented devices for the home can be successful! The use cases for the Cricut are similar to those of an FDM 3D printer, they are about a similar size, and the process of using one is similar. But Cricut machines require far less tuning and setup, making useful things on a Cricut requires much less specialized knowledge, and I would guess that Cricut designs have a higher success rate on average than 3D prints.

I think the type of community surrounding hobbyist 3D printing also plays a role. 3D printing fans are often willing to tinker with their printers, want the ability to customize them, and want to bring their own slicer and tooling. That's great for consumer advocacy, as evidenced by the failure of RFID-chipped vendor-locked filament among all but the most obscure printers. But it tends to direct the innovation in the 3D printing space towards chasing ultimate customization and featureset instead of building a cohesive user experience like Cricut has. I recognize that buying cheap hobbyist-oriented printers makes me part of this problem, but ultimately, that's the type of printer that fits most home users' budgets.

I really love owning a 3D printer! And I really wish I got to use it more. I am a huge fan of customizing my living space to fit my life with objects that exactly meet my needs, fit my aesthetic, and work with the other objects in my home. 3D printing allows me to do this, and I think that's wonderful.

I bought a Sony Mavica! And everything about this camera has been such a joy, I can't help but write about it :)

The Digital Mavica line of cameras saved images directly onto a floppy disk. The model I own is the MVC-FD90 which was released in 2000. It captures images at a maximum resolution of 1280×960, but can also shoot at 640×480 to save space on the floppy disk (which can store around 20 photos at that resolution).

Inspiration

Earlier this year at Anthro New England, my girlfriend Mia was taking photos on her Mavica and let me try it a bit! It seemed like such a fun and whimsical camera to keep around.

While I was intrigued by the camera, I was a bit intimidated by the floppy disks -- Mia has much more experience with retrocomputing than I do, and I worried that getting disks, a drive, pulling images off the camera, and keeping everything in good shape might be too difficult for me.

Swapfest my beloved

It was at the most recent MIT Swapfest that I picked up my beloved Mavica. I purchased it from some very persuasive transfems who had a table dedicated exclusively to Mavica stuff. I bought the model they recommended and they helped me get set up with a battery, strap, etc. I figured that, now that I'm getting settled into post-college life, now was a good time for me to pick up something new!

I'm a big fan of Swapfest: it's almost more of a community gathering to me than a place to buy stuff. Whenever I go I see so many friends, former classmates, and colleagues I know from work. You get to know the tables that come back every month and the people who sell regularly. All in all, it's a great way to spend a Sunday morning!

Meticulous unscheduled disassembly

Mavica experts will know that aftermarket batteries tend to be a bit oversized compared to the originals. The folks who sold me mine made sure I was aware of this and even gave me a ribbon to put around the battery. However, in my excitement to try it out, I totally forgot about this and immediately put the battery directly in. I had to disassemble the camera to get it out!

Big thanks to Sony for putting that plastic cover over the flash capacitor... that could've been a nasty shock otherwise 😬

My first disk

A few tables at Swapfest hand out free floppy disks to passerby. I had accumulated a total of three disks and found that, miraculously, one of them actually worked! I had a mild panic moment when I realized the camera couldn't always successfully read all of the photos on the disk, it turned out to be easily readable by a USB drive.

At the time, I didn't have any way to read files off of the disk, so I just took as many as I could until it filled up.

Getting serious

By this time, I had had enough fun with the camera that I figured I should actually get ahold of some disks and a floppy reader!

Mia advised against me buying a random floppy drive on Amazon, so at the recommendation of my friend Tris, I bought a Dell Latitude drive online for about $6. It's designed to be put into a slot on the laptop, but it actually just has a normal Mini-USB connector. I bought some disks from an Amazon listing that looked legit (it was Maxell brand and had good reviews).

The disks arrived just in time for me to take them on a trip to New York City!

Data recovery adventures

After getting back from NYC and finally being able to read the disks, I discovered that a few of the photos weren't readable. This wasn't a big deal (the photos I cared most about were fine), but it definitely taught me to be less careless with floppies. My backpack has dozens of tiny magnets in it and I struggled to find a spot far enough away from all of them!

Mia recommended I use ddrescue to try to recover parts of the images. It ended up not giving me anything more than I got from copying files over, but was interesting to learn. I could definitely see it coming in handy in the future.

sudo ddrescue /dev/sda hdimage mapfile
sudo mkdir /tmp/hdimage
sudo mount -o loop hdimage /tmp/hdimage
cp /tmp/hdimage/* .
sudo umount /tmp/hdimage

To reformat the disks, I used a tool called ufiformat to re-lay the tracks and then used the Mavica's built-in disk format tool to recreate the filesystem.

sudo ufiformat /dev/sda

More to come

Thus far I'm really enjoying shooting on the Mavica! Limiting myself to a specific number of shots per disk helps me be more thoughtful than with a modern digital camera, but unlike with film, if I take a bad shot I can just delete the file -- I haven't wasted anything.

Dealing with floppy disks as a medium is much more fun and less stressful than I anticipated! The disks themselves are cheap, devices for reading and writing them are cheap, software support is quite good even on modern machines, and with a little care they're easy enough to keep in working order.

I've been in a bit of a lull with photography lately, but I think I might've found something to get me back into it!

I've noticed that I can be far too quick to assume that the experience of others matches my own. This is one of the reasons I've held off on writing about being transgender fo so long: I figured I would just be telling my transgender audience things they already know. But over the past four years, I've realized that the way I experience gender has substantial differences to the way every other trans woman I've talked to experiences it, just as every trans woman I know has a unique and deeply personal experience compared to any other. I am to highlight this broad range of experiences when I portray my own.

The other reason I have held off on discussing my experience is that I often mistake the hesitance of my cisgender friends to ask questions as apathy. While I know many other transgender people are less open about their experience (which is understandable!), I have always been overjoyed to have conversations about gender and my existence as a trans woman with the cis people in my life. I hope this post can lay out the groundwork and serve as an invitation for people I know to start these conversations with me.

My wonderful girlfriend Mia has also written a blog series about being transgender, which you can find here. You should read that one after this -- I always look up to her wisdom when it comes to big topics like this :)

Update: My girlfriend Ava and my friend Tris have also written very insightful posts about this!

Early Transition

A commonality between most transgender people I know is that our decision to transition is defined by strong emotions. Deciding to transition is deciding between something very known -- your everyday life, as it is now -- and something entirely unknown. It is impossible to know beforehand what emotions you will experience after coming out. Nobody can tell you with perfect accuracy if transitioning is something that would make you happier.

The only way to know for certain is to try it in a truly immersive sense: name, pronouns, and the way you present yourself in every aspect of life. Unfortunately, we live in a society where this is often somewhere on the scale of impractical to impossible. And thus, most trans people are going in blind.

In my case, the strong emotion that determined my decision to transition was fear. Not fear of stepping into this unknown experience, but fear of not doing so. I was depressed in the years leading up to the start of my transition (for me, the last two years of high school). Had I not transitioned, I may not have made it to here today. Despite being a complete unknown, transition felt like my best chance of survival.

While I know my story is far from unique, it brings me happiness that there are those for which this decision is led primarily by positive emotions, too.

I chose to delay my social transition until starting college since moving cities let me avoid or postpone coming out to people I already knew. While diving headfirst into being "Brooke, she/her" was overwhelming at times, it was better than having to hold a "coming out" discussion with everyone in my life at the time.

It is a cruel irony that one of the most difficult parts of transition -- having to explain to everyone in your life what being transgender is and answer countless questions about something so deeply personal to you -- happens so early on in the process, before you have had a chance to build confidence in yourself or even grow to understand yourself. It makes me happy to see communities in which gender fluidity and questioning are more widely accepted and supported.

For the first year of my transition, I honestly didn't have the energy to think critically about my own experience of gender. I knew that presenting femininity felt nice, but the details didn't matter -- I was too focused on learning how to buy clothes that I like (and throwing out everything I knew about how clothes were "supposed to" fit), learning how to take care of long hair, and overcoming my fears, one step at a time.

I take my confidence for granted nowadays, but I still vividly remember my first time leaving my dorm room wearing a skirt. I remember noticing how much my hands were shaking when reaching for the elevator button, and trying to make myself small in any way possible. I am six feet tall, and while I enjoy being loud and confident nowadays, there was a long stretch of time when I just wanted to blend into the background.

Gaining confidence

The next year of my transition is when I finally started to feel like I could "take up space" in social gatherings. I dyed my hair bright pink for the first time and started to assert myself more. I finally started to feel okay with using the women's restroom instead of walking halfway across campus for a gender-neutral one.

While I made a few false starts at voice training, over time I realized that I had become comfortable with my voice as it is without deliberate training. While I'm less likely to be gendered correctly by strangers in public, I can still feel safe in that my friends, peers, and coworkers will not see it as evidence that I am any less of a woman. While I'm not ruling out voice training in the future, it would be on my terms.

My goal in transition is not to assimilate into cisnormative society. It is to wake up each morning and present myself to the world in a way that sparks joy.

Being transgender does not define my life nowadays to the extent it used to a few years ago. On one hand, the day-to-day stresses of my early transition years have subsided, and I am much happier as a result. On the other hand, I do think my trans-ness -- my experience of transitioning, my presence in transgender social circles, and the unique perspective I have as a result -- is one of the most notable pieces of my life.

I would say that in my day to day life, I am a very happy person. I would also say that my trans-ness is an aspect of my life that brings me happiness. This would have been unfathomable to my past self.

Finding stability

The word "transition" implies that the process of changing one's gender happens over a well-defined time interval. This is, in my eyes, inaccurate.

In one sense, my transition "ended" when I made the commitment to myself that I was a woman. The social steps -- changing the way I dressed, coming out to friends and family, updating my legal name -- were all just logistics.

In another sense, transition is never "done." As my body cannot naturally produce estrogen, I will keep taking hormone replacement therapy for as long as I live. The style of clothing I wear will continue to shift throughout my life. There are still people who knew me before transition and who I will need to come out to if we ever cross paths again.

It is only recently that I have felt stable enough to reconsider some of the fundamental questions of my transition. I enjoy the way I dress, the way people refer to me, and the effects that feminizing HRT brings to my body. But the label of "transgender woman" was less something I decided on and more something that was handed to me as a result of this.

I've begun to question whether "non-binary" describes the way I experience gender. I've always felt a bit of dissonance with being referred to as a woman, and for the longest time, I wrote that off as imposter syndrome. But four years in, I'm as confident as can be, and that feeling is still there.

Perhaps this questioning could only happen this many years in. It's taken time for "they/them" pronouns to stop feeling like a way for people to avoid acknowledging my transition and gender, and start feeling like them acknowledging the complexities of my gender beyond the binary.

Alternatively, perhaps this was down to a lack of representation of non-binary people in my social groups. While early in transition, I sought out friends with similar experiences to me, and ended up in groups primarily consisting of binary trans women. At present, several close friends of mine are non-binary, and I've been able to have many more interesting conversations about gender with people outside the gender binary. These conversations have undoubtedly been a catalyst for questioning my own gender identity.

Conclusion

There has been an ongoing bit on this website since three years ago in which, at some random probability, the pronouns displayed on the homepage will be either "she/her" or "she/they". I wish I could remember what was in my head when I wrote that.

One explanation is that labels have always felt unimportant in relation to my own experience. While I recognize how powerful labels can be in helping people explain their identity and connect to others, I find their imperfect fit frustrating. I care less about whether I am called "she" or "they," and far more about the motivations of those calling me this. Am I being called "they" in an attempt by the speaker to avoid recognizing or legitimizing my identity as a woman? Or am I being called "they" in recognition of the fact that my identity does not perfectly map onto the gender binary?

I hope that in five years I can re-read this post and find myself disagreeing with parts of it. I hope that my perception of myself, of transness, and of the way I fit into society will shift over time. I want to continue thinking about, exploring, and experimenting with my own sense of gender identity.

I have very thick, curly, bleached hair, which is about as "hard mode" as you can get. I have gotten so much bad advice from people about how to care for it. After a few years and some tips from friends, here's the system I've ended up with!

I've added products I use in the toggle elements below, since I think providing specific examples can be useful, but I don't want this to come across as too much of an endorsement. I know enough about this topic to solve problems, but I'm not 100% sure I've landed on the best set of products for my hair. You, dear reader, likely have different hair with different needs anyway.

Last year at the University Rover Challenge. I've grown out my hair quite a bit since then!

Requirements

To give some context, here are the requirements I have for my hair routine:

Regular maintenance

Whenever I shower, there are two main hair-related tasks I focus on. Detangling is the endless struggle I face, and what the majority of my routine is built around. Washing ends up essentially being a by-product of the detangling process, not the other way around.

I generally almost never use shampoo! My regular process consists of two steps: a washing conditioner (aka "cowash") and a leave-in conditioner. I first add the washing conditioner at the roots of my hair, scrub it in, let it sit while I wash my body, loosely detangle my hair with my fingers, then rinse it out. Then, I add leave-in conditioner, focusing on the roots of my hair, and use a wide-tooth comb to continue detangling.

Often when I mention that I don't use shampoo, the response I hear is "my hair is so oily, I could never," and this is often valid! I am lucky that oily hair is a problem that I do not face, and I find cowash to help with tangles much more, so it is the solution I use.

Less frequent maintenance

Very occasionally, I will use shampoo if I need to clean my hair more vigorously than washing conditioner allows. I've heard that sulfate-free types are better for "low-shampoo" routines like mine, since they don't leave oils as much.

I do also have a styling cream I can use, but I don't find myself reaching for it very much.

Coloring

Me in 2021 at the start of college -- I looked pretty different at the time!

Unfortunately, bright pink is not the natural color of my hair, so I need to dye it every month or so to keep it colorful!

I dye my hair DIY-style, for two main reasons. The most obvious is that I started off with hair dye as a student -- I can afford to get it professionally dyed now, but I couldn't back then. But more importantly, I view it as a really fun activity to do with people I'm close with. The joy of doing it with someone I know vastly outweighs the somewhat inconsistent results I get :)

I don't always use the same shade of dye each time, but the resulting color stays pretty consistent since it blends with the residual color from prior applications.

Every few dye sessions, I touch-up bleach the roots of my hair. The first initial bleach was a massive ordeal that took three boxes, but since then, I've just used one box every few months.

I also dye my girlfriend Ava's hair with the same setup.

An older shade of dye I previously used. Picture from 2023.

I use a semi-permanent dye (doesn't require developer) since it's much less stressful and easier to apply, and it seems to last as long as the "permanent" ones I've used. (I've accidentally fallen asleep with hair dye in and been fine!) I used to use a boxed "permanent" dye (and even once went on a 2+ hour bus journey into the a CVS in the Boston suburbs when the chain stopped stopped carrying it just so I could buy up the remaining stock), but in hindsight I didn't like the color it gave as much.

Color conditioner

A few people I know have recommended creating a color conditioner by combining my conditioner with a few drops of hair dye. I'm interested in this idea to supplement my regular re-dye process, but there are a few reasons I've held off:

That said, I will probably still try this at some point once the time is right!

Travel

Me this summer traveling for the University Rover Challenge.

I take a lot of overnight or weekend trips, and I like not having to worry about having my favorite hair stuff with me, so I keep my hair products in 3oz travel containers. It's also quite helpful for fitting all of the containers onto limited shower shelf space, and for dispensing small amounts of cowash (since mine comes in a tub).

I haven't found a good solution for labeling these yet, and I'm worried that I'll eventually mistake two similar-looking products, but so far I've been getting by with occasionally re-writing labels on them with sharpie.

Conclusion

This system has worked for me for the past ~4 years! I think it balances my needs well and I'm proud that I've found something that works.

A recent-ish picture of me having fun in the Rover lab, with a slightly oranger shade of dye.

I've developed a strong appreciation for small devices that solve problems for me. While larger versions of these tools exist, there are a few reasons why these tiny tools hold a special place in my heart.

The first is that I have spent the last several years living in small dorm rooms and now live in a relatively small apartment. While I wish I had the space for a dedicated electronics workbench, the practical reality is that my tools need to fit either on my normal-sized desk or be stored away in a drawer. The second is that, both for my work with the Northeastern Mars Rover Team and for my own projects, I do occasionally end up needing to fix, debug, or bodge something together in locations which are less conducive to this type of work (such as the middle of a grassy field).

The criteria I decided on for devices in this category include:

Acknowledgements: This post was loosely inspired by my friend Hunter's Devices of All Time post. Many of these devices were gifts from close friends!

iFixit Moray Driver Kit

Gift from Ari and Hunter!

This is a screwdriver kit including a small screwdriver and a large collection of bits. It comes in a hard plastic case, the lid of which doubles as a tray for storing screws.

I've used the Moray's older sibling, the Mako, extensively on the rover team, but haven't had my own iFixit kit until recently! I like to flip the bits I'm using for a project down to visually distinguish them from the rest and make them easier to grab.

Likes:

Dislikes:

FNIRSI DPS-150 DC Power Supply

Gift from Mia!

This is a benchtop DC power supply, which is an essential tool for electronics prototyping. This type of power supply is useful as it precise voltage control and current monitoring and limiting to protect your circuit. It provides power through "banana plugs" or directly to wires via the two terminals.

Likes:

Dislikes:

GL.iNet Opal (AC1200) Router

This is a "travel router": A small, lightweight router and wireless access point powered over USB-C. This one in particular is quite cheap and actually ran my home network for a bit until I was able to set up the pfSense unit I use now. It's also saved the day at Rover events a few times when I've used it to establish a wireless network in areas otherwise slightly too far from a nearby building's coverage.

Likes:

Dislikes:

RadioShack 22-182 Digital Multimeter

This was a gift from... my dad probably? I've had it as long as I can remember.

This is a multimeter: a device that measures the voltage difference between two points, the current flowing through itself, or the resistance of an electrical component. While some meters support "autoranging," this one requires that you specify the range of values you expect the quantity you measure to be in (e.g. 200V, 20V, or 2V). I honestly prefer manual ranging meters, since dialing in the voltage at the start is usually easy to do and the meter responds more quickly when it doesn't need to go through the autoranging sequence.

Likes:

Dislikes:

Pinecil V2

Gift from Ari!

This is a soldering iron: a tool that heats its tip up to a high enough temperature to melt solder (metal) and join electrical wires. Other potential uses include pushing heat-set metal inserts into plastic parts. This is a temperature-controlled iron, which is essential to allow heating solder efficiently without damaging components from excessive heat.

Likes:

Dislikes:

AIOC (All-in-One Cable)

This is from a run made by Mia, although I believe I paid her back for it? Mia if you're reading this and I owe you money, please let me know :)

This is a device with two purposes: to upload configuration data ("codeplugs") to handheld radios, and to connect a handheld radio as a soundcard to a computer. The former use case is nice for programming radio channels as it allows configurations to be easily created and distributed. The latter use case enables computer modulation techniques such as AX.25.

Likes:

Dislikes:

Conclusion

You'll notice I didn't put direct links to where to buy any of these (and some of them, like that Radio Shack multimeter, are likely impossible to get these days). My intent is to show how these tools fit the style of work that I do and enable me in ways traditional alternatives don't, and to maybe help you consider how the tools you use in your work shape the projects you take on and the methods you use.

I recently reorganized all of my home networking hardware into a 10" rack that sits in an IKEA KALLAX shelf next to my desk.

Why Mini-Rack?

I was motivated to take this project on by Jeff Geerling's Project Mini Rack website, which collects links to hardware and guides for rackmount builds in the 10" form factor. I've wanted to build out a rackmount system for ages, partly because I want a rigid frame for my networking hardware (as opposed to a loose collection of boxes sitting atop a shelf), and partly because I love the aesthetics of rackmount gear. While I could undoubtedly achieve the same goals in this project without using a rack at much less cost, I've wanted to play with something in the rackmount form factor for years. I had largely written it off as impractical given the small size of the apartment, but the proliferation of 10" gear has made it possible to set up a compact rackmount homelab system.

Goals and Improvements

My existing home network setup is designed for a small apartment which I share with my girlfriend Ava. We get internet service from a cable connection (no fiber here unfortunately), and we bought a basic Arris SURFboard modem. From here, we use a Netgate pfSense router gifted by my other girlfriend Ari, which also runs a site-to-site Wireguard VPN connecting to her home network. The router connects to a Ubiquiti U6 Mesh access point which both provides our personal Wi-Fi network and functions as a working eduroam service provider with client isolation (the details of which will remain a mystery).

The wired side of the network runs through a Netgear GS105 switch which I got at MIT Swapfest for five bucks. (My switch was actually, unbeknownst to me, featured on my friend Hunter's blog last year.) From here, wired connections run to my desktop and Ava's desktop.

My primary goal for this project was to reduce the clutter on my shelf and replace it with something that looked cool. I also wanted to finally have a good way to self-host things again. While I'm happy to keep most services hosted online, it would be nice to have some services running locally where convenient.

Networking Setup

I chose a RackMate T0 case, since it would fit within a single shelf of the IKEA KALLAX next to my desk (and because the next size up was 8U and I doubted I would have enough hardware to justify that). The case arrived with a broken top acrylic panel, but the folks at GeeekPi (DeskPi) shipped me a replacement quickly without any hassle! In the meantime, I got to start building out the rest of the rack.

No rack is complete without a patch panel, and mine needed some way to pass cables through from the front-mounted Ethernet ports on my router and network switch. I chose a 12-port model from DeskPi, designed for the rack I used. This provides more ports than I need for my modem connection, access point, and for our desktops, but the Ethernet ports can be swapped for any keystone block in case I want to use them for any other purpose.

One problem I wanted to solve with this design was the bundle of power bricks tangled up behind my shelf. I saw the DeskPi PDU Lite available, and decided to use it in combination with a 12V 8A power supply to power devices in the rack. Right now, it powers my modem, router, and GS105 switch, but there are 4 additional channels which could be used for an SBC, hard drive mount, etc.

Proxmox

After using the rack for some time, I finally broke down and spent the money on an actual server. I went with a refurbished Lenovo ThinkCentre M710q that I bought for around $75, since it had relatively modern hardware, would fit within a 10" 1U shelf, and uses relatively low power.

A few friends recommended I try Proxmox for this project, so I decided to give it a try! It supports full VMs as well as containers, which is nice. Previously I've relied on solutions like Dokku which only support container-level virtualization.

UniFi Network Controller

Shortly after I moved, I invested in a Ubiquiti access point for the new apartment, and I absolutely do not regret my decision. However, this AP has one minor annoyance for home use compared to a traditional residential access point. Instead of being configured via a web UI running locally on the device, Ubiquiti APs are configured by a central server. This approach is great for managing a large deployment of access points but slightly annoying for a home setup.

However, there's an upside: Ubiquiti's controller software can be self-hosted rather easily! I decided to do this first, and after spending 5 minutes trying to find where to install container templates from, I got the Unifi controller running on an Ubuntu container. After exporting and importing my config to transfer it to the new controller, I can finally manage access point settings without needing to boot my desktop!

HomeAssistant

Ava is a big fan of smart home devices, so we've ended up with a dozen smart light bulbs in the apartment. While running these through Google Home works great for basic usage, the lack of an extensible API is something we've missed dearly. Our home has already started to collect weird and wonderful homemade devices, and an open platform would give us much greater opportunity for fun shenanigans.

On the recommendation of a friend, we're trying out HomeAssistant to run our devices. Most of our devices are based on the Matter standard, which should (in theory) enable fully local control. The process of importing Matter devices went pretty smoothly, although we needed to tediously rename each device we added. We do have a few weird bulbs we bought to fit in an IKEA fixture that used the Tuya app, but HomeAssistant had an integration for those as well which seems to work perfectly.

Reverse Proxy

With all of these services on the same box, I'll definitely need to set up a reverse proxy for these services. My go-to for this is usually nginx, but I decided to try Caddy instead to avoid the hassle of setting up HTTPS certificates. While I did find it more annoying to debug, adding new hosts and proxying new services is turning out to be a breeze so far!

The reverse proxy also serves a basic static site, mostly as an excuse for me to make a cute drawing of the rack and my desktop.

Syncthing

For my "hot workspace" of files (documents, code, etc), I like to store things in Syncthing so they're available on each of my devices. It's the best cross-platform tool for file sync that I've found, and is really easy to self-host!

I created a container and followed the Syncthing setup steps for Ubuntu. The only snag I hit was that I couldn't remotely login to set up the GUI from a different device. Turns out the approach is to edit the config to replace 127.0.0.1 with 0.0.0.0, which is pretty straightforward:

nano /root/.local/state/syncthing/config.xml

The documentation for setting it up to run with systemd also worked perfectly.

NAS

I tend to keep only actively used files on Syncthing since my devices are constantly running out of storage space. For long term storage, Ava and I used to use NextCloud, but found it annoying to keep it running and stable. Thus, we switched to a basic Samba and WebDAV share for this.

While I don't immediately have hard drives ready to add, I was able to get started by just setting up a container and using a folder on its root volume. Installing Samba was straightforward. I do eventually want to install WebDAV as well, but that will have to wait for another time (I've been sitting on this blog post for far too long!)

OctoPrint

OctoPrint is a server for managing a 3D printer remotely. I recently dusted off my old Ender 3 Pro and started making stuff with it again, and now that I've graduated and work a normal work schedule, I have fewer opportunities to check on a print throughout the day.

I decided to use the popular octoprint_deploy for installation. Since the README warns against trying to deploy in an LXC container, I made this host a full Ubuntu Server VM. I was able to pass through the USB device easily by its vendor and product ID, and everything worked on the first try! I don't have a USB camera for remotely monitoring yet, but I might decide to add one if I find this useful.

General purpose Ubuntu and Windows servers

I find myself occasionally booting into Windows 11 for things, most recently for uploading a codeplug to my OpenGD77 radio. Since I almost never use this partition otherwise (the video games I play all support Linux at this point), I really only need a basic Windows 11 box with USB passthrough support. I got the idea for this from my girlfriend Ari who uses a Windows VM for similar purposes. I also set up an Ubuntu desktop machine at the same time, mostly because it was easy to do so.

Results and Conclusions

First off, I would like to think Ava for putting up with me occasionally disrupting our home internet to perform upgrades or maintenance. Thanks also to Ari, Hunter and Tris for helping me troubleshoot on the countless occasions I accidentally shot myself in the foot with Proxmox. This project veered farther into the realm of networking and system administration than I ever had before, and I couldn't have done it without help.

Taken as a whole, this is probably one of my more practical projects as of late: I'm finally able to clean up my convoluted file sync situation, I've gained the ability to remotely manage stuff on my home network, and I've already enabled a few helpful automations with HomeAssistant.

Sitting right next to my desk as I type this is an IKEA FADO lamp fitted with an RGBW LED bulb. Among friends, it's become affectionately known as "the orb" due to its unique shape.

Implementation

This project uses quite a few moving parts to pipe lighting data through from the user's browser into a smart light bulb. I'll trace the flow through each of these components.

The user chooses a color through a basic web UI written in vanilla HTML, CSS, and JS. There's a sketch of the orb that I made in Inkscape to provide a preview.

The UI makes calls to a Flask-based backend, which turns around and calls HomeAssistant. This layer exists only to allow unauthenticated users to call these two specific methods on the HomeAssistant API (since the Flask app keeps a bearer token around). I deployed this on Dokku, which is a stack I've developed a strong familiarity with.

While I initially tried to make this project work with Google Home, I couldn't find an API that would allow me to invoke commands programmatically. Ava and I had been looking for an excuse to migrate to HomeAssistant for a while, and this was it! We found the setup process to be quite straightforward: most of our smart devices are Matter bulbs, which could easily be added to both Google Home and HomeAssistant.

HomeAssistant gives each entity an "Entity ID", which makes programming easy. The orb light is just lights.orb! While finding the right documentation was a bit difficult, invoking actions on entities through the REST API was overall easy and straightforward.

We've deployed HomeAssistant to a Proxmox server running in our home mini-rack. Since our HomeAssistant instance is publicly routable at homeassistant.home.breq.dev, no special considerations were needed for piping together the Flask app deployed in the cloud with HomeAssistant running in our home network.

Results

In and of itself, the orb demo isn't that interesting, especially because you need to actually be in front of the light to see it working (and if you're already physically in my apartment, you can just use the Google Home to set the light). That said, it's a fun party trick, and I enjoyed finally being able to interact with my smart home devices easily from code! I'm definitely looking forward to doing more smart home projects in the future.

Over the years, I've developed a strong appreciation for the MOS 6502, mostly through my work writing an emulator (something I feel everyone should do). In researching this chip, I've stumbled across countless creative tricks and workarounds both in the design of the chip and the vast library of software written for it.

Zero Page

This one's more of an interesting design choice: the 6502 has an absolutely tiny register file. The only general-purpose register is "A", the accumulator. There are two additional registers "X" and "Y", but they are only used for indexing memory.

However, the designers of the 6502 realized that this was extremely limiting to programs and would require lots of reads and writes into memory to handle temporary values computed in a program. Using the "Absolute" addressing mode, this requires two bytes for the memory address and around 4 cycles depending on the instruction. The "Zero Page" addressing mode is a hack: when a Zero Page instruction is used, only the lower byte of the memory address is stored, and the processor can skip a cycle! This both reduces the program size and speeds up program execution. As a result, the first 256 bytes of memory are effectively "registers" of sorts!

The "CPS pin"

Like many architectures, the 6502 has a status register with flags like "zero", "overflow", and "negative" representing the last operation completed with the ALU. Unlike these architectures, though, the 6502 has a fun quirk: the chip has a physical "SO" ("set overflow") pin which, when the signal on it goes from high to low, sets the overflow flag in the status register. This flag can be used by subsequent branch instructions.

The usefulness of this pin is questionable, and it rarely saw usage in practice. It was apparently used in tight loops for routines interfacing with hardware, since a transition on the pin could cause code to jump out of the loop. The pin was originally called the CPS pin, short for "Chuck Peddle Special" (named after the main designer of the 6502, Chuck Peddle).

The BRK Instruction

On the 6502, opcode 00 maps to the "BRK" instruction. This is essentially used to trigger an interrupt from within the program.

In the 6502, the location in memory that code is executed from after a reset or interrupt is controlled by a table of "vectors" located at the very end of the memory map.

NMI and IRQ are for the two pins on the microprocessor of the same name ("non-maskable interrupt" and "interrupt request" respectively). RESET is for when the microcontroller boots up or is reset. BRK shares its vector with IRQ, meaning the code routine which services maskable interrupts is the same routine which services BRK instructions. So, when you run BRK, your code immediately jumps to the interrupt routine. Once the interrupt routine returns with the RTI instruction ("Return From Interrupt"), your code keeps executing.

BRK was intended for use in debugging (as a "breakpoint" of sorts). In some systems, this was what happened: the Apple II would launch the program monitor which would allow you to debug your program. However, this instruction has a weird quirk: despite BRK being a 1-byte instruction, when the interrupt routine returned, the program counter would jump forward by 2 bytes instead of 1 -- leaving a single byte after the BRK instruction unused. Clever programmers could exploit this by looking at the top of the stack and reading this value, using it to pass state between the code calling BRK and the interrupt service routine.

On the BBC Micro, BRK was used for error handling. To throw an error, the BRK instruction would be called with the error code stored in the byte immediately following it. Since returning from an error handler was not supported, an error string could be stored immediately after the error code byte additionally.

On the Apple III, Apple's Sophisticated Operating System used BRK for system calls. The system call opcode would be placed immediately after the BRK instruction, then a 2-byte pointer to additional parameters. This did mean the SOS routine responsible for system calls needed to increment the return address by 2.

Wozniak's SWEET16

The 6502 is an 8-bit processor, but addresses and other values are often 16 bits. While it's of course possible to manipulate these larger values using a combination of 8-bit operations, the resulting code is often quite verbose. Steve Wozniak, known for his obsession with going great lengths to save a small amount of resources, did not want to deal with this, so he created an interpreted bytecode language called SWEET16. A program could call into a subroutine, and the code after the subroutine call would be interpreted as SWEET16 instructions until the SWEET16 "Return" instruction was executed, at which point the following code is executed normally.

SWEET16 is an interesting tradeoff for code size versus performance. While it massively reduces the size of code, interpreted SWEET16 code is about one tenth the speed of native 6502 code. While optimizing for code size was usually preferred in the past, modern processors have improved slower than modern memory has, so this tradeoff is less relevant now than it once was.

SWEET16 supported several "nonregister operations". These included branch operations, which used a 1-byte signed offset as the branch target.

It also featured 16 "registers", stored in the zero page. These were specified using the second nibble of the opcode, allowing these instructions to take up only a single byte. Notably, some instructions supported an indirect addressing mode. Many operate on register R0, also called the accumulator.

SWEET16 stands out to me as a shockingly complete interpreted language implemented in a shockingly short amount of code. It's something I otherwise wouldn't have been expected to be feasible in this context.

Illegal instructions and their uses

The 6502 has 151 valid opcodes. However, there are 256 possible values that the first byte of an instruction can take. What happens if you try one of the invalid ones?

The technical reasons for why this behavior happens are described well in this blog post and this document.

A few of these instructions were used in a few NES games (mostly 2-byte NOPs). These also saw use on various BBC Micro games. Games used these for both copy protection (as an attempt to validate the hardware?) and for performance. My guess is that more serious software doesn't bother with these since performance is less of a concern than correctness.

Later revisions of the 6502, such as the W65C02S, replaced the opcodes in the "F" column with additional bit manipulation opcodes.

The Ricoh 2A03/2A07 and the NES

The 6502 was incredibly popular, and inspired a few clones and derivatives. One such clone was the Ricoh 2A03/2A07 chip used in the Nintendo Entertainment System. Apparently, Nintendo was not a big believer in copyright law back in this era. Oh, how the times have changed...

To see how blatant this was, first look at this image of the 6502 chip (courtesy of Visual 6502)...

Now, look at this image of the Ricoh 2A03 (also from Visual 6502)

Does that little bit in the lower right corner look familiar?

Interestingly, the Binary Coded Decimal (BCD) functionality of the 6502 were fused off before the chip was incorporated into the Ricoh clone. This feature allowed the 6502 to perform arithmetic operations on numbers stored as decimal, with 4 bits representing a decimal digit. It was covered under a patent held by MOS, which is probably why Ricoh disabled it.

The other parts of this chip are for things like the NES's sound generator, known as the "APU" and definitely worthy of its own post someday. It's also why there are two variants of this chip: one for NTSC and one for PAL.

Conclusion

The 6502 is one of the final bastions of an age where microprocessors could be contained in a pile of parchment in a dude named Chuck's desk. It was one of the first mainframes in a chip, and powered countless number of people's first experiences in computing. The number of programmers today who got their start on the 6502 alone makes the chip worldchanging in it of itself.

The article below was republished from the internal NURover Notion wiki. On the Rover team, we use a PCIe to USB card with four individual USB controllers to provide enough bandwidth for our camera streaming system (something I've posted about previously). As you can probably imagine, the USB system on the rover has very high current draw, and we've fried cards (quite dramatically) in the past.

Some PCIe to USB cards require a modification to pass through a high amount of current on the 5V rail. This page describes how to apply the modification.

Do I need to perform a 5V Bypass Mod?

There are two types of "quad chip" PCIe card available: “B” variant and “C” variant.

PI040202-7X2B (”B” variant)

PI040202-7X2C (”C” variant)

This modification is only required for the “C” variant. The “B” variant boards provide power from the +5V pin of the Molex or SATA power connector directly to the USB port (after some LC filtering). The “C” variant boards instead provide power from the +12V pin on the power connector, passed through a voltage regulator chip. This chip cannot handle the high current loads present on the rover.

Once performing this modification, the card can no longer be powered from the PCIe slot and will REQUIRE external power to operate.

Removing the Voltage Regulators

The first step is to remove the 12V to 5V voltage regulator chips. There are two ways to do this.

Hot Air

The least destructive method of removal is the use of a hot air workstation.

Notes on this method:

Cutting Tools

If the chip is burnt, you can use flush cutters to remove it.

Notes on this method:

Attaching Jumpers

You want to bridge the +5V input from the Molex connector to the two +5V outputs that would have been coming from the regulator.

Your first instinct may be to solder to the 5V output pad where the regulator used to be. This is possible, but difficult to do by hand. A much easier approach is to solder to the input side of the inductor!

Drop a blob of solder on the side of the inductor. The picture shows slightly too much, I cleaned it up after. Don’t worry too much about making it pretty.

Cut a small length of wire (I used 24 gauge) and strip a bit at the end. Tin the end with more solder.

Finally, use one hand to push the wire into position and the other hand with the soldering iron to heat the pad. It may take a while for the inductor to get up to temperature — be patient!

The reason we were liberal with solder before is it frees up one of your hands — you don’t need to hold the solder while attaching the wire.

Repeat the process for the second inductor, attaching a second wire.

Trim the wires to length — you want them to attach to the +5V pin on the Molex connector (furthest from the PCIe connector on the card). You can line them up perpendicular style like so to attach both easily.

Finally, add solder!

Testing for Shorts

There are two shorts that you need to check for the presence for. The GND and 12V pins heading to where the voltage regulator was are very close to the pad on the inductor that you soldered to, and bridging is very possible. You can test this by checking connectivity between your jumper wires and the USB connector shielding (for GND) and between your jumper wires and the output side of diodes D1/D2 and D3/D4 (for 12V).

If neither rail is shorted, your board is ready to use!

Schematic

This project recolors SVG exports of KiCAD circuit board designs to appear in realistic colors.

Motivation

I faced an annoying problem with KiCAD the other day. I wanted to export a realistic-looking image of a PCB design to use for an upcoming project. However, the existing options for doing this didn't fit my needs.

The 3D viewer, while very cool, isn't great for this use case. I would prefer a vector image to a raster one. Plus, with the way the lighting is set up, an "exactly head-on view" of a board can look very blown out. Here's a board with light-colored silkscreen as viewed head-on in the 3D viewer:

KiCAD also has the ability to export an SVG natively, but the result looks like what you would see in the editor: the colors match the editing layers, not the actual realistic colors of the board, and the soldermask layer is inverted. Plus, I wanted a way to generate separate front and back views, with the back view mirrored to how it would appear in real life.

Implementation

So, I built a tool to recolor the output from the KiCAD format to look more realistic. Annoyingly, KiCAD-generated SVGs do not put each layer into a separate SVG group or layer, so my code queries for nodes based on their existing color to apply the new colors. This would break if KiCAD exports things in an unexpected color, but since these colors are seemingly not configurable by the user, this should be fine.

I chose to write this app in JavaScript since the language already provides good utilities for parsing, querying, and manipulating XML documents like SVG files. The UI is pretty utilitarian, but hey, it works and loads quickly! I wrote this without using a framework for styling or JavaScript -- it was nice to take a break from the complex tooling of my usual React/Tailwind setup and go back to basics for a bit.

The whole app is based around some gnarly CSS selectors, especially since the format changes substantially between KiCAD versions and I was trying to target versions 7 through 9.

Here are some example outputs for a board I'm working on:

Try it yourself

If you want to try it out yourself, you can! First, in KiCAD, export an SVG with the relevant layers. In older versions, this is straightforward:

However, in the new version, they've replaced the "Export SVG" dialog with the "Plot" window, which exports a separate SVG file for each layer (which is not what we want!) The easiest way to get a single SVG image is with the KiCAD CLI (replace MyBoard.kicad_pcb with the filename of your board file):

kicad-cli pcb export svg --mode-single --layers F.Cu,B.Cu,F.Silkscreen,B.Silkscreen,F.Mask,B.Mask,Edge.Cuts MyBoard.kicad_pcb

Then, go to kicad-pretty.breq.dev and:

If you're having issues with the export format, feel free to reach out to me and I'll do my best to investigate! I've only been able to try a handful of SVG viewers and KiCAD boards so far.

My friend Jules made a fun suggestion on Bluesky recently:

Let's step through the search results for my domain name on GitHub and see what we can find!

eightyeightthirty.one

In 2023, some friends and I built a scraper tool called eightyeightthirty.one which attempted to map the entire graph of 88x31 buttons. Soon after, I wrote an article applying network science to the 88x31 graph.

A few people credited my project page for the 88x31 scraper tool. Please credit NotNite's portfolio instead, she did most of the work!

5F3759DF

A few people have linked to my 2021 article on the fast inverse square root algorithm in code over the years:

Flowspace

At least one person other than myself has set up a profile on Flowspace, a social network demo I made in 2021.

Accelcoin

In 2022 I was a teaching assistant for Fundamentals of Computer Science 1 at Northeastern. One of our course assignments was for students to build a working blockchain node implementation. I ran some infrastructure relating to this! Some students have published their code to GitHub.

Wordle

In 2022 I published a few implementations of the Wordle algorithm in TypeScript and Rust.

As many of you may know, I've been involved with the Northeastern University Mars Rover Team for four years now. We compete each year in the University Rover Challenge and occasionally the Canadian International Rover Challenge. I personally have competed in four URC events and one CIRC event, and will (hopefully!) be competing at the upcoming URC and CIRC events this summer.

Here's our latest System Acceptance Review video, which gives some context for the team structure and the subsystems we develop.

Rover has largely dominated my college experience. Late nights in the lab have been a staple of all four years of my college life. During my co-op searches, rover has consistently been the only thing any interviewer has ever wanted to talk to me about. While the club has occasionally gotten overwhelming, I have no regrets about pouring as much energy and effort into the team as I have.

This post is a collection of things I learned from my time on the team and reasons why experiences like this are valuable. This section is written from my perspective as a software developer, but the general principles apply across the engineering disciplines.

Thinking across disciplines

It's critical to work with engineers outside the software field, for two reasons: it helps you teach others about software, and it helps you learn about other forms of engineering.

As an early career software developer, you might never work with anyone outside the software field unless you seek those situations out. Even if classes require team assignments, they are typically done exclusively by students majoring in computer science. Furthermore, many internships on large software teams don't involve any communication with those outside the software team. During my 6-month co-op at Amazon Robotics, I can't point to a single issue in which I had to work deeply with someone with expertise outside of embedded software in order to solve a technical problem. That's a shame, since communicating across disciplines is a critical skill for solving problems in the real world.

The other advantage of an experience like the rover team is that it allows you to develop an understanding of fields outside your core strengths. While on rover, I did electrical work, replaced and installed mechanical parts, and even got involved in our filming team for our System Acceptance Review video. My greatest regret on the team is not learning SolidWorks. The base level understanding of electrical and mechanical engineering I developed has come in handy countless times over the years.

Hands-on experience

My university is (somewhat) unique in that it encourages students to participate in 6-month "co-ops" instead of 4-month internships. They pride themselves on this "experiential learning" approach, and one could assume that it diminishes the importance of developing hands-on experience through clubs. However, I would argue the opposite: co-op program participants benefit greatly from hands-on club experience at least as much as those on a traditional internship schedule.

One reason is that club experiences give members a much broader range of fields and topics to study. Co-op projects are often small, self-contained, and confined to a small field and area of the product. In clubs like the rover team, you can choose as many or as few topics as you like to learn about and dive deep into. Clubs like the rover team have countless subsystems and components, both hardware and software, which need to be integrated; with members leaving every four years, there are plenty of opportunities to become the team expert in a particular system.

During my time on the rover team, I became the "expert" on our command and control interface (since rewriting it happened to be my first project), our radio system (since I was in the right place at the right time to troubleshoot it at my first competition), our camera streaming system (since I arranged to take over development on it after the previous experts graduated), and our migration to ROS 2 (since I was in a leadership position at the time we migrated). Unless you work at an early-stage startup, it's difficult to develop this breadth of expertise in a corporate environment.

Similarly, engineering clubs allow members to be part of high-level design decisions and planning steps that are often opaque to interns and co-op students. The design of large-scale software systems is a critical skill often not taught well in classes. You can study design patterns all day long, but putting that knowledge into practice is a skill that takes, well, practice.

Another advantage of club experiences that it teaches you software development practices before you go on an internship or co-op. I am always amazed at the number of extremely smart Northeastern students who have no experience with Git, pull request workflows, how to leave a code review, or other critical steps in the software development process until their first co-op. While Northeastern does have a class which covers these, it is taught too late and in too little detail. While you could make the argument that teaching these skills is the purpose of a co-op, it often doesn't work like that in practice: small companies may employ poor software development practices, large companies like Google or Amazon use their own bespoke workflows unlike others in the industry, and students don't always get the chance to leave code reviews themselves.

Leadership and collaboration

Working on a 4 person team is quite different from working on a 40 person team, and a 40 person team is quite different from a 400 person team. My rover team had around 40 active members at a given time. I think this number is about perfect for a few reasons:

I'm sure I'm far from the first person to point out that personal projects are no substitute for building things with others, but it's true! Going directly from personal work to working 40 hours a week on a team is a jarring transition for many. Although software courses try to remedy this with teamwork assignments, there is no substitute for the experience gained working with a large team over multiple years.

It's fun!

My final reason is that rover development work and competition is fun as hell. If the only software development work I did was for classes or co-op, I worry I would lose interest quickly in the field. Personal projects can be fun to develop as well, but nothing compares to the shared sense of comradery, triumph, and accomplishment I experienced on the rover team.

This project is a wall-mounted LED matrix built to show album art, nearby transit and bikeshare information, and weather data.

Motivation

Almost four years ago, I made the first iteration of a wall-mounted LED matrix project to show weather and transit info, and it's one of the projects I've gotten by far the most use out of. It's been hung at my childhood home in Maine, in various dorm rooms over my college years, and now at my apartment I share with Ava.

Over that time, we've made only a few changes:

However, as we've added each of these features, we've needed to pack more information into a very small (only 16x32!) screen. This tips the balance away from legibility (mostly fine for us, but less useful for guests at our house).

Simultaneously to this, the two of us got a record player shortly after moving in and have been starting to build a record collection, which has reinvigorated our appreciation of album art. I thought the idea of a piece of wall art to display album covers sounded cool, so I started looking at getting a variant of the LED matrix with a square aspect ratio.

Technical Description

"Hardware Rev 2" is built around a 64x64 matrix panel with half the pixel pitch of the original, leading to a device with the same width and double the height. I decided to build it with a Raspberry Pi Zero 2W and a Matrix Bonnet from Adafruit with matching dimensions to shrink the electronics down from the original and let it sit closer to the wall.

This also meant that I could build this new revision without taking parts from the old build! I gave the old model to my girlfriend Mia.

Software

I also chose to start from scratch with the software of this version. I've learned a lot about software development since writing the first iteration of the code, and the old design included way too many layers of abstraction. (Features like the preview display and web server were cool, but saw little day-to-day use.)

The display supports four main screens with information pulled from Spotify, weather, MBTA, and BlueBikes.

The Spotify screen just displays the artwork from whichever album Ava or I am listening to at a given moment. While I was initially happy to see Spotify's CDN making a perfectly-sized 64x64 version of the image available, it seems to be too affected by JPEG artifacts to be usable here. So, I settle for downscaling a higher-resolution version.

Spotify data is pulled using the spotipy library.

The weather screen is pretty similar to the weather screen on the old version, showing an icon from Dhole's awesome weather pixel icons alongside the current time, temperature, and high/low temp for the day. (Thankfully, we now have labels corresponding to the F and C temperatures.) Data is sourced from OpenWeatherMap, using a lookup table I made to match their weather condition codes to weather pixel icons.

The MBTA screen is designed to mimic a station countdown clock, but with data pulled from several bus and train lines near my apartment. It merges schedules and realtime predictions from the MBTA Realtime API. I've written some custom logic to try to deduplicate schedule and realtime data based on trip identifiers, but it doesn't always work perfectly.

The BlueBikes screen is probably the one I'm most proud of. I hand-drew each of the icons for bikes, e-bikes, and docks that show for each station.

Bikeshare services publish realtime using the General Bikeshare Feed Specification format (GBFS), analogous to the GTFS format used for transit data. The BlueBikes API has excellent documentation. It's a bit inefficient (there's no way to filter to just a subset of stations that I've found yet), but it works great for this use case.

My friends and I have fun with a website called makea.fish, which generates an image of a fish at 11:11. (We've worked on various other fish generators, too!)

As a fun easter egg, the LED matrix will switch at 11:11 to showing a fish image generated by makea.fish which refreshes every 10 seconds.

Hardware

A view of the back of the device, showing the 3M command strips used to hang it on the wall

Just like the original, this LED matrix sign is contained within a 3D-printed enclosure. It uses a Raspberry Pi Zero 2W (my first project with a Pi Zero!) with Adafruit's Matrix Bonnet, using a DC jack "extender" I soldered to route the power input to the bottom of the device.

The case features a "chin", since the stackup height of the electronics is annoyingly tall. I also added ventilation holes, since there was a bit of discoloration on the wall from the old design. This was my first 3D printing project in a while (outside of my day job working with volumetric printing).

Quite a long time ago (was it middle school??), I bought a Monoprice Select Mini 3D printer. It held up surprisingly well over the years, including through two hotend replacements. This is the printer that the old LED sign case was printed with! It had a print volume of only 120x120x120mm, so I had to break up the piece into two separate parts.

A few years ago, I bought an Ender 3 Pro during a Micro Center sale. (Annoyingly, I was living away from Boston over the summer, so I ended up grabbing it and lugging it around for a full day of sightseeing with my parents -- in my defense, the sale was about to expire!) I set it up and ran a few prints, but could seemingly never get it to work as well as my old Monoprice one. While I used this for the rave choker project, it just didn't work well for printing larger parts. It spent a few years collecting dust at my parents' house.

Then, earlier this year, I decided to take the printer back down to my apartment and try to get it up and running again. I ended up doing almost a full rebuild, and found just two loose screws holding the Z-axis stepper in place (maybe vibrated out, maybe not tightened properly in the first place). After tightening those, the parts for this project came out beautifully!

The Ender 3 is large enough to fit the full size of the LED matrix, but isn't quite big enough to print the entire case in one piece. Thus, I broke the part up into the frame around the LED matrix and the "chin" below which contains the Raspberry Pi. The two parts are attached using heat-set inserts.

And as for why the parts are pink instead of black: after years of being stored badly, the pink spool was one of the only ones I had which wasn't made too brittle by the humidity. (Plus, the pink and black scheme is growing on me...)

Conclusion

I'm really happy with this! It fits nicely into our living space and does its job of both looking unique and providing useful info. I have thought of a few improvements to make, which will maybe happen at some point. There's a lot of empty space in the enclosure right now!

It also feels quite polished compared to a lot of the quick builds I've done, like I could do a small run of these. The parts costs are a bit high (mostly the LED matrix itself), so that'll probably never happen, but I can dream!

The amateur radio hobby is generally dominated by a particular demographic. If you look around at a typical large hamfest, spend any time listening in on a net, or ask anyone for their stories of family members, you'll easily spot it. As a young queer student, my friends and family members are often surprised to hear that I'm involved in the community. So why do I gravitate to it?

Projects

As my software knowledge has matured, many of my recent projects have been more focused on building things that I would find useful in my day-to-day life. Amateur radio is a great opportunity for projects like those. Radio hardware is often very open to hacking and experimentation, open-source custom firmware and custom programming tools are extremely common.

Personally, I enjoy connecting the hobby to my interest in software through projects like rolodex, a "contacts app" for storing callsigns and repeater information, or my work with the Northeastern Mars Rover Team involving autonomous mission control over AX.25.

Adventure and Practicality

When amateur enthusiasts talk about "off-grid communication," it's often in reference to an emergency scenario in which cellular networks fail. While amateur radio does often prove useful in these scenarios, it's infrequent enough that I don't find it particularly exciting.

Another situation where off-grid communication is desirable is in those few areas which are still unreachable by cell signal. I've personally experienced the usefulness of this at the University Rover Challenge, where the site has little to no cell coverage and the range of cheap FRS radios is often inadequate. I've also tuned my handheld radio to weather frequencies to get forecast information in remote areas of Utah and New Mexico.

Even in situations where communicating over cell phones is possible, radio can still pose advantages. It's much easier to communicate via radio while driving a car, while outside with gloves on, or in other situations where reaching for a phone is inconvenient.

Train autism

One of my favorite things to use my handheld radio for is scanning: listening to signals on railroad channels, marine channels, and other government or commercial bands. Listening to how radio is used in the world around me illuminates systems and processes like train dispatch, ship-to-ship communication, and more that are otherwise hidden from view. Discovering these frequencies is an interesting and rewarding challenge in and of itself, and seeing these processes firsthand shows so much about the infrastructure that makes our world run.

Friends

Amateur radio is a hobby more or less predicated on talking to people, so it's critical to find a group of people you enjoy talking to. Radio communication takes a few forms:

HF equipment is more expensive and generally incompatible with apartment life, which means I'm stuck with a handheld for personal use. That said, I've done some contesting with Northeastern University Wireless Club and my friend Philo is an avid FT8 enjoyer -- it's possible!

Regardless, this leaves me with nets as one of my only options. While I've tuned into a few on repeaters around Boston, I generally am not interested in the conversation that goes on there. (I can only take so much listening to boomers talk about their medical problems.)

Despite how it may seem at first glance, there are a lot of young people in the hobby! Younger radio operators tend to form much more insular groups with their friends or at their university or hackerspace. A good place to spot the younger crowd is often on university-run repeaters -- around Boston, good options include W1KBN or W1XM.

If not for my close friends in the hobby, I would likely have nowhere near the level of involvement that I do now. We often use radios as an interesting and unique way to coordinate adventures together. My friend Ari created her own bandplan of simplex frequencies that we often use to communicate.

Radio can be a daunting hobby to get into due to its social nature. Finding your people is critical to having a good experience -- I had a handheld collecting dust on my desk for about a year before I really started using it. The first step in the hobby for many is becoming licensed, which often means working up the courage to show up to an unfamiliar place and take an exam administered by an unfamiliar group of people. Part of the reason I recently became a Volunteer Examiner is to help reduce this barrier to entry among friends, and to give me a better answer when someone I know asks how to get involved.

As far as hobbies go, amateur radio has a lot to offer, and not fitting the stereotypical demographic shouldn't hold you back from carving your own niche in the world of radio communications.

Some of you might know that Ava and I recently moved to our new home in Somerville! As part of this, I finally got the space to start having fun in building a home network setup.

While the setup is pretty tame for now (and Ava has specifically banned me from purchasing any rackmount gear -- alas...), I hope to keep this post updated with notes I have as I work to build more.

Core Equipment

Modem

We get internet through a cable provider, so the network starts with a modem. I picked a basic Arris SURFboard with DOCSIS 3.2 support since I wanted the flexibility afforded by running a separate router.

This modem operates a little strangely: it gives itself 192.168.100.1, hands out 192.168.100.10 to the first device it sees on DHCP, then also forwards along the assigned public IP address to the connected device. While waiting for the pfSense router to arrive, we actually had to use a Gl.inet router since neither the modem nor our access point has no routing functionality.

Router and Switching

My router is a small Netgate box running pfSense that was gifted to me by my friend Ari. It has two LAN ports: one goes to the network switches, and the other goes to the Wi-Fi access point. This lets the router and access point use VLANs to separate out clients on each SSID.

I keep the travel router I was using in the interim around for any networking shenanigans that might require it.

I have two Netgear switches I picked up at MIT Swapfest a while ago: one sits near the rest of the equipment on my shelf, and the other sits on my desk to connect to my desktop and oscilloscope. One of these switches was featured in my friend Hunter's Devices of All Time blog post, which I somehow didn't realize until months later :)

Wi-Fi

After dealing with tons of Wi-Fi dropouts in our old apartment, I decided to invest in a Ubiquiti U6 Mesh access point. It's definitely overkill, but we get exceptional connection to anywhere in the apartment.

I was very excited to finally have a router that didn't look like an over-the-top gaming device. The U6 Mesh is very compact and fits nicely on a shelf -- it's about the size of a tall seltzer can. The PoE power is actually quite useful in a home environment -- the cable run is a lot tidier with only a single cable to the device. The only criticism I have so far is that gets pretty warm when it's running.

While it's theoretically possible to set up the device without a UniFi Controller, I couldn't figure out how to do so (the app kept crashing when I tried to go through the flow). So, I just installed the UniFi server software onto my desktop and ran it so I could configure the device, then closed out of it. The software allows creating any number of separate SSIDs, which we use to run our primary network, an SSID matching my parents' home in Maine, and an Eduroam hotspot.

Layer 3

Our IPv4 subnet runs on 10.0.0.0/24, mostly since it makes it possible to use shorthands like 10.2 instead of 10.0.0.2.

Our also ISP provides us IPv6 - yay!

I use pfSense's Dynamic DNS tools to point home.breq.dev at our IPv4 address. I couldn't get this to work for IPv6 (the router would use its own address, where I wanted to direct traffic to another point on the network). So I put our IPv6 prefix directly into the DNS dialog on Cloudflare and hoped for the best.

IPv6 Addressing

Getting the "suffix" of my IPv6 address to remain static ended up being a hassle -- I wanted my computer to continue to respect the prefix it was given, but to keep the same suffix part instead of randomly generating it.

I found ip token to solve this problem.

To try things out, I ran:

# Accept Router Advertisements to configure the network prefix
sudo sysctl -w net.ipv6.conf.enp37s0.accept_ra=1
# Set an IP token identifier (::bc for my initials)
sudo ip token set ::bc dev enp37s0

Then, to make it persistent:

# add net.ipv6.conf.all.accept_ra=1 in the ipv6 section
sudo nano /etc/sysctl.conf

# NetworkManager equivalent of the ip command
# My wired network is called "Ethernet",
# or by default it's something like "Wired Connection 1"
sudo nmcli connection modify Ethernet ipv6.addr-gen-mode eui64
sudo nmcli connection modify Ethernet ipv6.token ::bc

Split DNS

For IPv4, since we do NAT, I needed to use Split DNS to make sure services were available both on and off the network. (And since the DNS resolver built into pfSense was synthesizing records for these addresses, I also needed to have it synthesize AAAA records accordingly too.)

Services

We've got a few services hosted here, with more potentially to come:

In the future, I'd love to also spin up

Eventually, I'd love to have a dedicated server for lots of these things (Nextcloud, Syncthing, etc) and maybe a reverse proxy to point to other devices on the network? It's not a huge priority, but maybe after I upgrade my desktop machine I'll be able to scrounge together some hardware.

fissh.breq.dev ("make a fissh") is a terminal-based fish generator that you access via SSH terminal. At 11:11, try running:

ssh fissh.breq.dev

and you'll see a fish appear in your terminal!

Fish generators

My friend Ari introduced me to the original makea.fish site (HTTP only), made by weepingwitch. The site embeds an image rendered with PHP showing either a randomly-generated fish or the phrase "come back at 11:11." Getting the 11:11 fish has become somewhat of a ritual among my friends (we have a Discord channel specifically for posting fish screenshots!)

Over the last few months, we've made a few fun variations on this:

"Make a fissh" was written by myself and my girlfriend Ava based on an idea from my girlfriend Mia. It's our contribution to the weird and wonderful world of fish generators.

Wish, bubble tea, and lip gloss

I got the idea for an SSH-based application from terminal.shop (an SSH-based coffee beans store). My first idea was to create a user whose login "shell" is just the fish application, similar to how git-shell only allows running Git commands. However, I couldn't find a way to make OpenSSH require neither a password nor a public key. I suppose there's definitely a good reason for that, but it meant I needed to look elsewhere.

Looking into how terminal.shop implemented their solution, I stumbled upon Wish, a Go library designed for "SSH apps." I decided to build this within the Charm ecosystem, using Wish, Bubble Tea for layout, and Lip Gloss for "styling" (i.e., ANSI escape codes).

This stack imposed some requirements on the project structure: this was probably my first model-view-controller app since taking Fundamentals of CS 2 and I definitely would not have picked Go first. The MVC architecture works fine for something like this (although I would've preferred something a bit more component-based / React-y). I found the docs for the Charm projects pretty lackluster, and was often faced with undocumented behavior or limitations that I couldn't find reference of online. (I probably should've realized that the number of GitHub stars probably didn't correlate to the amount of actual use that the library gets...)

I won't pretend to have insightful commentary on Go after using it for a tiny toy project, but I honestly struggle to see myself reaching for it in any situation. My first impression is that it's the complexity of Rust without any of the error handling, memory safety, or performance benefits, with more than its fair share of strange syntax or stdlib quirks, coupled with tooling that makes strange decisions at times.

That said, for a toy project, I got a huge amount of learning out of this, so I can't really complain :)

A fishy dataset

With the stack out of the way, it's time to actually build this thing. The most important part of a fish generator is the fish it generates. I decided to go the easy route and source fish ASCII art from ascii.co.uk. One thing I like is that most artists leave their initials somewhere in the drawing. I wish I could link to some of the ASCII artists directly from the fish page, but unfortunately, the site doesn't give any other attribution.

Finishing touches

Getting the user's timezone right is another integral part of fish generation. Web-based generators get this easily from the JavaScript Date API, but for other protocols, it can take some creativity. The aforementioned SSTV fish telephone line, for instance, just accepts any time ending in :11.

With a terminal, though, we get a bit more information: the user's IP address. The app uses ipinfo.io as a geolocation database since their free tier seemed generous and they helpfully provide a timezone key in their output. As such, the server can ensure each user gets the fish in their corresponding timezone (assuming the GeoIP database is accurate).

No app would be complete without a short "about" page, let alone one with so many friends to credit. I took advantage of OSC 8 escape sequences to embed links to the fish dataset and various folks' websites. These work in many popular terminal emulators, but not all, and there's no method to query if a terminal supports OSC 8 and fall back to including full URLs otherwise.

Deployment

Deploying something that runs on port 22 on a server that you presumably also need to SSH into is an annoying situation, and I've had enough bad experiences messing up an SSH config and locking myself out of a VPS that I decided to just spin up a new one for this. (Oracle Cloud Free Tier my beloved!) That said, switching OpenSSH to port 2222 and running this on port 22 largely went off without a hitch.

One of my favorite quotes is part of The Cult of Done Manifesto by Bre Pettis and Kio Stark:

Those without dirty hands are wrong. Doing something makes you right.

So do it. Go write an emulator. Pick a programming language. Pick a basic CPU architecture: either something retro like the 6502 or Z80, or something modern but still simple like the RISC-V base integer instruction set, or something designed for teaching like Pep/9. Implement a basic model of its registers and a simple block memory interface. Start with a few opcodes, write some assembly programs to test them out, and then add more.

By the time I sat down and wrote an emulator, I had taken courses on the architecture of a CPU! And yet writing this one piece of software fundamentally changed my understanding of how computers work at a basic level. Computers are obviously far more complicated than the toy examples I'm suggesting, but advances like branch prediction and caching did not fall out of a coconut tree; they exist in the context of all that came before them.

Concepts like Return-Oriented Programming require a strong ability to reason about how machine code is executed. Working with computers at the lowest level through emulator development helped me develop this reasoning ability far better than a class would.

Another underrated skill in software development (that I especially think isn't taught well!) is reading a datasheet and translating that to a software implementation. Writing software in the real world involves working with lots of weird protocols with documentation of varying quality: on Rover I work with tons of random sensors and modules, at work I write code that communicates with lasers with protocols that vary widely between "very reasonable" to "moderately insane," and in personal projects I frequently find myself reaching to implement communication with a part that lacks a comprehensive library. While implementing my emulator, I reached for datasheets, schematics, and any documentation I could find. And where the docs weren't clear, I was forced to find ways to answer questions about the protocol experimentally.

When I started, it felt like almost all resources I found on emulator development tried to talk me out of it:

Good knowledge of the chosen language is an absolute necessity for writing a working emulator, as it is quite complex project, and your code should be optimized to run as fast as possible. Computer emulation is definitely not one of the projects on which you learn a programming language.

- "How To Write a Computer Emulator" by Marat Fayzullin

Writing an emulator was my first project in the Rust programming language, and I don't regret it at all. An emulator forces you to think deeply about your code's structure: in the real world, retro computers reused chips for different purposes, split functionality across different parts of the system, and were generally full of leaky abstractions. How do you write a program which encapsulates that structure? Mirroring the structure exactly a la MAME leads to convoluted code, but building too many abstractions leads to quite complex data flow. And if you're anywhere near as much of a perfectionist as me, you'll rewrite each component at least three times, agonizing over the small details and using every tool your programming language offers you. And each time you rewrite it, you'll learn something.

It is my belief that there is no single project which introduces you to the good, the bad, and the ugly aspects of a programming language than an emulator.

In classes, computer science assignments are typically rigidly defined (you're given an exact set of functionality to implement) and small-scope (you write code once and never touch it again). There's great value in just sitting down by yourself a few times a week and building something: not knowing what exactly you're making nor what "done" even looks like. Being self-guided is an incredibly useful skill: in the real world, you won't have assignment descriptions or TAs to guide you to a solution, and there's often no exact description of what a solution looks like. Going from nothing to a functional, complex emulator shows mastery over this skill.

Large-scope projects are also fundamentally different to work on. While I'm all for building small, toy projects to learn technologies or solve simple problems, they don't help you learn how to structure a program. While group work can help, I'd make the case that it's better to learn design patterns in a solo project since you'll never encounter a pattern you aren't comfortable with. An emulator is usually a large enough project that you can't fit the entire thing in your head at once, meaning you'll need to rely on good design to help you navigate it.

Realistically, the code you're writing probably looks nothing like an emulator. Maybe you're doing frontend development, or writing a backend CRUD app, or even doing some embedded work. That said, I hope I've made the case for how writing an emulator is an immensely useful exercise regardless of the flavor of software development you do.

88x31 Dungeon is a set of "games" that allow you to traverse the graph of 88x31 buttons, which are small, pixel art buttons that website owners include to link to related people and projects. It's based on data collected by eightyeightthirty.one, a scraper which originally mapped the full 88x31 network. The graph is used to generate a "dungeon" grid, with each room containing a particular website. This project can be thought of as an alternate way to visualize and interact with the graph created by eightyeightthirty.one.

Experiences

88x31 Dungeon provides a set of related experiences, each providing a different way to interact with the graph.

Birds Eye

The birds-eye view shows the full grid and allows the user to scroll across the map. This was the first and simplest experience I implemented, and is useful mostly to visualize the layout of the dungeon. Each square renders the site's 88x31 and domain name, and it's possible to "fly" to a specific site.

Navigate

Navigate embeds each site with an <iframe> and allows you to visit its neighbors using arrow keys. It's loosely inspired by the "dungeon crawler" format, but the arrow buttons at the top let you see the domain of the site you're about to navigate to. It's almost like a two-dimensional webring. I find it fun to move quickly through this one, getting a feel for each region of the dungeon by seeing each site for a few seconds.

Walk

Walk presents a draggable canvas on which rooms of the dungeon are embedded as <iframe>s. The buttons at the left allow you to switch between "drag mode" and "interact mode," where the latter allows you to scroll, click, and interact with each website on the grid. Unlike the other frame-based experiences, Walk shows up to nine frames at once, letting you see each site in the context of its neighbors.

A limitation of Walk is that each site is rendered in a relatively small viewport, breaking some sites. This is particularly an issue on mobile browsers where the viewport is already very small.

Crawl

Crawl is the most similar experience to a text-based dungeon crawler. At the top, the site is embedded in a 4:3 <iframe>, and at the bottom, the user can run commands in a terminal.

Crawl provides the following commands:

Technical Description

This project consists of two parts: the code to generate the dungeon, and the frontend experiences.

Dungeon Layout Generation

A full view of the generated dungeon layout.

The dungeon is generated using a breadth-first greedy algorithm, which is roughly defined as:

I'm happy with how this algorithm creates neighborhoods, and it places a good portion of the total graph. However, it struggles with cliques, since each node can have at most 4 neighbors.

Putting Sites in Frames

Each experience is implemented differently, but most required embedding untrusted sites using <iframe> tags. Iframes are a relic of the old web, and honestly probably would not become a web standard if they were proposed within the last few years. A few issues I ran into included:

React reusing frame elements

Previously, some parts of the implementation looked like this:

return <iframe src={`https://${room.domain}`} />;

The subtle bug here that creates undesired behavior is that when room changes, the iframe element is reused instead of recreated, so the old frame contents persist while the new ones are loaded. This leads to a confusing user experience, especially in Walk where frame positions are constantly changing positions and sources.

React provides a key parameter for lists to tell what elements to create, destroy, or reuse. The best solution I found in this case is to create a 1-element list and use this:

return <>{[<iframe src={`https://${room.domain}`} key={room.domain} />]}</>;

Dead links leading to frames not loading

If you scroll for any appreciable amount of time, you're likely to find either the "page crashed" icon (in Chrome) or a "Firefox can't display this page" error (in Firefox). Unfortunately, many targets of 88x31 links are dead pages. While I'd love to show a fallback UI containing the 88x31 button itself in these cases, IFrames don't send error events since it could be used to "probe the URL space of the local network's HTTP servers." Since there are many other ways to probe the URL space of the local network's HTTP servers from a website, I'm not sure what the merit of this policy is, but as web developers we must accept that we live and die at the whim of browser makers.

Amazingly, someone found a batshit insane way to solve this problem, but I'm hesitant to use a brittle solution like this.

Sites setting the X-Frame-Options header

A lot of sites set the X-Frame-Options header (or its modern alternative, the Content Security Policy directive frame-ancestors) to prevent untrusted websites from loading them as frames.

There is a legitimate security argument for some sites due to an attack called Clickjacking, in which trusted content is loaded into an <iframe> and buttons, input fields, or other interactive elements are overlaid on top of the frame to capture input. However, this isn't applicable to most simple personal websites.

While it might be possible to build and use a proxy which strips these headers from the site, doing so would violate the spirit of this header: if someone set this header, they would probably prefer their site not be embedded into other sites, and I don't want to override their preference. (I'd still really prefer to show a fallback... alas.)

Sandboxing frames

During early development, I found a few sites acting annoyingly, including one that would navigate the top-level window away (what the hell?). While I'd love it if everyone on the indie web played nice, I did implement some protections using the sandbox attribute.

I added the following flags:

This still lets frames cause problems (lagging the user's computer, autoplaying annoying audio, etc), but the worst of the problems have been dealt with. Plus, this project deciding what's annoying and restricting features of iframes runs counter to the spirit of showcasing the variety present in the indie web. As such, this approach strikes a balance between maintaining the availability of the site and allowing for creative expression.

Conclusion

While some parts are a little clunky, the intent behind this was less a polished end result and more an experiment into designing interesting and fun experiences which are part game and part visualization tool. I'm happy with the variety present in the end result. Undoubtedly I think I'm going to end up finding more ideas for experiences, and I'm excited to explore this idea further.

To paraphrase a close friend of mine:

Boston is like a Grand Theft Auto map. There's a lot to do, but it never really feels all that spread out.

I moved to Boston almost three years ago from today. What I love most about this city is how accessible it is. Here are, roughly chronologically, the ways I've picked up to navigate here, and my reflections on city life.

The Subway

By far the most obvious answer to "how do you get around?" is the subway. Despite distinctly remembering a CharlieCard shortage for around a month after I arrived in the city, I quickly started using the system to explore.

The subway gave me direct access to downtown, and somewhat efficient access to many other places around the city. But using it also feels kind of like "fast travel" in a video game: you hit a button somewhere, wait for the map to load in, and boom, you're somewhere else, bypassing all of the beautiful level design in between.

Walking

When I first moved here (and didn't have many friends), I would sometimes pass the time by just stepping outside, pointing myself in a random direction, and walking. When I would get tired, I would start looking for a T stop or just turn around and head home.

Shortly after, I started challenging myself to walk places without looking at any map: learning street names, orienting myself based on the skyscrapers and cranes overhead, and taking plenty of wrong turns in the process.

I love the chaotic street network of Boston, and how it always gives the sense that there's more to explore.

Walking puts me on the same scale as the world around me. I can look into any storefront, stop to peek down side streets, or take in the scenery around me. To this day I'll still happily walk impractical distances from time to time for no reason other than to take in my surroundings.

The Bus

It took me about a year before I started taking MBTA buses. Many friends I've talked to share the same experience. I'm not sure if it's because the bus network is harder to understand, buses are the more "looked down upon" form of transit, or if those who are new to city life, like my past self, are just less aware of the utility of local bus routes.

I've found the social dynamics on buses to be quite different to the subway. You'll see tons of people piling into the back doors on the Green Line to avoid paying the fare, but you almost never see that on buses. On the bus, there's a much stronger sense of "we're all in this together."

Buses also let me expand my radius of places I can explore. A few of my friends live in places vaguely accessible by infrequent bus routes, and while we'll usually opt for the more practical option of meeting at a subway line terminus, it's fun to occasionally take a long, winding transit journey.

Commuter Rail

While the most expensive transit mode on this list, the commuter rail is hugely useful both for going beyond the confines of the city and quickly getting to and from downtown (provided you time it right). Honestly, it's one of the modes I wish I took advantage of more.

Also: the $10 weekend passes are an insanely good deal. Use them.

BlueBikes

I didn't start biking in the city until recently, but I immediately fell in love with it. Biking immediately felt like the "just right" scale: when riding, you can see every detail of every street, you have full autonomy to stop and explore and detour to your heart's content, but you also get places fast.

When I was young, I used to watch those Casey Neistat vlogs where he lane-splits through NYC traffic on a skateboard, and I always wished I could experience a feeling like that. Biking gives me that excitement: navigating complex situations at high speed, finding miniscule ways to shave time off of trips, and playing my small part in the chaotic yet coordinated motion of traffic.

I expected biking in Boston to be scary, mostly because people tend to make it out to be. But coming from narrow, rural, two-lane roads with large trucks whizzing by at highway speeds, the city feels like a cakewalk by comparison. Sure, you're navigating way more interactions with cars than you would be otherwise, but you don't get nearly the speed differentials of somewhere more rural.

That said, if you take anything away from this post, wear a helmet. One of my most "I can't believe I didn't buy this sooner" purchases was a nice folding helmet that fits in my backpack, so I'm ready to ride safely whenever.

Cars

My girlfriend and I are looking to buy a car soon. While I'm excited to have easier access to places beyond the city, I'm simultaneously nervous that the convenience of car trips will make me lose out on the connection to the city that life without a car has allowed me to build over the past three years.

To end on an anecdote: Once, after a robotics competition in rural Utah, a close friend of mine picked me up to drive me to her home in New Mexico. Sitting on the center console was a notebook on which she had written the numbers: 70, 191, 491, 160, 84 -- the highway numbers we followed on our drive back.

I asked about the notebook, and she said it she was trying to use Google Maps less since she felt it made her less connected to the areas she drove through: instead of looking out the window to understand and be present in the places you drive through, it encourages you to just follow the arrows, ignoring the world around you.

Whenever I navigate a place by memory or intuition, be that the chaotic streets of Boston, the sprawling MBTA bus network, the back roads of small towns in Maine, or the Albuquerque airport I hadn't been through in a year, I feel, in a way, welcomed by the communities I pass through. I feel nostalgic for the times I passed through those same places before. I feel excited to explore, knowing that wrong turns can be less of an annoyance, more of an adventure. I feel grounded, able to anchor my experiences to the world happening around me as I traverse through it. Above all, I feel content, no longer stressed by the arrow on my phone screen, just present in the here and now.

My own card as it appears in my account.

Overview

Rolodex stores a virtual contact card with a callsign, name, and other metadata for each of your amateur radio friends. Cards can be viewed, created, and updated on a desktop or mobile device.

Motivation

My friends and I often use local amateur radio repeaters to communicate, so I often need to call one of them by callsign. I didn't feel like I had a centralized place to store this information: a text note would be difficult to navigate and annoying to sync between devices, a traditional contacts app lacks an appropriate field for callsign or DMR information, and I wanted something that I could search by callsign in case I can't quite remember who holds a particular call. The resulting app is a single place where I can track all of my friends' calls, and is something I can easily reference from my phone in the field.

I chose the name "Rolodex" because it conveyed the purpose of the app (storing contacts), and it sounded unique, functional, and a little playful.

Technical Description

The frontend uses the stack I tend to reach for for frontend work: Vite, TypeScript, React, and Tailwind.

Cards use a fixed aspect ratio to follow the ID-1 spec. I used the morse code version of the callsign as an artsy divider between the callsign and name -- this uses a specific morse code font file. The interface uses a mix of DIN and JetBrains Mono, two fonts I haven't worked with much before, since I felt they worked together well and gave each card a utilitarian look without being off-putting.

The "rolling" animation in the column view works by checking where each card is in the scroll view with a JavaScript event, then applying rotation in X and translation in Y and Z. This took me the most effort to get right, and I'm probably going to continue tweaking it to improve its behavior across various screen sizes.

The backend of the app is entirely done in Firebase. This was my first Firebase project, and I figured it was a good fit since the requirements are quite standard (it needs a simple authentication system and a basic way for each user to store and retrieve a small amount of data).

The app is a Progressive Web App, allowing me to install it on the home screen of my phone.

Results

You can try it at rolodex.breq.dev! At the time of writing, we have 5 total users, including myself! Personally I have more than a dozen contacts saved already, and am constantly adding more. I've already found it quite useful, and am looking forward to using it more as more of my friends get licensed and have callsigns issued to them.

Nostalgia

When I was younger, I had a PocketCHIP handheld that I would carry around. It's what I wrote the MakerGamer fantasy console for, and it's the device I first started experimenting with batman-adv mesh networking on.

I loved being able to have a Linux box with me that I could use for experimentation. The keyboard sucked, and the interface wasn't amazing on a tiny-resolution touchscreen, but that didn't matter for quick usage, and it was still miles ahead of pulling up Termux on my phone.

Part of what got me thinking about this is seeing a few of my friends pick up Flipper Zero devices. These have that quality of being a device for "real-world" tinkering and exploration, encouraging people to interact with systems in the world around them. The main reason I don't have one is just that I'm not particularly interested in RFID/NFC applications (but I applaud those who are).

About a year ago I picked up a cheap Baofeng handheld radio for amateur use. While it's a practical way to communicate with friends on a short range, I've found the most fun part is using it to explore the world around me -- finding repeaters and trying to communicate with them, listening on railroad or marine channels, and generally exploring to see what systems out there that I can observe. (See codeplug for how I've iterated on this process over time).

I want a device which combines the convenience of a Baofeng, the customizability of a Flipper, and the power and peripherals of a PocketCHIP. But more broadly, I want something that encourages that sort of exploration with a physical element to it.

There was a time recently when I was sitting out in a park downtown, poking at some interesting stuff accessible over Wi-Fi that I'm not quite ready to talk publicly about yet. While I was doing this on my laptop, doing so felt odd given all I needed was a basic environment where I could configure my Wi-Fi settings and invoke SSH. A dedicated handheld device would have been just as functional and a bit more fun.

My Dream Handheld Computer

Here's a rough list of requirements that I'd look for in such a device. This is based on my own interests and the areas of tech I tend to explore most.

Existing Products

Dead Stock PocketCHIP

Up until recently, the website pocketchip.co has sold old PocketCHIPs (and still sells CHIPs, albeit at 4x the original selling price). Even if they were still available, though, the lack of up-to-date software for them makes common tasks like flashing a hassle.

Clockwork uConsole

The Clockwork uConsole is probably the closest thing I've found to the device I want, but since it's designed more for "fantasy console" use, it falls short in a few ways:

It's also still limited to pre-order.

Whatever arturo182 Is Cooking

Solder Party has previously sold a "Keyboard FeatherWing" containing an LCD, QWERTY keyboard of the type you'd find on a Blackberry phone, and a place for an Adafruit Feather board. While a Feather doesn't provide quite the amount of computing power I'd like for such a device, I love the form factor and the pricing is reasonable.

Rooted Android Devices

What about carrying around rooted Android phone for something like this, or rooting my daily driver? This would give an unrestricted Linux environment to play around with, but it also falls short of what I'm after:

Designing My Own?

While I'm relatively comfortable with basic PCB design, this is something that's likely way beyond my skill level. Plus, I'd ideally like to have something that can have a community form around it (like PocketCHIP did, or like Flipper Zero does now). There's also the enclosure to consider -- PocketCHIP did this relatively well and the Flipper does it perfectly.

I'm largely ruling this out unless I suddenly meet lots of people who share this vision but have much stronger electrical/mechanical skills than I do.

Something Else?

If you can think of something that fits in this general category/vibe of product, please let me know about it! Even if it isn't what I'm personally looking for, I really want to learn more about projects in this space.

The codeplug generation tool in action.

Overview

Codeplug is a tool to automatically generate a configuration file (or colloquially, a "codeplug") for a handheld amateur radio.

The tool prompts users to select their geographic regions of interest and to choose which common simplex channels on amateur and other bands to include (such as calling frequencies, FRS channels, etc.), and builds a .csv file containing those channels and channels for radio repeaters in their selected regions.

Motivation

Many of my friends are licensed amateur radio operators, and when we travel together, we often exchange notes on which repeaters to program into our radios. However, as I've made more extensive use of my radio, my previous strategy of including every repeater I can think of in my codeplug has caused me to hit the channel limit.

Ari and I had the idea for a program which can automatically generate a suitable codeplug based on a subset of regions, channels, etc., allowing us to easily program in relevant channels to our radio based on our current interests, upcoming trips, and more.

Technical Description

The tool is a CLI app built with Inquirer for Python. It loads repeater and channel definitions from .yaml and .csv files, and assembles a .csv which can be imported into CHIRP, a common tool for editing and uploading these configs.

Results

Instead of continuing to maintain my codeplug by hand, I'll be using and improving this tool going forward. A few of my friends have already offered to contribute repeater information to it, and I'm planning on extending it to include AAR channels by region as well.

Picture this: you're developing a robot that you aim to control using a nice, clean user interface. Control goes through this UI on one laptop over a network connection to the robot, which runs its own SBC, all using standard ROS 2. You do a ton of testing, at first just with an Ethernet cable since you don't feel like dragging your radios out of the closet, and everything looks good! And then, you show up to an event with your robot, and you realize that whenever you switch views in the UI, suddenly your entire comms link grinds to a halt.

So, what gives?

Discovery Traffic and Fragmentation

This issue is a classic case of discovery traffic trashing your link. Whenever the UI subscribes to a new topic in ROS, the Data Distribution Service (DDS) middleware needs to figure out which node in the network is publishing that topic, so it generates a ton of multicast UDP traffic to send out to the network. This works fine over a high-bandwidth Ethernet link, but causes problems with a bandwidth-constrained radio connection.

When our team hit this, we took some Wireshark captures before and after. Here's the normal, working state of the network:

And here's after we generate a bit of DDS discovery traffic:

The unassembled fragments! The horror...

If you're not familiar with fragmentation, it's something that happens at the network layer to break large packets into smaller ones to fit your particular network's maximum transmission unit (MTU), which is usually 1500. This disassembly and reassembly is typically transparent to the user, and is handled at the OS level. The fact that Wireshark is showing these unassembled fragments implies that they couldn't be reassembled correctly -- some of the fragments were dropped, and now the message can't be recovered.

In other words, this is not cute, computer networks only do this when they're in extreme distress.

Tunable Parameters

The ROS 2 documentation suggests a few knobs which you can turn to hopefully improve performance:

Alternative DDS Middleware

One thing to try is changing the actual DDS implementation you're using. DDS is an open standard, and there are plenty of implementations to choose from. The popular implementations all have corresponding ROS middlewares written, and it's trivial to switch from one to the other.

The default middlewares so far have been:

In addition to FastDDS and CycloneDDS, there are ConnextDDS and GurumDDS, but those lack open-source licenses.

Interestingly, Cyclone is seemingly more loved by the community than FastDDS: The MoveIt inverse kinematics tool recommends Cyclone. For what it's worth, switching to Cyclone largely fixed our discovery traffic problems:

DDS Overhead

This slide deck by Charles Cross goes into detail about some techniques for optimizing ROS 2 traffic. One often-overlooked feature of networking in ROS 2 is the amount of overhead for each message.

Here's a packet containing a command that we sent to the left wheel of our robot. It's a single 32-bit floating point number, so the value itself takes up 4 bytes. How much overhead does transmitting this incur?

That's 138 bytes over the wire:

In other words, sending each message requires 132 bytes of overhead, regardless of the actual size of the message. With this in mind, we can greatly reduce the load over the network by combining as much as possible into as few topics as possible. In practice, here's how we applied this:

Custom Messages

Wherever possible, make use of ROS 2 custom messages. Each new system we build starts with a *_msgs package, which includes message definitions tailored to that stack.

There are lots of reasons to do this other than performance: it makes the development experience much better by giving meaningful names to values, it catches semantic "mismatches" caused by improperly hooking up a publisher and subscriber, and it helps make invalid states unrepresentable (can you tell I'm a Rust fan?).

For the above example, our team now uses a DriveCommand which encodes the speed of each of our six motors in a single message, each named semantically:

float32 left_front
float32 left_middle
float32 left_back
float32 right_front
float32 right_middle
float32 right_back

Large Messages and Fragmentation

Our team ran into another issue with ROS 2: heavy fragmentation with large message types.

While filming a demo video of our system, we wanted to include a visualization of point-cloud data from our stereoscopic camera (a Zed 2i by StereoLabs). Last year, we got the shot (video here):

We had no trouble doing this in ROS 1 -- as our communication link degraded, the point cloud data just started to slow down. However, in ROS 2, this demo completely broke down.

The key difference here is that ROS 1 used TCP, while ROS 2 uses UDP. Each point cloud message is big: our sensor is quite high-resolution and each pixel needs a color and position in space. With a TCP connection, congestion control takes care of things, slowing down the link and causing frames to be dropped to keep things working -- this slows down the framerate at the output but otherwise works fine.

In ROS 2 using a DDS middleware, messages are sent over UDP. However, remember that frames on most networks are limited to an MTU of 1500. When you send a large message using a DDS, the DDS layer just creates a single massive UDP packet, and the OS is responsible for fragmenting it appropriately. If the computer on the other end can defragment it properly, you're in business. In a congested network, the chances of successful reassembly fall to near-zero, meaning you'll get nothing out of the other end of the connection.

Here, you've got a few options:

QoS parameters

ROS 2 introduced Quality-of-Service parameters on subscribers and publishers. A QoS "profile" includes a few parameters (listed here roughly from most to least important):

While subscribers and publishers use the same profiles, the work in slightly different ways. A subscriber's profile is the minimum quality it's willing to accept, and the publisher's profile is the maximum quality it's willing to provide. This makes a bit more sense when you look at examples:

Zenoh

What if, however, you weren't at the whimsy of the DDS protocol at all? If you let go of the purism of the DDS standard and instead substituted a more efficient protocol which could better handle the congested, dynamic conditions of a lossy wireless network?

This is where Zenoh comes in. Zenoh is a pub/sub protocol designed with one goal: reduce network overhead as much as possible. Most importantly, it has plugins tailored at interoperating with existing DDS systems.

When working with ROS 2, Zenoh dramatically reduces the amount of discovery traffic generated (by up to 97%) compared to a traditional DDS middleware. We've also found it to be much better at communication in general even after the discovery state is finished. We're not the only ones: an Indy Autonomous Challenge team made the same observations.

So how is Zenoh some kind of miracle for communication? It's a well-designed protocol, but more importantly, it's not held down by the restrictions of the DDS standard. DDS is based on a protocol designed in 2003, and is designed foremost for interoperability, not efficiency. You can see this in the protocol's design: instead of defining a standard byte order, it's defined on each message based on a flags register; information about the DDS system vendor is sent with every packet; each node is required to maintain a full graph of the network; and entity IDs are duplicated between submessages in a packet instead of being sent once per packet.

ROS 2 was initially pitched as a DDS-centric version of ROS, replacing the bespoke TCPROS with DDS as an open, standard communication protocol. However, frustrations with the complexity of the DDS implementations and their poor performance on lossy wireless networks motivated OpenRobotics to search for an alternate middleware. Unsurprisingly, they chose Zenoh.

If you're reading this after May 23, 2024 (World Turtle Day), ROS 2 Jazzy Jalisco will be out, and hopefully rmw_zenoh with it.

In the meantime, we're stuck running a DDS on each host and using plugins to bridge the gaps. The best approach as of writing is to use zenoh-plugin-ros2dds, setting ROS_DOMAIN_ID to a different value on each host to prevent DDS communication over the network. Old tutorials used zenoh-plugin-dds, which works for ROS 2 applications but doesn't support ROS 2 tooling as well.

Zenoh ended up being the solution to our team's DDS dilemma. We plan to run Zenoh at the 2024 University Rover Challenge.

UDP vs TCP

When running over a network, Zenoh can run over either TCP or UDP. Typically, TCP is usually chosen as a network protocol for its reliable transmission guarantees, so it may seem like a poor fit for a system where some topics only need best-effort transmission. We've found, however, that our system tends to work much better over TCP than UDP, and we believe this is due to the TCP congestion control mechanism.

TCP uses a complex congestion control algorithm to avoid sending too much traffic over a congested network and causing undesirable behavior. With our radio network, the constrained link is specifically between our two IP radios, Each radio has a full 100 Mbps link to its computer (either the Jetson on the rover or the laptop in the base station), so the computers running ROS have no direct knowledge of network conditions. Then, when naive UDP protocols like DDS send way too much traffic over the network, it causes packets to be dropped on the radios, and the only feedback each host receives is a lack of acknowledgement after a timeout. On the other hand, TCP congestion control minimizes packet drops in the system, causing more reliable transmission.

ros2-sunburst

In true breq.dev tradition, this post leaves off with a tool I've written to help solve some tiny facet of this problem. In this case, it's a bandwidth usage visualization tool for ROS 2 traffic.

Here's a plot of our team's bandwidth usage by topic:

In ROS, topics tend to follow a hierarchy structure: your robot's drive stack might fall under /drive, commands might be under /drive/cmd_vel, etc. This structure isn't inherent to the transport protocol in any sense, but it does help you reason about the parts of your system which are using the most bandwidth.

Using this tool works as follows:

You can grab the script from this gist.

Radios are often an afterthought when designing a robotics project. Few robots are self-contained: almost all need to communicate with some external system, be that a basic remote control or a complex offboard system for advanced processing.

Most useful robots also need to operate in an uncontrolled environment. A Roomba needs to connect to a customer's Wi-Fi network, a self-driving car needs to operate reliably regardless of local RF conditions, and a photography drone can't fall out of the sky if an FPV racer sets up shop at the next field over.

My experience with this is primarily in the space of wheeled, "prototype-scale" robots of varying sizes. There is much to consider when scaling up a system beyond one or two prototypes, and the math tends to work out differently for flying 'bots compared to terrestrial ones. (And if what you're building is a boat, good luck -- RF and water do not mix well, and the Fresnel Zone is your enemy.)

Hardware

Radio hardware is built for a wide range of applications, meaning it's difficult to even enumerate the options available to you. Especially for a prototype or hobby project, expect to look in interesting places for what you need to make things work.

Consumer-Grade Networking

We've gotten quite good at building cheap radios to connect consumer devices over short distances, over Wi-Fi, Bluetooth, or other protocols. While these chips might have been designed for things like smart lightbulbs or wireless headphones, they're often a great choice for short-range networks and can often provide substantial throughput.

Your built-in Wi-Fi

Sometimes, the best radio is the one you already have. If your project uses a recent Raspberry Pi model, and your laptop has a Wi-Fi chip, then you're all set for short range communications!

With Wi-Fi, one device must be the access point and the rest are stations (clients). There are a few advantages of making your robot the access point:

This is a great option for running your robot at events where you won't know the setup beforehand, provided you don't expect too many other robots taking up their slice of the 2.4 GHz spectrum.

But it does come with a major limitation: It is generally quite difficult to create a setup in which your robot, as the access point, is able to access the internet via one of its clients. If your laptop is a station on your robot's network, then whoops, it can't be on your home network anymore. Even if you plug your laptop into an Ethernet connection with internet access, good luck getting your laptop to route traffic from your robot to the Internet and back. It can be done, but it takes some complicated network configuration that isn't fun to deal with.

Bluetooth

Bluetooth is great as a "it just works" system, requiring just the initial pairing flow before you're off to the races. I used it to send commands to the LED choker that I made. You can get up and running quickly with a microprocessor with Bluetooth support (like the nRF52480) or a drop-in Bluetooth Serial module (like the HC-05 available from tons of sketchy Amazon sellers).

Bluetooth modules are typically much lower power than their Wi-Fi counterparts, don't usually work in a scenario more complex than a point-to-point connection, and have substantially worse throughput. Different Bluetooth profiles support sending different amounts of throughput through, but something like the Serial Port Profile will struggle to transmit images or video.

Bluetooth also doesn't typically use the TCP/IP stack (although it can be done with a specific profile). This is typically fine for microcontroller-based robots, but can make development annoying if your robot runs Linux.

External Wi-Fi routers and cards

Still within the consumer realm, some off-the-shelf Wi-Fi routers and cards can be great for getting a bit more performance or range out of a Wi-Fi based system.

Consider a Wi-Fi router on your robot as an extension to the "robot as access point" idea in the prior sections. You get a physical piece of hardware with a sole purpose of providing a Wi-Fi network, along with Ethernet ports to connect any computers onboard your robot. Wi-Fi routers typically use higher power, have better antennas, and can provide more throughput than (ab)using your device's built-in radio.

Alternatively, you can put the router at a base station and connect it (e.g. via Ethernet) to give it Internet access. This gives you the benefits of a wireless network entirely under your control while still giving your robot and ground station Internet access for development.

On the station end, you can still improve compared to whatever built-in adapter you have. Some UAV enthusiasts working on the OpenHD project have identified the Realtek RTL8812AU and friends to be a strong contender, having the maximum allowed 500mW power output, two antennas for diversity and MIMO operation, and support for both the 2.4 GHz and 5.8 GHz bands (which we'll discuss more in the Band Planning section).

ESP32 chips

Back in my day, all we had was the ESP8266, a cheap board with no GPIO, bad hardware support, and acceptable 2.4 GHz Wi-Fi in a very cheap package. Nowadays, Espressif Systems has unveiled a line of ESP32 boards with ample GPIO, fast clock speeds, and both Wi-Fi and Bluetooth capabilities.

Choosing an ESP32-based design gives you lots of options for small robots, either as the robot's primary microcontroller or as a peripheral. For instance, you can use Bluetooth to set up the initial connection, then let the user input their Wi-Fi details and switch to that connection. Or, you can use Bluetooth for telemetry and controls, while keeping a Wi-Fi link during development for stronger visibility.

Of course, this series of chips is in the microcontroller realm, so they won't be able to push through as much throughput as a dedicated Wi-Fi router or card. for small and short-range projects, however, they can be a strong contender.

Drones

The world of UAVs has brought forth many unique solutions for radio communication. Unlike wheeled robots, you can typically assume that a drone will maintain line-of-sight with the control station at all times, giving signal penetration a lower priority. Drones also are extremely sensitive to weight, so solutions tend to be compact and simple.

One quirk of drone communications is that drones typically use two frequencies: one for control and basic telemetry, and another for FPV/video streaming, leading to two very different sets of solutions.

For controls, the RFD900 is by far the most well-known choice. It provides a basic serial connection which typically facilitates the transfer of Mavlink messages. Unlike any of the other solutions we've looked at thus far, the RFD900 works on the 900 MHz band.

The RFD900 uses frequency-hopping spread spectrum: the specific frequency used will jump around in the 900 MHz band to avoid interference. This is great in a scenario with other RFD900s (like an FPV drone racing event) since they'll generally stay out of each other's way.

FPV drones also typically use analog video systems. There are a few reasons why these have stuck around, even though they use spectrum far less efficiently than digital systems:

Because of this, analog video gear is generally commonly available. If you're working with a small robot with only a single camera over a relatively short distance, this can be a good solution. However, note that you can't perform any onboard processing with such a camera.

WISP Gear

Wireless Internet Service Providers, or WISPs, are companies that provide Internet access to customers using wireless links as opposed to wired (fiber or copper) connections. Equipment made for WISPs is built to be reliable, designed for long-term operation, highly configurable, and to operate right up to the power limits allowed by law.

Ubquiti products are cheap and work well, including the Rocket and Bullet series of radios. Each line comes in 2 GHz and 5 GHz variants, with 900 MHz variants discontinued but often found on eBay.

For stronger 900 MHz capabilities, consider the Cambium PTP 450 900. Note that these use a proprietary protocol. While Cambium makes some other variants, they're either difficult to get licenses for (like the PTP 450 on 3.65 GHz) or prohibitively expensive for most use cases (like the PMP line, which supports point-to-multipoint scenarios across many band types).

These radios provide an Ethernet port and can effectively make your network look like a flat Ethernet subnet, greatly simplifying your configuration. Alternatively, many Ubiquiti radios can be configured to operate as a standards-compliant Wi-Fi hotspot instead of in point-to-point mode, allowing you to combine a high-powered WISP radio with inexpensive consumer Wi-Fi gear.

Packet Radio

A promising method I've been looking for an excuse to use is low-cost packet radio modules. These implement a barebones protocol without any sort of pairing, connection, or scanning logic. They typically support relatively slow data rates, but can cover a relatively large area.

These radios tend to transmit in one of two bands:

An iteration of these uses LoRa, a modulation technique that enables much farther communication (although this increases cost).

Amateur Hardware

Another source of radios is hardware designed for amateur radio use. These operate on a huge range of bands, but typically focus on voice transmissions. The digital modes that do exist are intended for short, infrequent messages. This is typically not a good fit for a robotics scenario, but it could be useful.

On the absolute low end, you can always use a cheap Baofeng radio (usually ~$20 USD on Amazon) with your computer's sound card and some software for a basic digital mode.

Of course, amateur hardware also has restrictions (e.g., no commercial use) which we'll talk about more when we discuss amateur licensing.

Band Planning

When picking a radio, you'll often want to consider the frequency band it transmits on. Specific bands have certain innate characteristics which make them more or less suited to a given application. You'll also need to consider other radio systems within your robot and ensure you do not cause interference.

Different bands have different strengths and weaknesses. Generally, lower frequency bands will provide better signal penetration and non-line-of-sight operation (e.g., operating your robot from the other side of a wall), while higher frequencies are better suited to line-of-sight or short-range uses.

An advantage of higher-frequency bands is that they generally can provide much higher bandwidth, both via higher signal rates and wider channels. If your system involves streaming lots of high-resolution video, point clouds, etc., you will probably struggle on a lower-frequency setup.

If your robot has multiple communication systems, you'll need to make sure you aren't causing interference to yourself. This could take the form of multiple links to your base station (such as separate links for telemetry and video), or it could arise in a system with multiple interacting robots.

Generally, for multiple radio links to your base station, you will want to put each link on a separate band to minimize interference. For instance, many drone pilots use 900 MHz or 2.4 GHz for teleoperation and 5.8 GHz for video streaming.

Interference can also be tricky to identify the source of. On the Rover team, we noticed an issue with our 900 MHz radio gear interfering with our high-precision GPS receivers, preventing us from attaining a precise fix.

Your robot might also need to operate in an area with lots of interference, such as a Maker Faire, in which the sheer number of users on the 2.4 GHz and 5 GHz bands can make protocols like Wi-Fi almost unusable. While frequency-hopping protocols can work more reliably, they still have their limitations.

Amateur Licensing

If you're a hobbyist, you aren't just limited to the ISM or unlicensed bands: the Amateur Radio Service is there for you! Getting an amateur license gives you access to a much larger array of bands across a wide range of radio spectrum. While this varies by country, most countries do something similar to the U.S.:

The most important restrictions to keep in mind are:

Getting licensed is generally straightforward (Becky Stern's article gives a good overview of the process). A few tips from my own journey:

In the U.S., there are a few classes of license, and most countries follow a similar structure. At time of writing, I hold a technician class license, and it gives me almost all of the same privileges as the higher classes on the shorter bands (433 MHz and up).

Some amateur bands overlap with unlicensed/ISM bands: the 33 centimeter amateur band takes the same space as the 915 MHz ISM band, the 2.4 GHz amateur band partially overlaps the ISM one, and the 5.8 GHz amateur band is a subset of the spectrum set aside for Wi-Fi. In these bands, you can often use commercial grade hardware at higher power limits or with larger antennas -- for instance, running a Ubiquiti access point on 2412 MHz at maximum power with a high-gain antenna. This gear generally isn't designed for use above the unlicensed limits, so you won't always gain a significant boost, but it can still give you a bit more range.

Other bands are generally pretty quiet except for amateur users:

Conclusion

Based on my experience so far, I've developed a few rules for selecting which system to use:

Background

You might've noticed the tiny buttons/badges at the footer of this site and others, each showing a pixel-art design and linking to another website. These are called 88x31 buttons, and they're a de-facto standard of the indie web. Sources on this are few and far between, but The Neonauticon identified the start to be Netscape, which published "Netscape Now" buttons that sites could add to show their use for then-new web features.

People took this idea and ran with it, placing 88x31 buttons for themselves, their friends, and projects they supported in their websites, forum signatures, etc. Similar to webrings, 88x31 buttons provided a fun way for people to find related content to a given page.

My friends and I are working on mapping the entire 88x31 graph. The result of our work is the first ever snapshot of the entire 88x31 network, available in a single JSON file.

Relatedly, a few semesters ago, I took a course on network science. One of the main takeaways was that lots of real-world networks exhibit something called the small-world property, specifically:

We see this phenomenon across all sorts of network types: the original 1998 Watts & Strogatz paper uses networks of film actors, the electrical grid, and the neural network of a worm.

So: we've got a brand new dataset and some criteria to test. Does the 88x31 graph exhibit the small-worlds property?

Analysis

To analyze the behavior of the graph as it expands, I'll be using two datasets: one with 4428 nodes that we obtained partway through the scraping work, and the most recent dataset with 16023 nodes. Analysis was done using Gephi.

Clustering

Clustering is the measure of how locally-connected a graph is. Think about a social graph: if everyone had a set of, say, 20 friends randomly chosen from a group of 1 million, you'd have quite low clustering. However, typically, you're much more likely to be friends with your friends' friends, implying high clustering. Clustering is quantified by the clustering coefficient.

The formal definition of a graph is a set of vertices (nodes) $V$ and a set of edges (links) $E$, such that:

We write that edge $e_{ij}$ connects vertex $v_i$ with vertex $v_j$.

To define clustering, it's useful to define the neighborhood of a vertex $N_i$ as the set of all of its connected neighbors, considering both links "in" and links "out":

Let's go a step further and define the degree of a vertex $k_i$ as the number of connected neighbors it has, i.e.,

where we use the vertical bars to mean the cardinality (number of items contained within) the set of neighbors.

How many links could exist within the neighborhood of a node? Let's ignore self-loops, so the number of possible links is $k_i \cdot (k_i - 1)$ -- each node can connect to each of the other nodes.

The number of links actually in the neighborhood can be written as:

And thus, the clustering coefficient for a given node is:

We can use software to compute the clustering for each node, and take the average across all of them. The average clustering coefficient is:

Is this value high enough to indicate high clustering? For comparison, let's construct two random graphs with the same number of nodes and edges as each of our sample datasets: one with 4428 nodes and 17254 edges, and one with 16023 nodes and 57202 edges. Gephi supports the Erdős–Rényi model, which generates a graph based on the number of nodes and the probability of an edge between any two given nodes. Based on the real graph, we can determine the probability numbers as:

I needed to multiply these by 2 to get them to load into Gephi properly; perhaps there is some correction going on for undirected vs directed graphs.

Finally, we can get the average clustering coefficient for each of these graphs:

Note that the precision of these values is limited by the output of Gephi to 3 decimal places -- those values really are that low!

Our random graph has low (sometimes called vanishing) clustering as the number of nodes increases. This is the typical behavior for a random graph to have -- as graphs grow, the likelihood of a given node being connected to another node decreases, regardless if those nodes are linked by an intermediate node.

In contrast, our 88x31 graph has high (or nonvanishing) clustering even as the number of nodes increases. This is a notable property in and of itself, and brings us halfway to demonstrating the small-worlds property!

Distance

The average path length $L$ of a graph is the average of the length of the shortest path between each pair of nodes, ignoring node pairs with no connecting path.

We can compute the average path length of our graph to be:

For the network to exhibit the small-worlds property, we need the average path length to be proportional to the logarithm of the number of nodes -- that's why I've added it to the table above.

Between graphs, the average path length increased by a factor of $1.57$, while the number of nodes increased by a factor of $3.62$ (and its logarithm increased by $1.15$). This isn't exactly on the expected value for $L \propto \log N$ (where $\propto$ means "proportional to"), but it's also quite clearly less than the amount that $N$ increased, meaning $L \ll N$. Personally, I'm comfortable calling this bound satisfied.

A few reasons come to mind as to why we might not be getting the expected value here:

Hubs

Real-world graphs often have hubs. In a transportation network, hubs are points with connections to lots of other places; in the Internet, hubs are major ASNs like Hurricane Electric; in a social graph, hubs are just people who know a lot of other people.

We can see if our 88x31 graph exhibits this same behavior by looking at the degree distribution of the graph. For this, I generated the degree of each node in the full 88x31 graph in Gephi, then exported the table into Python to take a histogram and then into Google Sheets to make charts.

Above is a log-log plot of the degree distribution of the full 88x31 graph. Let's walk through it, since it's not the easiest thing to digest.

Each x-value represents a degree which a node could have, and the corresponding y-value is the number of nodes with that degree. Per the chart, there are about 10,000 nodes with degree 1, but closer to 500 with degree 10, and only a handful with degree 100.

The "clumping" taking place at the bottom is since we're working with a discrete number of nodes. For each of the higher degrees, we'll probably only see only 1 or 2 nodes with that degree, and whether it's 1 node or 2 nodes is up to random chance. Since there's nothing in between, we see those two lines at the bottom of the graph, and we can notice quite a bit more noise there as well. Similarly, towards the top-left of the graph, we're running into quantization there as the degree of a node must be a whole number.

So what are we looking for here? For reasons which will become clear, let's bring back our random graph from earlier and run the same procedure on it:

That looks very different -- sort of like a bell curve around some single most-popular degree value, giving the network a sense of "scale" or "size." The 88x31 graph, on the other hand, is scale-free in that hubs in the network get larger as the network grows.

Scale-free networks are more rigorously defined as a network in which the degree distribution follows a power law. That blue line in the graph is the power law function that best fits the data. Especially compared to the random graph, it's pretty clear that our 88x31 graph meets the definition of scale-free.

What are these hubs in our network? In our case, they're mostly sites which try to collect as many 88x31 buttons as possible into a single site, like the neonaut collection (which currently holds the #1 spot).

Conclusions

In this post, we looked at an entirely new network dataset through the lens of network science, discovering that it fits the same criteria that characterize social, communication, transport, and biological networks in our world.

Network science is a field that interests me greatly, but that doesn't often come up much in my work, so I'm grateful for the chance to work on this project. A lot of key insights about real-world networks from the field come up in unexpected ways.

We'll continue to track the 88x31 network over the forseeable future. Maybe we'll discover an unmapped part of the network and end up with even more data to sort through. If you'd like to follow the technical effort, feel free to watch or engage via the project's GitHub repository.

Of course, you can join the network as well, and it's as straightforward as it sounds: find a friend with 88x31s on their website, make up your own little 88 pixel by 31 pixel image in your favorite bitmap image editor, and get them to add it to their site along with a link to your website! A few tips:

Hope to see you on eightyeightthirty.one soon!

The entire mapped network of 16,000+ pages, as of 2023-12-26.

This project was a joint effort by myself and a few friends:

Overview

88x31 buttons are everywhere on the indie web -- they're those tiny buttons on the homepage or footer of sites like mine which link to friends, projects, etc. They've been around for decades and have spread all over webpages and forum signatures. However, until now, there has been no way to view the entire network of 88x31 links all at once.

My friends and I have implemented a scraper which can crawl a page for 88x31 links, a server to manage the queue of sites to crawl, and a web frontend to visualize the graph as a whole.

Motivation

NotNite provided the initial idea and implementation, and after adryd sent me a proof of concept, the three of us and some friends immediately hopped on a call to work out the details.

Technical Description

The implementation consists of three parts: the scraper, the orchestration server, and the frontend.

Scraper

The scraper receives a URL and is responsible for visiting the webpage to look for 88x31s which link to pages. We experimented with a webdriver-based solution (using Puppeteer), but ended up switching to static HTML parsing for performance, making the tradeoff that the scraper can't read client-side-rendered pages.

The scraper is also responsible for noting any redirects that happen, and for trying to identify canonical URLs for pages, just as a traditional search engine crawler would need to.

The scraper is written in Rust and keeps no state, so it can be scaled up horizontally as needed.

Orchestration Server

To coordinate the scrapers and provide access control, an orchestration server is used. This server also accepts URLs into the network (adding them to the queue) and produces the graph file.

The orchestration server is backed by a Redis database -- this was chosen to ensure the queue was stored in memory.

The orchestration server also handles inserting found links into the queue, and correcting records after a redirect is found -- both complex processes to handle edge cases that arise when webmasters change things about their site.

Frontend

The frontend is used to display the graph to the user and allow them to navigate it efficiently. It has some functionality for zooming to a particular node and highlighting its links, and it can show the badge images used to link from one site to another.

We struggled to choose an effective graph library implementation which could render this many nodes on the screen, but eventually settled on Cosmograph as it blew everything else out of the water in terms of performance. The end result still takes time to load the initial graph but feels relatively snappy to navigate even on mobile devices.

Results

The result is a map of over 16,000 pages, either linking to or linked from another using 88x31s.

One thing that we didn't expect to happen was webmasters noticing our user-agent in their access logs and inquiring about our work. Apparently folks aren't used to scrapers which specifically target 88x31 images! It was very cool to get to explain this project to those who asked about it, and we made sure to clearly identify ourselves and provide opt-out instructions in case any operators disagreed with our mission.

I also got to finally use everything I learned in a network science course last year to analyze our resulting data, which was quite fun. TL;DR: our dataset of 88x31 links is similar to other social networks which have been studied in the field!

Disclaimer: This article may or may not be an application of Cunningham's Law. I'd love to hear about how other teams work and why people like (or dislike!) their current way of doing things.

I've maintained an (albeit small) open-source library, led software development on a robotics team of >40 people, and worked on a professional software team at co-op. Through this, I've gotten to experience a range of code review styles and merge request etiquette. A common piece of wisdom I've seen tends to be: "keep your merge requests small, so they are easily reviewable." Conventional wisdom states that longer merge requests are more difficult to review -- obviously this is true, since there are more changes to look through. I'd like to argue, however, that in the long run, writing longer merge requests can save time and effort for a team.

Breaking features down is burning developer effort

Suppose you've set out to implement a feature, and you've ended up with a much larger implementation than you expected. Should you split that implementation into smaller merge requests to make life easier for your reviewers?

What makes breaking up merge requests tricky is that you'll need to verify that the state of the repository in between your two merge requests is valid. You need to create an intermediate version, then test that intermediate version. Semantically, this "in-between" version might make no sense, since it contains a partially-completed feature.

If you're using a tool like Rust's clippy to analyze your code, the intermediate version might need to be littered with #[allow(dead_code)] or your language's equivalent in order to pass CI. This clutters the commit history and provides no real value. And what if you accidentally leave in one of those #[allow(dead_code)] functions later?

Splitting a merge request into more than two parts makes this even more complicated. In many cases, it just isn't practical to break a single feature into multiple semantically reasonable and CI-passing intermediate steps.

More frequent code reviews means more "round trips" and greatly reduced speed

When you submit a merge request, you're essentially handing things off to your reviewer. A good reviewer will get back to you in 24 hours, but someone who's busy with other work might take even longer. In this intermediate time, what do you do? In most cases, you'll end up switching to another project, losing steam on the feature you were working on. Only when they get back to you can you resume work on the feature. The more of these "round trips" you have to deal with, the longer it will take you to implement useful features.

The good news is that there are ways to gather feedback from others in the middle of implementing a feature that don't slow you down! You can set up a 1-on-1 to talk through design decisions, exchange UML drawings, pair program, or just Slack your teammate a link to your branch. They can then respond at their own pace, without blocking you from doing your work.

If merge requests queue up, addressing concerns becomes nontrivial

If you're impatient, you might not wait for one of your merge requests to be approved before starting work on the next. After all, they're all part of one feature, so it makes sense to tackle them sequentially. So you start your feature-2 branch before your feature-1 branch is merged in.

In order to pull this off, however, you'll need to branch feature-2 off of feature-1, since the changes in feature-1 aren't in main yet. This works fine for now. And if your code is good, it's not a problem, since you'll just merge feature-1 into main first, then merge feature-2 in after.

This falls apart as soon as your reviewer points out an issue with feature-1. You switch to that branch and commit your fixes. Then, while waiting to hear back from your reviewer, you go back to developing on feature-2. But feature-2 doesn't have those changes, meaning you'll have to either rebase it onto feature-1 (which is annoying if any teammates also checked out that branch) or merge feature-1's most recent commit into it (which clutters up history). Then your reviewer requests more changes, and you need to do the song and dance once again. You probably can't just merge in all of your changes into feature-2 once feature-1 gets merged, since they're so tightly coupled.

This gets quadratically worse if you have a longer chain of merge requests. I've seen situations like this up to four layers deep, where each change to the oldest MR require rebasing every other MR in sequence. This is an awful experience for everyone involved, and it leads to comments on merge requests like "Fixed this in [later merge request], please approve this one now," which is definitely not the way things should work.

Reviewers need to consider a certain amount of context regardless

When you're reviewing code changes, you aren't just thinking about the code itself. You need to consider everything else in the system that the code interacts with.

Suppose you're incrementally replacing a control stack for a device written in ROS. Conceptually, the new and old implementations will be quite different.

You could go about this by rewriting each node in turn, each as its own merge request. You'll need to plan an order to migrate nodes in that allows you to reach the "target" system architecture, but you'll also need to not break the existing system along the way, since the system should be functional at each point in between the merge requests. If you want to rename certain topics shared between nodes, or change the message type passed between nodes, you'll need to carefully plan when you do that as well.

Then, your reviewers will need to consider each change in the context of both the existing system and the future/new system. Again, you can't break things in between merge requests, but you also want to make sure you're architecting things properly for the final system.

Or, you can just replace the old system with a new system in a single, atomic merge request, allowing your reviewers to focus exclusively on reviewing the new system.

For a different example: consider implementing a program that bridges a serial, I2C, or SPI connection and a Redis pub/sub channel, for instance forwarding messages from serial into Redis and vice versa. Suppose you want your program to handle several data types and have a consistent interface on each hardware interface.

You could break this feature into separate merge requests for each interface. Then, when your reviewers went to review each merge request, they would need to understand the management of the Redis connection, the datatype system used throughout the codebase, the differences in protocol between each of the planned hardware interfaces, and the handling of the specific hardware interface actually implemented in the MR.

This isn't too bad if these merge requests are reviewed one after the other, but what if a week or more goes by between each of them? Your reviewer will need to remember each of these things, every time, ultimately leading to spending more time on review compared to just reviewing everything all at once.

Long diffs should take longer to review

There's a common refrain that "you can't possibly catch everything in a patch of more than X lines."

Consider this: would you rather read one chapter of a book every week or two, or spend a few days reading as much as you can? Which would allow you to better follow the story? Keep track of the characters? Understand the book at a deeper level?

I see no fundamental reason why a reviewer would have a lower success rate at spotting issues per line of code for a larger diff as opposed to a smaller one. It will require more time to read and understand, and it will require a reviewer to potentially be more deliberate in their review, but it ultimately leads to the same quality assurance with substantially reduced effort for the development team.

Let's say you're building a live camera feed for something. Maybe it's a 3D printer, or an RC car, or a Mars Rover analogue (website hasn't been updated in years). Here's a pretty typical requirements list:

With one stream, you're in pretty good shape! Just plug in any off-the-shelf USB camera, run a quick GStreamer command on both ends, and boom, you're good to go. But as you add more cameras into the mix, you'll run into some hiccups at seemingly every point in the pipeline.

This blog post is the culmination of 2 years of work into camera streaming pipelines on NU ROVER, Northeastern University's team for the University Rover Challenge. Over that time, we've evolved from an unreliable six-camera setup requiring four separate devices to an efficient, easy-to-use, reliable system encompassing 14 cameras on a single computer. This post will summarize the experimentation we did, the decisions we made, and the lessons we learned, starting with the individual camera modules themselves and working up through the pipeline all the way to how the feed is displayed.

Cameras

A camera module connects some image sensor to some interface. The choice of image sensor doesn't impact much; NU ROVER chooses camera modules based purely on FOV and resolution. The interface, however, greatly affects what hardware you can use for streaming.

CSI

Camera Serial Interface (CSI), sometimes called "MIPI," is the most barebones protocol for connecting a camera: the protocol is specifically designed for cameras and connected directly to the CPU on boards like the Raspberry Pi. A good example of this type is the Raspberry Pi Camera, but other manufacturers supply various cameras with different FOVs and sensors.

Pros:

Cons:

In the past, we used multiple devices (2 Raspberry Pis, a Jetson Nano, and a Jetson TX2) to connect many CSI cameras into our system. This dramatically increased complexity and required installation of a network switch, taking up precious space inside the rover. Our new system does not make use of CSI cameras.

IP

IP cameras are often used as part of home security systems, making them readily available. They directly connect using Ethernet, removing the need for a separate device for encoding. Our team purchased a few, and I spent some time testing them, but I ruled them out pretty quickly. They're physically much harder to mount and integrate into a system, and since they're typically designed for recording and not live viewing, they often have very high latency.

Pros:

Cons:

If you're working on a physically large, stationary, or spread-out system, IP cameras might be worth considering. Otherwise, the added bulk is a substantial drawback.

Analog Video

This year, our team partnered up with NUAV, Northeastern's unmanned aerial vehicles organization, to utilize a drone in part of the autonomous challenge. Their camera feed (used only during emergency teleoperation) was a 5.8 GHz off-the-shelf system common in FPV (first-person view) drone operation. These systems use an analog video signal to send video to the operator's goggles. While analog video is tempting as a self-contained system, it doesn't scale effectively to a multi-camera setup.

Pros:

Cons:

We considered installing an analog video camera to use if our primary network link dropped out, but given the characteristics of the 5.8 GHz band, there are few situations in which this would be helpful. (If we can get a 5.8 GHz signal to the rover, we can almost definitely get a 2.4 GHz signal to it as well.)

USB

This brings us to USB cameras, arguably the most common type in consumer use today. These seem like an obvious choice, but there are a few complications when you scale beyond one or two cameras (which we'll get into).

Pros:

Cons:

It's no surprise USB was our first choice when we began developing the system. Even if you rely on CSI cameras for some systems, you're unlikely to be able to build a low-cost and practical system that incorporates 10 or more of them, meaning you'd have to mix in some USB cameras as well.

USB Bandwidth

We're all used to just plugging in USB devices arbitrarily and having everything work. Cameras, however, tend to push the USB 2.0 connection to its limit, and using several of them on one computer can be surprisingly difficult.

A bit of bandwidth math

Consider the bandwidth requirement of a typical video stream: 640x480 pixels, 30 fps.

Quick note: If you've worked with images, you might assume we need 8 bits per channel to represent each pixel in the frame. This isn't quite true -- we can save a little extra space with some smart encoding. Most raw video from USB cameras is in "YUYV" format, and contains three values: Y (luminance, or "brightness"), U (color), and V (also color). The eye is more sensitive to small variations in luminance than to similar variations in color, so we can save space by using 8 bits for luminance and 4 bits for each of the color signals, giving a total of 16 bits per pixel per frame.

The requirement for a single frame is thus:

And the requirement for a single second of video is:

The theoretical maximum speed of USB 2.0 is 480 Mbit/s, and subtracting signal overhead, we're only left with 53.2 MByte/s. For a single camera, this is fine! We've got plenty of bandwidth left over.

What happens when we add multiple cameras? It depends. Let's first consider the case that the cameras are on different USB controllers from each other -- we'll go over what this means in a second. This is also fine.

Now what if we try to put multiple cameras on a single USB hub? Here's where we start to run into problems.

In a setup like this, two of the cameras will work fine, but the third will not work. Specifically, you'll be able to open all three device files simultaneously, but you'll only receive video frames from two of them. The third camera will not raise an error, which can make troubleshooting annoying.

If this seems weird for a reasonable USB setup to fail like this, that's because it is. Most devices don't come anywhere near to reaching the USB bandwidth limit. And if they do, they'll typically handle it gracefully: your SSD might copy files over a little bit slower, or your printer might take a bit longer to load the document you sent it.

With a USB camera, though, there's no good way to handle a lack of bandwidth: it doesn't have the memory to delay frames and send them later, it can't send a smaller-sized image since the software expects a specific resolution, and there's no way for it to coordinate with other cameras to share the bandwidth by slowing down framerate. Thus, you get nothing.

Let's take a step back and look at what this means in practice. A USB controller is the "root" of the tree of USB devices. It's a chip that connects the USB bus to the CPU, either directly or via a protocol like PCIe. In practice, most computers only have one or two controllers, then use internal USB hubs to provide more physical USB ports.

For instance, a Raspberry Pi 3 has four physical USB ports but only a single internal USB controller, meaning all of the ports share bandwidth. As a result, you're unlikely to get more than two USB cameras working in YUYV mode at this resolution simultaneously on the Pi.

USB 3.0

However, most modern devices have USB 3.0 ports. Can those help us out? USB 3.0 has a theoretical maximum rate of 500 Mbyte/s, so we're good, right?

Unfortunately, this is not how this works.

USB 2.0 uses a single differential pair of wires for signaling. USB 3.0 adds extra wires to the cable and connector, for a total of 3 differential pairs. The two additional pairs are used for "SuperSpeed" signaling (the fast data rates), but the original pair is still just a regular USB 2.0 connection. USB 3.0 traffic is handled completely separately from USB 2.0 traffic.

In essence, a "USB 3.0 cable" is a USB 3.0 cable and a USB 2.0 cable in the same housing. Consequently, a "USB 3.0 hub" is a USB 3.0 hub and a USB 2.0 hub in the same box, wired up to different pins in the same connectors. And you guessed it, a "USB 3.0 controller" is just a USB 3.0 controller and a USB 2.0 controller in the same chip.

Here's the output of lsusb -t on a Raspberry Pi 4:

/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M

You can see that despite the Pi 4 having a single USB controller chip, it appears in Linux as two entirely separate USB buses -- one for USB 2.0 (the "480M" is the speed of the port in bits) and the other for USB 3.0 (listed as "5000M").

Here's a diagram that more accurately captures what's going on:

Since the cameras are all USB 2.0, all of the data they send stays on the USB 2.0 pair all the way to the controller. (USB 3.0 cameras exist, but are expensive and rare outside of industrial settings.)

MJPG encoding

Your webcam probably runs at much higher than 640x480p. The popular Logitech C920, for instance, supports 1080p video at 30 fps and runs over USB 2.0, which seems like it'd use up more bandwidth than we have, right? Something doesn't add up.

These cameras support higher resolutions and framerates by compressing frames before sending them to the computer, using a technique known as MJPG (motion JPEG). Frames are captured by the camera, compressed using the JPEG algorithm, and then sent along the wire to the computer, at which point they can be decompressed.

Different cameras will use different compression ratios depending on the resolution and framerate, meaning we don't have hard numbers to go off of. The improvement from motion JPEG in a multi-camera setup is unfortunately not that great: compression ratios are set assuming a given camera is the only device on the bus, so the bandwidth reduction at smaller resolutions is much less significant. In practice, I've found that up to 3 cameras can coexist on a single bus at this resolution and framerate, meaning it's a pretty minor improvement.

Your Toolbox

Here are the tools you have to try to solve the USB bandwidth problem:

Encoding

You've gotten an image from the camera. Now what?

You'll probably want to use GStreamer to link together most of your pipeline. GStreamer is a framework that allows you to chain together different video elements, such as sources, encoders, payloaders, and decoders, into a pipeline that is executed all at once. The concepts in this post aren't GStreamer-exclusive, and I won't be providing many "ready-to-go" pipelines since they vary based on hardware encoding support and system, but I will at times assume that you're linking together your source, encoder, and payloader using GStreamer.

The most common codec for video compression is H.264, which is generally good enough for most purposes. Other codecs like H.265, AV1, and VP9 can offer lower bandwidth, but the benefits are minor at smaller video resolutions. The most important factor is hardware encoding and decoding support; if the encoding is handled by the CPU, it may not keep up with many video streams simultaneously, especially on lower-end devices.

The most important factors to tune are bandwidth and keyframe interval. Bandwidth, in this case, is the amount of video data sent over the network per second. If your bandwidth is set too high, you'll start to see artifacts as packets are dropped whenever your network link slows down (for instance, if you have a wireless link and your robot drives behind a wall). If it's set too low, your video quality will suffer and your feed will be full of compression artifacts. If you're operating over a wired Ethernet connection, you can probably divide the bandwidth of the Ethernet link by however many streams you plan to have simultaneously, but take into account any other network traffic on the machine, too. NU ROVER uses a bandwidth of 1 Mbps for each 480p stream.

Keyframe interval, also known as "I-frame interval," refers to the time in between I-frames in the video signal. H.264 encoded video consists of I-frames (frames which contain a complete image), P-frames (which reference previous frames), and B-frames (which can reference future frames and are only relevant in the context of a pre-recorded video). The purpose of P-frames is to improve the video compression by not sending the same data in every frame -- if your camera feed is mostly a still image, it is wasteful to send that entire image on every frame when you can instead only send the specific areas of the image which changed.

If you have too many I-frames, you will devote too much of your bandwidth to them and have worse picture quality as a result. However, if you have too few, you may have periods where you can't see the entire image if an I-frame gets dropped due to network conditions. This usually manifests itself as a gray screen in which moving areas gradually return to color while motionless areas remain gray until the next I-frame. NU ROVER uses 1 keyframe every 30 seconds.

Protocols

Now that you have your encoded video feed, you'll need to find a way to send it over the network. There are a few existing protocols that can help you here. Of course, you'll need to use the same protocol on both sides of the connection.

Motion JPEG

Motion JPEG, also known as MJPG, also works for sending video over a network, frame by frame, usually over HTTP. Each frame is compressed individually, making it one of the simplest protocols. However, because there's no inter-frame compression, this protocol uses much more bandwidth to stream video at a given quality than others.

HTTP Live Streaming

HTTP Live Streaming (HLS) is a protocol used to stream H.264 videos over the internet. It works by dividing the video into chunks, then serving each chunk in turn using HTTP. This chunking process introduces a high amount of latency, making HLS unsuitable for a low-latency feed.

Real-time Transport Protocol

Real-time Transport Protocol (RTP) is one of the simplest ways to send video over a network. It sends data over UDP, meaning network errors and dropped packets are not corrected like they would be with a TCP-based protocol. This is ideal for low-latency applications -- if one frame isn't received correctly, you don't want to spend time retransmitting it when you could instead be transmitting the next frame. However, RTP could be a poor choice for streams where quality is more important than latency.

RTP works by payloading data into packets and sending them to a predefined UDP host and port. This payloading process will look different depending on the codec you choose -- for now, let's assume we're using H.264. GStreamer provides the rtph264pay plugin to payload H.264 data, as well as the rtph264depay plugin to depayload it on the other end.

For example, let's assume the video server is on 192.168.1.10, the client is on 192.168.1.11, and the video stream should be on port 9090. We can set up an RTP stream by running this GStreamer pipeline on the server:

gst-launch-1.0 videotestsrc ! video/x-raw,rate=30,width=640,height=480 ! x264enc tune=zerolatency ! rtph264pay ! udpsink host=192.168.1.11 port=9090

Then, on our client, we can run this to decode the stream:

gst-launch-1.0 udpsrc port=9090 ! application/x-rtp,payload=96 ! rtph264depay ! avdec_h264 ! autovideosink

To scale this to multiple camera feeds, we just need to choose different UDP ports for each feed. There are plenty of available port numbers for this.

Real Time Streaming Protocol

The solution above using RTP leaves a few things to be desired:

Real Time Streaming Protocol, or RTSP, is a protocol built on top of RTP which can solve these shortcomings. RTSP at first behaves much like an HTTP server, and RTSP URLs look similar to HTTP URLs.

Let's look at what happens when a client tries to load rtsp://192.168.1.10/video:

RTSP provides a few other methods, including PAUSE, which can be useful for interruptable streams.

To play around with RTSP, you can use the gst-rtsp-launch project:

gst-rtsp-launch "( videotestsrc ! video/x-raw,rate=30,width=640,height=480 ! x264enc tune=zerolatency ! rtph264pay pt=96 name=pay0 )"

Then, connect with GStreamer's rtspsrc plugin:

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/video latency=0 ! rtph264depay ! avdec_h264 ! autovideosink

Or, connect with VLC by entering the RTSP URL into the "Open Network Stream" window.

If you're building out a system with multiple video streams, you'll want to use the GStreamer bindings to define your own RTSP server's request handling logic. Here's an example of how to use the Python bindings, but bindings for C++ and other languages also exist. NU ROVER has our own in-house implementation which we may decide to open-source once it stabilizes, assuming we have time to do so.

Extra Signaling Logic

One limitation of RTSP is that by default there is no way for a client to list the streams available on the server. If you don't want to maintain a list of streams on both sides of the connection, you may want to add some extra logic to handle this.

One approach would be to try to send this data over the RTSP connection somehow, say, by returning it in response to a DESCRIBE request for a certain URL. Doing this would require the ability to embed this handling logic into whatever RTSP client and server you use, which could be difficult.

The approach NU ROVER took instead was to use another communication channel which we already had (ROS messages) to send this data. However, you could rely on an HTTP API, a database, or some other method. You might want such a protocol to handle:

Clients

What client are you going to use to decode and display the video signal? You'll likely want your video feeds to be integrated into whatever control UI you use for other operations, so this will depend on your situation.

VLC

VLC Media Player is a common program for playing various media files, and it can play video from an RTSP source. Go to "File" → "Open Network Stream" and enter your RTSP URL.

VLC is typically meant for watching movies or other prerecorded content, meaning it buffers video to smooth out playback, dramatically increasing latency. While there are ways to reduce the latency (I couldn't find a good comprehensive source, so just search for "VLC RTSP low latency"), I haven't had much success. I'd suggest keeping VLC around to help with troubleshooting streams, but not to use it in your "production" system.

GStreamer

GStreamer's rtspsrc can also load video data over RTSP, and its glimagesink plugin can display video data in a window, making it functional as a standalone RTSP viewer. (Remember to specify latency=0 for rtspsrc.) However, if you want your video feed to be displayed in the same window as the rest of your UI, you'll likely need something more complex.

Qt, etc.

Bindings often exist to display media from a GStreamer pipeline in applications built with frameworks like Qt (QtGStreamer), React Native (react-native-gstreamer), and others. You can find a pipeline that works by launching with the gst-launch-1.0 command, then copy it into your application.

Browsers

Web-based user interfaces are becoming much more popular, and NU ROVER has our own based on React. Of course, you can't run arbitrary C++ code inside of a browser sandbox, nor can you access UDP/TCP ports directly, so embedding a GStreamer widget directly in the browser isn't feasible.

Browser Specifics

Sending low-latency video to a web browser is more difficult than it would seem. Browsers are built for media consumption, meaning they often buffer video to smooth out playback at the cost of higher latency. There are quite a few ways to include videos in a web browser, none optimal.

HTTP Live Streaming

HLS seems like it's exactly right for this use case, but the required "chunking" of video data adds a high amount of latency, making it unusable for a low-latency video feed.

Raw H.264

It is possible to directly send an H.264 stream to the browser. However, pre-existing solutions for streaming H.264 are difficult to come by. One example is the ROS web_video_server package, which implements its own HTTP server to handle this.

The benefit of this approach is that video can be embedded in a website with a <video> tag, and the video is compressed when entering the browser, reducing bandwidth use and allowing the browser to choose the most efficient decoder for the platform. The important drawback is that there is no way to control the amount of buffering that the browser does to the video feed, and browsers like to buffer a significant chunk of frames to ensure smooth playback. NU ROVER experimented with using JavaScript to automatically seek the video to the most recent frame in the buffer, but found this difficult and unreliable.

Motion JPEG

One potential approach is to run a GStreamer pipeline on the client which decodes the H.264 data, then compresses each frame as a JPEG, then sends those into the browser.

Sending a series of images to the browser can be done using the multipart/x-mixed-replace mimetype: the server will send an HTTP response with Content-Type: multipart/x-mixed-replace;boundary=--<boundary_name>, then send a series of MJPG frames, separated by --<boundary_name> lines.

This is the approach NU ROVER currently uses. We've found that encoding and decoding JPEG images adds a small but acceptable amount of latency. We use an in-house project to serve these images, starting and stopping the underlying GStreamer pipelines and RTSP streams on demand.

A tip for using GStreamer with an application like this: to send buffers from a GStreamer pipeline into your application, you'll want to use appsink. Here's an example of code that uses this.

Motion BMP?

In theory, you could send a multipart/x-mixed-replace payload with any image format, not just JPEG. Sending an uncompressed bitmap image (.bmp), for instance, would remove the need to encode and decode the JPEG but would increase the data sent to the browser. In our case, we're running the client on the same machine as the browser, meaning this added data transfer is largely irrelevant.

This approach is a bit harder to implement than Motion JPEG: effectively, you just need to add the proper headers to make each frame a BMP file, but we haven't found a good GStreamer plugin that does this. In theory, this could be done in the HTTP server when the GStreamer buffers are read.

Stream Limit

If you try to run more than six concurrent Motion JPEG streams into a browser, you'll notice that beyond the first six, none of them load. To prevent creating too many HTTP connections when a site loads, browsers will restrict the number of concurrent HTTP connections to a single host to six.

In Firefox, it's possible to change this setting by going to about:config and changing the network.http.max-persistent-connections-per-server setting from 6 to a higher value, say, 20. However, there is no way to change this setting in Chrome.

One workaround, if you have a DNS server handy or can modify /etc/hosts, is to configure multiple subdomains for the same server, as described in this StackOverflow answer.

An example /etc/hosts file could look like this:

cameras1.local 192.168.1.10
cameras2.local 192.168.1.10
cameras3.local 192.168.1.10
cameras4.local 192.168.1.10

Alternatively, a DNS server could be used with a wildcard record to allow any subdomain to work.

Then, ensure that no more than six streams are loaded using the same subdomain, and all streams should work fine.

WebRTC

One potential approach is using WebRTC, which is built for low-latency applications like video conferencing. We haven't investigated this approach yet for a few reasons. WebRTC is primarily aimed at browser-based applications exchanging data with each other, not receiving data from a non-browser application, so example code for non-browser environments streaming into browsers is hard to find. WebRTC is also built around the assumption that it needs to traverse NAT using a STUN server -- not only is that not necessary for us, it's impossible as our network runs completely offline. That said, the use of WebRTC for low-latency streaming holds potential. I'd love to see this investigated, and hope I'll get the time to play around with it myself at some point.

Conclusion

We've walked through the steps in building a camera streaming pipeline, from the choice of individual camera modules to the framework used to display them to the user. This guide is the resource I wish our team had when developing our streaming pipeline, and covers many of the pitfalls and decisions that will need to be made. It's far from complete; there are many techniques and topics in streaming to explore that simply fall outside of what we've worked with, but I hope it's useful in whatever system you're building nonetheless.

A few weeks ago, I achieved one of my dreams: being on a team that publishes a paper from doing something cool. Here it is in full (shoutout NIME for being open-access):

You can see my cameo as a hand model in Figure 5.

Our Vision

The project was pitched to me by my advisor roughly like this:

My Work

Our approach was to circumvent the JVM entirely by rooting the phones, building binaries for them on a host computer, and executing these binaries directly. This gave us more direct access to the hardware, at the cost of a more difficult setup process. I focused on two main goals: our sensor I/O and our build system.

Sensor I/O

This work involved working with libandroid, Android's internal library used for handling device functions. I spent a ton of time in Android Code Search to investigate how everything fits together, and started to put together some test functions for adjusting sensor sampling rates and reading data, which my advisor then turned into LDSP's user-facing sensor API, providing functions like sensorRead -- taking heavy inspiration from the Arduino IDE's simple functions like analogRead.

Build System

Our build system draws heavily on the Android NDK, or Native Development Kit. This toolchain is designed to build smaller native components within a larger JVM-based app, so we needed to circumvent the intended way of doing things a bit.

Initially, we just used manually invoked the g++ binary included with the NDK to compile our executables. However, as the project grew and we began pulling in more and more libraries, it became clear another solution was needed. We decided to start with Makefiles, due to their popularity and simplicity. This worked for a while, and even let us include some other scripts in the file (e.g. we set up make push to push the binaries to the phone).

This solution worked well as we continued development, but started to stretch when we wanted to make our build system more user-friendly. We wanted to provide a configuration file for each phone we supported, so users would not need to configure compiler settings for their phone manually. Many phone model names contain spaces. Furthermore, we wanted to let users specify a path to their project, and compile some of their files alongside our files. Allowing user project names to contain spaces was a hard requirement for us.

As it turns out, Makefiles handle spaces extremely poorly -- not only do they have to be escaped differently in different places, but many functions treat a variable with spaces implicitly as a list, causing issues. Many make builtins simply could not handle spaces in filenames, so we were forced to use $(shell ...) and rely on shell commands (which aren't portable across operating systems). Our solution was still plagued with bugs.

Eventually, we gave up on pure Makefiles and I started to investigate other build systems. Ninja seemed promising, and has the backing of projects like Chromium and LLVM. We rolled out some CMake files across the repo, and set the output format to Ninja files, and... success!

Of course, CMake isn't necessarily built for running random commands in the way pure Makefiles are, so I created a few wrapper scripts: ldsp.sh for Linux and macOS and ldsp.bat for Windows. Each script allows configuring the project, building (with incremental builds), pushing the build to the phone, and running the build.

We got to see everything come together in an all-day workshop we did with many students from NU Sound. While the install process was a bit convoluted, most participants were able to get a toolchain up and running on their computer!

Future Goals

The biggest hurdle with our current setup is complexity: participants need the Android NDK, CMake, Ninja, ADB (Android Debug Bridge), and more installed on their computer. If we could package everything into an easy-to-install application, we could make this project more accessible to those unfamiliar with the command line or programming in general, and make it much more fitting as an educational tool. Furthermore, if we could install this package onto the phone itself, we could remove the need for a host device altogether, making our work accessible to those without a laptop or desktop.

I've been working on cross-compiling LLVM to run natively on an Android host. Admittedly, I haven't had much luck so far, but it's still early days. Once we have this crucial portion working, we can pivot to developing a web UI which runs on the phone to support editing and running code from any connected device, then package everything up into an .apk for people to install.

Thanks

I want to thank my advisor, Victor Zappi, for giving me the opportunity to work on this amazing project.

This blog post follows my journey to learn about reverse-engineering on Android over a few months. Unlike a traditional project writeup, the structure of this piece matches the process of discovery I took. Dead-ends, useless tangents, and inefficient solutions have been intentionally left in.

The Idea

I live in Boston currently, which has a wonderfully extensive transit network. To navigate between unfamiliar areas, I use an app called Transit. Transit (or TransitApp, as I tend to call it) tracks bus, train, and subway departures in realtime to provide directions and time predictions.

TransitApp also has a "gamification" system, in which you can set an avatar, report the location of trains from your phone, and show up on leaderboards.

Instead of public-facing usernames or profile pictures, TransitApp's social features work off of random emoji and generated names. You are given the option to generate a new emoji/name combination.

For reasons I won't get into, I really wanted the 🎈 emoji as my profile. It's not on the list of emoji that the shuffling goes through (I shuffled for quite a while to confirm). If you pay for "Transit Royale" (the paid tier), you can choose your emoji yourself. However, I was bored, so I decided to see if I could find a more fun way to get it.

My hypothesis was that the app is doing this "shuffling" logic on the client side, then sending a POST request to the server with the emoji to be chosen. If true, this would mean that I could replay that request, but with an emoji of my choosing.

Methods

Android Emulator

Because messing with a physical Android device seemed tricky, I decided to try using the Android emulator built into Android Studio. (No need to create a project: just click the three dots in the upper right and choose "Virtual Device Manager.") After picking a configuration that supported the Google Play Store (I chose the Pixel 4), I installed TransitApp and logged in without issue.

Burp Suite

Burp Suite is the standard tool for inspecting and replaying HTTP network traffic. Burp Suite creates an HTTP proxy, and is able to inspect traffic via this proxy.

I began by setting Burp to bind on all interfaces, then set the proxy in the emulated phone's settings to point at my computer's IP address and proxy port.

Burp Suite's analysis tools effectively break HTTPS, since it's the literal definition of a man-in-the-middle attack. In other words, TLS protects the connection from my phone to any cloud services, and for Burp to mess with that, it needs a way to circumvent TLS. One approach is to install Burp's certificates as a "trusted" certificate, effectively making Burp Suite able to impersonate any website it wants.

Most guides for manually installing CA certificates on Android require a rooted operating system, but with a reasonably recent OS, it's possible to install them in the phone's settings, by going to: Settings -> Security & privacy -> More settings -> Encryption & credentials -> Install a certificate -> CA certificate. This adds a "Network may be monitored" warning to the phone's Quick Settings page.

This works perfectly! Now let's just fire up TransitApp, and... nothing. It looks like TransitApp doesn't respect the user's proxy settings.

A few apps claim to be able to use a VPN profile to force all traffic over a proxy, but this seems to not work properly for any I tested.

Wireshark

Wireshark is a much more general-purpose network traffic capture tool. It supports capturing network traffic and filtering, making it relatively easy to inspect traffic.

However, Wireshark cannot circumvent TLS. As a result, even though the presence of traffic is visible, it cannot be actually inspected. When filtering for DNS traffic, though, we do get something helpful: the domain names that TransitApp uses. We see a few:

With the exception of RevenueCat, which seems to manage in-app subscriptions, all domains are *.transitapp.com.

Burp Suite Invisible Mode

Burp Suite's Invisible Mode allows the proxy to work for devices that aren't aware of its existence, by relying on the host OS to override DNS queries for the relevant domain names and send them to the proxy instead. To get this to work, we can run the Burp .jar with sudo, then set up two proxies: one on port 80 for HTTP and one on port 443 for HTTPS. Make sure to enable "invisible proxying" for both.

Since we're using invisible proxying, we'll need to explicitly tell Burp where to forward the traffic -- the HTTP(S) requests themselves won't have enough information for Burp to route them onwards. We can make a DNS request to find the IP address of the serer hosting api.transitapp.com:

$ dig @1.1.1.1 api.transitapp.com

; <<>> DiG 9.10.6 <<>> @1.1.1.1 api.transitapp.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38753
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;api.transitapp.com.		IN	A

;; ANSWER SECTION:
api.transitapp.com.	248	IN	A	34.102.188.182

;; Query time: 55 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Fri Jun 16 18:27:14 EDT 2023
;; MSG SIZE  rcvd: 63

Configure both proxies to route traffic to that IP address, keeping the ports the same.

If you've played with networking before, your first instinct will probably be to mess with the /etc/hosts file to override DNS for the domains you want to intercept. This is a pretty common technique for web-based attacks, so let's give it a try. One hiccup: we can't put wildcards directly in /etc/hosts, so we'll have to list each one individually. Worse, we'll actually have to only do one at a time, since Burp Suite can only forward traffic to one IP at a time. Start with api.transitapp.com.

Aaaand... it still doesn't work. Android is using something called Private DNS to automatically route traffic over HTTPS, meaning it can't be tampered with as easily as traditional DNS. But even if you turn that off in the emulated phone's network settings, it still doesn't work, because the emulator doesn't respect the /etc/hosts file. You'll need to run a DNS server.

First, find the IP address of the emulated phone and your computer. In the phone, go to the network settings page, then look for "IP Address" and "Gateway": those are the IPs of the phone and your computer, respectively.

Install dnsmasq on your host computer and run it (here's a nice guide for macOS). Set the Android DNS settings to point to your computer's IP. And then update your /etc/hosts entry to point to the IP address of your computer on the network instead of 127.0.0.1, since otherwise the emulated phone would attempt to connect to itself.

Even still, this doesn't work. If you try to inspect traffic, relevant features of the app will simply stop working and you will not see any connection trying to be made.

Static Analysis

Maybe some static analysis could help? We can download an APK from a definitely legitimate source, then unzip it (.apk files are just .zip files).

The first thing we can do is to start looking for URLs. We already know that some URLs start with api.transitapp.com, so let's try looking for that:

$ rg "api.transitapp.com"

No results. What about just "transitapp.com"? Still nothing.

It could be substituted in somewhere. Maybe we could look for the beginning "http" or "https" part of a URL?

$ rg http
assets/cacert.pem
9:## https://hg.mozilla.org/releases/mozilla-release/raw-file/default/security/nss/lib/ckfw/builtins/certdata.txt

okhttp3/internal/publicsuffix/NOTICE
2:https://publicsuffix.org/list/public_suffix_list.dat
5:https://mozilla.org/MPL/2.0/

META-INF/services/io.grpc.ManagedChannelProvider
1:io.grpc.okhttp.d

google/protobuf/source_context.proto
3:// https://developers.google.com/protocol-buffers/

google/protobuf/empty.proto
[many similar protobuf results omitted]


META-INF/MANIFEST.MF
494:Name: okhttp3/internal/publicsuffix/NOTICE
497:Name: okhttp3/internal/publicsuffix/publicsuffixes.gz

META-INF/CERT.SF
495:Name: okhttp3/internal/publicsuffix/NOTICE
498:Name: okhttp3/internal/publicsuffix/publicsuffixes.gz

res/56.xml
2:<resources xmlns:tools="http://schemas.android.com/tools"

res/sd.xml
6:<resources xmlns:tools="http://schemas.android.com/tools"

Okay, that's interesting. "okhttp3"? This seems like it could be related to how the application makes HTTP requests to the API. But this still leaves us with a few questions:

Certificate Pinning

OkHttp is a library made by Square for making HTTP requests on Android (or other Java platforms). The fact that it was developed by Square, a payment processing company, is a clue that they might be taking steps to secure the connection that most apps wouldn't.

Here's a snippet from the front page of the OkHttp documentation, emphasis mine:

OkHttp perseveres when the network is troublesome: it will silently recover from common connection problems. If your service has multiple IP addresses, OkHttp will attempt alternate addresses if the first connect fails. This is necessary for IPv4+IPv6 and services hosted in redundant data centers. OkHttp supports modern TLS features (TLS 1.3, ALPN, certificate pinning). It can be configured to fall back for broad connectivity.

Certificate pinning sounds relevant to what we're doing here, considering our approach is to supply an alternate certificate. So what is it? According to the OkHttp docs:

Constrains which certificates are trusted. Pinning certificates defends against attacks on certificate authorities. It also prevents connections through man-in-the-middle certificate authorities either known or unknown to the application’s user. This class currently pins a certificate’s Subject Public Key Info as described on Adam Langley’s Weblog. Pins are either base64 SHA-256 hashes as in HTTP Public Key Pinning (HPKP) or SHA-1 base64 hashes as in Chromium’s static certificates.

Our monitoring setup is such a man-in-the-middle scenario: we use our own certificate authority (in this case, Burp Suite) to "re-secure" the actual connection from TransitApp, after we've monitored and tampered with the connection.

So, how do we get around certificate pinning? We'll need to get quite a bit more serious about our decompilation efforts, since we'll need to remove the hash of the existing certificate in the code and replace that with the hash of our own certificate. I roughly followed this guide for this step.

APK Analysis

Android Studio helpfully includes an APK Analyzer tool which we can use to peek a bit deeper into the app. Create a new project,then drag and drop the TransitApp APK into the main window.

Unfortunately, method names have pretty much all been minified, so you'll see a bunch of u4, a5, o4, etc. Android Studio also doesn't let us view or modify the Java code within each method. However, note that even the OkHttp3 code seems to be minified, and I couldn't find any reference to CertificatePinner (although rg -a CertificatePinner returned a few matches, so maybe there's hope?)

An open-source tool called APKtool might save us. APKtool allows us to essentially decompile an APK into Smali (essentially an assembly listing for Java bytecode), make modifications, then recompile it. To start, let's see if we can find the certificate hash.

First, download the .jar from here, then run it with:

java -jar apktool_2.7.0.jar d Transit\ Bus\ \&\ Subway\ Times_5.13.5_Apkpure.apk

Finally, dive into the directory named after that APK. We're looking for the code that invokes CertificatePinner.

Reading Smali Bytecode: Working Up

rg CertificatePinner

This seems to give quite a few results within the okhttp3 library: the implementation of certificate pinning, special handling in the connection class, and a few other uses. However, there is one result outside that library:

smali_classes2/com/masabi/justride/sdk/platform/AndroidPlatformModule2.smali
91:.method private getCertificatePinner()Lokhttp3/CertificatePinner;
107:    new-instance v1, Lokhttp3/CertificatePinner$a;
111:    invoke-direct {v1}, Lokhttp3/CertificatePinner$a;-><init>()V
206:    invoke-virtual {v1, v3, v4}, Lokhttp3/CertificatePinner$a;->a(Ljava/lang/String;[Ljava/lang/String;)Lokhttp3/CertificatePinner$a;
215:    invoke-virtual {v1}, Lokhttp3/CertificatePinner$a;->b()Lokhttp3/CertificatePinner;
250:    invoke-direct {p0}, Lcom/masabi/justride/sdk/platform/AndroidPlatformModule2;->getCertificatePinner()Lokhttp3/CertificatePinner;
258:    invoke-virtual {p1, v0}, Lokhttp3/OkHttpClient$a;->d(Lokhttp3/CertificatePinner;)Lokhttp3/OkHttpClient$a;

The result on line 111 looks perhaps the most interesting to us: invoking the constructor on the CertificatePinner class. However, it doesn't seem to be passing in any sort of hash. Let's consult the OkHttp documentation as to how CertificatePinners are constructed:

String hostname = "publicobject.com";
CertificatePinner certificatePinner = new CertificatePinner.Builder()
  .add(hostname, "sha256/AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=")
  .build();

OkHttpClient client = OkHttpClient.Builder()
  .certificatePinner(certificatePinner)
  .build();

Request request = new Request.Builder()
  .url("https://" + hostname)
  .build();

client.newCall(request).execute();

Huh, okay, so we're really looking for a CertificatePinner.Builder. But rg CertificatePinner.Builder gives no results.

Here's where a Java quirk comes into play: Nested classes like CertificatePinner.Builder are represented internally using a dollar sign in place of the dot. So we're really looking for CertificatePinner$Builder. Make sure to escape it properly:

rg 'CertificatePinner\$Builder'

And... still no results. But don't lose hope yet--remember our experiments with APK Analyzer? Perhaps the name Builder got minified. Let's try to find any nested classes within CertificatePinner:

rg 'CertificatePinner\$'

It looks like there's a CertificatePinner$a, CertificatePinner$b, and CertificatePinner$c. The docs seem to show three nested classes: Builder, Companion, and Pin. Out of the three, it seems like Builder is the only one that should need to be used externally. Looking at the one result outside of the okhttp3 package:

smali_classes2/com/masabi/justride/sdk/platform/AndroidPlatformModule2.smali
107:    new-instance v1, Lokhttp3/CertificatePinner$a;
111:    invoke-direct {v1}, Lokhttp3/CertificatePinner$a;-><init>()V
206:    invoke-virtual {v1, v3, v4}, Lokhttp3/CertificatePinner$a;->a(Ljava/lang/String;[Ljava/lang/String;)Lokhttp3/CertificatePinner$a;
215:    invoke-virtual {v1}, Lokhttp3/CertificatePinner$a;->b()Lokhttp3/CertificatePinner;

We're looking for the call to Builder.add(), since that will pass in the pin as a hash. Again, the name .add() will be minified, so we'll need to be clever. Now that we've narrowed things down to a single file, we can start to read through and look for anything interesting. This method stands out (.line directives omitted for brevity):

.method private getCertificatePinner()Lokhttp3/CertificatePinner;
    .locals 8

    iget-object v0, p0, Lcom/masabi/justride/sdk/platform/AndroidPlatformModule2;->sdkConfiguration:Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;

    invoke-virtual {v0}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->getCertificatePins()Ljava/util/List;

    move-result-object v0

    new-instance v1, Lokhttp3/CertificatePinner$a;

    invoke-direct {v1}, Lokhttp3/CertificatePinner$a;-><init>()V

    invoke-interface {v0}, Ljava/util/List;->iterator()Ljava/util/Iterator;

    move-result-object v0

    :goto_0
    invoke-interface {v0}, Ljava/util/Iterator;->hasNext()Z

    move-result v2

    if-eqz v2, :cond_0

    invoke-interface {v0}, Ljava/util/Iterator;->next()Ljava/lang/Object;

    move-result-object v2

    check-cast v2, Ljava/lang/String;

    iget-object v3, p0, Lcom/masabi/justride/sdk/platform/AndroidPlatformModule2;->sdkConfiguration:Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;

    invoke-virtual {v3}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->getHostname()Ljava/lang/String;

    move-result-object v3

    const/4 v4, 0x1

    new-array v4, v4, [Ljava/lang/String;

    const/4 v5, 0x0

    new-instance v6, Ljava/lang/StringBuilder;

    invoke-direct {v6}, Ljava/lang/StringBuilder;-><init>()V

    const-string v7, "sha256/"

    invoke-virtual {v6, v7}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

    invoke-virtual {v6, v2}, Ljava/lang/StringBuilder;->append(Ljava/lang/String;)Ljava/lang/StringBuilder;

    invoke-virtual {v6}, Ljava/lang/StringBuilder;->toString()Ljava/lang/String;

    move-result-object v2

    aput-object v2, v4, v5

    invoke-virtual {v1, v3, v4}, Lokhttp3/CertificatePinner$a;->a(Ljava/lang/String;[Ljava/lang/String;)Lokhttp3/CertificatePinner$a;

    goto :goto_0

    :cond_0
    invoke-virtual {v1}, Lokhttp3/CertificatePinner$a;->b()Lokhttp3/CertificatePinner;

    move-result-object v0

    return-object v0
.end method

Let's step through what this is doing, using our intuition to bridge the gaps:

This means we'll need to search a little bit deeper to find the hashes we seek, starting with getCertificatePins().

$ rg getCertificatePins
smali_classes2/com/masabi/justride/sdk/converters/config/SdkConfigurationConverter.smali
454:    invoke-virtual {p1}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->getCertificatePins()Ljava/util/List;

smali_classes2/com/masabi/justride/sdk/internal/models/config/SdkConfiguration.smali
762:.method public getCertificatePins()Ljava/util/List;

smali_classes2/com/masabi/justride/sdk/platform/AndroidPlatformModule2.smali
99:    invoke-virtual {v0}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->getCertificatePins()Ljava/util/List;

The result with .method public is the definition of the getCertificatePins() method -- let's start there.

.method public getCertificatePins()Ljava/util/List;
    .locals 1
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "()",
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;"
        }
    .end annotation

    iget-object v0, p0, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->certificatePins:Ljava/util/List;

    return-object v0
.end method

Okay, so we're just returning the certificatePins field. It's just a classic "getter method." We can track down the field definition:

.field private final certificatePins:Ljava/util/List;
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;"
        }
    .end annotation
.end field

Okay, so it's a private final List<String>. That's pretty standard. This essentially gives us two options: either these options are set in the constructor, or they're added later through another public method. Let's check the constructor first.

.method private constructor <init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Z)V
    .locals 2
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "(",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;",
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Z",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Ljava/lang/String;",
            "Z)V"
        }
    .end annotation

Oh god, that's 32 lines and we haven't even gotten to an implementation yet. Here's the part of the implementation that deals with the certificate pins:

move-object v1, p4

iput-object v1, v0, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;->certificatePins:Ljava/util/List;

So the pins are passed in as a list, in the fifth parameter (p4 is indexed starting at zero.) Let's keep following this wild goose chase: where is the constructor called?

$ rg 'SdkConfiguration;-><init>'
smali_classes2/com/masabi/justride/sdk/internal/models/config/SdkConfiguration.smali
211:    invoke-direct/range {p0 .. p18}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;-><init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;Z)V

smali_classes2/com/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder.smali
379:    invoke-direct/range {v2 .. v21}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration;-><init>(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;Ljava/util/List;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$1;)V

Oh, duh, it's another builder. At least this one isn't minified? Let's take a look at the SdkConfiguration$Builder.smali file. Alongside brand, country code, and other parameters, we see a few points of interest.

Here's the definition of the certificatePins field on the builder. Note that it isn't marked final, meaning it probably gets assigned to somewhere.

.field private certificatePins:Ljava/util/List;
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;"
        }
    .end annotation
.end field

Next in the file is the .build() method. It does some checks for null, but other than that, seems pretty boring. Scrolling down a bit, though, we see a definition for setCertificatePins():

.method public setCertificatePins(Ljava/util/List;)Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;
    .locals 0
    .annotation system Ldalvik/annotation/Signature;
        value = {
            "(",
            "Ljava/util/List<",
            "Ljava/lang/String;",
            ">;)",
            "Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;"
        }
    .end annotation

    iput-object p1, p0, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;->certificatePins:Ljava/util/List;

    return-object p0
.end method

There's no surprise here regarding what this method does (it just assigns the argument to the certificatePins field). However, now that we know the name of this method, we can try to look for it elsewhere.

$ rg setCertificatePins
smali_classes2/com/masabi/justride/sdk/converters/config/SdkConfigurationConverter.smali
199:    invoke-virtual {v2, v3}, Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;->setCertificatePins(Ljava/util/List;)Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;

smali_classes2/com/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder.smali
770:.method public setCertificatePins(Ljava/util/List;)Lcom/masabi/justride/sdk/internal/models/config/SdkConfiguration$Builder;

It looks like the only place setCertificatePins is called is within SdkConfigurationConverter. This is an interesting class, and it's not immediately clear what it's doing. A few method names give us a clue, however:

It would be really easy if the certificate pins were just stored in a JSON file somewhere... But find . -type f -name "*.json" comes up empty.

One quick sanity check: Is SdkConfigurationConverter even constructed? It could be that we're looking in the complete wrong part of the code here. Maybe our assumption about certificate pinning isn't correct after all?

Note that SdkConfigurationConverter has a private constructor and a public static create() method. Therefore, we should be searching for SdkConfigurationConverter.create().

$ rg 'SdkConfigurationConverter;->create'
smali_classes2/com/masabi/justride/sdk/jobs/config/ProcessConfigurationDataJob.smali
1096:    invoke-static {}, Lcom/masabi/justride/sdk/converters/config/SdkConfigurationConverter;->create()Lcom/masabi/justride/sdk/converters/config/SdkConfigurationConverter;

Okay, so it's used somewhere at least. Is ProcessConfigurationDataJob invoked anywhere? It looks like it has a create() method, just like SdkConfigurationConverter.

$ rg 'ProcessConfigurationDataJob;->create'
smali_classes2/com/masabi/justride/sdk/AndroidJustRideSdkBuilder.smali
121:    invoke-static {v0}, Lcom/masabi/justride/sdk/jobs/config/ProcessConfigurationDataJob;->create(Lcom/masabi/justride/sdk/platform/crypto/PlatformSignatureVerifier;)Lcom/masabi/justride/sdk/jobs/config/ProcessConfigurationDataJob;

And just to finish going up the chain, where is this instantiated?

$ rg AndroidJustRideSdkBuilder
[...]

smali_classes2/com/thetransitapp/droid/shared/TransitApp.smali
524:    invoke-static {}, Lcom/masabi/justride/sdk/AndroidJustRideSdk;->builder()Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;
532:    invoke-virtual {p1, p0}, Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;->application(Landroid/app/Application;)Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;
540:    invoke-virtual {p1, v0}, Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;->configuration(Ljava/io/InputStream;)Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;
548:    invoke-virtual {p1}, Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;->build()Lcom/masabi/justride/sdk/AndroidJustRideSdk;

[...]

Here we are: TransitApp.smali, which seems like the entrypoint to the application. It seems like it passes some sort of InputStream to the builder--maybe this is the JSON data we're looking for?

Here's the method that invokes this (it's just labeled j, since we're back into minified code):

.method private synthetic j(Ljava/lang/String;Ljava/lang/Throwable;)V
    .locals 2

    if-eqz p2, :cond_0

    return-void

    :cond_0
    const/4 p2, 0x0

    :try_start_0
    new-instance v0, Ljava/io/ByteArrayInputStream;

    invoke-virtual {p1}, Ljava/lang/String;->getBytes()[B

    move-result-object p1

    const/4 v1, 0x0

    invoke-static {p1, v1}, Landroid/util/Base64;->decode([BI)[B

    move-result-object p1

    invoke-direct {v0, p1}, Ljava/io/ByteArrayInputStream;-><init>([B)V
    :try_end_0
    .catch Ljava/lang/Exception; {:try_start_0 .. :try_end_0} :catch_1
    .catchall {:try_start_0 .. :try_end_0} :catchall_1

    :try_start_1
    invoke-static {}, Lcom/masabi/justride/sdk/AndroidJustRideSdk;->builder()Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;

    move-result-object p1

    invoke-virtual {p1, p0}, Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;->application(Landroid/app/Application;)Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;

    move-result-object p1

    invoke-virtual {p1, v0}, Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;->configuration(Ljava/io/InputStream;)Lcom/masabi/justride/sdk/AndroidJustRideSdkBuilder;

    [...]