We all make mistakes in our professional lives, but some of those stay with us more than others. Here are some mistakes I’ve made while developing software that to this day still make me cringe whenever I think about them. And I think about them A LOT.
1. Four Processes to Rule Them All
In the eternal quest of sorting out complexity using those shady CPU instructions, software people developing applications typically find themselves in the following conundrum: should they implement several threads executing inside the same process, or should they implement separate processes and use some IPC magic to couple them together? There are pros and cons to both options—which I will not discuss here to avoid triggering the nerds that surely love either side—and it also depends on what you’re trying to solve, but somehow back in 2009-ish, as I was working with some simulation environment, I chose to use separate processes. Four in this case.
The whole thing was hideous from conception: it used a shared memory space that all processes could access, and I had the marvelous idea of using the operating system's signals (yes, signals), to somehow keep the synchronism between the processes. Signals are very basic mechanisms to, well, signal basic things such as process exiting or a segmentation fault, and for performing some critical error handling. But they were never, under no circumstances whatsoever conceived to sync things up. But I could not care less. Additionally, I wrote a shell script that would spawn all these processes in the right order (the order of appearance mattered A LOT), and said script would act as a watchdog of sorts, monitoring that all processes were up and running and restarting them in case one of them would have crashed. Ah, and hear this: the watchdog script would BEEP A SOUND in case of a crash. Horrible. I remember having to disable the beeping feature upon the request of a guy who happened to be working next to the computer rack that ran this 4-process abomination. He hated the beep.
One thing’s for sure: you never “discover” shitty software architecture at the very end of the process and as a sort of big, shocking surprise. No; you usually see bad software architecture slowly forming and growing in front of our eyes. But our sunken costs always keep us going: we have already suffered enough with it so the only way is to suffer until the end and be loyal with our subpar choices.
All in all, this architecture looked so brittle, so clumsy, that I still have nightmares about it every now and then. Beep beep.
2. There’s Always Time For Doing It Over
It’s the year 2016-ish, and I find myself in the situation of having to handle an increasing amount of time-series data stored in small databases. The data is satellite telemetry, and as such it is quite plain and unsophisticated: readings from sensors, power telemetry, attitude data, and the like. Tabular information. There are some initial challenges with the formatting, the merging of the databases, and the need to do some wrangling of the data—there seems to be a certain diversity of timestamps and floating-point precisions—but nothing too difficult.
So, to help with the analysis of this time-series data, I go on and create a rather contrived Python module to form data tables, lists, and fancy dictionaries, including features like sorting by value up and down, downsampling, and exporting to CSV and other handy formats.
It is surely fun to work on, and I happen to use it a lot—especially during anomalies for diagnosis. But at some point, I realize I had basically reinvented Pandas. Sure, Pandas was not as mature as it is today, but according to this, it had been around for quite a while. When I finally used Pandas for the first time, I felt incredibly dumb for having spent all that time and energy to recreate the exact same functionality.
This lesson reinforced something I kind of knew: when facing rather generic problems, most likely there’s already a Python module out there solving the wheel you are just about to reinvent. So don’t.
3. They See Me Scrollin’, They Hatin’
It’s around 2010, and I am venturing myself into the muddy waters of web development. The reason? A bit of masochism for sure, but also there is the objective need in the company I work for to control some special equipment using a web page for ease of use. So, there I go to code in PHP using a text editor, all locally hosted and deployed in a LAMP environment.
The web page I come up with is more or less fine, I mean nothing too complex and it does the trick. It has one problem though: it is a rather long page and it requires some intensive scrolling, and somehow every time someone has to refresh the page, the scroll level resets to the very top. This is surely a bit annoying for the occasional user, so the UX expert that lives in me—and I didn’t know it did—tries to solve this by doing the usual: hoping to find a magic JavaScript code snippet in StackOverflow that would sort it out. To my surprise, I cannot find anything that would work right away, and the problem proves to be a bit more complicated than initially thought. As the pressure from the users increases, I come up with a brilliant idea: I will handle this from PHP itself, on the server side. I spend a full morning doing this, and by noon it is more or less working. I am so proud of myself. Of course, until I realized the sheer stupidity I had just committed. If the scroll level is handled from the server side, then what would happen if two users are operating the page and scrolling up and down? Who would win? It didn’t take more than a few minutes of my new feature being deployed until someone called me to complain the page was loading with a weird scroll level.
I ended up removing the offending feature and it remained unsolved for the rest of my time in that company. This ranks quite high in my list of “stupid things I’ve done that I initially thought were brilliant”.
4. There’s No Sequel
Today, ChatGPT is the darling of the programming world, where you can go and ask it to sort regular expressions for you and you don’t need to suffer any more regex’s famous torturing, obscure syntax. Back in the day, all we had was StackOverflow. There, you would use to your advantage the fact that your problems were probably not more original than the next guy’s, so most likely another person had probably suffered your current misery before and shared a code snippet that would sort your current problems out.
While I was working on the website I described in mistake number 3, I needed to store data in a MySQL database. I knew some basic SQL1 from before, but my knowledge of it was rather shallow. This time, I needed to do some more complicated stuff, and StackOverflow somehow took me down the dark path of nested queries, and of course, I showed no resistance. I wish I still had the code with me to share here the obscurity of the highly convoluted query I came up with. It was like a rotten onion, with so many layers and parentheses that you could barely understand what was happening. Once the monstrosity was deployed, it kind of worked. But as the database started to be filled with data, the obscure query started to run slower and slower. So the time came when I had to sit down and figure out how that hideous nested query worked in order to fix it. Believe me, it was not pleasant.
Every now and then, I dream of having to fix this query all over again, just like the recurring dream of going to school in your underwear.
5. Friends Will Be Friends
There is an antipattern in software development called “new best friend” (also known as “Golden Hammer”). Basically, it means that when a software developer reads about a concept, a pattern, a framework, or technology, it just must be applied. Like a new best friend you need to show around everywhere. When all you have is a hammer, everything looks like a nail.
My new best friends back in 2010-11 were wrappers. I had read about the possibility of packing software made in low-level languages (C or C++) that could be called from interpreted languages for scripting purposes or similar. Fancy. And I was thrilled. So I had to find an excuse to use this, to show my new friend around. The possibility finally came up (I can’t really remember what was the actual application) so there I went to use a wrapper called SWIG, a rather convoluted thing honestly. Ironically, the “S” in the acronym stands for “Simplified”, but believe me there wasn’t anything really simplified about SWIG at all. In the way it works, SWIG generates some really obscure 'glue code' that is required for the target languages to call into the C/C++ code. And which target language I chose? TCL. Why on earth TCL you ask? It was another friend I had discovered recently (Python was too mainstream so I wanted to be a contrarian). So there I went to create a half-baked contraption out of two things I had only recently gotten to know about, and whose combination was unsurprisingly bad. I remember endlessly battling with SWIG’s glue code, absolutely unable to understand why it was not working, or why it was working.
This lives rent-free in my head. Why? Why suddenly lose all my critical judgment for something I have *just* learned about?
6. 16-bit Supercomputer
It’s maybe the year 2005, and I work for a company that makes LED digital billboards and those signs that call you by number and play a funny ding-dong when you go to the pharmacy or the bank.
The company wants to launch a new product: a new queuing management system. And the time had come to move on from the legacy 8-bit microcontrollers we’d been historically using and embrace the thrill of—drum roll—16-bit MCUs. It was the PIC24F family from Microchip if I recall correctly.
With so much more “computing power” on our hands, so many extra bits, and in perhaps one of the most rampant cases of featuritis I can remember, my boss and I propel each other into pushing so much functionality to that poor 16-bit bastard, that by the time the programming is done we barely have any space left on its tiny flash memory. We want everything configurable, we seek to optimize waiting times and collect metrics about server performance. We want 4096 RGB tones for each 7-segment display, we even kind of code a filesystem from scratch in that poor MCU. We also want the system to automatically discover new displays being connected to the bus so I come up with a clever system where the main CPU would sends heartbeats over the RS-422 bus and increments counters inside the displays’ MCUs (which are initialized with random numbers in the factory) and once the counters overflow, the display would send its information to the bus to be collected by the main board. I am so proud of this feature; I feel like an inventor.
I do remember one guy came at some point and said “You know, you could just do all this in Visual C++ on a PC and just use TV screens over WiFi or LAN”. Of course we ignore his absolutely on-point comment. Who are you to bring your common sense here, mister?
When the coding frenzy has finalized and the product is undergoing some pre-release testing, we realize it is slow and a nightmare to configure, with so many options and with one of the shittiest user-experience ever as the whole thing is still a goddamn embedded system which are not precisely famous for being a charm to deal with (you must connect an RS-422 cable to the QMS main board and use a converter to a laptop running some CLI black magic to configure the product).
Needless to say, the product never sees the light of the day. If there’s any morale of this mistake, it is surely around being aware of when to stop to add features taking into account the capabilities of the underlying computing plaftorm.
7. The Final Countdown (Wake Me Up When September Ends)
It’s the year 2006 or 2007. I am still in the company making LED billboards and am tasked with the job of working on some countdown signs—clocks— that will be used to inform about how long before the Buenos Aires Marathon starts. So, the requirements were rather simple: input a date, and the system will need to display how many days, hours, minutes, and seconds between now and that target date. Sweet. I mean, you don’t always get such clear requirements so when you happen to really know what’s needed, it’s a great feeling. So there I went to code the clock.
Somehow I choose to over-engineer the calculation of the amount of seconds between now and the target date, which of course I need to know before converting it into days, hours, mins and secs. The formula looks quite convoluted, with some shady type casting here and there, but it probably checks out. But it did not check out. For reasons I cannot fully describe because I do not recall the specifics, my calculation is not accounting for a change of month between the current and the target date. So, a change of month would screw the numbers so badly that the display would show just absurd times or even increasing times. An increasing countdown. Lovely.
And, knowing my luck, of course, the scenario includes a change of month: I am working on the clock in mid September, and the Marathon is scheduled for early October. And because I am happily unaware of the bug because my testing in September could not reveal it, I finish my work and tell the guys in production to “ship it”.
As dozens of these clocks are shipped and installed across the marathon path while branded with the sponsor of the marathon a few days before the event, all of them are showing rubbish as they are turning them on.
Debugging code in a weekend with a client angrily looking at you is one of the most uncomfortable tasks ever.
8. Old News
To finalize the list, another one from my time doing LED billboards. It’s crazy how that company did not fire me after all this. This time, I am working on one of those big LED signs that go to highways and display information such as driving conditions and the like. I’m doing routinary firmware coding and testing work, and when you don’t have real highway messages to display, you need your occasional Lorem Ipsum. My style back in the day is to go to the main sports newspaper in Argentina and copy some text from some article to use it as a placeholder to test the code. It’s the year 2005, and I am using text from an article that is about Boca Junior’s coach “Chino” Benitez discussing the line-up for an important Copa Libertadores match his team is about to have. The blurb works like a charm for my testing, so I decide to keep the text from the article as a default message for testing in the factory, so the guys from manufacturing don’t need to call me every time for a blurb to test the display. Needless to say, we are supposed to delete the test blurb before the displays are shipped and installed on the highways and replace it with an empty string as per the client’s requirement, but somehow I don’t do it.
So there they go, dozens of billboards installed on highways across the city displaying the same sports news over and over, which looks current for a day or two, but after a few weeks, it looks ridiculous.
I am thankful social media is not a thing in 2005, otherwise it would have made some waves.
Bonus: Not my mistake, still lives rent-free in my mind
Years ago, a colleague of mine whose background was not software embarked on the quest of interacting with some API from a domain-specific SaaS. I can’t really remember what he was trying to achieve, but I do remember him struggling with his Python code. At some point, he calls me to his desk to ask me some questions, and he walks me around his code. In the conversation, he explains how much he has spent time and energy not really dealing with actual API requests and transactions but with the strange characters the API is framing its replies with, and somehow complains about how much he had to burn brain power to parse the myriad of colons and curved brackets that seem to infest the API replies for reasons beyond him. The guy had simply discovered JSON by accident. We had a good laugh when I told him.
Here we pronounce it ess-queue-ell and not “sequel”