The Key to a Strong Engineering Culture: Managing Mistakes

By
Anupya Pamidimukkala
October 2, 2023

After working at seven different companies in the tech industry, ranging from startups like Sym to giants like Oracle, I’ve learned that company culture is as unique to a company as a fingerprint to a human. 

Company culture dictates everything from the process of working on a project to how people in the organization react to mistakes. This is a story about what happened at Sym the first time I made a mistake, and how your team can emulate this response to mistakes if you want to build a supportive, productive engineering culture.

The Process

We are a small team of six engineers and one PM here at Sym. Unsurprisingly, each member of our team carries a high degree of responsibility. For new projects, that process looks something like this:

  1. Our rockstar PM, Max Baehr determines new projects by interviewing our customers
  2. If there is sufficient demand, he puts together a project brief. 

Then, two engineers volunteer to take on the project. At Sym, this entails thinking through all the technical requirements and potential hurdles that we may run into in this project, and writing all the code and other implementation details required to deliver to production.

Rather than spending weeks of asynchronous back-and-forth, we go through new project briefs in our daily sync and often arrive at an implementation-ready state by the end of the hour. Moreover, this daily sync is the only meeting of the day 🤯 This was the biggest surprise for me when I was interviewing at Sym - everyone in the engineering team has a grand total of 1 scheduled meeting per day (the only exception being Tuesdays when we have all-hands). 

The Mistake

Ever since I joined Sym, I was looking forward to the time I would lead a project. I got such an opportunity with the Vanta integration

Vanta is an automated security monitoring platform that helps companies get SOC2, HIPAA or ISO 27001 certified quickly. With Vanta + Sym, you’re able to manage temporary access, audit your team’s access, track access changes, and prove that only a set group of admins have access to change access workflows and reporting.

Now, to build that integration, Vanta has several API endpoints that we can integrate with so our PM decided that for V1, it was sufficient to sync all user accounts to Vanta (PUT <https://api.vanta.com/v1/resources/user_account/sync_all>). This “state-of-the-world” sync will help Vanta become a one-stop hub to determine whether users have: a) appropriate levels of access and, b) secure authentication mechanism for their audit. Additionally, since Vanta only needs this information for compliance purposes, the sync can be done hourly instead of real time.

The precursors are the most difficult parts to think through because this is the first instance of implementing OAuth flow at Sym. While Ari (our tech lead) worked on the OAuth flow for a Sym user to grant access to their Vanta account, I sunk my teeth into the hourly sync job itself.

After two days of work, I put up a pull request. There was no way to test it until the precursor work was complete but I wanted to get some early feedback from the team. I was in the middle of going through said feedback, when I came across this.

😕 indeed. In all my prior jobs, if Auth0 was a component of the system, then all users in the organization existed in Auth0. However, Sym has various types of users: admin, member and guest. And of those three, only an admin must create an account in Auth0. Everyone else can simply call /sym and request access to a resource without having to login to Sym. Since admins need to set up Sym in Slack, they need to be able to authenticate themselves and therefore only they exist in Auth0 🤦‍♀️

This was not written down in the brief because Ari assumed I had considered the case of members and guests but I assumed they had to exist in Auth0 so I didn’t think it was important enough to bring it up.

At this point, I was sweating bullets. Where would I get all users if not from Auth0?

How big of a refactor would this take? How many days of work has my oversight cost the team? At a startup of our size, every day counts.

I had made my first big mistake at Sym. 😰

The Solution

Having worked at Sym for a bit by then, I had a grasp on our culture. I knew that if I asked for help and came clean, the team would not punish me. I posted a separate thread in our project Slack channel soliciting opinions for workarounds. My residual trauma from past jobs where mistakes were treated much worse had me mentally preparing for the worst.

And then a fellow engineer at Sym pointed me in the right way:

The DB? You mean to say we have everyone using Sym (i.e. everyone in our customers’ Slack) stored in our database? Of course! In my panic, I had not considered that this information had to be stored somewhere as we needed to keep track of who had access to what as part of our logging/auditing. Luckily for me, this is a few straightforward lines of code. Instead of querying the Auth0 API, we can query the database and process the users that way. I heaved a sigh of relief and made the necessary change.

The rest of the PR review process was not smooth either. After it was merged, I checked the total comment count; it was over 100 comments. It remains my most chaotic PR at Sym and I got the big learning experience I was yearning for. 

The Aftermath

One part of the process I left out earlier is that we have biweekly retros. This is where we discuss what went well, what went poorly and ways to improve the latter. What’s unique to Sym is that we also talk about our feelings. In this phase, we turn our audio and video off, join a Google doc in incognito mode and express our true feelings about the past 2 weeks anonymously for 5 minutes. 

Unsurprisingly, I used this opportunity to express that I was feeling remorseful for my oversight, even though it ended up being a quickfix. My coworkers were very empathetic and instead of letting me take the fall for it, we blamelessly discussed why it had fallen through the cracks. Only by doing this, did we uncover the underlying issue: the team was stretched too thin across multiple projects. It slipped the other engineers’ minds during brief review due to juggling context across the number of open briefs. In order to prevent this from happening again, we vowed to soft cap the number of open briefs that the team is actively reviewing/working on and not spread ourselves too thin.

As I’ve mentioned before, I’ve worked at companies where an engineer faces worse repercussions after a mistake like this. This might be implicitly by coworkers not trusting them with big (or high impact) projects anymore, or explicitly, by placing them on PIP. At Sym, we treat each other with compassion and foster a growth mindset. It increased my confidence to continue to try new things, lead projects and rely on my teammates for support whenever I am stuck.

Since the Vanta project, I have become better at leading projects (check out reason for approve/deny and KnowBe4 SDK Integration) due to a better understanding of the codebase, which resulted in better anticipation of roadblocks and thorough brief writing. Shoutout to my manager Nickolas Means for trusting me to lead and Ari Spikerman for being my partner-in-crime and north star for this project.

Recommended Posts