The cybersecurity discourse tends to be adversarial, but more often than I’d wish, I see the most animosity coming between security engineers and developers – not between technical teams and attackers.
When you look at current tools and practices, such as manipulative phishing tests, forced reboots to install patches, and draconian access procedures, you can start to see why. The animosity is baked in.
There’s a reasonable justification, of course: All these practices protect companies (at least in theory) against very real threats. But even if developers agree with these security efforts on a broad level, they rarely agree at any given moment – "Of course, we should keep our devices patched” vs. "No, I’m in flow right now and can’t reboot my machine” or "Of course, no one should have access to everything" vs. "I need AWS access now on the off-chance I need access tomorrow."
We’ve assumed, developers and security engineers alike, that security slows down or delays the “real work.” Downstream of this assumption are some unsurprising results: Developers who resent security engineers; security engineers frustrated by developers; and companies that push security and productivity but achieve neither.
The argument I want to make here is this: Security and efficiency (or velocity, productivity, quality, etc.) are not necessarily contradictory – they can actually reinforce each other. And luckily for us, this isn't just a theory – it's backed up by data and by leaders in the field, such as Jez Humble. It’s an idea that’s time has come and we, as an industry, are on the cusp of fully realizing it.
In 2014, Gartner debuted the bimodal system, which explains why enterprises should split IT into two kinds of systems:
Gartner split software in two to show that the traditional, one size fits all approach doesn’t work. The systems of record involve greater risks and warrant a waterfall development approach and the systems of engagement require exploration, experimentation, and an Agile development approach.
But as Jez Humble, technology advocate at Google Cloud and “godfather of DevOps,” writes “Gartner’s model rests on a false assumption that is still pervasive in our industry: that we must trade off responsiveness against reliability. The conventional wisdom is that if we make changes to our products and services faster and more frequently, we will reduce their stability, increase our costs, and compromise on quality.”
Research, however, proves this assumption wrong. “Companies like Amazon and Google haven’t beaten all-comers by building flaky, insecure systems,” writes Humble. Instead, he argues that DevOps has posed a paradigm-shifting change that many companies have yet to realize.
Defining and measuring development productivity is a perennial problem. Most companies and managers have ditched measuring productivity by lines of code but from there, methods vary.
Many companies take velocity – a tool for agile capacity planning – and twist it to measure productivity but this doesn’t account for how developers actually work and how they actually generate value. Other methods, such as DORA and SPACE, have also emerged but companies often struggle to implement them because they can’t resolve the contradictions between speed and other concerns.
We want to measure and improve productivity but speed inevitably means a drop in quality, security, or both. No matter how we twist our metrics, they remain in a knot of misaligned incentives as long as any improvement seemingly entails a cost. Right? Well, wrong
In Accelerate, Humble – along with co-authors Nicole Forsgren (Microsoft research partner) and Gene Kim (The Phoenix Project co-author), walks through research that shows “there is no tradeoff between improving performance and achieving higher levels of stability and quality.” Instead, the research proves that “high performers do better at all of these measures.”
This pattern applies to security too: “High-performing teams were more likely to incorporate information security into the delivery process” and “[infosec personnel] did so in a way that did not slow down the development process, integrating security concerns into the daily work of teams.”
There’s an irony here – Humble himself frames this as “astonishing” and many project managers would likely feel the same, but as he writes, “This is precisely what the Agile and Lean movements predict.” Remember, lean software development emerged out of the Toyota production system, which was revolutionary because there were no tradeoffs. Toyota figured out how to produce cars faster, cheaper, and better than other companies.
It’s reasonable to feel skepticism. These changes wouldn’t be paradigm shifts if they were easy to internalize and implement. Here, confirmation bias comes back. Alongside any initial skepticism is the real experience of being slowed down by security teams or security experts. But as the research shows, the form of security we’re familiar with isn’t all security can be and when companies implement DevOps processes, security isn’t merely faster; the whole system changes.
It’s not about security vs. velocity (or efficiency or quality). Developers can have their cake, their tea, and their ice cream, and eat them too.
The research done for Accelerate shows there isn’t a choice between security and efficiency but that there is a choice between modern software development and traditional, dogmatic software development. Much of the latter approach, the authors write, “still rests on the false assumption that moving faster means trading off against other performance goals, rather than enabling and reinforcing them.”
If we move past this assumption, we can see that speed and efficiency don’t demand tradeoffs against performance, quality, and security, and we can see that speed and efficiency actually enable and reinforce performance, quality, and speed. Downstream from this realization, numerous problems – many of which seemed unavoidable – eventually dissolve.
It’s tempting to apply a speed vs. safety metaphor to velocity and security. On the road, for example, it’s intuitive and true that enforcing lower speed limits reduces accidents (though research shows too-low limits can also be dangerous). But in software, increasing development productivity and speed can actually increase security.
Focusing on speed from the perspective of “Work harder! Work faster!” won’t help speed or security. But if we extend the ethos of platform engineering, particularly the focus on building golden paths and making the right way the easy way, then we can see how speed actually enables and reinforces security.
No one likes to have dependencies delay them or clunky tools slow them down, so the rise of shadow IT has been the almost inevitable result. Developers are notorious for finding shortcuts to getting things done – both because of how companies measure them and because of how many tools and how much access developers tend to need.
The default workflow has to be the fastest one. If it’s not, you risk the proliferation of tools and shortcuts that will leave your company less secure.
One reason developers sometimes resent security work is because of speed. But a slow pace isn’t inherent to security work – that’s something we can improve.
There are two measures of speed to consider here: Sheer speed and perceived speed.
Vulnerabilities are a great example of sheer speed (or the lack thereof). According to SlashData research, for example, only one-third of organizations can resolve security exploits within one day. You can imagine the team-wide chaos and panic after the announcement of a zero-day.
You could catch glimpses of this dynamic when Log4Shell hit and everyone was scrambling. On the slow end, a federal cabinet department reportedly took 33,000 hours to address Log4j, and on the fast end, cloud computing provider Nutanix took under four days.
Log4j was particularly bad but you can’t chalk up the severity of the consequences purely to the severity of the problem. The ineffectiveness of vulnerability scanning is almost an open secret among security engineers.
According to Veracode research, the average organization is scanning three times as many apps per quarter as it was 10 years ago. And according to Tromzo research, 42% of developers are “seeing more false positives and noise than ever before.”
Threading through this research and these examples, we can see how just one type of security problem can feel like a productivity-jamming interruption. When an exploit is discovered, developers aren’t equipped to address it efficiently, and even with vulnerability scanners deployed and active, there are many more false positives than helpful results.
That said, the implied advice here – “just get better tools!” – isn’t helpful in the short term. But speed is also a matter of perception and there are ways to improve that too.
UX designers, for example, know that page load times are never instant but that there’s a spectrum between seemingly instant and slow. Nielsen Norman Group research shows that if a page loads in under one-tenth of a second, it feels instant – which is ideal – but if a page takes about a second, users still feel in control and if a page takes under 10 seconds, then users can still pay attention.
An example closer to home is code compiling, which developers know can take a while but experience more as a natural annoyance – or even a meme – than an actual frustration.
(Source)
A similar principle applies to security and cross-team collaboration: If the development team is isolated from the security team, delays from the latter team will be frustrating if the former team doesn’t know the reasons behind a delay or have a timeline.
Research bears this out, with the same Tromzo study showing that almost 30% of developers and security engineers are siloed and have “little communication between them.” Worse, is that only about 17% of teams have developers and admins perform security reviews together.
There’s a higher-level frustration above the frustration from day-to-day work: Companies and thought leaders are encouraging philosophies like shift left security and DevSecOps without actually enabling developers to pursue the goals those ideas imply.
Rachel Stephens, RedMonk industry analyst, writes that “If we are asking developers to be increasingly responsible for building secure apps, we have to make it as frictionless as possible for them to do so.” She cites building software with security defaults, embedding principles of least privilege, reducing configuration work, and increasing automation – in other words, she writes, “We need developer experience.”
To be clear, it’s not as though no one has proposed creating a better developer experience before. Nicole Forsgren, Microsoft research partner, has said that “DevEx isn’t necessarily a new idea; it’s just something that almost no one has paid attention to before, and rarely approached with appropriate rigor, whereas Google and a handful of other top companies have leveraged and benefited from these methods for years.”
And therein lies the gap: aside from a few companies handling developer experience really well (and reaping the rewards), most companies handle developer experience either through anarchy or bureaucracy.
Startups often embrace anarchy both because they don’t consider security to be a high priority until a later stage of growth but also because a frictionless developer experience is more attractive than the slow, bureaucratic, locked-down experience most enterprises offer.
But this isn’t fate. Companies have slower development experiences as they grow because the industry tends to counterpose productivity against security and because the primary approach to increasing security is to add tools and processes. Without collaboration, throwing a tool or process over the wall becomes the only available solution and a development process that requires many extra steps becomes inevitable.
SlashData Research on developers working in enterprises proves that pattern, with 25% of enterprise developers saying that one of their most significant challenges is managing security or adding security into the development pipeline.
Here, it’s worth remembering that developer experience, like speed, is as much about perception as literal reality. In a “by developers for developers” post, Sourcegraph CTO Beyang Liu writes that developer productivity conversations should “start from direct experience.”
In one section of the article, Liu argues that there’s a loop inside the traditional SDLC loop. The latter maps onto a higher level of development work, including sprints, projects, and releases, but the former maps onto development work that non-developers don’t always recognize – primarily the “read-write-run loop” that occurs throughout writing and testing code.
You can see how two teams could nominally agree and substantively disagree if security engineers consider a security step a natural part of the SDLC outer loop but developers perceive a security check as an interruption to the inner loop.
The name of the game is integration – integrating security tools into tools developers already use, like Slack, and integrating security processes into development workflows.
If we rethink the relationship between security and efficiency, we can see how efficiency reinforces security and how we can build security processes with, alongside, and for developers.
Change management is one of the biggest barriers to adopting new security tools but that’s because companies are thinking about replacing heavyweight systems and processes with other heavyweight systems and processes. Instead, companies can and should be finding ways to leverage already available skills – coding chief among them.
Security is what you get but coding and engineering are how you get it.
With Sym, for example, we’ve focused on building an access management system that companies can use to make incremental improvements to their security posture over time and as they roll Sym out. If companies can iterate, they can adapt their access policies as their needs change and as they find ways to customize and fine-tune.
The goal is to show more companies how to make security lightweight, customizable, and automatic – another have-your-cake-and-eat-it-too moment. Developers and security engineers can build security tools and processes rather than merely pilot them.
