Matthew Kennedy
Product Security Senior Manager
Zendesk
My story moving from App Security to Product Security
Current State & Background
How would you describe Product Security at your organization today?
Product Security within our organization is the link between our Product Development organization and the Security organization. We maintain the closest relationships with our engineering and product leaders, and try really hard to deliver with them oftentimes bringing in specific stakeholders when we need to (legal, compliance).
We champion high impact work. This is how we build trust with our engineers. They know we will only surface vulnerabilities that are actionable and on work that has the potential to affect the greatest % of our development organization. We work hard to determine the root cause of problems and make sure our foundational components are “choke points” for our security controls.
Before Product Security became a focus, how was security typically involved in your development lifecycle?
It started with scanners & software, which produce a bunch of findings that may or may not be looked at. Often we became involved at or post development, as a checkbox.
Inflection Point
What started to feel misaligned that made you think the focus on holistic product security is needed? Was there a specific moment, pattern, or type of issue that made it clear code-level security wasn’t enough?
There was a shared level of frustration that led to misalignment. Security was frustrated by perceived inaction and accountability, and engineering was frustrated by a lack of transparency and collaboration. Specifically, we were noticing alert fatigue.
How did those issues show up internally - was it delays, friction with engineering, uncertainty, or something else?
Yes, friction is probably the correct categorisation.
Organizational Response
When you first raised the idea of formalizing earlier or design-stage security, how did leadership respond?
Our internal security awareness is strong and we are lucky to have partners that are very receptive to “shifting-left”. It was straightforward to describe the benefits we expected.
What concerns did you hear most often from product or engineering teams at the beginning?
The concerns came more from our own people actually. Product managers wanted to get us involved, but it was not scalable for us to have engineers in every single design and planning meeting, especially before things like problem statements and customer impact assessments are complete.
Also, our people aren’t experts in functionality or usability, and security flies in the face of those concepts often, so we try and establish what the goals are for functionality, then work towards delivering that functionality securely. Our customer base is very diverse in their scale and risk tolerances so we often discuss compromises between perfect security and enabling growth.
Execution & Impact
How do you measure whether the focus on design stage risk management is actually working versus just box-checking?
If you have anything that works, let me know. We are still in the infancy of being able to successfully measure this. It has always been a problem for us.
The first thing we are doing is standardizing the outcomes and output of a review, because you can’t measure it if you can’t predict what it might look like. This should hopefully allow us to track the number of control gaps or actions we are requesting our team fix, and from there start drilling down into trends of this class of work vs other streams such as our bug bounty program, or our DAST tooling.
Ultimately the goal would be to see that this class of work results in quicker total remediation time on average, and a decrease of the volume of vulnerabilities that are identified via other methods.
What changed once security started engaging earlier - either in planning, design, or architecture discussions?
The collaboration is definitely more fruitful.
Engineers love solving problems, so being able to give them problems at the design stage, when they are still in that mindset, as opposed to once they have written 500 lines of code and are in the mindset of getting things out the door, makes a huge difference.
I think the biggest difference I have seen is the shared uplifting of both parties' skillsets – our people know and understand more of our product architecture and the customer experience, and our product managers and engineers are starting to pre-empt security controls or questions before we can even get to them, meaning we can all be more efficient and also get deeper quicker.
Reflection
How long did it take to go from "we're doing this" to "this is how we work now", and what were the milestones in between?
It’s still happening.
Our biggest upcoming milestone is to shift the responsibility of engaging security off of engineering and into a fully automated process. At the moment we still rely on relationships between team members to engage us early and it doesn’t always happen. We have found that just having a process documented doesn’t mean it will be followed. So we are looking for opportunities to work within the bounds of what engineering already does and integrate into their processes.
I think once we hit the point of being able to ingest each and every new feature or significant change that happens, that will be the "this is how we work now" moment. This goal has been in place for 3 years and it took us about 12 months of active work to get to where we are today.
Looking back, what was harder than you expected about making the shift - and what turned out to be easier?
Edge cases and the variability of a product development org of 2000. It is very easy to define the happy path, assuming everything is being completed as it is documented, but that is rarely true, for many reasons. It is really complicated to create a process that is consistent and repeatable with input that varies so widely.
Getting buy-in from people was easier than I thought it would be. I expected much more resistance to change, but overall the response has been positive and enthusiastic. We spent a significant amount of time listening to people and understanding their frustrations and then used them as our objectives for instituting change.
Advice to Peers
For teams considering a similar transition, what would you recommend starting with?
My two tips are to start by establishing what security wants, and what engineering wants, and then looking for common ground. There is likely more overlap than you think. And I also recommend meeting engineering and product where they are already. The less systems and processes everyone has to work in, the better.