DevOps NG++
I've done a lot of public speaking lately; something that talk about quite often is what I think good development looks like. A multitude of definitions exist for the term "DevOps" and now we're seeing "DevSecOps" or "SecDevOps" and even "SecDevSecOpsSec" whoa!
DevOps was the solution for common friction between development and operations, a new cultural movement to create a fusion between the two units or just outright collapse them all together.
I made this point recently in a talk I gave; DevOps will undoubtedly make things faster, however, it won't fix quality and if you have a development quality problem, it's just going to accelerate the mountain of shit you're piling up and remove the "Deployment Problem" you've been hiding behind or blaming for the last 18 months.
DevSecOps is the realisation that Security can no longer be an afterthought, it's way too important, you've only got to google the latest stats on open source projects, CVE's or reported breaches to see why.
Will the DevSecOps movement change anything, sure people might start thinking about security the right way, shift left and defend a few attacks, but is that enough, are we still missing a trick.
Here's what I know, if you think about your development pipeline, and for the sake of explanation lets we're doing kanban and apply a WIP limit of 3 to our development pipeline.
Now, at this point, everything looks good but as we so frequently see, it's the root of a multitude of sins, development on credit, "scrumbut", literally having no real idea of a teams capability or cycle time.
Take a look after a few weeks:
We can see that for every feature we're shipping out of development, poor testing is leading to acceptance criteria not being met and testing or UAT rejecting what the team has built. Now I could easily go off on why I think UAT and testing are totally wrong ( but let's save that for another post) and if you're already embraced the product owner properly you've maybe already removed one or both of those hops from your pipeline.
We ship one feature that's broken, we get 5 defects back, we fix 5 defects, they raise 2 more. You can see the pattern occurring right, and security is even worse, even if you're doing continuous development badly, I expect you are testing before you release which means you're getting a measure of early feedback. Security is being thought of last though, so when we get our failed pen test results back and raise all those defects, all that's going to happen is another round of Testing/UAT cycle followed by a cascade of defects, Madness!!!
You've now created the software development equivalent of gremlins and at every stage, you push the poor little mogwai through it's like pouring water on gizmo, so we get 5 gremlins back.
The answer is pretty clear, testing, test early test often, let's talk about test coverage, 100% is the minimum you should be aiming for, why? because it's an easy low hanging fruit to hit, technically you're only testing for conditional coverage and path based coverage, ensuring that every logic branch is hit by your tests.
Heres where the hard work really starts, your tests are as good as the developer that dreamed them up, we don't want to be testing for silly things but we do what to be testing stuff that might break our code and cause regression, whenever we get a defect back we should test first then fix the problem.
Let's look at security testing, for example, testing that your nodeJS express server isn't returning the "X-Powered-By" header isn't going to give even 0.1% more coverage but instantly we've reduced the information we're giving away thus increasing the amount of time a bad actor is going to have to spend to figure out what is powering our web server.
How many times have you heard this during the last leg of programme delivery "but we can't affect velocity", let me tell you something, the only thing that won't affect velocity is doing nothing and even that isn't a guarantee if you're already in a bad state.
My advice is this, software development isn't complicated, it's simple and we have bad habits that make it complicated leaving a legacy of failed programmes and once bitten twice shy business.
Take the hit and establish good code quality upfront, yes your velocity will suffer and your team is going to go through a learning curve, but eventually, you will gain a nice startup growth graph hockey stick velocity chart.
Everything you ship will be tested in development, fewer things will come back from testing/UAT, everything that does come back will get a postmortem, your development testing will get stronger, UAT will soon have nothing to do, you can invest in a RED team, continuous security, things will be going great. The RED team will find a vulnerability, you will hold a postmortem and party, no one has failed, this is a great success. Finally, a massive gold statue will be erected in your development team's honour, you have achieved greatness and shipped a product that not only works but ensures it's protected from regression and security mistakes and supply chain vulnerabilities because you did development right!
That might not be how it goes and maybe you're in a hole already and development isn't going so well and all this stuff seems a million miles away and good for greenfields, but don't despair, the beauty and simplicity of agile methodology is here to save you.
BML, that's not a fetish or new type of sandwich, it stands for Build-Mesure-Learn, it's the most basic cycle we use to get better, it's fundamental to making a positive change during any point of software development and here is my recipe for change.
- Stop doing whatever methodology you're doing badly
- Switch to kanban, use 4 columns, (Backlog, Todo, Doing, Done)
- WIP limit of 1 per person for week one, this is a team exercise
- Take one feature and add it to your backlog
- Now with the team break that features into tasks, you should be aiming to deliver this feature by the end of the week, as a team, remember, one task per team member, stick to trunk based development and short-lived feature branches.
- Your definition of done includes security and testing, deployment.
- Got no pipeline? there is your first feature, run it through.
- At the end of the first week Mesure, did you reach your definition of done? Chances are you didn't and that's completely ok.
- Run a postmortem, we usually do this anytime we affect our customer, so let's think of the business as our customer for now.
- Figure out why you didn't get to" done" in a week, this isn't a witch hunt, it's an opportunity to grow and establish greatness (and a gold statue).
- Document your findings and commit the postmortem to your new postmortem repository for everyone in the business to see, frame this correctly and socialise it to the business "We're being transparent, we fell short on a couple of things as a team but we've already committed to fixing those things this week".
- Now, find owners within the team for all those tasks, we take personal responsibility to ensure they are actioned.
- Finally run the exercise again and with those changes you've just made and the process you've just build you're going to improve week on week, deliver week on week and if you don't you will the next week.
It sound's like a lot when you write it down but it takes 2 weeks to see a measurable difference in quality and success.
So after all of that, what is DevOps to me? I'm redefining engineering, I'm doing the NG++ version of this, my engineers are awesome, we embrace failure because it leads to a stronger success, we're focused on the value.
If your development process is a "Schrodinger's cat" and you don't know if you can deliver until you hit the date then simply run the exercise sooner rather than later and establish your capabilities, then estimate that delivery date.
Got the wrong thing from DEV? then your product owner isn't close enough to development, ditch the BA and embrace true product ownership and make them the nucleus and dev the electrons.
Last thought, I promise, if you want to stop doing development on credit and start building products and operating a platform it means development owning code, monitoring, tracing, logging, security and testing. we have already proven Adam Smith economics have no place in software development so let's be brave and redefine how we build software.
Ping me on social media, LinkedIn or leave a comment if you have questions and I'll do my best to help you our and answer.
Bree xoxo