In this episode we’re joined by medical devices specialist Jeff Gable, whose mission it is to „drag embedded development kicking and screaming into the 21st century“.
He tells us about the specific issues when applying DevOps principles to embedded systems, from the technical to the cultural.
In this episode, Luca Ingianni and Dierk Söllner interview Jeff Gable, an embedded software consultant in the medical device industry, about the challenges and opportunities in applying agile and DevOps principles to embedded systems development. Jeff highlights the unique aspects of embedded systems, such as hardware involvement and slower iteration cycles, and delves into how these can be addressed through modern software techniques. He also discusses the cultural and regulatory challenges in the medical device industry, stressing the importance of agile documentation, risk management, and continuous improvement within these constraints. The episode concludes with a focus on the potential benefits and future prospects of implementing agile and DevOps in hardware and safety-critical industries.
Inhalt
- Introduction of hosts and guest, Jeff Gable
- Jeff’s background in embedded software for the medical device industry
- Discussion on DevOps principles and their application in embedded systems
- Differences between DevOps in web and embedded systems
- Challenges of implementing agile and DevOps in hardware and regulated environments
- Agile documentation and risk management in medical devices
- Leveraging modern tools and platforms in embedded systems
- Cultural challenges in adopting agile and DevOps in traditional industries
- Potential of agile and DevOps in safety-critical environments
- Jeff Gable’s resources and recommendations for further learning
Shownotes
Transkript (automatisiert erstellt, wer Fehler findet darf sie behalten)
Welcome to a new episode of DevOps auf die Ohren und ins Hirn, or in English, DevOps from your ears straight to your brain.
Usually, this is a German-language podcast, but we sometimes make exceptions for international guests.
Today is one such exception.
My name is Luca Ingianni and I host this podcast together with my colleague Dierk Söllner.
We’re both DevOps consultants, trainers and coaches, trying to get teams to work together better and create better products for their customers.
Today we’re joined by Jeff Gable.
Jeff is an embedded software consultant for the medical device industry.
And his mission is to drag medical device software development kicking and screaming into the 21st century.
Jeff, welcome to the show.
Hi, Luca. Hi, Dirk. It’s great to be here. Thank you.
Hi, Jeff. Great to have you here. I want to give you a warm welcome.
Thank you.
Could you please give us a short impression to introduce you?
How would you introduce yourself?
Sure. So, as Luca said, I’m an embedded software consultant.
I’m an embedded software consultant and I’m focused on the medical device industry.
And the medical device industry in general is way behind the times in terms of adopting modern software development techniques and principles.
And so it’s, as Luca said, it’s my mission to drag the medical device industry kicking and screaming into the 21st century.
I’m trying to introduce agile software techniques, DevOps principles.
And, you know, starting at the technical level and moving up the organizational level.
And, yeah, I’ve been a solo consultant for three years and it’s been a fun journey.
Awesome.
Since you spoke about DevOps, we usually have this tradition on the podcast of having our guests explain to us their definition of DevOps.
So what’s yours?
Sure. So DevOps is obviously the amalgamation of development and operations.
And I think at its most fundamental level, it’s about breaking down the silos and treating the production of a software product in a more system, with a system level view in a more holistic way, as opposed to the way it was done before DevOps was introduced, which was in a very siloed way.
And there’s certainly, you know, there’s lots of other principles that come along with it, but I think that’s the most fundamental level.
For me, at least.
Okay. Are there some difference between DevOps in your industry and the normal industry for Luca and me?
Sure. So my understanding is, you know, you guys are focused in the web development world and my focus is in the embedded device world.
And I’m not an experienced web developer. I’ve done some.
But I’ll say the two most obvious.
The two most obvious differences to me are, one, in embedded systems, there’s hardware involved.
And hardware is just trickier to do in an agile context.
The iteration time is fundamentally slower.
There are, Luca and I have talked about this on our own podcast, that there are ways to speed that up.
But fundamentally, hardware adds this complication that just isn’t there in the world of web development.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
Okay.
you know we’re not doing this well we want to be more agile we want to start doing this in a more
modern way they don’t have that really mature set of tooling and practices that they could just draw
from they have to figure out a lot of things along the way and cut themselves on a lot of hard edges
um and and so that those two things i think the heart the fact that hardware has a slower
iteration time and the fact that the tooling is less mature are the two main differences
yeah i was just going to ask like what are you even doing on this podcast you’re doing hardware
aren’t you um how does that like does does devops even apply to your line of work absolutely so so
i am not a hardware engineer myself i’m a i’m a software engineer so i write uh embedded software
and firmware uh and so very very much i i try to apply devops principles in my daily work
and and and and and and and and and and and and and and and and and and and and and and and
convince the others around me to do the same uh but it can it can be an uphill climb how so
like why is it hard to convince people to do devops and hardware
uh because essentially it it requires a different architecture maybe in terms of how you actually
write the software so you can’t just take kind of a traditional embedded software application
uh that was written without this in mind and just kind of do devops
on it like there’s uh in order to even um do take the most fundamental technical steps like
setting up a continuous integration server doing automated builds uh starting to do unit testing
uh your your application has to be architected in a way that supports that and that’s often not
the case for someone who didn’t write an application with that in mind so it often
you know especially if you come into a legacy program where there’s a lot of code built up
it’s really hard even to take that very first step uh and that’s if you’re motivated to do so
a lot of people just don’t see don’t really get it and don’t understand the value in it
and so that’s that’s where it just takes longer to convince them you have to you have to do those
architectural changes yourself get it going start to start to uh demonstrate some benefits before
people really see the value in it okay let me let me ask about this again just because you
pointed out that you’re not a developer you’re not a developer you’re not a developer you’re not a developer
iteration speeds are slower you know everything’s maybe more complicated and clunkier
can you even reap the benefits of agile development and of and of devops in uh you know especially
maybe in safety critical industries like medical devices or something or are you just fundamentally
shackled to maybe a half year or yearly release cycles and you know and you really have no chance
to iterate quickly as agile would want you to do sure so so again because my focus is in embedded
software and because i’ve architected my software to uh be as uh divorced from the hardware as is
practical um my software development definitely takes advantage of agile principles and i can
iterate very quickly um i i think from it if you look at it so you can you can have kind of an
agile software department within a organization that is not agile and i’m maybe you guys have
have touched on this in the past where uh you know if someone has a software that’s not agile
your audience you know they’re a mid-level software manager say and they they want to start
making things better but they’re stuck in this organization that is say more waterfall driven
um you can still do a lot within the area over which you have control you just can’t you know
maybe the organization will never come along uh the way you would like it to but you can still
get a lot done in the in your own department and so in my sense you know even if even if i’m working
with a medical device company and i’m not a software manager i’m not a software manager
and the hardware releases are six months apart i can still do lots of software iteration in the
meantime um and i can even get the software ahead of the hardware and demonstrate that it’s working
in simulation or emulation and then inform the hardware what changes they need to make
encouraging them do it more quickly that can be more of a challenge
so in the end i suppose it comes down to that principle principle that we talk about
in quote-unquote ordinary devops as well which is um autonomy right you you need to be sort of
insulated from from the hardware and conversely i suppose the hardware from you so that everybody
can move along at their own pace would that be the right kind of understanding sure i i i think
that’s a that’s a really good way to put it certainly yeah the software needs to be
insulated from the hardware it’s it’s i haven’t really considered whether the
hardware needs to be insulated from the software in that way um i suppose in the in the sense that
the hardware team should be they should get input from the software engineers obviously
but they should not feel constrained from making a change because it will cause problems in the
software and that’s where um actually having a good software architecture that’s decoupled
and divorced from the hardware the hardware should be able to change out from under you
you as the software developer and
it shouldn’t cause you a lot of problems and that because you’ve architected your software in that way
now the hardware team is free to change things more rapidly um if they find a new sensor or new
component uh that has better performance no no longer is it going to be well don’t change that
because the software can’t support it you know or it’ll take forever in software to write a new
driver for that so so yes actually i think i think it does it works in both ways it helps both the
hardware team and the software team to have that
uh good software architecture that minimizes the dependence between the two
very interesting so even even though maybe the circumstances changed all the underlying
principles are you know just as applicable aren’t they right i mean if you look at you know kind of
gene kim’s three ways of devops and and uh looking at things from a systems perspective to try to
move the you know increase the the speed of the flow of work through the system all the way to the
uh increasing the speed of the feedback feedback loops to get information back upstream um and
practicing continuous improvement those you know those three principles are valid in almost any
context and can like the way you apply them may differ but but i absolutely think those those
principles are are just useful in any context you said that that um embedded is 10 15 maybe 20 years
behind the times what like what what does that mean for the practical work of developers or
engineers right so it means that essentially if you want to take this this agile transformation
journey uh as an embedded developer you know there are not as many um best practices figured
out for you you’re gonna like i said you’re gonna cut yourself on some sharp edges
of the tooling the the tooling in particular uh there are starting to be services coming online
that i think will provide a lot of power and utility and take care of a lot of the really
messy details um services that come to mind uh memfault which makes remote uh monitoring and
debugging very easy uh platform io which is a you know just a general embedded iot platform
takes care of a lot of the the cloud service details for you uh and over-the-air updates
that kind of thing um renode is a very good simulator emulation platform rather that makes
that much easier it’s kind of docker for embedded emulation you know docker is now how old is docker
10 years i guess so right so so docker is behind you know and and again because i am not an
experienced web developer you know maybe i say the word docker and that already dates me um you know
and and luca’s nodding has said yes um so anyway so so and even you know maybe in the web
development world i’m sure there’s a lot of controversy on the bleeding edge you know
people are really into kubernetes and other people very anti-kubernetes whatever
uh the point is is that that industry has has progressed really to the next level in terms of
enabling smaller development teams to accomplish more and more and that is just now and that’s
been the case for a long time and that’s been the case for a long time and that’s been the case for a long time
time in the web development world that’s really just now coming online in the embedded world
and so it’s an exciting time in the embedded development world because all of these services
are just now becoming available and getting mature uh but even so simply setting up a
continuous integration server a lot of it is built for web development and to apply it to
embedded development just takes some a lot of fiddling and configuring and um you know you have
uh integrated development environments and debuggers that are designed to work with the
hardware and you know there it’s hard to get the usb connection to your to your sandbox you know
things so it’s uh that’s what i’m saying is the the practical different effect on the day-to-day
work is that there’s going to be a lot more uh an embedded developer will have to um figure a lot
of these things out on their own okay um jeff you’re talking about difference between embedded
and web development
absolutely the culture is the same way um especially especially in the regulated
industries uh so where i come from medical devices and i’m certain you know aerospace
automotive um nuclear power plant development that kind of thing uh they’re going to be even
farther so i i haven’t actually done very many um consumer electronic projects uh so
i would imagine that’s a little more stratified in terms of there are still people doing
consumer electronics that have no idea uh and there are people doing consumer electronics who
are quite modern and are taking advantage of all of these services that i just mentioned
or developing themselves uh so yeah so the the the culture of these organizations um i i think
is also very behind again there’s a bit of that chicken and egg problem like it’s if you want to
change the culture you’ve got to show the benefits but if you want to change the culture you’ve got to
like actually accomplish the technical tasks and the tooling makes that difficult which means you
don’t have anything to show for it which means it’s hard to change the culture um and so on and so on
and so on and so on and so on and so on so you gotta you have to be a source of excellence and
just uh drive your organization forward in whatever way you can uh regarding um the the silos
hardware yes and software um what what do you think or do you know some
some teams which are uh cross-functional so where you develop the hardware and the software
within one team uh yes and and you know in the medical device world there are a lot of
startups uh where it’s basically just a few people so it’s really the entire company is
only a few people it’s all one product team working on one product uh and so in in such
companies the uh the electrical engineer and the firmware developer are going to be working on
very close to each other and what you would hope from a cultural perspective is they’re they’re
extremely collaborative um you know the the the classic interaction is they’re both blaming each
other the problems i think it’s in hardware no i think it’s in software i that really doesn’t
happen quite as often as as you might fear um but yeah i i guess um that’s something i i like
and especially with my background my degrees are actually in mechanical engineering and i’ve done
um i haven’t designed any pcbs but i’ve done a lot of pc pcb schematic review and so you know
everyone having to um and then again i’ve tried to teach uh software skills to my
double e and me counterparts so they at least understand a little bit of my world so
um certainly the more you can do to infuse the skill sets that will also
help to infuse the mindset and add some empathy and enable you to be more cross-functional
yeah but i’m not sure whether that mirrors your experience like i’ve often experienced that
you know even though everybody’s meaning well and trying really hard to work together and be
cross-functional especially like mechanical engineers and software engineers come just
come from very different worlds in in terms of like their traditions even off of maybe
expressing technical ideas um
how much of a stumbling block has that been for you i i think it is a stumbling block in the
industry i i can’t speak to it that intelligently just because i happen to have that cross-functional
background that’s my that’s my own personal engineering background and most of my work
is with small teams uh but i would certainly imagine in larger larger medical device
organizations say um like i don’t know johnson and johnson or edward scientific or you know
these really really big organizations
there is there is probably a lot of that where you have silo development you have the
mechanical department and the electrical department and the and the firmware department
they probably a lot of companies don’t form these product development teams where you have
a cross-functional group working on a single product line yeah i imagine it is it is a real
a real issue so there’s another question that i’ve been meaning to ask because
now you’ve persuaded me that okay devops works for for embedded systems works for for hardware
adjacent products i suppose or you know products combining hardware but now you come from an
even maybe more difficult world which is the world of medical devices you know a regulated
industry a safety critical industry where you can’t just do whatever you want and iterate
whichever way you want and then looks good let’s let’s deploy it and then you know somebody’s
pacemaker fries and that’s like not a good thing does devops does agile work in safety
critical environments in regulated industries
yes yes it does there and there’s uh it’s a hot topic in medical devices right now
uh and and the fda has you know issued guidances on this and there’s been technical information
reports that kind of try to clear up this misconception that um agile development is
is incompatible with with being safety critical trying to get concrete as i like to do the the
way that you the things that separate say regulated industries or safety critical industries from
those
that are not so fundamentally it’s uh risk analysis and risk management um so like when
you say you know you you’re putting a pacemaker in someone and if you make a mistake it could
kill them essentially the the way that you do that is you know you first come up with this
concept of this pacemaker how it’s technically the work how are you even going to regulate
someone’s heartbeat and then you very methodically and thoroughly think of everything that could go
wrong and the probability that it’s going to happen and the severity
of of what it does and then you introduce additional requirements into your engineering
requirements into your system to mitigate those risks and to bring them down to an acceptable
level and you know the fda in in particular for medical devices um gives you this this
high level process to accomplish that but they don’t necessarily specify the details that’s up
to your company but that’s that’s what you need to do to produce a safety critical device that’s not
harm to a patient you can still do that in an agile context the i would say the difference is
is that what you think of as a sprint where you where you actually release a device to a to a
customer there’s going to be internal product development sprints but if you think of an
overall iteration of you come up with a concept and you you clear it past the fda and you put
it to market that broadest definition of an iteration that will be more slow that will be
slower than say a web development project but it still can be a lot faster than it has been in the
past and essentially you do that by really being ruthless with the number of requirements that go
into that version one and then you still have to do all the steps you still have to do the risk
management you have to specify requirements for each of the engineering disciplines you have to
verify them on the back end you have to validate the device but going to market with a truly
minimum viable product absolutely
works in in that context you just have to follow all the steps you you just don’t you don’t do this
multi-year uh development effort you can do it in one year it still may drive people nuts that you
know you you’re not uh you know maybe people in your audience that are are used to much much
faster iteration times and you can still do those very fast iteration times within the product
development cycle like as you’re like i said as you’re actually writing the firmware for your
device you can realize oh it’s not going to work that way you’re not going to be able to do that
the way that we think it was we need to change and you make that feedback loop as fast as possible
but your overall iteration you can certainly cut from years to a year but that still sounds to me
like you you have some kind of a waterfall start to the process anyway where you do your safety
analysis and you write down your requirements like in the like in the bad old days you write
it all down up front and then you pass it on to the developers and they you know do their thing
is that really what’s happening or do you even
iterate on like your safety case and and whatnot absolutely yeah you certainly iterate on on the
even within that that shortened product development cycle you certainly iterate on the requirements
you iterate on the risk management and and you need to set up your tooling and so at that point
it’s not it’s when i say tooling at that point it’s no longer just the engineering tooling you
know your your continuous integration server thing but it’s it’s the software they’re using
for requirements management the software they’re
you’re using to manage test cases and link the two together to verify that all your requirements
are actually tested those tools need to be set up to support this rapid iteration of requirements
and that that is where the medical device industry is way behind and a lot of people are still
writing word documents and excel files to manage these things and linking them by hand which is
horribly error prone and takes a long time and so that’s great if you write it once and never
change it but as we all know it changes constantly and so that’s great if you write it once and never
change it but as we all know it changes constantly and so that’s great if you write it once and never
continue so you should automate excel
also
um
should automate is the creation of documents so in the end there needs to be a pdf
that goes to the fda that lists out all your requirements and knows which requirements you’re
required to semicolon um and both do that but at that point it just activates a blockchain mechanism
that goes to the fda that lists out all your requirements and knows putting yourself
resources and sometimes you too way we it takes more time u need x my hours x multiple times
で to make this possible and what you should automate is the creation of documents so in the end there
needs to be a pdf that goes to the fda that lists out all your requirements and knows which certain
requirements and lists out all your test plans and shows the results of running those
individual tests, those documents, the production of them from this tool that you actually work
in, you absolutely should automate the production of those documents rather than working in
Word directly.
Because then, as those inevitably change over the course of a project, you don’t have
this parallel effort of trying to keep them up to date.
They’re kept up to date naturally and linking all of the wonderful things that databases
enable behind the scenes can be automatically reflected in your documents.
And that’s something really interesting that I think is worth pointing out, that you have
to modernize, not just make agile, not just your product development in terms of maybe
code, but also really the…
The requirements management side, because traditional agile, quote unquote, maybe is
fairly light on requirements management.
You have a bunch of user stories and you keep them in a backlog and that’s it, essentially.
Like there is no tracking, there is no, what’s the word, I’m blanking on it, traceability.
But of course, the FDA requires these kinds of things.
So you have to almost separately agilize.
It sounds like…
It sounds like the requirements engineering side of your product development.
Absolutely.
Yeah.
So the regulatory and quality teams in, say, a medical device company need to get on board
with this.
And it’s funny in my work, so I know a lot of medical device quality and regulatory people.
One woman I’m thinking of in particular is extremely competent, extremely well-knowledged,
and she hates agile because…
Teams that she’s worked with in the past that have said they were agile used as an excuse
to not do the necessary work.
They used it as a crutch to basically say, oh, we’re agile, we don’t actually need to
write requirements or like, you know, we’re going to take our JIRA user stories and just
kind of like literally print it from the screen.
And the format is completely unsuitable.
It’s not useful for anyone else.
So they’re not thinking it’s…
The problem is, is the development team she’s worked with weren’t thinking from a system
perspective.
Because in medical devices, the system includes generating documents that are going to be
read by the FDA.
You don’t get to sell a medical device without them.
So it’s a necessary component.
You can’t say, oh, we’re agile in our development process and we’ve made this device in half
the time that we used to or whatever.
It does no good if you can’t get it past FDA and sell it.
You know, I told her recently, I can’t wait for the two of us to work on a project together
so that she can see how good agile documentation can be.
And she said, great.
Can’t wait.
So when we do finally work together, I’ll have to put my money where my mouth is.
Yeah, that’s interesting.
Like, this is something that I also encounter in other contexts quite frequently that,
you know, your idea of what your product constitutes needs to change.
Your product is not just the widget that you build.
It’s not just the pacemaker.
It’s also the documentation for the FDA.
And so you also need to, as you hinted at before, you need to create that.
Not in an agile way.
Correct.
And recognize.
So the problem is, is that people, because they still have this waterfall mentality,
they don’t invest in improving that tooling or that process.
And so then changing it is a real pain.
It’s full of a lot of friction.
And so then they do that classic vicious cycle where they don’t want to do it as often.
And that’s, you know, sends you down this road path of very waterfall based development
when it’s just.
Not real, whereas we all know that, you know, if something is painful, do it more often.
So if it’s painful to update your risk management and then update your requirements and then
update the traceability matrix, automate it so that it is painless and systematize that
process so that, you know, every time, oh, I just thought of something that needs to change
in our risk management, like from that aha moment to the documentation is updated and that
not just for the documentation sake, but that as an engineering organization, you’ve
you’ve done your due diligence and you are confident that your device is safe, that
you have not introduced any risk or that you have mitigated any additional risks that
you’ve introduced.
You satisfy yourself first and then satisfy the FDA.
Anyway, from that aha moment of realizing something needs to change until it is actually
changed and is ready to go to the FDA that you need to smooth and automate as much as
possible.
And a lot of people just don’t go.
Through that effort.
Sometimes I think automation is even a question of of mindset.
If I look to my to my customers, I see the younger guys, they are too.
Oh, I don’t know the English word for foul.
You can’t smoke with half and lazy, lazy.
They are too lazy to do the second thing again.
So they automated and the older people.
They are not too lazy.
For them, they are busy and so maybe it’s a little bit a little bit black and white
looking.
But what do you think?
Do you do you see that too?
It’s so I’ve often heard that like, oh, software software developers do all this
investment because they’re lazy and they don’t want to keep doing things.
And and for the easy things, that’s true.
But when when automation requires a fair amount of investment.
I, I don’t think that whether you’re lazy or not will will determine which way you go.
It’s it’s really it’s having that faith that like sometimes automation requires an
investment of effort.
And if I’m lazy, I’m going to avoid doing that, you know, in one sense.
So like if it so so that making that short term sacrifice for long term gain, you know,
cutting your development velocity down in the short term.
Knowing that.
In three months, it’s going to be much faster, that takes a certain amount of forward
thinking and faith that that process is going to turn out, turn out.
And and it takes a management and leadership team that’s going to support that.
Either they have faith that it will pay off or they’ve created a culture where failure,
occasional failure is okay, that that a culture of experimentation.
We’re going to try this and maybe this specific effort won’t pay off, but overall, the fact
that we as an organization.
Are experimenting with new ways to improve that will pay off, so it’s fine.
Yeah, I’ve I’ve often heard the lazy trope and I I I think it’s it’s it’s very it applies
to things that require very low investment, then your laziness will clearly drive you
to make that quick win and that quick script.
But when something takes more time, it’s I don’t know that that’s the best way to to
think about it.
Yeah, that’s the thing, right?
Also with with technical debt, for instance, in general, and and I suppose you could
talk about automation in terms of process that, you know, you’re trading future velocity
for present velocity, right?
Yeah, absolutely.
If you have that, go ahead.
Go ahead, Dirk.
No, it’s it’s it was just one one remark, because if I’m if I’m working as a coach,
it’s not very cherished to to say you are lazy.
So just just a small footnote.
Got it.
Got it.
So but but I mean, you are making a very.
Compelling argument for why Agile and and DevOps really should be in every engineers
and and every business’s best interest.
And it should really be, I suppose, the industry standard.
So why isn’t it?
Or is it?
I don’t think it is, is it?
So again, I I can only speak to the medical device industry, but it’s certainly not there,
though it is it is becoming more accepted.
There is more and more talk about there’s official guidance document, which, you know,
a lot of the.
The legacy companies kind of always look to there’s now official cover from the FDA to
allow them to pursue this, though, I will say so so another way to divide up the maybe
the audience or the people who have to actually handle this in in real life, there’s more
established companies that maybe have a suite of products or quite a few different product
lines and they’re coming out with new versions every few years and then there are
startups who spring into existence.
One of the bigpeda will become solely for the purpose of releasing the single product and
they’re just hoping for an acquisition.
Those those startup teams that are there are often sometimes there are none of them are
full-time employees.
That’s actually a project that I’m working on right now is the two founders and a bunch
of consultants.
These these teams that spring into existence for a single purpose and then disband, they
are going to have a harder time.
Applying DevOps.
organization as a whole has less time to continuously improve.
You can think of a larger organization that really believes in these principles.
They’re going to learn from one project
and apply it to the next and then apply it to the next.
And yes, even within each project, of course, they’re continually improving.
But sometimes some lessons aren’t learned until the end of the project.
And I think that that’s not my way of thinking of Agile or being Agile
and DevOps, because you have to build up teams who are working for a long time together.
Right, right.
That’s where you’re going to get the most
wins, is where your team has time to continuously improve.
And in the embedded world and especially in the medical device world,
there are startups that are literally only together for three years
before their technology gets acquired by a larger company and off they go.
And so there’s just
not a lot of time there to really improve.
And so that’s, you know, as a consultant, that’s the kind of project I’m involved with.
And so I bring in, you know, I’m working on applying DevOps to my own work.
But in terms of coming into an organization
and really affecting change at the organizational level,
it is much more challenging and sometimes I just can’t do it.
I have to work within the constraints of the organization I’m in.
And there’s not that time to, say, build that credibility
to then
leverage it into organizational change.
Luca and I not too long ago talked with a man named Joe Schneider.
He runs an embedded consultancy here in the States.
But for a long time he worked for John Deere, which is this agricultural company.
They make tractors and agricultural equipment.
You know, think of a legacy company.
But he was with them when they started
their agile transformation of their firmware development group.
And this was a large company.
And there were 400 firmware engineers by the time he was done.
And it took you know, he was there for 10 years.
And over that time, the organization as a whole made huge improvements.
And they started with this little kernel of technical practices and slowly over
time transformed that into or that that evolved into an organizational transformation.
And if you’re a startup, you just may not have time to do it.
Yeah, that’s actually an important point to make, I guess, that there’s just so much
more to think of in embedded systems than in quote unquote ordinary software work.
Because you have all the challenges of software.
Plus you have the challenges of hardware.
Plus you have the challenges of documentation, of regulation, potentially anyway.
Plus you have the challenge of not being able to continuously deploy your code.
Or at least I would feel a bit uneasy about over the air updates for pacemakers.
Is that actually a thing?
So not for pacemakers.
I would say for so the FDA kind of has different risk level classifications,
class one, two and three and pacemakers are class three, which meaning that a they
could cause severe injury or death, you know, class two, which is they could
cause moderate injury or class one is minor or no injury.
Class one is a Band-Aid, class two is a diagnostic device that’s returning a
a diagnosis.
Or or some kind of treatment device that, you know, is not treats your eye maybe
or something, but is not life critical.
So certainly medical IoT is also a hot topic and people are doing over their
updates, but I imagine they are not doing over their updates of pacemakers for that.
Because of that risk analysis, the risk is that something goes wrong and someone
dies and that’s there’s really not a I can’t think of a strong way to mitigate that.
Maybe you could and if you can, then by all means, you know, like
a long time ago we talked about Tesla and I don’t want to open a can of worms talking about Tesla too much, but
they they do over the air updates and they have done over the air updates of things that are safety critical, like the suspension performance and the ride height.
You know, if they if they got that wrong, they could cause a lot of crashes.
So they I hope behind the scenes went through a process to verify that to a level where the risk was was low.
I think, you know, the can of worms is
their risk tolerance is higher than a lot of us are comfortable with.
But that’s that’s all I’ll say about them.
But you’re absolutely right.
I mean, not
over the air updates are rare in the medical device industry.
And and that makes you less agile as an organization because it is more
difficult to iterate on your product after it’s out in the market.
You have to release a new version.
And then now you’re dealing with a headache of configuration management.
Yeah. And also maybe you need to cut open your patient.
Right.
Which is kind of a bummer.
Right.
So that’s you know, that’s the kind of thing where
when you can’t do updates of your product in the field, you have to.
And I’m advocating to release a minimum viable product like you release a pacemaker.
It needs to work as a pacemaker.
And then maybe you the next year you introduce an improved version that
maybe a pacemaker is a bad example, but
it’s
a better pacemaker in whatever access that is.
And, you know, new patients are getting that.
And that’s based on feedback from the initial cohort of patients.
But they they still have a safe and effective product.
Maybe your new pacemaker has like implements new functionality,
has additional requirements that are based on market feedback.
Again, pacemaker may not be the best example, but we’ll run with it.
All right. That, I think, has been very, very interesting.
Dick, is there any.
Are there any questions you would like to to ask of Jeff?
Oh, no. Just for the second.
No, it was it was really gave me a good a good impression of for the
problems you have or the issues you have using HR, using DevOps in those kind of industries.
And I hardly and I hardly have no answer for that, because
my last question maybe you say hardware is that you have more longer sprint.
How much would you would you guess that a sprint in hardware development is?
That’s an excellent question.
Luca and I have discussed this previously, so I would say typical is several months.
So say, say, you know, you’re developing an embedded device and.
You discover a problem that requires
hardware fix, I would say typically you get a bunch of those problems to stack up
and then they’re all fixed because the the you know, this again, this is people
not leaning into the pain of what’s difficult.
They let a bunch of those things stack up for a few months and then they do a big
revision and that process of once we say, OK, these are the things we’re going to fix.
It’ll be four weeks minimum until the board is delivered.
But but it is so common to wait
on those kind of things for months.
And Luca has proposed to me in the past and I’m certainly intrigued by the idea
of having essentially a fixed hardware iteration where, yes, OK, whatever,
whatever time from you start the new schematic, which then leads to new layout,
which you order the boards and you have them delivered, whatever that iteration
time is becomes your hardware sprint or maybe something just barely bigger than
that, so that actually you have a new set of hardware coming out every four weeks.
Or six weeks, whatever it is.
I’m I have never seen that done.
I am intrigued by the possibility.
I think it could be really, really powerful.
Sounds good. Thank you.
You’re very welcome.
All right, Jeff, I think this is an excellent place to leave it.
Thank you so much for being on the show.
Where can listeners find more find more about you?
And this is something maybe you would like to brag about.
Sure. So so you can just go to my website.
It’s Jeff Gable dot com.
That’s J E F F G A B L E dot com.
And you can also find me on LinkedIn.
Obviously, all my content and
writings are in English,
not German.
But and you can also Luca, you and I obviously have a another
podcast called the Agile Embedded Podcast.
And if any of you out there are are interested in and how.
More learning more about how Agile techniques apply to the embedded world,
you can I definitely recommend listening to that podcast.
You can find it on any major podcast
publishers, Apple, Google, Spotify, et cetera.
So the Agile Embedded Podcast is what you would look for.
Awesome. Thanks so much, Jeff.
So again, thank you again for being on the show.
This was a really fun and interesting episode.
Thank you. Thank you so much.
Thanks, fellas.
Ciao.