hello everyone welcome to another
episode of the cloud show today we're
going to talk about distributed systems
and we're going to talk about it in the
in the terms of figuring out the most
sort of underrated and hard things about
building distributed systems and for
that the star of the show is the
distributed systems expert Leila
[Music]
hello hello he would have called you
there the distributed systems expert
yeah i'm like "Oh my god he's overdoing
it again." No no that's true it's true
thanks hey it's good to see you welcome
back to the show this is your second
appearance yes thank you for having me
again it's good to see you absolutely
you didn't have enough for the show last
time that's good apparently you keep
convincing me so here I am here you are
all right that's great and and um I know
that you know uh distributed systems is
a is a big deal uh for a lot of people
um and and they're trying to build
something really great and it's hard um
there are some problems and some
challenges to tackle in this space so
tell us first
like who are you why why do you know
anything about distributed systems if
you will
yeah great so my name's Laya uh I work
for a company called Particular Software
where we build in service bus and I'm
basically uh mostly focused and
specialized in messagebased systems
event- driven systems if you will and
I've been building these systems for
over a decade now so there's one or two
things that I picked up along the way i
guess just a little bit just a little
bit and but you're based out of you're
from Belgium right yes absolutely
waffles and fries and ice cream and
lovely things all good things all right
now actually I was I was there just
recently right for the for the
conference it's a really good conference
down there at the um um Techama yeah
yeah yeah that was absolutely one of my
favorites in fact we had already
scheduled this uh this session at that
time so we didn't schedule it then we
had we had we had this on the books for
a couple of months right yes you're
you're a busy person and you you're like
"Can I schedule you in oh June
sorry but I'm here i'm here that's all."
Now you're here and that's it's all our
to our benefit so So let's dive into it
let's go into distributed systems some
underrated hard things about building
distributed systems i mean it's all the
floor is yours take it away what are we
talking about thank you well one of the
things that um I like to bring up is
that we spend so much time uh especially
in event- driven systems talking about
oh should this be a command or should
this be an event and what are our
service boundaries and which service is
going to communicate with what and no
that service should call that service we
don't want that type of coupling should
we do orchestration or choreography and
it's like so many discussions and I mean
they're valuable discussions right um
they're important discussions They shape
how the system is going to look like but
there's this one thing that is like an
oversight over and over and over again
and it's basically how are we going to
version this system like you know the
system is going to evolve over time
right it's not going to be perfect from
day one business requirements change our
understanding of the domain changes and
then things are going to have to change
so how do we do that in a way that
doesn't break the rest of the system oh
that's interesting so so the the the
over focus is on the shape of the thing
uh but they're not understanding or
maybe not putting enough emphasis on the
fact that there is an evolution here you
will you will have revisions of this
thing yes so I think it's really
important to think about it that way
from the start like how are we going to
prepare ourselves for that type of So
there's a version you know 01 or version
one like you have to have a version from
the beginning right not as an add-on
later yes so to start some of those
things also tie into how do you design
the system because I think we have to
design for evolvability as well and one
of the things that I um like to
emphasize uh whenever I can is to say
it's important to start thinking about
public facing events and private facing
events from the beginning because when
private when you have private events
then basically the subscribers are
within the boundaries of that service
right yes they're not going to be
outside of that hence the word private
clearly
but the thing is that when you have
private events they're more easily
changed in the sense that you control
the subscribers right you can just
change them because you know who's going
to use them and you can Yeah yeah yeah
makes sense exactly but with public
events well then as a publisher the
whole idea of popups is that you're
unaware of your consumers that you don't
care who is subscribed and why they are
subscribed and what they're doing with
that information and
One of the biggest mistakes is putting a
lot of information into that payload and
then you're like "Ah that's probably not
a good idea oh these teams are doing
what with that data and
and then it's kind of like really tricky
to take something away because that
breaks your consumers and it's
especially tricky in these large
distributed systems where you don't even
know who the consumers are right that's
true so you kind of have to think about
these things up front so you don't don't
just like expose everything and now
they're taking all and you're like what
did are you taking all you weren't
supposed to take all why are you doing
this exactly so one of the things that I
always like to compare this to is
whenever you think about public facing
events think about it's like it's like
posting a picture on social media like
are you going to regret it later like
how much do you want to give away
basically fair enough i may have had a
couple of couple or two of those in
incidents myself maybe
yeah it's it's a good way to get people
to think about it but of course that's
just the first step is right just how do
you design it um I think a good way to
also think about it is uh you know
something for the audience to dig a
little bit deeper into summer events for
example you could have multiple private
events but then your consumers don't
necessarily care about super granular
level of events those may be very
helpful and meaningful inside your
service boundary but it might be helpful
to basically summarize them into a more
overarching event so let's say that
stuff is added to your shopping basket
all the time and you could be publishing
item added item removed item increased
whatever it is right so um the thing is
that most of your consumers don't care
about that and at checkout you just want
to know like okay what's in the basket
now i don't necessarily want to have all
of that information all of that traffic
and all of that complexity of
understanding what happened before and
all of that and ordering issues and
things like that so then it can be
useful to have that service publish a
summary event of this is the shopping
basket at checkout right and then that
would give you like the view that you
want from a consumer perspective oh man
so that's one way to go about it yeah
but then of course that's not all right
things are going to change whether we
want it or not like I always say change
is inevitable right um so then we have
to prepare ourselves in in terms of how
are we going to version these events and
like a lot of attention is paid to that
in terms of APIs but I don't see that
same attention being spent on on our
events but events are are the API right
in terms of of event driven systems so
how do you prepare yourselves for that
and and the first thing to do is
basically to have a schema to have a
schema Now for that data how is that
going to look like how is that
structured that can also already provide
like additional insights into how your
consumers are even supposed to treat
that data consume that data like a good
example is temperature right okay you
have a field in there that's temperature
okay what does that mean like is it
Celsius or is it Fahrenheit or is it
Kelvin and is it the temperature of
light right like it it could be anything
if it's science then it could be Kelvin
right and and a schema allows you to add
that information to add that sort of
what we call metadata so that it's
easier on the consumer side to
understand how that data should be
interpreted but what's especially
helpful about schemas is that they
enable versioning
ah and then then we can start to talk
about okay how is this schema going to
evolve and what type of compatibility
strategies do we want to have for this
schema
okay
so um there are a couple of different
strategies you could have forward
compatibility backward compatibility
full compatibility and you can even make
that transitive can apply to any of the
previous back uh compatibility
strategies forward backward and full and
transitive basically means that um for
example let's say that you want to use
forward compatibility which would make
sense probably if you're using Azure
service bus for example as a broker then
it allows you to delete optional fields
and to add fields whether they're
required or not and consumers can still
use the previous version of the schema
to basically receive messages that were
constructed with a newer version of the
schema i see yep yep right so when you
use backward compatibility it's kind of
reversed and you can delete fields and
add optional fields and then consumers
can use the next version of the schema
to still be able to process messages
that were constructed with the previous
version right now when you make that
transitive it doesn't only apply to the
previous or the next version but to all
of the versions of that schema that you
have so it becomes a lot more
restrictive
um and the thing is that that basically
allows you to communicate to your
consumers how can this change like how
can I expect this thing to evolve and
how my and it allows consumers to make
better decisions on how to write their
code and how to uh basically prepare
themselves better and gives you a lot
more flexibility from a producer
perspective because as long as you
fulfill the compatibility promise that
you set out to to basically a dear to
like you can keep upgrading to new
versions of the schema without breaking
consumers
now I also want to caveat that right
because if you add something to a schema
well your consumers will still have to
upgrade to use that thing makes sense
right yes but at least it won't break
them
they're not going to fail because oh
something new was added and they don't
know what to do with that right and the
reverse could also be true if you delete
an opt let's say an optional field well
if your consumers were aware that that
thing was optional which is indicated by
the schema then they would have been
prepared to deal with that being removed
because it will just not be there and
they can deal with that yeah yeah yeah
yeah it does make sense because I can
see how the client side or the
integrator or whomever like the the
who's calling um can can have code
dependent on the exact nature of the of
the API or the the schema from before
right uh if they've done it right then
they can they can survive if you have
declared how you are how you are revving
this
exactly so by setting up that what I
always tend to say is that a
compatibility um strategy that you set
up it's basically a policy decision
right it's it's communicating how is the
system going to evolve for also from the
producer side but what can our consumers
also expect from that and and and and
this is definitely one of the choices
that need to be made earlier when
building the system it's not like this
afterthought oh yes and now we need to
version
No this is something that needs to be
discussed up front you need to agree on
the strategy uh how we're going to apply
this how do we see everything evolve so
that you can make as little breaking
changes as possible basically yeah yeah
so one place where I encounter a lot of
um a lot of API schemas and a lot of
versions is or API versions is in um in
the Azure management uh endpoint uh
there's a you know there's a hundreds of
services right and they all have they
all have a version and then you can
query you can ask okay which versions
are you supporting still if you will
right and and sometimes there has been
there has been some warnings come up as
oh this is going to this is going to be
changed by so and so date this is not
going to be compatible anymore
that sort of thing so how do you how do
you go about like can can even is that
even a thing you can do how do you go
about removing an old version like okay
we can't support this old anymore
what are we going to do that's a really
good question so that's for me like the
next step right so first we need to try
to set up a system that allows you to
make some evolution that is not breaking
okay um and then the next step would be
okay like really there's nothing we can
do about this we really need to break
this you have a breaking change well
then it then it kind of becomes a
completely new thing because the schema
has changed so basically the the meaning
of that event has now changed and now
needs to be re-evaluated by your
consumers and the best strategy to deal
with that is to basically first notify
your consumers like you know this this
is not going to be supported but to also
not go and delete it immediately to
basically work with a deprecation uh
warning is like like we do with public
APIs right um so basically to say okay
this will be supported for the next
three months and throughout that sort of
deprecation period what is important
from a producer perspective is let's say
that you have an old order placed event
and a new one right is that throughout
the deprecation period you continue to
publish both both of those events of
course and that that allows your
consumers to gradually start basically
upgrading to that new order placed event
in in in favor of the old one but even
that is quite complicated right because
the thing is that especially in an
event- driven system well let's say that
you're using Azure Service Bus okay so
you're subscribed to this topic and now
there's a completely new topic because
it's a completely new event it has
semantically changed therefore it goes
to a different topic at least if you
chose to use a a topic per event type of
topology of course assuming that for a
moment now the problem is that there's
inflight messages right how do you not
lose anything when you're trying to do
that yeah exactly and that that requires
a really sort of step-by-step type of
upgrade strategy so the whole idea is
that during this dual publishing period
when when the producer is publishing
both of the versions of the event you
basically as a subscriber have to go and
subscribe to that new type of an event
and as close to that as possible
unsubscribe from the old one yes now
unsubscribing does not mean that your
inflight messages are lost like in Azure
service bus when you subscribe to topic
you get a virtual queue or subq whatever
you want to call it right so your
messages are still there so you need to
keep the handling code in place so that
you can consume all of those inflight
messages and you don't lose them
right and so you then you'll have to
check is there any is there any more
data in this for me at all or can I stop
using it and then on the other side if
you're if you're the publisher you you
you probably want to check I have this
break here uh how many are still using
the old one and then how many are now
using the new one is do we see a trend
of that's the tricky part that's the
tricky part is as a producer you can't
really do that because like we mentioned
in the beginning sometimes you don't
know who your consumers are and that's
why purely from the producer perspective
I think we need to be a little bit
flexible with the deprecation period
like don't make it a week don't make it
three years right
but maybe like three months five months
depending on how quickly your system can
evolve right but but what I al also
always say from a producer perspective
is that if you have a way that your
subscribers can get notified of this
change right and you have set a
deprecation time well for you it's not
your responsibility to then go and check
the consumers right this is one of the
agreements that you can make within the
system is like okay when the deprecation
time has basically been elapsed and
we're done we've given it six months
we're done then you can basically stop
publishing that old version of the event
and at that point assume that all of
your consumers have upgraded if they're
good citizens of the system landscape
right
yeah this is hard of course well this is
all brilliant and I guess we could be
talking about this for a long time still
but we don't have any more time to talk
about this so we have to stop doing that
which is a shame but it's it's been a
real pleasure having you on the show to
talk about these very complicated
matters of of versioning and and
handling distributed systems thank you
for being on the show thank you for
having me good to chat with you yeah
totally my pleasure and audience see you
next time on the cloud show bye-bye
[Music]