Pratica di Shadowing: Microservices with Databases can be challenging... - Impara a parlare inglese con i video

microservices bring a lot of benefits.

There's no doubt about that.

But at the same time, it gets kind of tricky when you're trying to deal with storage and databases within your microservice architecture,

because the question is, how do you actually properly couple them together?

Well, it turns out there are different design patterns for that that we're going to learn in this video.

And to start with some anti patterns, meaning how not to do that.

And of course, we're going to learn very important things such as eventual consistency, see deadlocks, transactions, and so on over the course of this video.

So if you're ready, buckle up and let's get started.

All right, folks, so we're going to start with the very first pattern or rather an anti -pattern, and I'm going to explain why, which is called shared database pattern.

And I'm going to use Eraser's AI diagrams here to save time because I'm lazy.

And I'm simply going to tell it that it's a cloud architecture and paste this text.

So I'm going to create two microservices, payment and user services that are easy to instances that talk to the same shared database separately.

Let's click generate and let the AI do its thing.

So as you can see, we have a shared database and our services are talking to it separately.

Now, why is this considered an anti -pattern?

Well, we're losing one of the best features of microservices, which is modularity and scalability and, or you can say isolation instead of modularity.

So usually whenever you have a separate database attached to your service, you can scale much easily.

You can simply create new instances automatically, which is auto -scaling.

But in this case, we're bound to the same database.

So apart from the human problems or human aspect, which is two teams trying to manage the same database and

which is going to lead to worse consistency and design because yeah, maybe sometimes it's not that easy to align with different teams on how to design your database.

But apart from that, working on the same database with different microservices can actually introduce technical issues as well.

For example, deadlocks.

What's a deadlock?

Let's read about it.

In a database management system, a deadlock occurs when two

or more transactions are waiting for each other to release resources such as locks on database objects

that they need to complete their operation.

As a result, none of the transactions can proceed leading to a situation when they are stuck or deadlocked.

So basically, when a payment service tries to make like a big update on the database, it's going to lock the table so that there's no other service

that kind of modifies this data while the update is happening, which can lead to data inconsistency or loss.

That's why it's going to lock this table.

And then at the same time, user service tries to update this table and it fails because the database or the database table is under a deadlock.

So as you can see, there are many reasons why not to use a shared database for your microservices.

Of course, you can go with this if you have a smaller application and not that much load on your platform.

But if you're trying to scale in the future, definitely don't use shared database.

All right.

Instead, what you can do is the following.

So let's clean this up a bit.

I'm going to move this here and I'm going to remove my lines and arrows.

So what we can do is actually Eraser allows us to modify this AI architecture

or use this AI tool to modify our diagram by simply editing a prompt.

So I'm going to say now use a separate database for each microservice and let's see what it's going to do.

All right, it recreated the diagram and it created separate databases.

I'm somehow getting fascinated by how smart the AI is when drawing diagrams.

So now this is a better pattern because we can scale separately.

So we can scale this service and the database separately.

Now it's probably not depicted here, but payment service and payment database are going to be living in separate Docker containers.

So usually it's not a good practice to pack them all together within one Docker container.

But aside from that, now we have better decoupling.

So the requests coming here are going to take their own time.

Even if this database gets deadlocked, there's no issue that user service is going to suffer from this.

All right.

This is a better architecture when you're trying to deal with microservices.

Now there are some things called transactions.

As you can see, this diagram is kind of cool we have separation, but how do we actually deal with transactions?

So let's say we have another service.

So let's modify and say order service is going to call both payment and user service.

So let's generate another service that's going to connect to our two services like this.

Now imagine this diagram, order service wants to update the payment information and the user information at the same time.

This is called a distributed transaction.

Now, how do we deal with distributed transaction?

Because if we were having a shared database, we could easily do a transaction within one shared database and maybe lock it, of course, which introduces a problem.

But now we cannot, the order service cannot simply do a transaction saying, hey, user service, do your thing.

And at the same time, payment service, do your thing and we're just safe, all right?

Because what if this transaction that is atomic, so follows the asset principle,

the atomicity means that whatever started can be rolled back, all right?

So if we're trying to assure that both of these databases are going to be eventually updated, what if one of them actually errors out but user service proceeds?

So the user database is going to be updated but the payment information is going to be old.

So we don't want that.

But how can you ensure this when you're having microservices.

This is one of the issues that comes up whenever we deal with distributed transactions.

And there is one workaround for this called a two -phase commit.

What a two -phase commit means is

that we're going to have two phases in order to ensure consistency within our transaction, all right?

So what this is going to look like is the order service first is going to issue one phase, which is called prepare.

So it's going to tell user service, hey guys, prepare for an app database update.

And the payment services are going to start a transaction to begin.

So if we look here, I'm looking at Postgres.

So Postgres lets you do transactions in the database like this.

So you can start with a begin query, and then you can define your query.

And then before doing an, or you can define your query with an insert and then committing is already the second phase that we're talking about.

All right.

So before committing, if there are no errors happen until this point, this payment service can tell back to the order service that, hey, everything's looking good.

There are no errors, there are no deadlocks.

So I can actually commit.

As soon as the order service knows that the user service and payment service are ready to commit, it issues the second phase, which is the actual commit phase.

And then the second phase lets the user service and payment service know that, hey, you have had these, you already have these queries ready, just commit them.

All right, we're going to commit them and everything's good.

But if one of them fails, the user service says, hey, I'm not ready to commit.

So what we're going to do is we're going to roll back, meaning we're going to close this transaction.

Okay, how cool is that?

So this is one of the ways of dealing with distributed transactions.

But there's one problem.

So in our use case, we have only two microservices.

What happens if we have 10 other microservices that the user service is talking to?

So imagine we have another inventory service, we have another, I don't know, customer service, and so on.

It's very hard to orchestrate all of that.

So order service is going to issue kind of first phase to all of them.

100

And then it's basically going to be a mess until you wait for the second phase.

101

That's why people came up with another pattern called a saga pattern.

102

So I'm not talking about Harry Potter or Star Wars, but it's literally called saga pattern kind of means that these transactions are going to be related to each other.

103

So for this, I'm actually going to create some microservices.

104

Let's say this one is order and this one is payment.

105

This one is going to be user.

106

So we're going to update data in all of those databases.

107

And that's one is going to be inventory.

108

All right.

109

So user and inventory, and they of course have their respective databases.

110

So let's take a database here and place a database for all of them here.

111

So they're going to be living in the bottom of these services and here and here.

112

So the saga pattern is going to look the following way.

113

And this is by the way, our second pattern that we can use.

114

So the saga pattern is going to have a message broker.

115

So something that we already covered in one of our previous videos this can be a rabbitmq all right

116

so we're going to have a broker such as rabbitmq

117

and this is our broker

118

so what's going to happen is the order service is going to issue an event

119

that hey i need to update something to the message broker

120

the payment service is subscribed to this it's going to pick it up

121

and as soon as it's processed it's going to issue another event

122

that the payment has been processed then the user picks it up

123

and then the user issues another event as soon as it's done updating its database here.

124

And then eventually the inventory picks it up and updates its database here.

125

So as you can see, all of them are updating their databases one after another, and they're doing this in an event -driven fashion.

126

So how do we deal with rollbacks then?

127

Well, the rollbacks are gonna work the following way.

128

If this one fails, if the payment fails, then of course, user and inventory are not going to be able to update their databases, which is good for us,

129

but we can also roll back the order updates.

130

So the update on the database is going to be rolled back, but not so fast because it's not so easy to implement that.

131

This literally means that you need to implement all of those rollback functions yourself.

132

So there's some overhead, additional overhead is not always good, but this is one way of ensuring an actual distributed transaction consistency.

133

All right, and this is the second pattern that you can do with your databases.

134

Basically, use Saga pattern.

135

Just keep in mind that it can get a bit complicated.

136

All right, so far we implemented two or looked into two different patterns.

137

The shared database, which is an anti -pattern, so we're not talking about that.

138

The separate databases and Saga pattern.

139

Now, there's another one which is called a composition.

140

So let's take our diagram here to make it clean.

141

So whenever we're looking at this one, actually, so this is already called a composition.

142

So let's change the title of this that we can easily do.

143

So API composition.

144

And you might be asking, what does it actually mean?

145

So we are apparently already using API composition, which is if we have two services here, payment service and user service,

146

and we need to query data from their respective databases, it's quite easy.

147

So querying is much easier than updating.

148

What we're going to do is we're simply going to have an order service that can query payment service first of all, and then it can query the user service

149

and then wait for the events to finish and then simply aggregate all the results here in the order service.

150

Okay.

151

So it's called an event or API composition, meaning you can always have a parent service that tries to aggregate data from other sources.

152

And this is totally fine.

153

This is not an anti -vend.

154

All right.

155

So going further, what can we actually do in order to improve our throughput, so to say?

156

All right.

157

So let's say we don't have the order service anymore, or let's say we have the order service, but no other services.

158

So I'm going to remove two other services.

159

And as you can see, this is super easy with razor .io.

160

And let's keep the database maybe, but call it order database.

161

And we are going to connect the order service to order database like this, and we're going to delete the other connection.

162

All right, so like this.

163

So what we're having here is that the order service is going to talk to the order database.

164

But whenever we're dealing with a lot of data, it's actually good practice to separate your view database, meaning whenever you're doing selects compared to your update.

165

So what we're going to do is we're going to create another database like this.

166

So we're going to say order view database.

167

And the second one is going to be order write or rather not view, but read.

168

All right.

169

So we have now two databases.

170

And what we can do is probably delete this.

171

And we're going to say order service reads from order read database.

172

And it's also going to connect to order write database.

173

Oops.

174

So order service is connected to or the write database.

175

So whenever we're reading the information, we're going to read it from here.

176

And whenever we write, we're going to write it here.

177

And as reads happen more often than writes, we can separately scale this database many times.

178

So we can literally take this database and scale it one time and two times.

179

All right, now we have more throughput whenever we try to read the information from our API, which is cool.

180

And this is called CQRS.

181

So command query responsibility segregation, because we're dividing the responsibility between our reads and writes.

182

And now you might ask, but how do we actually ensure consistency?

183

Well, there are separate ways.

184

All right.

185

So whenever you were writing to the order database, what we can do is at the same time, write to all of our read databases.

186

As you can see, this is probably not efficient

187

because we need to make sure that we write right to all the replicas

188

so this is a bit of a overhead

189

but we can also kind of make sure

190

that there's an automatic replication from write to read database it's going to happen within a millisecond probably

191

so there's an automatic script

192

that can do every time you write to the right database it's going to update the read databases as well

193

but as you you notice this is also might not be perfect in some case.

194

Why?

195

Because if the updates of your database are not that critical, so if the fact that as soon as the right database is updated,

196

updating your read databases at the same time within the same second is not crucial, maybe there's something else that we can use

197

because otherwise updating all the read databases as soon as the

198

right database has been updated also adds some load to your server, okay?

199

Otherwise, this is quite legit.

200

But if our...

201

If the time is not so critical, for example, imagine an e -commerce website where users are updating their shopping cart

202

and it's not so critical to update the shopping cart of a user as soon as the user clicks on plus, but one or two seconds of delay is totally fine.

203

Maybe we shouldn't be updating our read databases so quickly.

204

But what we can do instead is use an event sourcing pattern.

205

So what is an event sourcing pattern?

206

Event sourcing pattern is especially good in some of the use cases, such as finances.

207

I'm not talking about stock market because stock market needs real -time updates, but finances like banks where you want to see all of your transactions in the past.

208

So we're going to keep literally the lock of every transaction or some other governmental organizations.

209

There can be many use cases.

210

So what is the difference?

211

So let's create a diagram and this one is going to be an entity relationship.

212

So let's say create a user table with three different users as rows.

213

Okay, I changed it to ID name and surname and let's generate it now.

214

And we're going to get a database with three different rows.

215

All right.

216

So whenever we update our user, let's say we want to update its surname or name.

217

The update is in place, meaning we're losing the track of the old name.

218

But what if for compliance reasons, we wanted to see their older name as well?

219

Maybe we are some kind of a governmental crucial organization that need to track all the changes within the system.

220

Well, that's why you can use an event sourcing system, all right?

221

So event sourcing is something that's going to basically append events to the database.

222

So one event, which was create user, and there's some associated data with this.

223

Another one was update user.

224

So we are already going to be able to find the information when the user was created.

225

So maybe the name was Tim and then later it got updated to John.

226

So if we go back into the events and see that whenever the user was created, it was Tim, we can easily check this back.

227

So it's not that easy to do it with traditional database.

228

That's why it's good to have an event log.

229

So how's this going to look within the architecture of your database?

230

So let's say you have a service backend service and it's going to, or let's say it's actually a user service.

231

So like this, it's going to update or connect to an event log and event log can be actually Kafka stream.

232

And we're going to write all the logs in here, such as, oops, let's remove that, such as update user and then again, update user.

233

Then maybe we want to delete the user.

234

So if our business requires to save all these changes within our database,

235

we can save them within the Kafka stream and they can be stored there as long as possible.

236

All right, so here's our first connection.

237

But what happens if we want to read that data?

238

All right, this is very, very tricky.

239

So reading the data will be from an actual database.

240

All right, so we're gonna have another DB here and this is going to be a read DB.

241

So what happens is we can have changes on this Kafka stream and every time a change happens, we kind of stream it to our read database.

242

So whenever there's another service, maybe it can be a user service or some other service, we want to read the data, we're going to read it from a materialized view.

243

So first of all, the question is, how do we update the read database at the same time?

244

Well, you can actually listen to different events, all right?

245

Not only in MongoDB collection, you can listen to different events, triggers in MySQL, all right?

246

You have triggers, meaning it kind of listens for events but you also have the same similar thing in Kafka.

247

So as soon as the event happens, we can stream it to a database, all right?

248

We're gonna look into the code and the actual implementation in the future video, but for now, just keep in mind this concept.

249

Okay, so we have a read database

250

and the read database is basically going to take all the snapshots that happened in between.

251

So it's not going to store all the events because otherwise it would be replication, but it's going to save all the snapshots,

252

meaning the states of the application at a certain point of time.

253

Okay, one last thing that I want to show you is actually something that has to do with documentation and Eraser.

254

Actually, we have this tab called Document.

255

And Eraser recently, they actually let me know

256

that we can now use their AI features fully because it's so great as you saw.

257

We can actually even generate documentation here.

258

So as you know, we talked about microservices and databases, so I can click on Generate Outline and I can kind of create documentation for our video.

259

So I'm gonna say create a document that talks about micro services and databases with their different best practices and pattern.

260

Also include the anti -patterns.

261

And let's say what it generates.

262

So the technology stack, I'm gonna say SQL, I'm gonna say Kafka and so on.

263

So let's click generate.

264

What is the desired level of detail?

265

For now it's just an overview.

266

What's the primary goal of the document?

267

To offer the private a comprehensive overview.

268

What's the intended audience?

269

Let's say developers and architects.

270

What aspects of SQL in relation to Microsoft should be covered best practices.

271

Basically it's going to ask you different questions that you can specify and then it's going to generate this document.

272

So let's click on generate and now our documentation is going to be generated.

273

Basically I even could have used this to go as an outline for our video.

274

How cool is that?

275

So guys give it a try.

276

Eraser is amazing and they're actually my friends that I'm also in contact with and I've been enjoying this application since then.

277

So this was it.

278

If you enjoyed the video, always, as always, give us a thumbs up and I'm gonna see you guys in the next video.

279

Goodbye.

✨ Video consigliato

49 frasi

Redis Crash Course - the What, Why and How to use Redis as your primary database

University StudentsBusiness English+1

364 frasi

My Vibe Coding Workflow (Step by Step Workflow)

University StudentsBusiness English+1

177 frasi

Mini Project - Implementing a Simple Web Identity Federation Application

University StudentsBusiness English+1

277 frasi

Distributed Transactions Explained: 2 Phase Commit vs Saga Pattern

University StudentsBusiness English+1

114 frasi

Design a File Upload Service Like a Senior Engineer

University StudentsBusiness English+1

4 frasi

What is Event Driven Architecture (EDA)?

University StudentsBusiness English+1

Cos'è la tecnica dello Shadowing?

Shadowing è una tecnica di apprendimento delle lingue supportata da studi scientifici, originariamente sviluppata per la formazione dei traduttori professionisti e resa popolare dal poliglotta Dr. Alexander Arguelles. Il metodo è semplice ma potente: ascolti un audio in inglese di madrelingua e lo ripeti immediatamente ad alta voce — come un'ombra che segue il parlante con un ritardo di solo 1–2 secondi. A differenza dell'ascolto passivo o degli esercizi di grammatica, lo shadowing costringe il tuo cervello e i muscoli della bocca a elaborare e riprodurre simultaneamente i modelli di discorso reale. La ricerca dimostra che migliora significativamente la precisione della pronuncia, l'intonazione, il ritmo, il discorso connesso, la comprensione dell'ascolto e la fluidità del parlato — rendendolo uno dei metodi più efficaci per la preparazione alla prova di speaking dell'IELTS e per la comunicazione reale in inglese.

Tecnica dello shadowing: leggi la guida completa passo dopo passo →

Pratica di Shadowing: Microservices with Databases can be challenging... - Impara a parlare inglese con i video

What Are Microservices Really All About? (And When Not To Use It)

Design Patterns in Java Theory

Database Sharding! Designing Data-Intensive Applications chapter 6

Advantages and Disadvantages of AI

Armchair Architects: Resiliency in the Cloud – Core Patterns

System Design Was HARD - Until You Knew the Trade-Offs

Redis Crash Course - the What, Why and How to use Redis as your primary database

My Vibe Coding Workflow (Step by Step Workflow)

Mini Project - Implementing a Simple Web Identity Federation Application

Distributed Transactions Explained: 2 Phase Commit vs Saga Pattern

Design a File Upload Service Like a Senior Engineer

What is Event Driven Architecture (EDA)?

Informazioni su questa lezione

Cos'è la tecnica dello Shadowing?