Shadowing Practice: Databricks Interview Questions by Sr Data Architect - Learn English Speaking with Video

Elige un modo de práctica

Cada modo abre una sala de práctica a pantalla completa.

724 oraciones

Si las oraciones son demasiado cortas o largas, haz clic en Editar para ajustarlas.

Hi, I'm Narendra Kumar.

I'm currently working as a senior data architect.

I have been working in IT for 12 years with 8 years in data engineering.

I have completed all the certifications in Databricks and Microsoft related to data engineering.

Recently, I became a Databricks certified solution architect champion as well.

And taking interviews in Databricks, it is part of my day -to -day life

so I'm creating this video to help out other aspiring data engineers

or data engineers to understand Databricks better

and to answer the frequently asked questions in interviews about Databricks

that I actually ask so that you are more confident about the Databricks interviews

and you know what kind of real questions are asked in the Databricks interviews

I have tried to cover as much as questions I could

but if still I missed something put them in the comments I will try to create another video.

Before we start with the question I want to inform that I have also created playlists about complete PySpark course, Databricks complete course which includes CI,

CD process and an end -to -end project as well.

I have also created interview videos about Pi Spark and Delta Lake as well so feel free to check them out.

We will start with Unity Catalog because Databricks it is incomplete without Unity Catalog.

If there is any interview so there will be questions about Unity Catalog.

So let's go through them one by one.

First question which can come is what exactly is Unity Catalog.

So you see it is a centralized data catalog that provides access control,

auditing, lineage, quality monitoring and data discovery capabilities across multiple Databricks workspaces.

So what it means is that before we had

Unity Catalog, so in each Databricks workspace, it had its own separate user management, separate metastore and separate compute.

But with Unity Catalog, because users are normally common between different workspaces, environments in an organization.

So instead of maintaining them separately, now we can maintain them in one unity catalog similarly the meta store part

we create a central meta store where the different catalogs can be created for different workspaces so that

all our data it is visible in one place

and it is easy to maintain access wise also

we can control the access to meta store in unity catalog itself

and the compute part

that is separate for each of the workspace

that's how these things are related now the next question may come up is like how tables

and databases are arranged in a unity catalog

so this is how the catalog structure looks like in unity catalog

so topmost entity it is meta store

which we can create only one meta store in one region like

if we have east us or central india whatever region is there

so in one reason we cannot create more than one meta store

that is how databricks allows it and within

that one meta store we can create multiple catalogs normally we have one catalog for dev one for prod

and depending on different business units they have their separate catalogs as well within catalog

we have something named as schema so schema it is like our database

which is there in sql server or other databases

so we can create multiple schemas under catalog

and then we have the lowest entities which are our tables

and similar to tables we also have views volumes functions whatever these things we create

so this is how tables are organized in databricks the next question

it can be like how is the how does the three level namespace works in databricks

so the answer is simple because we have this three level hierarchy.

Metastore it is universal in the region so we don't need to mention whenever we are querying any table.

We mention it as catalog name dot schema name dot table name.

So that's how it follows this three level namespace.

The next question we are going to cover is about Databricks computes.

So we may ask like what are the different types of clusters

that we can create in databricks

or what is the purpose of different types of cluster in databricks

so normally people are aware about all purpose and job cluster

but there are different criterias based on

which we can create different types of clusters so

if we explain it in like a very detailed manner then it provides confidence that we

know everything about all the different types that are available.

So at a high level, like there are three types of cluster like all purpose, job and SQL warehouse.

And all these three are combinedly called as classic compute now in Databricks.

And the newer cluster type it is serverless.

So we will understand the difference between all these in more detail if asked.

And then whenever we create a all purpose or job cluster, we may also have either single node or multi -node cluster

and depending on access also we may have two types whether it is single user

or shared cluster

so this is high level we can tell what are the cluster types the next question

or generally normally what is asked is what is the difference between all purpose

and job cluster or a job cluster and sql warehouse

so this difference how much in deep we know

that is very important and it provides a really good confidence

if you are able to explain this one in too much detail.

So that's why we will be I will be explaining this part in more detail.

So these are the high level differences like in all purpose cluster the programmings

which are supported are basically whatever is there in Spark.

Spark, SQL, ML, Python, Scala, R everything is supported.

Job cluster also everything is supported.

SQL warehouse as the name suggests it is to run only sequels.

And the primary purpose for all purpose it is for interactive development and exploration purpose.

And job cluster it is for running automated jobs in production.

And SQL it is for SQL based analytics and BI workloads

and typical users for these clusters so all -purpose it is data engineers

and data scientists

because they mainly work on development job cluster it is mainly used to run jobs pipelines

and schedulers a sequel warehouse it is mainly for bi users analysts

and reporting tools and in terms of cluster life cycle

so all -purpose cluster they start on demand whenever we want to start them

and they stop when they are idle for 60 minutes or whatever minutes we have configured.

Job cluster they start whenever a job start and terminate immediately after the job is completed.

SQL warehouse they start on demand and auto stop when they are idle.

In terms of cost the cost of all purpose is high across these three

because it has some idle time and also the rate is also high.

It is 0 .5 dollar per dbu and

because all purpose it can be used by multiple users at the same time

so that's how the cost is divided between the users

but in general the cost is high and job cluster the cost is low

100

because there is no idle time

101

and the rate is also low like it is 0 .33 dollar per dbu only sql part

102

it is optimized for sql queries so

103

if you are running sql queries then it is a very efficient

104

so these are all the differences between these type of cluster

105

now we are left mainly with the serverless quest cluster

106

so the question

107

that may come up is like what is serverless compute in databricks how it is different from other type of computes

108

so these are the differences which are there and

109

when i say classic it means it is it covers all purpose

110

and job cluster both types and sql warehouse

111

that has its own serverless version as well

112

so these classic computes they run on a customer's cloud cloud sorry cloud account

113

so whenever we are running the cluster

114

so whatever cloud we are using let's say

115

if it is azure then it runs virtual machines in

116

that cloud environment and there the actual code runs wherein serverless it runs on a databricks managed account

117

which is not visible to us databricks they manages everything and

118

that is where the infra management it is uh like managed by customer

119

so as a customer we need to manage networking whatever quota

120

that we want to set up whatever limits we want to add on

121

so those things we need to manage serverless it come it comes as a fully managed by databricks

122

and customization level that is the power of classic

123

that we are in control we can control number of nodes type of node

124

or whatever libraries we want to install everything we can do in serverless we lose control

125

and it is up to databricks how many nodes internally they are running what kind of nodes they are running and everything

126

and with more more like control we get more responsibility also

127

so operational overlay overhead it is more in classic

128

because we need to maintain the cluster and node configurations

129

and we need to upgrade run times whenever we want to upgrade to newer spark version

130

and all but in serverless everything in is managed by data bricks

131

so we don't need to manage anything at all and

132

that is where if we are managing things the cost is low

133

if databricks is managing everything for us the cost is high in terms of data privacy

134

and visibility classic we have complete visibility complete control

135

so it is high serverless we are losing the control

136

and we don't know where exactly those vms are running so privacy and visibility part is low here

137

next we will jump to the questions related to notebooks

138

so the first question that can come up is like what is a magic command

139

and what are the types of magic command we can use in databricks

140

so magic command it is a specific key set of keyword

141

or command that we mention at the starting of any cell

142

and depending on that command the programming or the behavior of the cell is decided

143

So that is a magic command and these are the different magic commands we can use.

144

So for example if we type percentage python then the code is considered that it is a python code.

145

Similarly sql is for sql code, scala is for scala code, r is for r code and if we put percent md then it becomes markdown

146

which is used to write a description about the code

147

this cell it is never executed it is not actually a code

148

and sh command it is used to run shell scripts fs

149

is used to interact with file system like checking out the directory files

150

and all and pip command is used to install python packages

151

so these are the different types of magic commands which are available the next question it is about running databricks notebooks

152

So the question can be like how can we run one Databricks notebook from another Databricks notebook.

153

So the answer is that we have two options.

154

One is using a %run magic command and second way it is using dbutils notebook .run method.

155

And the next question can be like when shall we use, how shall we choose between these two methods of calling.

156

So the answer it depends on their behavior here the primary use case for percent run

157

it is to import the functions and variables

158

which are reusable we keep them in a separate notebook

159

and then hold them in the main notebook so

160

that we can use those functions for example here is a dummy function

161

which is calculating the area of circle and here is a test variable defined

162

so if we want to use them in the main notebook then we use percentage run

163

so those function and variable will be directly accessible in this notebook

164

and if we want to run the entire notebook step by step

165

and we are actually doing some operation inside

166

that notebook pass some input parameter return some output do exception handling for all those scenarios

167

we use this dbutils notebook .run method

168

and here are the differences between these two in some more detail

169

so main use case as i already told it is for percent run it is about importing variables

170

and functions wherein for a dbutil notebook .run it is about

171

running code separately and then getting the output

172

and a percentage run it has to be the first

173

and only line in the cell wherein it can be in any line

174

it is a programmatically we can call it anywhere we want

175

and if you use this one so it doesn't trigger any additional job

176

or something because it is just importing the functions

177

and variables wherein notebook .run it will trigger it as a separate job itself

178

that job will run we can track separately

179

that job will complete then it will come out

180

and then we will it again the main notebook will start

181

that is how this one works input parameters are supported in both output parameters are not supported in percent run

182

because it is not intended for

183

that where in the notebook dot run it it can return output parameters

184

and exception handling also it is possible in notebook dot run percent run it is not supported

185

and it is not required also because we use it only to import the functions and variables which are reusable.

186

Now the next questions can be about secret scopes.

187

So the question can be like, how can we store the credentials securely in Databricks and how can we access them?

188

So the answer lies is in secret scope.

189

So we create an entity named secret scope in Databricks.

190

Wherein we create a secret scope and inside that secret scope we can create multiple key value pairs,

191

which will be our secrets or the passwords that we can access.

192

And for creating the secret scope also we have two options.

193

One, we can create it backed by Databricks where the credentials are stored by Databricks itself

194

or in production scenarios when we work with any other cloud.

195

So there it can be backed by any cloud provider technology as well.

196

Like we have Azure Key Vault.

197

So there the actual secrets are stored in Azure Key Vault.

198

And this secret scope, it is mapping to that

199

Key Vault where we can store the secrets and then read them programmatically.

200

Now the next question can be how can we read any secret from a secret scope.

201

So this is the function we use.

202

At dbutil secrets .get it takes two parameter the name of the scope

203

and the key or the name of the secret which we want to read.

204

Based on that it will provide the secret.

205

The next question it can be about connecting with any storage account

206

so in azure it can be like how can we connect with the adls storage account in databricks

207

and in aws it can be how can we connect with the s3 bucket

208

or they it the question can also be like what is a databricks volume

209

so the answer is like we create a databricks volume

210

which maps to a adls account or any other storage location

211

and to set up this storage account we first create a

212

credential in unity catalog and then using

213

that credential we provide permission to the adls account

214

then we create something called as external location in a in in databricks unity catalog

215

and finally we create a volume using that external location

216

so all these steps are normally done by admin team

217

that way we can tell but once we create a databricks volume then

218

that volume is mapped to a container in an ADLS account

219

and we get this kind of path for example here

220

this is a catalog this is a schema this is a volume

221

so we get this kind of path so within Databricks

222

when we can access that particular container

223

which is mapped to this volume using this path

224

so this is the path we will use everywhere and we don't need to worry anything about the ADLS account anymore

225

So that is how the connection works.

226

The next question it can be like how can we list the files in a directory

227

or how can we copy some file, move some file or copy paste entire directory in terms of ADLS or S3.

228

So the answer again it lies in volume plus this DB utils.

229

So let's say if we have created a volume then in the DB utils we have this FS

230

which is file system library.

231

and which has multiple methods so ls method is used for listing down all the files

232

and then a cp function it is used to copy the file from one location to another one

233

and we can even copy entire directory as well using cp

234

if we provide recursive as true as a additional option and

235

as cp copy the file similarly mv is used to move the file

236

and rm is used to delete the file these are some common functions

237

and we have functions for creating directory also which is mkdiirs

238

we have head function also

239

which shows some initial set of bytes from a file

240

then we have a put function also where we can put some hard coded

241

or like dynamic dynamic provided text we can directly write to a file

242

and the path we can mention in this put function

243

so these are some functions which are available in the db utils which we can use to manage files

244

next we will cover some questions related to databricks jobs

245

so the question can be like how can we orchestrate different notebooks in databricks

246

or how we can schedule different entities in databricks like running the notebooks then refreshing a dashboard all

247

that so the answer is

248

that we create a job in databricks the next question is

249

how can we pass any parameter value from a job to a notebook

250

so here is a simple job

251

so at the job level we can create job parameters

252

which we can edit manually or during runtime also these values can be passed

253

These parameters are automatically propagated to the notebook, whichever we are calling in that particular job.

254

Automatically, these values are pushed down.

255

And then on the notebook side, we can receive these values using widgets.

256

And this widget, it can also be an important question

257

that how can we receive any parameter or how can we parameterize a notebook?

258

So the answer is we can do it using widget.

259

So we have this dbutil .widgets .text function which takes the name of the widget and default value also we can provide.

260

It will create a widget for us.

261

Once it is created this kind of widget will appear.

262

Then we can receive the value of this widget programmatically using dbutils .widgets .get function.

263

Here whatever widget name that we pass its value will be received in this particular variable

264

so this is how we can pass any parameter from databricks

265

job to a databricks notebook the next question can be how

266

can we receive any value back from the databricks notebook to the calling job

267

so the answer is using these task values so we have this function

268

dbutils dot jobs dot task values dot set wherein it takes a key value pair

269

so whatever key we are providing

270

and in the value part we can pass any value at the runtime

271

so this based on this key these values will be provided as a output to the job

272

and then the job will be able to receive it

273

so if we want to check it out in our job

274

so if we go to the job and this is our task

275

which was running and

276

if we go to the next step this is how the values we can receive

277

that like tasks dot the name of the task

278

this is the name of the task dot values dot name of the key

279

which we had provided in the notebook here so based on

280

that key we can receive those values back in our job

281

and the next question about parameters only

282

it can be how can we pass a parameter to a element in for each loop in a job

283

so uh the answer is uh like within the for each loop the inner element the parameter

284

it can be passed simply as input

285

and based on the input

286

it will receive the value whichever is provided in the outer for each loop as a array

287

so here we need to pass as a array as a input array

288

or list and the inner element we need to simply mention input

289

so that for

290

that input it will receive the value whichever was provided in

291

the loop the next questions those are about handling failures in databricks jobs

292

so the questions here can be like what are the different mechanism to handle a failures in databricks jobs

293

and how we can set up retry for different different tasks in a job

294

and how we can rerun any failed task from the job without rerunning entire job from the start

295

so about retry

296

so different mechanisms right before going to retry those are like task level we can set up a retry mechanism

297

and then uh

298

we can also like rerun only the failed task we get

299

that that as a option in databricks we can set up email notification on failures

300

and we can also send do some custom operation on the failure of the previous task

301

so let's check out all these one by one

302

so about retry part

303

so for each of the tasks we get it as a option by default also it is setting three retries within

304

here like four attempts it is doing it is configurable how many retries we want with

305

how much weight duration has to be there between retries all these things we can configure here

306

the next question like how to rerun a failed task in a job

307

so for that one if we go to the run we get a repair run button

308

if we click that it will rerun the job only from the failed point even

309

if there is a for loop and there are multiple iteration of the same element

310

so whatever was successful it will not run again whatever was failed only those it will rerun

311

that is how this repair run works

312

and the next question like how we can set up automated email on in case of failure of any job

313

so for every job we get this as a option which is job notifications

314

here we can add any notification we can put up any email address

315

then we can configure whether we want the message on start success failure duration warning all these options we can configure

316

so these are all the main questions related to jobs next

317

we will jump to the questions related to auto loader

318

so here the first question can be like what exactly is a auto loader

319

so auto order it provides a functionality for to for incrementally

320

and efficiently processing new files as they arrive in any cloud storage without any additional setup

321

so we get a ready to use library where we can just provide a path

322

which we want to monitor whenever there will be a new file in

323

that path it will automatically process those new file

324

and it will also track which all files it has processed

325

so that next time it will like know what are the new files

326

and we don't have to do any like complex setup

327

there all this functionality we are getting out of the box

328

so that is the power of autoloader and

329

if somebody asked like what are the supported sources for autoloader

330

so we have supported sources from all the three cloud services on azure side it supports data lake storage

331

and blob storage both azure sorry amazon it supports amazon s3

332

google it supports google cloud storage and if we are asked about supported files

333

which we can use you ingest using autoloader

334

so simply we can ingest any kind of file anything

335

so some files where it it understand the schema

336

and track the schema also those are like json csv xml parquet avro orc

337

and text files and

338

if the files are not about a structured data

339

they are about unstructured data also then those also can be ingested as a binary file format using autoloader

340

the next question about autoloader it can be like how does it keep track of new files

341

or how does it come to know what are the new file new files

342

so there are two ways uh the older method it is a directory listing method

343

which is the default as well

344

so here it identified based on listing the entire input directory

345

which all files are there on the checkpoint location it will track

346

which files it has ingested and that's how it will keep track

347

but it is less efficient because if a source directory they keep putting more

348

and more files so it has to keep track of all the files check all the new files

349

that way it has to process and the newer

350

and the recommended way it is using file notification mode

351

so this leverages a file notification and a queue service in our cloud infrastructure account

352

so whenever a file source they put any file in a cloud storage

353

then it generates a push notification into an event broker

354

and it creates a queue with list of the files

355

which are like a put up newly a base as compared to the previous run

356

and then auto loader it will be directly be knowing

357

that these are the new files

358

and then it can process only those new files from this queue

359

so that way this function is newer it is very very efficient

360

so it is recommended to set up

361

but to set up

362

this one some on the infra side from the infra admin some initial setup will be required

363

and the next question about autoloader it can be what are the different types of triggers

364

which we can create in autoloader

365

so there are three types of trigger we can create available now processing time

366

and once so available now it will process whatever new files are available in the source

367

or like whenever we trigger this job it will ingest those new files

368

and existing file if not already ingested

369

and it will stop processing time it is for like a real time

370

or near real time ingestion where we want to run the process continuously

371

so it will it the job it will never stop it will continuously run

372

and based on the interval we provide here so based on

373

that interval it will keep checking for new files

374

and then keep processing those files and once it is not much used

375

so it is like it will process all the data once

376

and it will exit so it is kind of a one -time load

377

if you want to do the next question about autoloader it can be like what is a schema location

378

in autoloader and why it is important so schema location we provide using cloud files dot schema location

379

on that location whenever it is reading new file

380

so it will automatically keep track of schema of the files whichever it is receiving

381

so that it can know whether the new files it is matching with the schema of our existing loaded data

382

or not if there is any mismatch then it will a resin alarm

383

and before this the basic question can also be like how do we use autoloader in databricks

384

so we use it using like spark .readstream we always mention readstream even

385

if it is a batch processing for autoloader that is how it works

386

and in the format we need to provide cloud files

387

and for the writing part we use write stream instead of read stream

388

and then we mention a checkpoint location

389

and the output location where we want to store the data

390

and similarly the another question can be what is a checkpoint location

391

so this is a location

392

that we provide where internally it will create some files to keep track what all files it has processed

393

so far so

394

that next time based on these these checkpoint location it will know what are the new files

395

that it need to process now the next question can be about handling schema

396

when using autoloader so it can be like what are the options to handle schema in autoloader

397

so there are two options

398

so one is providing the schema we can create our own schema

399

and we can provide that schema which is preferred for production loads

400

because we want to we are expecting the data to be in a very specific format

401

and the second option it is not providing any schema so

402

if we don't provide any schema then by default all the columns will be of string type

403

that is how it will do it will infer the data it will infer the column names automatically

404

but data type by default it will not infer automatically and

405

if we want to infer the column data types also based on the data

406

then we get this option like infer schema infer column types true

407

if we mention this option then the data type also it is inferred

408

and we get the right data type based on the provided data

409

and now the next question can be like how can we handle corrupted data in autoloader

410

and what is the rescue data concept in autoloader

411

so the answer to that is

412

that autoloader it has a automatic functionality of handling any corrupted

413

or unmatching schema changes in the source file

414

so here is the here is the option like

415

which is cloud files dot schema evolution mode which is rescue

416

which we can provide even if we don't provide i think this is the default value also

417

So if we provide that option then automatically it create a new column

418

which is underscore rescued underscore data and whatever whenever it find any corrupted data

419

or it find that the schema is not matching then

420

that those particular rows it will put in the rescue data column itself.

421

So that is how it can automatically rescue our corrupted data.

422

These were the questions about autoloader next we will go towards debugging related questions

423

so the question can be like how do you debug your code whenever you are doing any development

424

so the answer to that is that in databricks

425

or notebooks we can add break points and then debug our code

426

so while running any notebook we get a debug cell option using

427

that we can do a debug and

428

while debugging the data we get different options like step in go to next line step out all these

429

and going further the next question can be like what is the difference between step in

430

and step out so step in it takes us inside a function

431

if we are on on a line where we are calling a function

432

so if we do step in it will take the execution inside

433

that function and if we click on step out whenever we are inside a function

434

then it will complete the execution of that function

435

and come out of the function that is step in

436

and step out and also we have a go to next line option as well where

437

if we click

438

that then it will go to the next line wherever the like just the next line of the code

439

and we have this continue execution also like

440

if we press this it will take the execution to the next like break point wherever

441

we have added this breakpoint in our code

442

And apart from these options, we also get a debug console where we can run any small code, a line of code as well.

443

And we get this variable voucher as well where we can monitor the value of different variables as

444

and when the code is getting executed.

445

And the next question, it can be like, how do you use AI for your development purpose within Databricks

446

so the answer to that is that we can use this genie code in Databricks

447

so whenever we click this icon here we can write anything

448

or do anything and if we are asked about the modes of this genie code

449

so there are two modes one is chat mode where we can ask any question

450

and it will reply back as a answer

451

and then second mode mode is agent mode

452

which is recommended also where whatever requirement we provide based on

453

that it will write the code it will make changes to our existing notebook

454

if required it will execute the code also itself it will fix any error

455

if it is encountering anything

456

so basically whatever we want to do with data this genie code will be able to do that

457

and apart from this genie code within our notebooks also

458

we get uh ai based options like using a edit code with genie we can ask it to modify

459

or write a new code within a cell

460

and within this we get different commands as well like doc

461

is it is for adding comments explain is for explaining the code renaming the cell fixing any error

462

so these are some hot commands that we have the next question it can be about cost calculation and job optimization

463

So let's say if the interviewer asked that if you are working on a on optimization on any job or task,

464

then how do you track cost and how do you know actually how much cost it has optimized or not optimized.

465

So the answer to that is

466

that we can calculate the exact cost of any job run in Databricks using SQL

467

so we can build this kind of sql query

468

which can take a job run id based on the job run id

469

it can fetch the details from two tables one is a system .billing .usage table from this table

470

we get information like start time end time how many dbus are used

471

and what is the sku name sku name mean type of compute

472

which is there and then for the given type of compute we have this second table

473

which is billing .list prices which has the rate of those debuts like

474

if we spend one debut then if it is job cluster the rate

475

which is provided in this table it is 0 .3 dollar

476

and if it is a all -purpose cluster then it will be 0 .55 dollar

477

or somewhere around that so

478

that kind of price is available here we join both the tables

479

and then we can get the exact cost of our job run in database

480

so this way if we are working on any optimization

481

so before doing the optimization we run our job

482

we calculate the exact cost we note down the time of execution also

483

and then we perform whatever optimization we are trying to do in our code then we run the job again

484

and again calculate the cost using this kind of query

485

and again note down the time and cost so that way we are able to

486

explain what how much optimization we have done in terms of time by comparing the time

487

and how much we have done in terms of cost so both time and cost are very important

488

uh next we will cover questions about lake flow connect

489

and lake flow spark declarative pipelines so

490

if anybody ask like what is lakeflow connect and

491

when shall we use that so lakeflow connect it is a data ingestion service from databricks

492

so whenever we go to job

493

and pipelines we get three options ingestion pipeline etl pipeline

494

and job job part we have already covered so ingestion pipeline is nothing

495

but lakeflow connect

496

and for what purpose do we lakeflow connect uses we need to whenever we need to connect with any source

497

and we need to ingest the data into our Databricks environment then we use Lakeflow connect

498

and when shall we go for Lakeflow connect versus our like custom JDBC

499

or autoloader kind of code is

500

if we want any managed connector where we don't want to write any code we just want to connect with the source

501

And then after connecting with the source, we simply want to just select our table that we want to ingest.

502

And based on whether we want full load or incremental load or change data capture, those kind of configurations we can make.

503

And then that's it.

504

We just configure these options once in Lakeflow Connect.

505

And then ingestion is taken care automatically.

506

Then we go for a Lakeflow Connect kind of use case.

507

and we may ask like uh when uh shell

508

when can be a scenario where a lake flow connect may not be a good fit

509

that is currently like

510

that is for whenever we want to build any reusable metadata driven framework let's say

511

if we have hundreds or thousands of tables

512

so instead of selecting those tables uh like in a in

513

a managed way in a lake flow connect like selecting the tables

514

we can build a reusable framework by using a zdbc or any other kind of connectors or even with additional

515

services like adf and all and using

516

that reusable framework we can ingest more tables just by adding metadata so

517

that is the strength of using anything else any other way

518

of ingesting the data versus what we can do with the

519

lake flow connect the next question can be like what is lake flow spark declarative pipeline

520

and when shall we use those

521

so this spark declarative pipeline this is like a newer version of delta live table

522

that we had earlier

523

and whenever we create etl pipeline then it takes us to creating those spark declarative pipeline

524

and why do we need to use them is

525

if we don't want to write like a complete code like ourself

526

or whatever transformation we want to do

527

and we want we don't we want some connected doors

528

which are already available like reading the source writing the source

529

and also if we want some like ready to use functionalities related to data quality monitoring

530

and the monitoring and auditing capabilities of data bricks

531

which they have put lot of efforts so

532

that we don't have to write complete code we don't have to manage each

533

and everything ourself then we can use go for the spark declarative pipelines

534

so these sdp these are like somewhere between our managed sql based transformation

535

and the like spark structured streaming so if you use spark structured streaming

536

that can also work similar to these managed pipelines

537

but there we have to write so much code manage each

538

and everything like throughout the flow ourself and we don't get the uh detailed auditing and logging

539

properties which we get in these pipelines

540

and these pipelines can also be like deployed using asset bundle in a very managed way

541

so that is also another added advantage of using these declarative pipelines

542

so here instead of like writing the entire code we create them as declarative tables views

543

and also let's say here we are creating a declarative table then this table we are referencing in

544

while creating another materialized view

545

and then this materialized view we can use in another materialized view

546

or table that we are going to create so that way we just uh

547

like worry about the transformation or business logic that we want to do

548

and rest of connecting the dots and managing the things monitoring everything is taken care by databricks itself

549

so next we will focus on the asset bundles

550

and the cicd process so if somebody asks like what is a asset bundle

551

or declarative automation bundle and in which scenario we shall use those so uh

552

so basically declarative auto automation bundles uh those were previously known as the databricks asset bundles also

553

this automation bundles is the newer name

554

so they enable us enable the software engineering life cycle on data

555

and ai objects we can create different version of our code

556

and ai objects we can migrate from one environment to another environment easily

557

and in a managed and automated way so that is the purpose

558

and intention of these automation bundles the next question can be like what objects

559

we can embed into a declarative automation bundle

560

so it can we can embed so many things so like workspace settings cluster related configurations notebook and python files which

561

where our code is there obviously.

562

Our ML models can also be part of that.

563

Integration and unit test cases that we are writing that can also be part of this.

564

Lakeflow jobs and declarative pipelines that we just now discussed that also we can use these bundles to migrate.

565

We can even our dashboards

566

that we are creating those also we can put it in a bundle and then migrate from one environment to another environment.

567

So this is about automation bundle the next question can be

568

like how does the ci cd process works in your project end to end

569

and this is a very common question

570

and most probably it will be asked in almost every interview

571

so let's understand this one in bit detail so we can say

572

that when we are using automation bundles so we have integrated

573

or connected our github repository with our databricks workspaces

574

and there we have our main branch which is our production branch

575

and from the main branch our dev branch is created for the development environment

576

so whenever we are working on any new feature or new development

577

then we create a feature branch from this development branch after creating the feature branch we test our changes there

578

and then we create a pull request which is reviewed by any architect

579

or any senior person

580

and then the changes are merged into the dev main branch sorry dev final branch

581

and from the dev branch the changes are deployed to the dev workspace using

582

asset bundles so that is how like the development environment works

583

and once we are done with our development then we create a pull request from dev to our main branch

584

so that whatever changes are there we can take those to production

585

that pull request again it is reviewed by anybody and

586

after review the changes are merged into the main branch

587

and then while merging into main branch we have configured github actions

588

which deploy the artifacts automatically into our prod workspace so

589

that is how the flow works end to end

590

and another question about this can be like what is a hot fix

591

and how do you use it so hot fix it is some urgent

592

and quick issue that we can fix quickly without going through the entire cicd process

593

so if we need to fix something quickly in production

594

what we do is from our prod

595

or main branch directly we create a small hot fix branch we apply the fix there test the fix there

596

and from there itself we merge the changes back to the main branch

597

so that the production issue it can be fixed quickly.

598

Now let's cover some Databricks architecture related questions.

599

So the question can be like what is a control plane in Databricks?

600

So control plane it is the web application

601

that we log in in Databricks and whatever interface we are using so that is called as control plane.

602

Apart from the web application it also like take care of the orchestration of the compute

603

that is there like setting up the required nodes connecting with our cloud setting up vms

604

they are using those vms unity catalog also with all the access management lineage everything

605

that is also part of a control plane and whatever queries

606

and code we are writing so that is also part of control plane

607

so basically the entire data bricks application itself that is the control plane which is the central unit that can

608

drive what users are doing that can control what shall be done on the cloud environment

609

and everything and with that control plane users and applications they interact here

610

and the next question can be like when

611

if we are using a all -purpose

612

or job cluster then how does the a cluster is set up internally

613

or how it is linked to our Azure cloud account or whatever cloud account that we have.

614

How does it utilize that account?

615

So whenever a user is creating all -purpose or job compute in control plane, then the definition is created here.

616

And whenever we start the cluster, the underlying virtual machines are set up in the cloud provider like Azure

617

or AWS and those virtual machines are managed by Databricks control plane

618

that is how it is and in terms of billing also the for the virtual machine

619

that run so for that duration our cloud provider will charge us separately

620

and because Databricks is managing this compute so Databricks will charge separately in terms of db use that is how it works.

621

And the newer compute that is there which is serverless there the virtual machine

622

or the infra setup it will not be there on the cloud side.

623

It will be within Databricks itself.

624

So there is no management.

625

There is no two separate costs which are involved.

626

Everything is charged by Databricks itself.

627

The next question it can be about data quality like what is the data quality

628

or ready to use features available in Databricks.

629

So there we have these two type of feature which are available.

630

One is anomaly detection and one is data profiling.

631

So in terms of anomaly detection, it checks two parameters currently.

632

One is freshness and one is completeness.

633

Freshness means like when was the table last updated

634

and as per the normal schedule whether it has been updated as at the expected time

635

or not and completeness what it means is

636

that normally on a daily basis whatever number of rows we are getting

637

or did we get expected number of rows in last 24 hours

638

or was there a sudden increase or decrease there so then

639

that is what is checked by completeness

640

and in data profiling it prepare a very detailed summary about statistics of table like min,

641

max, median, distinct values, all those kind of metrics are gathered by data profiling.

642

And based on this anomaly detection, those are automatically applied at the table level.

643

And at the table level, we can see these kind of marks as well.

644

Green means there is no issue.

645

And this orange, it means like there is some issue with any of the data quality checks.

646

The next question it can be about delta sharing like what is delta sharing, in which scenarios we shall use it or what are its capabilities.

647

So delta sharing as the name suggests we can share our delta tables with other users

648

that is the purpose of delta sharing

649

and to share this the users don't have to be part of the same workspace

650

so we can share our delta our delta lake table with any users we want

651

and it is not only within databricks

652

that we can share we can share the these with outside also

653

because whatever sharing protocol is used that is built on open standards

654

so the data it can be directly shared with power bi spark pandas table java also can access it

655

and it can be accessed by all the different cloud providers as well.

656

So whenever we want to share our data with the people who are not part of our Databricks environment, then we use Delta sharing.

657

The next question it can be what is Lakehouse Federation.

658

So how it is used is whenever we have any external source

659

and we don't want to ingest the data from that source into Databricks,

660

but we want to simply access the data as it is without any ingestion.

661

Then we make a connection to that particular source in Databricks as a foreign catalog

662

and this process it is called as Lakehouse Federation

663

because we are not ingesting any data

664

but simply we are connecting with the source

665

and the data it is available in Unity catalog as if it is part of the Unity catalog itself.

666

So that way we are able to use the data from all these different sources like Synapse,

667

Oracle, Teradata, Snowflake directly from AZ Unity Catalog tables itself without ingesting any data at all.

668

The next question it can be about system tables like what are system tables in Databricks

669

and what kind of data it is stored there.

670

so we can say that databricks it has out of the box system tables

671

which provide like very deep visibility into our platform operation like

672

what operations are being done related to governance lineage usage billing everything we can monitor using these system tables

673

so we have multiple system tables for example audit table it include all the access whatever user is accessing anything

674

that will be logged here lineage are logged here list price

675

and usage table we show how we can use them for cost calculation

676

and observability everything all the cluster related information are stored in this warehouse events

677

and classification data classification results are stored here lake flow

678

or whatever jobs

679

that we are running are stored here whatever sql queries user are running those are stored here

680

so each and everything it is stored in a very granular level

681

and we can even create create dashboard on top of this system table so that we can monitor

682

overall how things are going within the databricks environment

683

The next section we will cover about Genie.

684

So we may be asked like what is Genie in Databricks and for what purpose it is used.

685

So Genie it is a AI powered service where we can

686

ask natural language based question on top of our tables or whatever data is stored in Databricks.

687

And creating Genie it is pretty simple whenever we click on new we simply select the tables

688

or views whatever is there and click on create that's how easily we can create

689

and once it is created we can ask any question in natural language like a general english

690

and it will understand our queries generate sql queries based on our ask fetch the results

691

and summarize the result for us

692

so all these things genie can do it for us

693

and next question it can be like how we can make any genie workspace more efficient

694

and how we can get the right answers based on our data

695

so the answer to

696

that is in in the instruction section we shall configure proper instruction what kind of data is stored

697

and what is a what are the dimensions what are the effects

698

and some generic instructions also we can put we can provide it ready to use join conditions

699

so that it use right join condition when asked we can provide common

700

asked measures and dimensions we can define in sql expressions

701

and we can also provide like normally asked sql queries so

702

that like based on those kind of queries it will use the ready to use query

703

and provide the answers quickly

704

and it also comes comes up with a benchmark where we can define some benchmark

705

and test against those benchmarks so that we can evaluate how our Gini workspace it is doing.

706

And the next section it can be about Databricks 1.

707

So we may ask like what is Databricks 1 and what is its purpose in Databricks.

708

It is a new feature maybe not everybody may be aware about it

709

but we normally ask it to know how much up to date the user is.

710

So Databricks one it is a dedicated ui created by databricks especially for business users

711

which is a very simplified and there is no need to write any code or anything

712

so the purpose of creating databricks one

713

it is to provide a very simple interface for business users

714

where they can access their dashboards genie workspaces apps these things they can access

715

and on top of these things they can simply ask questions also in natural language

716

and they will get the answers like which are generated through AI similar to Gini.

717

So these are all the questions that I wanted to cover in this video.

718

I hope you got pretty good idea now you are more confident in terms of Databricks with these questions and answers.

719

If still you think there are some questions which are unanswered or you have come across those in some interviews, put them in the comment section.

720

I will try to create another video with those questions.

721

So that's all for now.

722

Thank you.

723

Bye bye.

724

Happy learning.

Descargar app

Todo lo que necesitas para hablar con fluidez

Pronunciación con IAEvalúa cada oración

Práctica de IPADomina cada sonido

VocabularioConstruye tu banco de palabras

Juego de vocabularioAprende mientras juegas

Escanea el QR

Download on theApp Store Get it onGoogle Play

FOR YOU

Recomendar Video

186 oraciones

Excellent 2 Years AWS DevOps Engineer Mock Interview with Questions and Answers | Cloud | Docker

Business EnglishInterview EnglishUniversity Students

26 oraciones

🔥 Java Multithreading Rapid Fire | Can You Answer These Interview Questions? #shorts

Interview EnglishUniversity Students

324 oraciones

Distributed Task Scheduler: System Design Interview (Stripe & Amazon Offers)

Business EnglishInterview EnglishUniversity Students

Top 30 IICS interview questions and answers | Informatica cloud interview questions full video #iics

371 oraciones

Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

Business EnglishInterview EnglishUniversity Students

188 oraciones

IT System Administrator Interview Questions and Answers | How to Pass an IT System Admin Interview

Business EnglishInterview EnglishUniversity Students

257 oraciones

A/B Testing in Data Science Interviews by a Google Data Scientist | DataInterview

Business EnglishInterview EnglishUniversity Students

239 oraciones

Learn how to SOLVE a data analytics case study problem

Business EnglishInterview EnglishUniversity Students

Vocabulario y notas de pronunciación para esta lección

Esta lección de conversación se basa en el vídeo “Databricks Interview Questions by Sr Data Architect”. Las palabras que más se repiten: databricks, file, create, table, code. Este vídeo tiene 724 frases y 8991 palabras para practicar shadowing. La parte hablada dura 58:10. El hablante habla a un ritmo natural de unas 155 palabras por minuto, cercano a una conversación cotidiana. El 83 % de las palabras está entre las 3.000 más comunes del inglés; conviene repasar el resto antes de empezar.

Vocabulario clave de este vídeo

15 palabras del vídeo que vale la pena aprender, con su pronunciación y significado:

Palabra	Pronunciación	Significado
cluster sustantivo	/ˈklʌstɚ/	aglomeración, agrupamiento
notebook sustantivo	/ˈnoʊtˌbʊk/	cuaderno, libreta
catalogue sustantivo	/ˈkæt.əˌlɔɡ/	catálogo
cloud sustantivo	/ˈklaʊ̯d/	nube, nimbo
connect verbo	/kəˈnɛkt/	conectar, empalmar
unity sustantivo	/ˈjunɪti/	unidad
pipeline sustantivo	/ˈpaɪpˌlaɪn/	oleoducto, tubería
bundle sustantivo	/ˈbʌn.dl̩/	haz, atado
parameter sustantivo	/pəˈɹæm.ə.tɚ/	parámetro
spark sustantivo	/ˈspɑɹk/	chispa, centella
query sustantivo	/ˈkwɪɹ.i/	consulta, pregunta
delta sustantivo	/ˈdɛltə/	delta
variable adjetivo	[ˈvæɹ.i.ə.bl̩]	variable
monitor sustantivo	/ˈmɑ.nɪ.tɚ/	monitor
input sustantivo	/ˈɪnpʊt/	aportación, aporte

Phrasal verbs que vas a escuchar

Palabra	Pronunciación	Significado
come up verbo		salir a relucir
check out verbo		investigar, chequear
go through verbo		atravesar
put up verbo		colgar
come across verbo		pasarse a
come out verbo	/ˌkʌm ˈaʊt/	revelarse, salir a la luz
come up with verbo		ingeniar, ingeniarse
get out of verbo		salirse de, ponerse fuera de

Gramática en este vídeo

Las estructuras que más usa el hablante, con las palabras exactas del vídeo:

Estructura	En el vídeo
Oraciones condicionales if + oración, will/would + verbo — una condición y su resultado	if we click that it will rerun · if we do step in it will take · if we press this it will take
Present perfect have/has + participio pasado — una acción pasada que sigue importando ahora	have completed · have tried · have also created
Voz pasiva be + participio pasado — importa lo que ocurre, no quién lo hace	are asked · can be created · are arranged
Oraciones de relativo who / which + oración — información extra sobre una persona o cosa	question which can · entities which are · differences which are

Pronunciación a tener en cuenta

El hablante usa 19 contracciones y formas reducidas, como don't, I'm, doesn't. Dilas en su forma corta, tal como las oyes.

Los sonidos de “sh” y “zh”: automation /ˌɔ.təˈmeɪ.ʃən/, optimization /ˌɑptəmaɪˈzeɪʃən/, execution /ˌɛk.sɪˈkjuː.ʃən/, notification /ˌnoʊtɪfɪˈkeɪʃn̩/, efficient /ɪˈfɪʃənt/
Palabras largas — cuida el acento: parameter /pəˈɹæm.ə.tɚ/, directory /dɪˈɹɛktəɹi/, automation /ˌɔ.təˈmeɪ.ʃən/, automobile /ˈɔː.tə.məˌbil/, optimization /ˌɑptəmaɪˈzeɪʃən/

Sonidos difíciles para hispanohablantes:

s + consonante al inicio — sin añadir una “e” delante: spark /ˈspɑɹk/, scope /skəʊp/, schema /ˈskiːmə/
/v/ — no es /b/: los dientes tocan el labio inferior: variable [ˈvæɹ.i.ə.bl̩], virtual /ˈvɜɹt͡ʃuəl/, visibility /ˌvɪz.əˈbɪl.ɪ.ti/, whichever /wɪt͡ʃˈɛvɚ/, provider /pɹəˈvaɪ.də(ɹ)/
/z/ — sonora, no /s/: optimization /ˌɑptəmaɪˈzeɪʃən/, visibility /ˌvɪz.əˈbɪl.ɪ.ti/, visible /ˈvɪzɪb(ə)l/, mechanism /ˈmɛk.ə.nɪ.zəm/, materialize /məˈtɪɹiəlaɪz/

Cómo practicar con este vídeo

Escucha el vídeo entero una vez sin hablar y anota las palabras que no conoces.
Empieza a velocidad 0,75×, haz shadowing frase por frase y vuelve a la velocidad normal cuando te resulte fácil.
Grábate y compara con el original, prestando atención a palabras como cluster, notebook, catalogue.

¿Qué es la Técnica de Shadowing?

Shadowing es una técnica de aprendizaje de idiomas respaldada por la ciencia, desarrollada originalmente para la formación de intérpretes profesionales y popularizada por el políglota Dr. Alexander Arguelles. El método es simple pero poderoso: escuchas audio en inglés nativo y lo repites en voz alta de inmediato, como una sombra que sigue al hablante con solo 1-2 segundos de retraso. A diferencia de la escucha pasiva o los ejercicios de gramática, el shadowing obliga a tu cerebro y músculos de la boca a procesar y reproducir simultáneamente patrones de habla reales. Las investigaciones muestran que mejora significativamente la precisión de la pronunciación, la entonación, el ritmo, el habla conectada, la comprensión auditiva y la fluidez al hablar, convirtiéndola en una de las metodologías más efectivas para la preparación del IELTS Speaking y la comunicación en inglés en el mundo real.

Técnica de shadowing: lee la guía completa paso a paso →

Shadowing Practice: Databricks Interview Questions by Sr Data Architect - Learn English Speaking with Video

Elige un modo de práctica

Descargar app

Databricks Interview Questions Part 1

The Essential UX Interview Questions (And How To Answer Them!)

Uber Data Engineering Mock Interview - Ride-Sharing Data Warehouse Schema

10 Data Analyst Interview Questions and Answers - Senior Data Analyst Explains

Excellent 2 Years AWS DevOps Engineer Mock Interview with Questions and Answers | Cloud | Docker

🔥 Java Multithreading Rapid Fire | Can You Answer These Interview Questions? #shorts

Distributed Task Scheduler: System Design Interview (Stripe & Amazon Offers)

Top 30 IICS interview questions and answers | Informatica cloud interview questions full video #iics

Anthropic's CCA Exam as a Field-Guide for Agentic Engineering — Frank Coyle, UC Berkeley

IT System Administrator Interview Questions and Answers | How to Pass an IT System Admin Interview

A/B Testing in Data Science Interviews by a Google Data Scientist | DataInterview

Learn how to SOLVE a data analytics case study problem

Vocabulario y notas de pronunciación para esta lección

Vocabulario clave de este vídeo

Phrasal verbs que vas a escuchar

Gramática en este vídeo

Pronunciación a tener en cuenta

Cómo practicar con este vídeo

¿Qué es la Técnica de Shadowing?