Shadowing Practice: Incremental Cooking in UE 5.7: A Dive Into the UE Cook Pipeline | Unreal Fest Chicago 2026 - Learn English Speaking with Video

Hi, my name is Matt Peters.

I'm on the Foundation Core Data Pipelines team.

I'm responsible for most of the code in uCook on the Fly server, which runs the Cook.

My team has been working on the incremental Cook performance feature for a while, and it's ready for use in 5.8.

I'm describing today what it is, how to enable it, and how to write code that complies with its requirements.

We'll start off with concepts and then how to enable it.

From there, move on to how to write your classes and work with it, and then what you should expect from the code now and in the future.

Therefore, starting with the context.

Unreal is a runtime game engine and a toolset.

Our most important two guiding principles are, it just works.

You don't have to enter a lot of settings and work around engine limits and content or game code.

The engine teams have solutions in the engine layer that solve almost all problems well.

And maximal tech, maximal art.

Best in the world graphics, audio, performance, and engine features.

These two principles conflict.

Having the best performance sometimes requires content-specific tweaks.

Having giant data sets makes analyzing those data sets too expensive.

Doing both at once is a goal with a score rather than a binary pass or fail.

The approach that we've taken over the past few years in the cook is a hybrid approach.

We make the default path for content constrained and as fast as possible.

But we allow class authors to deviate from those constraints with the caveat that custom code requires custom compliance.

Class authors have to write hooks and boilerplate to work with our constraints.

The example relevant to this talk is manual declaration of dependencies rather and auto detection, I'll discuss how to do that in this section on writing classes.

Cooking is part of the data build.

It is responsible for transforming content from an editor format into something consumable by the user.

In Unreal, this transformation is accomplished by what is in principle a simple and straightforward solution.

Load packages, run transform hooks, and save them.

The hooks apply transforms described by the authors of content types in Unreal.

Here on the right are the most common hooks for the transformation.

begin cache for cook platform data and is cached cook platform data loaded are used for the most expensive and flashy transforms, texture compression, mesh simplification, and having cached in DDC,

derived data cache for years.

But there are other transformations that are not cached, and besides transformations, there's object and byte manipulation work necessary to load and save a package.

Most of the cooker's work is spent doing those load, transform, save operations.

We're experimenting with changes to this basic model, but in 5.8, this is the way.

That's cooking, long time behavior, straightforward.

What is incremental cooking?

It is an optimization.

I put up the costs of the steps of a Lyra cook, it takes 145 seconds.

Loading and saving every package that should be staged to the runtime dominates the cook.

The remainder is relatively minor bookkeeping.

This is simple and straightforward, but also redundant and wasteful.

During project development, project teams don't just cook once, they iterate and cook repeatedly, and loading and saving every package every time is redundant.

Cutting that time out and just verifying that everything is up to date only takes four seconds.

So we should follow don't repeat yourself.

Don't redo load save of a package that you already loaded saved in a previous cook.

In a word, caching.

In three words, caching, AKA memoization.

Cache invalidification is one of the two hard problems in computer science.

It requires knowing the inputs.

If your transformation is a mesh simplification with some heuristics and constants, then those constants are part of the input.

And every asset is at least one dependency beyond the bytes of the source package, the C++ code that executes the transformation.

That code could in theory change at any point to say, I want to insert this extra byte here.

And in general, you have a giant soup of C++ that is reading data from anywhere, and you need to record all of those dependencies.

We tried caching in a form of incremental cooking before.

At that time, we called it iterative cooking.

Incremental is a rename that we use to distinguish the two algorithms.

And iterative cooking's general flaw was that it didn't capture all of the inputs.

It only captured package dependencies and config dependencies.

In this image on the right, I have illustrated its most common failure.

Detecting that the mesh U asset on disk changes is insufficient.

You also have to detect when the use static mesh serialized function changes and recook the mesh in that case as well.

Despite missing some inputs, iterative cook worked well for some licensees.

They didn't change code or other dependencies frequently

and it was a game changer to recook just one package when you're editing rather than recooking the entire game.

But for many other licensees, working in some but not all cases is as bad as working in no cases

and most game teams abandoned iterative cooking for the slow but sure full recook.

Incremental cooking, which we want to distinguish in that previous more naive version of iterative cooking, is a promise to capture all of the dependencies.

The C++ code most obviously, but also config values and asset registry queries and command line arguments and non-package files and others.

Some of these we can capture automatically.

U properties in C++ classes drive the auto-generated serialization and those are known to reflection.

We can hook into t-object pointer dependencies for package dependencies and into the config API for reads of config values.

But others we can't.

Custom serialization code, cached pointers and manager systems, reads of disk data outside the Unreal package system.

For those, we will have to capture the dependencies through maximum effort.

They have to be manually declared, and we will do so for all engine classes.

Part of that manual effort, however, does end up in licensee code.

This is what I meant earlier by custom code requires custom compliance.

But we hope these will be few and easy to declare.

Here's an example class that demonstrates a dependency we can't auto-collect due to custom C++ code.

The calls to GConfig here internally call our hooks and automatically report the config dependency.

And this works fine for the first call at the top of the function.

But the second call to GConfig is made once and the results stored in a static function variable.

Since the call to Gconfig is made just once, we record it as a dependency of the first instance, it gets saved, but then we don't see it and don't record it for any other instances,

meaning we don't recook their packages when it changes.

This class therefore requires manual effort.

We can declare the config value manually in the class's append to class schema.

We want to capture automatically wherever possible, but when it is not, this kind of manual effort is required.

So capturing dependencies is the basis of incremental cook.

First, a note about the two types of dependencies.

Most obviously there are runtime dependencies.

Runtime dependencies cause other assets to be pulled into the cook.

One part of the cook process I haven't mentioned yet is the graph search.

The cook loads and saves packages, but which ones?

The answer is the project settings in the asset manager tell it an initial list of assets, the levels, the characters, the global inventory items.

And all of those assets have dependencies of other packages that they rely on.

A material references textures, a level references quests and conversations that can occur in the level.

The cook does a graph search over the package vertices and the runtime dependencies are its edges.

This is independent of whether we are cooking incrementally or not.

The other kind of dependencies are build dependencies.

100

Build dependencies are dependencies that cause a package to change.

101

I and I settings that parameterize transformations, shader files that have to be compiled into the package.

102

An instance level package that gets embedded into the cooked version of a level that refers to that instance.

103

These are the dependencies that we have to capture to know when we need to recook a package.

104

Side note, there is one interesting detail about incremental building.

105

The build dependencies are metadata about the package.

106

They are the list of things that can cause a change and need to be recooked.

107

The interesting point is that those build dependencies can also cause themselves a change.

108

Does that cause a chicken and the egg problem

109

because we need the new build dependencies to know that the old build dependencies will change?

110

No, because the list of build dependencies cannot change unless one of the old build dependencies changes.

111

In my example here, for a texture synthesis asset that transforms its input texture based on some settings, it might have an I and I value specifying the input texture in the settings.

112

We capture those settings and that I and I value as the build dependencies.

113

The jungle synthesis settings can change and cause a recook of the texture.

114

But it's not possible

115

that we need to consider the new urban synthesis settings as a build dependency unless one of our old build dependencies, the I and I values, changes to point to it.

116

So we're safe.

117

We can rely on a change in the old list of

118

build dependencies to decide whether we need to recook to gather the new list of build dependencies.

119

This is a relatively internal implementation detail, but I thought it was an interesting one, and we do rely on it, so I wanted to mention it.

120

I've mentioned some examples of build dependencies.

121

What's the list we've encountered so far?

122

I'll show the current list later, but to sum it up, any build dependency from any source can be described in a special function that a class author writes.

123

That's our catch-all.

124

Additionally, we have a short list of the types we've encountered used by engine classes.

125

The source package itself, the C++ class schemas that we captured from an Unreal build tool, the C++ serialization code, which has to be manually declared,

126

INI values, and the bytes of other packages.

127

There are a few extra types I'll mention later, but this is a good set to understand the gist of what we are collecting.

128

Another breakdown of dependencies is time of runtime dependencies.

129

Runtime dependencies are independent of incremental cooking but are important for an understanding of cooking in general.

130

There are multiple types of runtime dependencies in Unreal.

131

Hard dependencies are required for the package to function and are automatically loaded.

132

Soft dependencies are required to be available on disk but can be loaded later in response to player action.

133

Both of those are declared to the cook true soft dependencies are not required to be present in the staged game.

134

Something else can add them to the game if it wants them, and those are hidden from the cooker.

135

Besides that hard versus soft access, there is one more, editor only versus used in game.

136

Editor only references include preview content and build dependency only content.

137

They are marked by Unreal Build Tool or native serialization and are not included in the cook.

138

That's it for the type of dependencies, now back to how we gather build dependencies.

139

So with incremental we promise to gather all dependencies either through automation or through maximum effort.

140

What so far can we gather automatically?

141

The easiest to gather is class schemas.

142

We have reflection data of classes U properties from Unreal Build Tool.

143

We can hash their names and types.

144

That's an automatic solve for the previous pervasive problem for iterative cook.

145

Changes to class U properties now automatically calls re cooks of the packages using them.

146

Other easy dependencies are those read through a relatively small API that we can hook into and record.

147

Config files, console variables, f command line get.

148

One caveat with those API recordings, caching is a problem as I mentioned before.

149

Storing the results of a command line option in a function static bool is a common small optimization

150

and doing that prevents us from noticing the read of that dependency on the next instance of the class that gets saved.

151

We have a plan to detect those.

152

I'll mention that in future work.

153

A type of dependency that wasn't originally behind an API was the use of data from U objects and other packages.

154

We knew that would be an issue since some of those are sneakily loaded by string and undeclared.

155

So we created the T object pointer wrapper for u object star.

156

We now can hook into the dereference of T object pointer to detect reads of other packages during the cook.

157

Other dependencies are difficult or nigh impossible to capture automatically.

158

A change to C++ serialization code is the primary example.

159

C++ doesn't provide reflection, so how can we capture that?

160

We have some ideas for heuristics, but for now we rely on manual recording of those.

161

Fix it or fail is not the only option for classes with hidden or undeclared dependencies.

162

We have an opt out system that we call hybrid incremental.

163

If a package opts out of incremental, then it will always re cook, even if its dependency evaluation determines that it is unchanged.

164

This is done on a native class by native class basis.

165

Packages contain multiple U object instances.

166

For example, a material package has a U material along with U material expression and U material function inside it.

167

The serialization of any of those classes could include hidden dependencies that impacts the bytes of the package containing them.

168

So on a class by class basis, you can specify which classes are known to have hidden dependencies, or at least are not known to not have hidden dependencies, and opt those out.

169

And this applies for each U object class, not just each asset class.

170

A material package could be opted out if it uses an opted out U material expression, even though the only asset in the package is the top level U material.

171

Opting out is a default for all non-Engine classes, but most projects will need to change that since many of their packages contain project-specific classes.

172

Instructions on how to do that later.

173

Here are the two classes in Engine that are currently marked as not skippable.

174

Both are relatively new classes that we are still working on making compliant.

175

To sum up how we improved the previous legacy iterative attempt at incremental cooking.

176

Previously, we had dependencies from packages as calculated during editor save and recorded in the asset registry.

177

And a recording of every config setting that was read during cooking.

178

Besides those, no other dependencies were tracked.

179

Most importantly, class schemas were not tracked.

180

Improving that model required slow and broad changes.

181

T object pointer, gathering reflection data from property serialization, recording the list of class instances per package in the asset registry,

182

creating the Zen server op log to store the dependency data and the increased sophistication of the cookers search and load code,

183

including skip only editor only and the ability to make other decisions like it.

184

Put that all together and we have the plans for a fully robust system.

185

We still have some edges to smooth however, and it will require maintenance forever.

186

No more maintenance than any other core feature, hopefully, but it is another one on the pile.

187

That's it for background, let's talk about how to turn it on.

188

First, you need to set up some prerequisites.

189

These are on by default in 5.8, but if you've turned them off during experimentation and earlier releases, you need to re-enable them.

190

Derived data cache, caches the most expensive transforms, which for most projects are dominated by shader compilation, texture compression, and mesh transforms.

191

A cold, full, or recook is usually several times slower than a warm full re-cook, where cold and warm refer to the population of DDC.

192

Incremental cook caches more than DDC does, but they work together and DDC is required for good performance even when cooking incrementally.

193

DDC has been around since before UE5 and their instructions for managing it online.

194

It should be easy to set up and of course it is on by default.

195

A newer piece of our tools, but that has still been present for several releases is Zen server as the storage for DDC.

196

Before Zinserver, the binary bulbs stored in DDC were stored as loose files on disk.

197

Zinserver instead aggregates them into its internal database and reduces the per file IO cost.

198

That is doubly important when the DDC storage is on a network drive rather than the local disk.

199

The settings for DDC that configure it to be stored in Zinserver are stored in engine.ini.

200

Zinserver is required for incremental cook, but configuring it this way as a storage for DDC is optional.

201

optional but highly recommended and the default.

202

Build machines are a special case and there's one point specific to them.

203

They can share a common close by network DDC server rather than having a local one

204

that copies up to a shared DDC server.

205

The instructions for setting that up are described at these pages.

206

This helps with sharing of cache data between machines that trade off the responsibility for continuous integration builds.

207

In addition to storing DDC, Zen server can also store the output of a cook, the U-asset files.

208

This system was in beta in 5.7 and is now the default in 5.8.

209

It is required for incremental cook.

210

There might be some issues integrating it into your workflow.

211

Notably, the intermediate cooked files are no longer available on disk, and you may have some tools that expected them to be there there.

212

You can disable it for an immediate fix when integrating by overriding this INI value, but we recommend that you switch back to it as soon as possible.

213

You can get the files back on disk if necessary by running the Xen export command,

214

or you can change your tools to talk to the Xen server directly without needing the files present on disk.

215

Switching to XenStore is highly desirable not just because it enables incremental cook, but also because it reduces I.O costs and because our future tools will be relying on

216

If you have any problems using it, let us know on Epic Pro support.

217

Incremental Cook and ZenStore also require staging your files as IOSTore.

218

This has been the default since 5.0, and we are relying on it for all our future tools.

219

In case it is relevant for Buildfarm workspaces or any other reason, the disk location of the ZenStore data for the Cook is next to the DDC data,

220

and that's determined by the environment variable UE local data cache path.

221

The format of the data in that location is a private implementation detail.

222

We won't provide deprecation when it changes, and you should access it through the Zen servers API.

223

And some notes about why incremental cook relies on Zen store.

224

The primary reason to use it is reduction of IO costs.

225

We could have implemented incremental cook to store its extra data in loose files,

226

but we decided not to make that fallback path and spend the time instead on advancing cook features for our expected workflow.

227

Cook package data and the incremental cooks metadata is stored in a container called an op log, specific to the project, the project's workspace on disk, and the target platform.

228

Op log is short for operations log and is a common technique in database design to efficiently avoid data races.

229

Operations are given a sequence order and the current state is defined by the in order traversal of those operations.

230

From the perspective of the cook, this implementation detail is hidden and the op log is used as a key value store, store where the key is package name and the value is the cooked package plus its metadata.

231

The most important metadata for incremental cook is the list of dependencies.

232

We will add other data in the future and there's an interface for writing your own metadata as well through iCook artifact.

233

The API for all this reads and writes data as compact binary

234

which is similar to JSON but is binary encoded instead of text encoded.

235

Our preferred tool for viewing the data in XenServer is the HTTP interface that serves up.

236

You can click through from the dashboard to your op log

237

and see the list of packages

238

and click on each one to materialize the files on the disk or view the compact binary as JSON.

239

That's it for prerequisites.

240

Proceeding to the config values specific to incremental cook.

241

Incremental cook relies on the collection of dependencies.

242

This is enabled by default in 5.8 and the legacy iterative system that skips it is toggled off.

243

There's a small cost to this collection.

244

It costs a few percentage points of CPU time and increased storage usage.

245

The percentage value is about the same on small projects and big projects.

246

Collection of the dependencies is on by default, but cooking incrementally using those dependencies is not.

247

By default, all packages are recooked every cook as they were in 5.7 and earlier.

248

The recommended way to cook incrementally is to pass in the command line argument dash cook incremental.

249

This will make that single invocation of the cooker cook incrementally.

250

There is an I and I setting to manage the same behavior toggle as the command line, but we are recommending not turning that on for 5.8.

251

Wait until we have finished bulletproofing every edge case before turning it on automatically for all cooks.

252

For local cooks to the editor, rather than typing in the additional command line option, there are some settings you need to click.

253

These can be left on most of the time for local work, but you will need to uncheck them manually if you notice any suspicious behavior that might be related to incremental cook.

254

Suggest that workflow to your content creators with caution.

255

You do not want to train them to always try a full recook.

256

There is standard guidance that we expect to be given for a long

257

while yet until incremental cook has proven itself conservative for most projects over a long period.

258

Do not cook release builds with incremental cook.

259

Even a 0.1% chance of stale data being sent out to end users is too high.

260

Incremental Cook should be used for local Cooks, CIS, and QA builds.

261

Use a full Cook when making your release candidate.

262

I mentioned before that by default all of the C++ classes in your project opt out of being incrementally skippable.

263

This is to prevent us from making a bad assumption

264

that from day one all of your project's types have fully declared their dependencies.

265

Once you've added or validated those, here's how you opt your types back in.

266

Incremental class script package allow list and editor.ini, it uses a special token project root.

267

The Lyra sample project already has an example of doing that.

268

If some of your types are compliant and others are not, add the incremental class script package allow list setting and then add a deny list setting for each non-compliant type.

269

Opting out classes is usually only needed for a short time because it is relatively easy to add the hidden dependencies, but sometimes it can be complicated and need to stay around.

270

UWorld and UBlueprint, for example, took us a while to implement.

271

When you've done a cook that you wanted to be incremental or when you were unsure whether it was, how can you tell whether it is on?

272

There are two log statements that indicate incremental cooking.

273

The first is at the beginning of the cook.

274

either log cook full cook or log cook incremental cook is logged.

275

The second is at the end of the cook and reports a number of packages cooked, either new or recooked, and a number of packages skipped.

276

Other than logging by design, the output of the cook is supposed to be indistinguishable between recooks and incrementally skipped cooks.

277

One of the desired features of incremental cooking is having a farm do most of the cook cook

278

and having developers sync that down and incrementally cook on top of it.

279

With ZenStore and incremental cook, this is done by exporting a snapshot from the Zen server

280

that cooks the initial build and importing the snapshot into the Zen server that is cooking incrementally.

281

The farm machine cooks and after it is done your build script calls op log export.

282

ZenStore cooks still include some files written to disk rather than the op log.

283

We expect to reduce these in the but always support them.

284

To transfer cook results between machines, you need to include those in the Zen server snapshot.

285

Zen op log export does the export with an optional arguments to embed those loose files before doing so.

286

And the arguments of the export command allow you to direct the results to a file, cloud-based storage, or another Zen server.

287

On the developer's local machine, you use the corresponding Zen op log import command pointed to the same storage that you exported to,

288

and your state becomes the same as if you had previously cooked locally.

289

You can now cook incrementally and incrementally skip any package with no dependency changes from the farms environment.

290

That's it for how to turn on incremental.

291

It is intended to satisfy the principle of it just works

292

with the one dial being whether you pass dash cook incremental or not to the cook commandlet.

293

But custom code requires custom constraints.

294

And so we need to talk about the API you can use to make your C++ types compliant.

295

If your types don't override, serialize, postload, pre-save or other functions used are in cook or don't add hidden dependencies, then you won't need to use this API.

296

But scrutinize your types carefully before you draw that conclusion.

297

Sometimes uClass or uStruct code can be called from the cook entry points on other classes.

298

First, a project management caveat.

299

If your data build has false skips, bugs, developers can fix those, fix a bad build, by doing a full re-cook.

300

If that becomes a common problem-solving technique because it happens frequently,

301

then it will train yourself or your team to always try it whenever there is a weird error in the cooked results.

302

And that training means that you pay the price for a full cook, full recook at times even long past the point of the missing dependency being fixed.

303

Avoid getting into that situation.

304

Unlearning those habits can take a long time.

305

Air strongly on the side of caution before opting your types in to incremental cooks as part of your developer's local workflow.

306

In your native class there are two primary entry points for declaring dependencies.

307

If your dependencies are global shared by every instance of your class, then you can hash the value of your dependencies and the static append to class schema function.

308

Like changing the classes you properties, any change to the hash created by this function causes a re cook of every instance of the class.

309

This is where you should add a version and bump it to manually record changes to the C++ serialization code.

310

For dependencies that only occur on some of your instances or are parameterized by data on the instance,

311

you implement the virtual onCook event function and handle platform Cook dependencies event.

312

This event and that function is called after a successful package saved during Cook while collecting all of the automated dependencies.

313

It is more expensive than Append to Class Schema because it runs for every instance and stores some data for every instance, but is still cheap compared to loading and saving the package.

314

Dependencies declared in onCookEvent are constructed through static functions on the FCookDependency class.

315

These are the types that class has available in 5.8.

316

File for non-package files on disk, package for dependencies on other packages, console variable in config, and native class in AssetRegistryQuery.

317

The AssetRegistryQuery compares the list of package names returned by a query.

318

It does not automatically add those packages as package dependencies.

319

And finally, for everything else, a function type.

320

You write a function, define its arguments, and report the function name and arguments to the cooker for calling again later in future cooks.

321

That API for the function dependency touches on an important part of the algorithm that we need to make clear.

322

When you record a cook dependency, you're not just recording a hash value, you're recording an explanation for how to calculate the hash value.

323

You provide the explanation by passing through the function name and the parameters that should be passed to it.

324

For a file dependency, you would provide the hash file function and the file name, and the cooker then knows to call the hash file function to read that file from disk and hash its content.

325

At the beginning of the next cook, when deciding whether to skip the package, we will load those arguments, find the function name from the op log,

326

run the function and compare the function's new value

327

or current value to the stored value that we also load from the op log.

328

If those are equal, skip.

329

If there's a different re-cook.

330

These dependency functions are run every cook for every instance.

331

You should make them use as little data as possible and be as fast as possible.

332

Append the class schema on the other hand is similar in principle to DDC keys.

333

It is called once per process and does not need to be parameterized.

334

Gather all the data you want in the order you want it, pass it to the hasher and it is thrown into the class's schema.

335

That schema is included in the overall hash for any package containing instances of the class.

336

And if it changes, we re-cook.

337

Native class schema hashes are collected automatically for any class instances in the package.

338

You can also record them manually if necessary.

339

In on cook event, you can create a native class dependency and record the class name parameter.

340

And at evaluation time, we read that parameter and look up the class's current schema hash for comparison to the stored version.

341

Here's an example of append to class schema, the latest version used by UStaticMesh and UE5 main.

342

This version has some changes beyond the version in 5.8.

343

Those changes were added because the render

344

and Nanite teams found they were already bumping some DDC versions every time they changed C++ code

345

and they knew they needed to invalidate the incremental cook as well when those changed.

346

Adding GUIDs and versions used in DDC keys is a commonly useful technique that we recommend for append to class schema.

347

These two APIs together are extensive of the commonly encountered requirements for managing incremental cook code.

348

There's one more point to consider that is relevant to some projects, cook artifacts.

349

Cook artifacts manage runtime data that is loaded outside of Unreal's linker load system, outside of packages or bulk data.

350

Examples on engine code include shader libraries and asset registry.

351

The API for that is iCookArtifact.

352

Artifacts need to support incremental cook as well because they are commonly implemented by collecting data from packages that load and save,

353

and in incremental cook, packages can be present in staging without being loaded and saved during the current cook.

354

Therefore, iCook artifact has some functions that interact with incremental cook.

355

In general, iCook artifacts manage files of your own formats

356

that are saved into the cook output directory or the ZenStore op log.

357

iCook artifact has functions that incrementally load and invalidate those files.

358

There are examples of using this API in the artifacts for the shader library validation and the cookers global data.

359

That's everything for the new code you need to write a lot to take in.

360

But I think when viewed in the development environment with some engine classes to use as examples, it will be straightforward.

361

And to the extent that it does not turn out to be the case, let us know on EPS and we'll continue documenting and improving it.

362

Now let's talk about the diagnostic diagnostic tools we have so far.

363

The primary expected problem with incremental cook is what I mentioned before, stale data at runtime.

364

This is caused by an event we call a false incremental skip.

365

A package should have been recooked because if we were to recook it, it would be changed, but we didn't know that and skipped its cook and kept the old version.

366

The primary tool we have for detecting these is incremental validate.

367

This is a cooker mode that is intended to run repeatedly in a persistent workspace that keeps syncing new changes.

368

You can invoke it after syncing by passing dash incremental validate to the cook commandlet

369

or by running the incremental validate build graph task.

370

It works by calculating whether it should skip each package as normal for incremental cook,

371

but then saving the skippable packages anyway and comparing the new version against the old version that it would have kept.

372

It is built on top of the diff-only technology we've had for the past 10 years for investigating indeterminism

373

and provides the same feedback about what is different in a new package, C++ call stacks, and new property names along with other data.

374

Use that information to find the code that has hidden dependencies.

375

A common failure that incremental validate reports is that C++ serialization changed without a bump to append to class schema.

376

By default, incremental validate does not modify the workspace, but you can change that and let it write new versions after diagnostics with the dash incremental validate allow write parameter.

377

Unfortunately, the determinism problems

378

that cause the creation of the diff only tools still exist in various places today

379

and we continue fixing them as we find them.

380

And they cause a problem for incremental validate.

381

The algorithm gets confused as a change to the package that it didn't predict.

382

It's a good idea to fix indeterminism issues along with false incremental skips.

383

You should use incremental validate job to diagnose both of those.

384

Here's some sample output from incremental validate.

385

This is a complicated case.

386

An export changed the U objects and F names it serialized and that creates several knock-on diffs in the packages header.

387

This one was caused by indeterminism in a field that lists optional imports.

388

The field was not needed at runtime and removing it fixed the diff.

389

This one was more typical.

390

It shows that a single value in an export has changed.

391

This one was again indeterminism, a single integer in a U object that was calculated indeterministically.

392

We fixed it by updating the calculation of the value to be deterministic.

393

The next tool, incremental compare, gives less diagnostic information, but is simpler and gives one extra piece of information that incremental validate does not.

394

It runs Cookr twice back to back, first incrementally and then a full re-cook and reports adds, modifies, and deletes.

395

Incremental validate does not report the adds and deletes, you need incremental compare for those.

396

Some bugs can cause missing or added runtime dependencies in the incremental cook, and incremental compare reports when those results in a different set of staged packages.

397

Incremental compare runs outside the cook, so it is only available as a build graph script.

398

After the two cooks, it runs the diff cook commandlet to compare the two sets.

399

Those are the tools we have for diagnosing false skips, but what about the opposite problem?

400

A false recook is when incremental cook decides it needs to recook a package, but the package turns out to be identical.

401

False recooks are less problematic than false skips because they are only a performance call so rather than the incorrect runtime behavior, but fixing them is important for optimizing your cook.

402

Recall the end of cook log message I mentioned that reports how many packages were incrementally skipped.

403

If the skip number is lower than you expect for a small set of changes, then you should investigate whether false recooks occurred.

404

Run the incremental cook with the command line dash cook.diagnostic.modified.

405

It will write out a file with the end of cook with an explanation for why each recook package was recooked.

406

For each one, it prints out the top level reason for the recook, either the target domain key changed or a more edge case reason, such as not previously cooked.

407

And if the target domain key changed, it prints out which dependencies caused the change.

408

In this case, the BP SkySphere package recooked because U-static-mesh's class schema changed and the package contains U-static-mesh.

409

You should investigate the code making a dependency if it seems spurious.

410

For now, this metric doesn't report whether the package ended up different, it just reports why it recooked.

411

Another tool for understanding the dependencies of packages, dependencies for a package in the metadata in the op log.

412

The op log is viewable in the HTTP dashboard provided by Zinserver.

413

This is the recommended mechanism for viewing package data and metadata stored in Zinsstore cook results.

414

Navigate to the dashboard at the Zinserver's hostname and port, by default localhost 8558.

415

The projects available in Workspace is reported to Zinserver listed there.

416

Click on the project in the workspace you want and then click the target platform link for the cook you want.

417

And then this displays a list of cook packages.

418

Search and click on the package you want.

419

This, that page is where you could download the package data.

420

It also has the cook artifacts that contain the dependencies, which is what's on the screen to the right.

421

Here we see the bweaponfire package has various dependency types, function, package, config, native class, and redirection target.

422

It has further data for build dependency sets, a performance feature we are still developing.

423

This view will remain the recommended view for the dependency features we continue to add.

424

Viewing those dependencies can give you a sense of how broadly some of your packages gather build dependencies compared to others.

425

False recooks are one drain on performance of incremental cook, but there is another, the overhead that occurs every cook.

426

We call this the null recook time because it is the time observed when no changes have occurred.

427

It includes a prologue, dependency evaluation, and an epilogue.

428

The prologue is engine and cook commandlet startup time.

429

The epilogue includes updating cook artifacts and calculating chunk assignments.

430

And in the body of the cook, we have to evaluate all of the incremental dependencies, which takes seconds to minutes when the package counts are in the thousands to millions.

431

We are continuing to profile and optimize the null recook, both to cache more of its calculations and to reduce the time taken for the operations that can't be cached.

432

You may want to do the same to look at engine issues

433

that are disproportionately large on your project

434

or to look at some of your own functions that you have in the prologue and epilogue and optimize those.

435

Here's my favorite profiler view of the Lyra nullcook.

436

It's from Superluminal, a third-party profiler which uses event tracing for Windows.

437

It's the standard child display in the butterfly view that is a feature of many profilers.

438

In this case, we can see 22 seconds for the body of the cook commandlet, and of that, nine seconds are in startup packages, five in global shaders, two in asset registry,

439

and 2.7 in dependency evaluation.

440

Dependency evaluation is the lowest hanging fruit here, but reducing unnecessary startup packages will get the biggest long-term gains.

441

Unreal Insights is another profiler we use, and it sometimes provides information that the third-party profilers do not.

442

It relies on manually coded events, but we have thousands of those now, and it gives a pretty complete picture.

443

In the case of this Lyra Nolcook, it's giving roughly the same information as third-party profiler, but you should check it out as a first pass for investigations that often directs you immediately to the hotspots.

444

That's it for the diagnostic tools.

445

Going back to the topic of robustness testing, what kind of errors can you expect?

446

The most difficult to diagnose is a false skip after an undetected change to native serialization.

447

The symptom is a crash in package serialization with the victim often different than the cause.

448

When an incrementally cooked build crashes in serialization, start by checking out change history and source control to see

449

if any class has recently changed native serialization without a version bump.

450

Stale warning replays, on the other hand, are the easiest to diagnose.

451

A warning existed, there was a fix for it, but the warning is still being printed when cooking and the warning is prefixed with incremental replay.

452

Incremental replay means that we are skipping the recook of the package, but when it last cooked, it logged warnings or errors, and we recorded them for replay every time the package is skipped.

453

If the problem should be fixed, but the incremental replay remains, then you have a missing dependency that you need to add to detect the presence of the fix.

454

The other class of difficult problems are the ones that you overlook.

455

A mesh was modified, but it still displays the old value.

456

Or behavior of an enemy was supposed to be changed, but it is not.

457

These could be caused by, for example, an an undetected change to a config value.

458

To find these, we recommend strongly relying on incremental validate to give you the rigorous bytes are different information.

459

And to repeat the guidance from earlier, do not cook release builds with incremental cook.

460

Don't allow the possibility of overlooked stale content going out to end users.

461

Do full cooks for release candidates.

462

Some discussion of the results we've seen on internally on our test projects and production projects.

463

Lyra, designed to be the smallest project that demonstrates everything, cooks around 4000 packages and even a full cook is quick at 2.7 minutes.

464

The meaning of quick is context sensitive, however, what is quick for CIS is not quick for local iteration.

465

A null cook is 28 seconds, a five times reduction, a better workflow for local iteration.

466

City sample has eight times as many packages and five times the full cook time at 14 minutes.

467

A null cook is only 2.5 times the null cook time of Lyra at 68 seconds, so better relative improvement.

468

And Fortnite, 1.5 million packages, 10.5 hours of single process cook time.

469

Fortnite is big enough to benefit from multi-process cook.

470

I've shown the timings for one, four, and eight cook workers.

471

The eight cook worker time is 1.5 hours, a dramatic improvement over single process, but still far too high for frequent feeders, back on the build farm.

472

A no cook that brings that time brings that time down to 30 minutes, ironically a few minutes longer in the multi-process case because of the time to spin up cook workers.

473

Note that the no cook is not an empty cook in Fortnight's case because some packages, the classes I mentioned earlier, are not incrementally skippable for now.

474

So part of that 30 minutes is a recook of 2500 packages.

475

I mentioned so far the full cook and no cook times, but those don't cover the case of invalidations whether genuine from churn or spurious.

476

For your effective performance under production churn the typical cook how

477

much of the cook surface you are commonly causing to recook from the changes going in is what's important.

478

On fortnight that typical cook is 611,000 packages about one-third of the full cook number of packages

479

and that cuts our one-hour savings down to only 35 minutes savings.

480

On your projects hopefully the recook count will not be

481

so high because you will have fewer engine changes and many of your asset types will rarely need to be recooked.

482

For us on Fortnite, this invalidation rate is a high priority to optimize.

483

It's still second in priority after the null cook, however.

484

We think that the null cook is more important for making feasible some improvements to local workflows.

485

For robustness results internally, we've been using incremental cook on large test projects with production churn

486

and scrutinizing it for errors that are caused by incremental cook.

487

We also run incremental validate on that churn.

488

In 5.8 incremental validate reports 1600 of our test packages that have false skip errors out of millions of packages, but most of those are due to indeterminism.

489

The errors that are confirmed to be caused by hidden dependencies, which we know because they go away when we force a recook, occur rarely around one in a thousand submissions.

490

Again, we expect licensee results to be at a higher robustness level

491

because of a reduced number of changes to the code that drives your engine only issue assets.

492

That's the summation of everything you should expect from incremental cook and 5.8.

493

We're still working on it for multiple reasons.

494

We have some known improvement ideas that I'll talk about next.

495

Beyond that though, we expect to continue maintaining and improving the system.

496

Here are the ideas that are immediately on our minds.

497

For robustness, patching some of the holes I mentioned.

498

The biggest source of undeclared dependencies is C++ serialization changes.

499

Is there a way we can detect those?

500

A simple idea for a heuristic is to add markers, macros or otherwise, and C++ files and functions that indicate when the source code of the function changes,

501

there should be an automatic bump to the class schema for classes using the function.

502

And that can be enforced by Unreal Build Tool.

503

For example, the virtual uObject serialized function on a class, maybe we can even automatically add those serialized functions to the class schema hash.

504

Command line tracking is a current whole in dependency tracking.

505

Command line flags are parsed for many classes, and some of those change cook behavior.

506

We have a plan designed for a new API to replace the manual calls the fparse param on f command line git.

507

Using that API will automatically add dependency on the parsed token to the package that is active when it is called.

508

And for the static function variables that I gave you an example of earlier, we haven't thought of a way to automatically handle those, but we have thought of a diagnostic we can use to detect them,

509

and then manual review of the detection reports can lead to adding the required manual dependencies.

510

The idea is to cook twice in the same process and report any differences in dependencies recorded between the two cooks.

511

For performance, our immediate next plans are around improving no cook time.

512

We could evaluate dependencies in parallel.

513

They're currently evaluated in serial.

514

That's two seconds in a Lyra cook, but multiple minutes in larger cooks.

515

We also plan on profiling and optimization of the single threaded calls for system-specific dependency calculations, new materials and new blueprints are the highest cost currently.

516

And we plan to change the engine's iCook artifacts,

517

asset registry and shader libraries to do more of their finalization incrementally and avoid redundant calculations every cook.

518

Unrelated to CPU time, but still an efficiency feature of the cook, we plan to fix a case of unbounded disk space usage in the incremental cook op log.

519

As changes are made to the project over time, packages are marked deleted or no longer referenced.

520

Currently, those deleted assets remain in the op log, taking up disk space if nothing else.

521

This is not yet a significant issue because full Oplog clears still happen from time to time and clear those out.

522

But as full Oplog clears become rarer, pruning these unused assets will be more important.

523

We plan to add a pruning step to ZenServer's Oplog based on timeouts.

524

And lastly, improvements to the tools.

525

Incremental validate and incremental compare give good feedback, but they rely on being run ahead of time or on the problem being reproducible.

526

What about examining a staged incremental cook for false skips after it's already been made.

527

The opportunities are somewhat limited because the intermediate data is not saved, but there may be some good diagnostics we can find anyway, and maybe there's debugging data we can save to enable even more.

528

More concretely, I mentioned that incremental compare provides less diagnostic data than incremental validate, which it does because it examines only the on-disk output of the cook,

529

rather than reading data from memory during the cook.

530

We want to try improving it by getting call stacks for the bytes that differ based on reading the packages

531

into a buffer, and the same way

532

that incremental validate gets those call stacks by monitoring the differences when the packages are written out to a buffer.

533

And some improvements planned to the C++ API.

534

The current function dependency requires writing custom martialization code for the arguments that you want to record for your function.

535

We want to replace that with the use struct containing your arguments, which then gets automatically marshaled using Unreal Build Tools Reflection.

536

All of that is currently on our backlog, but we're also interested in hearing from you about how incremental cook works for your cases

537

and what we should prioritize for improvement.

538

Please give incremental cook a try on your project and let us know how it goes.

539

That's all for my presentation.

540

Thanks for listening.

What is the Shadowing Technique?

Shadowing is a science-backed language learning technique originally developed for professional interpreter training and popularized by polyglot Dr. Alexander Arguelles. The method is simple but powerful: you listen to native English audio and immediately repeat it out loud — like a shadow following the speaker with just a 1–2 second delay. Unlike passive listening or grammar drills, shadowing forces your brain and mouth muscles to simultaneously process and reproduce real speech patterns. Research shows it significantly improves pronunciation accuracy, intonation, rhythm, connected speech, listening comprehension, and speaking fluency — making it one of the most effective methods for IELTS Speaking preparation and real-world English communication.

Shadowing Practice: Incremental Cooking in UE 5.7: A Dive Into the UE Cook Pipeline | Unreal Fest Chicago 2026 - Learn English Speaking with Video

Understanding the Context: A Tech Talk on Game Development

Top 5 Phrases for Daily & Tech Communication

Step-by-Step Shadowing Guide for This Video

What is the Shadowing Technique?