Kruithne
patreon
WoW 3D: Importing Continents
🕑 Added 2021-06-01 18:36:33 +0000 UTCHave you ever wanted to take an entire continent from World of Warcraft and stick it into some 3D software? You're in luck, as this in-depth guide is going to focus on just that.
As with most of my 3D guides, I'm going to be using Blender since it's free and widely accessible, however a lot of the concepts will be applicable to other applications and workflows. Additionally, you'll also need the latest version of my wow.export toolkit as well as the Blender add-on that comes with it installed.
Chapter 1: Exporting
The first thing we need to do is export the data for the continent we'll be using. Open up wow.export and switch to the Maps tab. This is going to show you a list of all maps available in your game client.
Start by selecting the map of your choice from the list. For the purposes of this guide, I'm going to be using Eastern Kingdoms (Azeroth).
A map preview will appear on the right side allowing you to select individual tiles. Since we want to export the entire continent, hover your cursor over the preview and press CTRL + A, this will select every tile.
Before we export, let's evaluate the exporting options and configure them as necessary.
Export WMO - This exports World Model Objects for each tile. These are generally large objects, such as buildings or cities, that can be seen from a distance.
In a later chapter, we're going to look at using some of these to improve our map fidelity, so enable this if you're interested in that. If you're just after the terrain topography, disable this for a much faster export.
Example: goldshireinn.wmo (left), goldshireblacksmith.wmo (right)
Export WMO Sets - This option includes doodads (M2) models inside WMOs. This is primarily used for interiors, which we're not focused on here, so disable this.
Export M2 - This exports doodads for each tile, which are smaller objects such as chairs, bushes, fences and so on. Given that we're setting up an entire continent, we definitely do not want this, so disable this.
Example: elwynnwoodfence01.m2, barrel01.m2, lamppost.m2, elwynntreemid01.m2
Export Foliage - This option exports foliage, which is all the little details covering terrain primarily consisting of grass. For our purposes, we don't want this, so disable it.
Example: elwgra01.m2 - elwgra08.m2, elwflo01.m2 - elwflo03.m2
Export G-Objects - This exports dynamic game objects that players can interact with such as benches and signs. Just like M2 models, we're not interested in this, so disable it.
Example: postboxhuman.m2, humansignpostpointer03.m2
Include Holes - This option includes holes in terrain which are used where large models intersect the terrain and allow players to move underground, such as caves. Since our focus is not on what the world looks like up-close, we want to disable this.
If we skip WMOs completely, this prevents our terrain being filled with holes, and even if we do end up including some WMOs, we don't care about intersections from a distance.
Textures - This option controls if textures should be exported for models. Since we may be making use of some WMOs, make sure this is enabled.
Texture Alpha - This option controls if textures (for models) include the alpha channel. This doesn't really matter, but we'll leave it enabled.
Terrain Texture Quality - This is the most important option for our export. We're going to be exporting an entire continent, which is a lot of terrain to cover. Using the lowest pre-bake setting of 1k would result in around 2GB of terrain textures for Eastern Kingdoms alone.
While this isn't the end of the world for some people, we're instead going to use the Minimap (512) option, which uses the in-game minimap tiles as a texture. Not only does this cut our texture budget down to roughly 300MB~, but the minimap tiles are satellite renders of the game world itself, therefore include all of the map details, which is both a gain of fidelity and a massive time saver for us later.
Once everything is configured, click the Export X Tiles button and then sit back and wait.
The export is going to take a fair while to load and export everything necessary, but here are some tips on how you can speed it up:
Local Installation - If you have access to a local installation of World of Warcraft, using that to export a continent will be multitudes faster than using the CDN, since the latter needs to download everything from a Blizzard server.
SSD - If possible, make sure you're exporting to an SSD (Solid State Drive) rather than a HDD (Hard Disk Drive). This will noticeable improve speeds, especially for large exports like this.
Disable File Overwrite - By default, the Always Overwrite Files option is enabled in wow.export. Disabling this can help tremendously on large exports, however I recommend doing this on a fresh (empty) export directory to prevent conflicts between export settings.
Use Shared Textures - By default, the Enabled Shared Textures option should be enabled in wow.export. If you've disabled it, I'd consider enabling it for this export as it will not only speed up the export, but it will also preserve a lot of disk space.
Chapter 2: Importing (Manual)
Now that everything we need is exported, let's work on importing it. To do this, use the wow.export Blender add-on (File → Import → WoW M2/WMO/ADT (.obj)) to import one of the tiles.
Which tile you import doesn't matter right now, just pick one at random. Be sure to disable the importing of WMOs, WMO Sets, M2s and GOBJs, regardless of what you exported.
Once imported, we should now have a single terrain tile with our in-game minimap tile mapped over the top of it.
Despite the low-resolution, even from a moderate distance the minimap tile holds up really well as a texture for our terrain.
The first thing we're going to look at changing is the UV extension. UV extension controls how (or rather what) is rendered for an image when a UV expands outside of the images boundaries. The three main extension methods are repeat, clip and extend.
By default, Blender uses the repeat extension mode, which seems fine when we have just one tile, but when we place multiple tiles side by side, you can see what's referred to as UV bleeding.
To understand how UV bleeding works, let's take a look at how the three UV extension modes look in practice.
Repeat is great for tiling textures, but here it causes the textures to bleed onto the opposite sides due to interpolation. Clip would cause a similar issue however the bleeding would be black.
We'll be using the Extend mode, which repeats the edge pixels infinitely in every direction, this means that any bleeding caused by interpolation will be effectively invisible.
Setting the UV extension is done on the Image Texture node of our material shader using the Shader Editor panel.
Don't panic! Once we switch the UV extension mode over to extend, you're probably going to notice that your tile turns either a flat colour or some variation of stripes.
The reason for this is that the tiles exported from wow.export have UV layers that are offset by the tiles world position. To fix this, switch over to the UV Editor panel with your tile selected.
You may have to look around to find your UV layer depending on how far your tile is from 0, 0. Once you've found it, we can translate it as necessary. Here, we're offset by 1 on the Y axis, so using the keyboard combination G, Y, -1, Enter, our UV layer is now normalized.
The last thing we're going to take a look at is the terrain geometry itself.
A single tile of terrain consists of 16x16 (256) chunks. Each chunk is an 8x8 grid of triangles, for a total of 256 triangles per chunk. That puts us at 65,536 triangles for a single tile of terrain.
In Eastern Kingdoms, we have 839 tiles, which means for the entire continent, our terrain geometry will weigh in at 54,984,704 triangles. Depending on your intended use-case, you might want full-resolution terrain, but for a long-distance shot, we can definitely optimize.
Before we optimize the geometry, we need to merge the 256 chunks of our tile together, as by default they're all separate.
Select the tile and switch to Edit Mode. From the top bar, choose Mesh → Clean Up → Merge by Distance. On the modal dialog that appears, set 0.01 as the Merge Distance.
Now our terrain tile is one big happy mesh. Let's go ahead and look at optimizing it. To do this, we'll be adding a Decimate modifier to the tile.
If you set your Viewport Shading mode to Wireframe and adjust the Ratio of our Decimate modifier, you'll be able to clearly see the effect it's having on our terrain as you adjust it.
The final value that you decide upon is entirely up to you, depending on how much you want to optimize the terrain - if at all. For this demonstration, I'm going to settle on 0.05, as it brings the total faces down to 3275~ while maintaining a decent topology.
Now that we know what we're doing with the tiles, we've got to go ahead and import all 838 of the other tiles for the continent and do all of that to each one. Since doing that by hand would take a considerable amount of time, let's look at automating the process.
Chapter 3: Importing (Automatic)
To automate the process of importing our continent, we're going to be using Python. I'll walk through the individual steps of the code to explain, but I'll also include a link to the full version.
Our first step is to iterate over the directory that we exported our map to. Specifically, we want to find all of the 3D tile files, which are prefixed with adt_ and use the .obj extension.
For each of those tiles, we want to import them using the wow.export Blender add-on. To do this, we can invoke bpy.ops.import_scene.wowobj and provide the parameters that we would normally configure on the dialog if we were doing it manually.
After the tile imports, we can access it by checking which objects are now selected. An important distinction is that our active object does not change, only our selection.
The next thing to automate is the step where we merged the tile chunks together using the Merge by Distance operator. For this, we need to be in Edit Mode, so let's set our tile as the active object and then switch contexts.
Now that we're in Edit Mode, we need to select everything on the mesh, achieved using bpy.ops.mesh.select_all operator, and then run Merge By Distance which can be called with bpy.ops.mesh.remove_doubles, providing 0.01 as our merge distance just like we did manually in the modal dialog earlier.
Once merging is done, we switch back to Object Mode so we can add our modifier.
The next step on our list is to set the UV Extension of our image textures to extend. To do this, we iterate over the materials of the tile, and then iterate over the nodes to find the Image Texture node and set the extension, as shown.
That's a lot of looping, but we know there's only one material slot and that the material should only have one image texture, so let's reduce that logic a little.
We could access the Image Texture as nodes['Image Texture'], but we can't always guarantee the name will match, so it's best to filter by type.
After that, we need to normalize the UV layer. This is a little trickier to do automatically than it was manually. To normalize it, we subtract the value of the top-left UV (which is the second UV in the data, due to the way WoW terrain is rendered) from all of the UV.
Note that we offset the Y axis by -1 since the top-left corner of the UV layer is 0, 1 not 0, 0.
At this point, you need to decide if you want the full-resolution geometry or to optimize the geometry. If you want the full resolution geometry without optimization, you can skip this part and just run the code we've got so far.
Be warned that without any optimization, a continent like Eastern Kingdoms is going to require around 20GB of RAM with breathing room and around 4-8GB of VRAM, but it definitely looks majestic.
For most use-cases however, you probably don't need the full-resolution geometry, so let's look at how we can include some optimization into our import process.
In the manual demonstration back in Chapter 2, we used the decimate modifier. The problem there is that if we decimate each tile individually, the edges are no longer seamless.
The most common solution to this problem is to merge the tiles together and then decimate them together. After a lot of system crashes (and having to rewrite this entire guide due to losing it), I can tell you that trying to run a decimation modifier on a merged mesh of 54,000,000+ triangles isn't an option.
To this end, we're still going to decimate on a per-tile basis as each one is imported (which will help speed up the import, too), but rather than decimating the entire tile, we're going to preserve the edge vertices as they are.
This still results in a 60%+ optimization overall, and means our tiles seamlessly connect with one another. Achieving this however requires a bit of work. The first thing we're going to do in our code is switch to Edit Mode and deselect everything.
Now we want to select just one of the corner vertices.
To do this in code, we need to return to Object Mode, manually select the vertex, and then switch back into Edit Mode.
With our corner vertex selected, we can easily expand our selection to all of the edge vertices by using the non-manifold selection operation.
Once we have the edge vertices selected, we can use the inverse selection operation to switch to having everything but the edge vertices selected.
We can achieve both of these things in code by simply calling the operators, as shown below.
Next, we want to create a vertex group on our tile that will hold the selected vertices. We don't need to bother specifying a name for the group. Once created, we can assign our selected vertices to it.
Now let's return to Object Mode and add a Decimate modifier to the object. As discussed in Chapter 2, I've settled on a 0.05 ratio, however the final number you use is up to you. Additionally, we provide the vertex group we just created to the modifier as well.
Just adding the modifier isn't enough, as modifiers are (until applied) non-destructive, meaning our full mesh data is still in memory. We want to make sure to finalize our optimization by applying the modifier.
Last but not least, let's remove our vertex group now that we no longer need it.
We should be all set! As mentioned above, you can find a complete version of the script over here. To run it, switch to the Text Editor panel, create a new document, paste in the script and hit Run.
Once you've executed the code, you'll need to wait for it to finish. If you haven't got anything better to do in the meantime, head on outside and look at some birds.
After it's done, you'll be left with a magnificent continent. Optimized at a ratio of 0.05, our geometry sits at 2,748,564 triangles with a much more comfortable 2GB RAM/VRAM, and for long-distance shots, you really can't tell the difference.
Chapter 4: Ocean
The thing that stands out the most right now with our continent is the ocean, or lack thereof. While it exists on the minimap tiles that we use for texturing, it doesn't exist in the geometry itself, resulting in strange looking depths.
In addition to this, on the Eastern Kingdoms map specifically which I used for my example, we have the unique case of Vash'jir - an underwater sub-zone - which is all kinds of chaos right now since we're using the surface minimap tiles on underwater geometry.
To resolve this, we're going to turn to a quick little bit of maths and Python. Let's take a look at the following snippet of code, which can also be found on GitHub.
This code iterates over each vertex on any selected object and runs the max function on the Y axis against 0, meaning that the highest number of Y or 0 is chosen.
This effectively means that any vertex below 0, which is our sea level, will be pulled up to 0.
This type of quick snippet is the perfect candidate for the Python macro add-on that I recently released in another guide; check that out over here.
We could select the entire continent and run the code, but that's probably not what we want. There are some tiles, such as the one holding the Dalaran Crater, where the terrain naturally sinks below sea level.
We can simply just not select those tiles and it's not a problem, however there are some tiles, such as this one in Westfall, where we have part of it we do want to fix and part of it we don't.
To fix this, we first enter Edit Mode and select just the faces that we want to fix, leaving everything else on the mesh un-selected.
Switching back to Object Mode, we can run another snippet of code which is almost identical to the last, but checks the selection state and only adjusts selected vertices.
Once again, I've posted this snippet on GitHub. Running this, we can now level out just the vertices that we had selected, leaving the crater untouched.
With the code snippets in hand, we can very quickly bring the sea up to sea-level, leaving us with a much richer looking topology of our continent.
But wait, there's more! Check out the next chapter for how we can take our continent to the next level with actual 3D buildings!
Chapter 5: WMOs
The continent we have looks great so far, but there's one more thing we can do to take it to that next level: WMOs. WMO stands for World Model Object and is Blizzard's format for large structures such as buildings or cities in the game, often that can be seen from a distance.
One example of a WMO that can be seen from a long distance is the Bastion of Twilight spire in Twilight Highlands. If we take a look at our current map, it's a flat pile of mess.
We could just go ahead and load in every WMO on the continent, which for Eastern Kingdoms that would be 3296 models. That might not sound like a lot, but keep in mind that some of them are massive - like Stormwind City, Ironforge and Undercity.
Each of these WMOs is going to come with potentially hundreds of textures, let alone massive amounts of geometry. Instead, we're going to cherry pick the ones we want.
To do that without having to manually crawl through over 800 CSV files, we're going to import empty markers for all of our WMOs, which will show us what goes where without importing them. First up, let's create a collection for them to keep things organized.
Just like we did before with the OBJ files, we're going to iterate over all of the CSV files which wow.export exported the placement data into and parse each one. We only care about WMOs, so we can check the Type field of each row and skip the rest.
For each WMO, we're going to create an empty and apply the necessary location/rotation/scale transformations that our WMO will have. Additionally, we'll also store the model path on the empty. This means that when we import a WMO, we have all the necessary data and don't need to re-parse the CSV files.
There's one more thing we need to do before this is good to run. Due to the way that WoW loads terrain tiles, WMOs that overlap onto more than one tile are registered for each tile.
To prevent loading duplicate markers for the same WMOs, we need to cache the ModelId of each WMO and skip loading it if it's already loaded, the same way the game handles it.
As always, I've posted the full source code onto GitHub for easy copying. Just as before, create a new document in the Text Editor panel, paste in the code and hit Run.
This is going to take a couple of minutes to process, but once it does, you should find your continent now covered in markers for where WMOs should be.
Let's go ahead and find a WMO that we want to import. To help us, we can enable the names for them in the view-port. To do this for all markers, we can use a bit of Python.
The name of each WMO will now appear above the marker, which makes it much easier to identify where/what each one represents.
Going back to my previous example of the Bastion of Twilight entrance, I'm going to use that as an example as it's a prominent landmark which can be seen from anywhere.
Over at the location, we can easily see the WMO marker in question: twilightshammer_raid_entrance
With our marker (or multiple) selected, let's execute some more Python code, which as always I've posted on GitHub for easier access.
This code iterates over every marker we have selected and then imports the associated WMO, applying a quick rotation fix to make sure it's aligned properly on our map.
Once executed, within a couple of seconds we should see our selected WMO pop up into existence.
Before we go ahead and start peppering the world with models, let's take a moment to look at how we can optimize it to keep our resources in check.
The first thing to note that even this simple spire consists of 10 textures. Since we're not up-close with the object, we can optimize this by baking it into one, much lower-resolution texture, which from a distance will still look fine.
I recently published an in-depth guide on texture baking, which includes complete one-click automation for the process too. Using what we've learnt from that article, I'm going to bake this spire with a texture resolution of just 256 x 256.
Despite being such a tiny texture, it's more than enough for our spire to be perfectly recognizable from a distance. Remember, you don't have to optimize things if you don't want to, but it will come at a heavy resource cost.
Another thing we can do to optimize the model is to reduce the polygon count. Just like we did with the terrain, this can be done using the decimate modifier. I found that somewhere between 0.1 and 0.3 was a good ratio, depending on the size of your model.
Remember, as you're optimizing and throwing things out, be sure to keep purging your orphan data in the Outliner to keep memory usage in-check.
Using this method we can quickly cherry-pick the objects we want to include in the scene without any hassle. Adding large structures and cities that can be seen from far away really makes a lot of difference.
Chapter 6: Conclusion
That's it for this guide! Thank you for reading, I hope that it was either useful or insightful in some manner. Using the methods outlined in this guide, you can create some seriously intense backgrounds for your 3D works, and if you do be sure to tag me, I'd love to see!
As always, if you have any issues with the guide, wow.export, Blender or beyond, feel free to ping me in the #champions-lounge channel on my Discord.
