Today I took a really close look at some SM64 maps in an attempt to get a better understanding of how they were made and how I can avoid further mistakes when making my own maps for the engine. Perhaps it could prove useful for you too?
Let me first say that I’m pretty surprised at how inconsistent and sometimes downright lazy the mapping seems to have been done. Seeing as this was the first true 3D game on consoles, the tools available at the time may have had something to do with it.
Here are the things I discovered (long post!):
All holes for canons and rolling cannon balls, etc. seemed to have been simply cut out with boolean functions (placing a mesh through another and “cutting out” the shape”). This leads to a mess of triangles and very bad topology, so it’s not something anyone uses much anymore (it causes bad lighting when rendering and bad transformations when animating). However, it is very economic in terms of polygon usage, which makes sense when you consider how huge maps SM64 was able to use on the N64.
In fact, topology doesn’t seem to have been a concern at all. Here’s a clear example of how the creator has simply collapsed a bunch of edges to save polygons (all the lines bunched up in a single vertex). I have no idea how this works, as Mario simply clipped through the floor when I did something similar with my map. Perhaps it depends on how sharp the vertical angles are?
Extrusions also seem to have been done pretty haphazardly. This is particularly strange, because it creates more polygons, and doesn’t seem to have any sort of benefit. In the example below, you can see how an object has been placed on top of a flat surface, instead of having it extruded from it. Both seems to work just fine, so why extrude?
Often objects are placed on the ground like this. There doesn’t seem to be any problem with collision detection, even if two separate meshes are joined together this way. There are some peculiarities to consider, though: The edges on these objects are always marked sharp. I have no idea why. The edges also never clip through the surface. They’re placed PERFECTLY aligned with the ground, never going through it. This is strange, considering the example below:
Here, the plank is clearly clipping through the ground, but it doesn’t seem to be an issue. Why go through the trouble of aligning the edges in the previous example, if it doesn’t matter for collision detection? Perhaps it just doesn’t matter for the plank because it is a fully enclosed volume, with no open bottom?
Here’s a smart use of the same technique. The blue line shows that an edge is marked as “sharp” (like the bottom of the previous rocks). Here, it means that instead of having the ground be one continuous surface, a perfectly aligned incision has been made across the section where the bridge begins. This saves polygons, which would have had to be drawn from the corners of the bridge legs if the surfaces were connected.
The same sharp edge disconnection is used on the side of stairs throughout the other maps as well. But why go through the trouble of making such extreme polycount optimizations, and then waste a bunch of polygons on needless extrusions? A lot of things seem very inconsistent…
Another point of inconsistency is the texturing techniques used. Here’s a really clever example, where the artist has taken two surfaces and used flat projection to unwrap them as two UV islands, but still managed to align the texture seams. He has done this by letting both islands have one part each which is of equal size, and letting this part be exactly 1:1 with the texture map. Sadly though, someone has later come and adjusted one side of the bridge mesh and made it a slope, which sort of ruins the seam a little.
This technique isn’t very relevant today, as you can simply use a UV unwrapping function to equalize the sizes of several island groups simultaneously.
Here’s a very dumb example of UV unwrapping. The ground and side slopes here have been projected as one surface seen directly from above, which has lead to very badly stretched textures on the slopes (which have a much larger surface than what they’ve been allotted in the UVs). It’s easy to tell that this is what has happened, because of the 90 degrees angle on the corner in the UV map.
A final little thing I found a bit confusing is that the back faces of fences are rendered in this map. I was expecting to find “double” fences with one mesh for each side, since everything I’ve seen imported to SM64 has only been able to render the front of any object.
Another thing to note: Yes, walls have to be 100% flat for Mario to be able to wall-jump off them. Ledges can be grabbed if the wall below is completely 90 degrees, or if the angle of the wall is inverted (so the wall slopes inwards). All walls which are angled outwards, even by a single degree, will be counted as slopes by the engine. All walls that are angled inwards by a single degree will function more or less like walls, but Mario will play a sort of negative sound and fall off it immediately if you try to wall jump.
Hope this autistic write-up is of use to someone!