Skip to main content

[TUTORIAL] Finding Invalid Meshes with tools

Submitted by Ormin on

Hi guys,

First and foremost - I am not certain whether people here found any way of searching for those crashing meshes simple way already. I just discovered a method which can be used to identify those meshes. Mind that it's quite time consuming, but accurate, also involves using a lot of low-level tools. I am going to try to explain the guide as simple as it's possible, so everyone ( or almost everyone ) can make use of it, more people can do it - the better.

1. Tools Required

  • Olly Debugger v2.01 ( http://ollydbg.de/odbg201h.zip )
  • PrGrep searching software ( http://prgrep.com/dwnld.php?lng=en&pg=61 )
  • NifSkope, 3DS Max etc. ( for fixing the meshes, search 4 downloads ) [optional]

2. Configuration of the Tools

2a. Configuring PrGrep

Make sure you configure your PrGrep to search for binary files ( the box on the right to the ,,containing" field ) and searching in subdirectories and other options. Set "directory" path to your destination mesh directory ( in this case i've chosen Meshes\tes4, because I wanted to search for invalid meshes from Skyblivion ). A correct configuration looks like this:

[External image]

( http://i42.tinypic.com/auduew.png full resolution )

2b. Configuring Olly Debugger

Nothing to be configured, just make sure you run it as an administrator.

2c. Configuring the Game

In Launcher, tick all the .esp / .esm you want to try ( in my case I had Skyrim, Skywind, Skyblivion and Skyblivion Beauty/LOD ticked ). Make sure your game runs in windowed mode ( otherwise debugging might get really hard )

3. Starting the game from the debugger.

Start the Olly Debugger. When the big window appears, drag on TES5.exe from Skyrim's folder. Wait, until it ends the analyze ( you can see it being analyzed in the bottom bar below the window ). Hit the f9, it should say "Running" in the lower right box. In case it stops cause of one reason or another, for now, just keep pressing f9 until Skyrim menu starts ( in the another window that is ).

Now, start whatever you would like to do. Run a game, run a COW command from a terminal or whatever you wish to find a crashing mesh. I just did COW Tes4Tamriel 5,5.

When the game is about to crash ( the invalid mesh is being hit by the game engine ), the debugger will stop and you will see something like this ( or similar, but from what I've observed, the crashes are always in similar places ):

[External image]

( http://i42.tinypic.com/n12p28.png full resolution )

the black address on the left shows the place the crash happened. It's the int3 command.. see the next command ( jmp xxx) ? Click on it and hit enter. You should land in a place similar to this:

[External image]

( http://i40.tinypic.com/fjkl91.png full resolution )

The thing here now is to see the three ,,mov xx,byte ptr ds:[REGISTER+something]. IN my screenshot it is EBX ( you can see it three times in a row ). Now , search for it on the right panel ( on my screen - EBX =156D9A1A )

Now, in the left down corner, hit the big space ( where zeros on my screenshot are ) and press CTRL+G. Write the offset you've readed prior here, and hit enter. The window here should change and there should be a lot of nonsense characters and random bytes. Copy , let's say, first 8 bytes from this:

[External image]

( http://i42.tinypic.com/6ss2tj.png full resolution )

That's it - you actually copied a small small part from a mesh which was causing the problem. Now, back to PrGrep, in the ,,Containing" Field paste what you just copied. Remove the paddings ( unnecessary post-spaces etc so its like AA BB CC and nothing else ) and hit search. If you've done everything right, eventually the full path of model ( with an ascii representation of what you've been searching ) will be shown in the window.

4. Fixing [optional]

Crashes which I am describing here are based on the MOPP collision bug which affected quite a few meshes from both skywind and skyblivion. To fix it you can remove the appropiate branch using nifskope or just add a new collision by hand.

That's it!

 

 

Karma: 0 (0 votes)
Last edited by grumpycat on 4 Jun 2013 - 22:27.

Thanks, but I highly doubt there are people interested in Skywind/Skyblivion and don't afraid of assembly language (except you and me of course) :)

Deleting collision is not a fix for problem, we need a proper collisions. There are guides available on how to, for example search for "collider meshes" here http://www.darkcreations.org/forums/topic/4249-100-custom-tree-pyandonean-islands-tuto02/

But it involves a lot of manual work, and considering there are probably hundreds of meshes without proper collision in Skyblivion, I don't know how we can fix them.

Those bad meshes don't always crash the game, but they don't collide with anything either and sun shines through them. For example look at shadows here

https://www.youtube.com/watch?feature=player_detailpage&v=3JVG65ZFKBY#t=134s

https://www.youtube.com/watch?v=46VzRbIuoLQ&feature=player_detailpage#t=151s

Karma: 0 (0 votes)

So while I'm learning Payprus, this seems over my head, but really excites me at the same time.
From what I gather, Olly debugger/your setup is allowing you to see the game's code as you play. That's allowing you to search the code to find not only the bad mesh but the bad part of the bad mesh, that caused the game to crash.

Is this sort of thing used to create SKSE functions? Is this how they find variables to write getsomething() commands?

On another note: every mesh crash I've ever run into with my own custom made meshes was the result of a trishape not having a name in Nifskope. In Nifskope it looks like it has a name but then when you click to rename it, it's blank. Are we sure the problem has to do with collision, and doesn't just need to be renamed?

Karma: 0 (0 votes)

@zilav

That's why i tried to make the thing as simple as possible, so even someone who didn't see assembly at all is able to find the invalid meshes. That's also why I've done screenshots and explained it step-by-step without getting into what is happening. Yeah, I know collision deletion is not a fix, I just pointed to WHAT should be fixed :))

@Ice

You got the basics - yeah, indeed that's the case, the place you are highlighting here are exactly the bytes causing the game to CTD ( or actually, it's not a crash but an infinite loop ). If I recall correctly what SKSE is, then you are correct on this matter, too - reading assembly can provide a lot of valuable info, but mind it that it's already doing so-called Reverse Engineering and if i recall correctly it's disallowed by EULA - therefore I do not condone it :P

 

 

Karma: 0 (0 votes)

Hey, this is simple enough. Its not hard people. its time  consuming on the grounds of having to go cell to cell in game and finding the crashes. Making a Collision model is easy enough. Iv already fixed a few models and noticed the crashes around ruins have dropped significantly.  In all honesty most the models are fine its just the dungeon tile sets that are crapping out for some reason. 

Karma: 0 (0 votes)

All plants meshes are bad, they have collision reusing the same high poly mesh and causes severe FPS hit in wilderness, crash the game or crash when you approach some of them.

I don't know how to say properly, but they should have low poly collision so you can activate them, but at the same time you must be able to walk through them. Considering plants and mushrooms are everywhere, they are the major issue currently.

Karma: 0 (0 votes)

Just leaving a link about making proper collisions (better leave it here instead of adding to private favorites)

http://forums.bethsoft.com/topic/1464441-quick-questions-quick-answers-thread-21/page-4#entry22821350

Karma: 0 (0 votes)