32bit unsigned long in OBSE script?

Update: so you can do math operations on ref instead of long and it almost seems to be behaving as a 32bit unsigned integer... except the bit operations seem to be choking still... see post below.

Does anyone have any experience generating 32bit unsigned longs for use as reference IDs in an Oblivion script?

I'm currently getting crapped on by the script engine. For More-Armor-Slots, I am doing bit arithmetic to generate a 32bit unsigned number that I then convert to a ReferenceID. Unfortunately, OBSE/Oblivion Script does not natively support unsigned long, so I'm try to manipulate a signed long so it turns into the bit-equivalent of the 32bit unsigned long value I want so I can convert into a RefID. Example:

If I try to generate 7F00:13E1 (which uses 31 bits and leaves the 32nd bit = 0) OBSE handles the number perfectly and I can pass it to pluggy to create a Reference ID.

But:

If I try to generate FF00:13E1 (the 32nd bit is now = 1) OBSE will now do stupid things like trying to print out a 64bit hex string when I specify formating for 32bit hex string and it is always the following string no matter what the actual result of my math should be: FFFFFFFF80000000.

I'm about to look through the OBSE source code, but wanted to ask here just in case this is something that's been solved years ago.

Last edited by ponyrider0 on 14 Jun 2017 - 13:37.

Alright, I now have a partial

Alright, I now have a partial solution to my problems: instead of declaring variables of type long, I can declare variables of type ref. When using type ref, I can do a left-shift for 16 bits and get a correct result.

However, LogicalOr operations on ref values that have the 32nd bit set seem to give incorrect results. example: script result is 800013E1 when the expected result is FF0013E1.

And adding two refs together appears to be going through a int->float conversion, because the result is rounded off: example -> script result is FF001400 when the expected result is FF0013E1.

Update: If I do two bit operations like this, then it seems to work but then more crap happens -- it appears to be stuck as a number and will not work as a reference unless I convert it into a reference with Pluggy's LongToRef.

This math breaks:

LogicalOr FF000000 13E1 --> 800013E1

This math works:

LogicalOr 7F000000 13E1 --> 7F0013E1

LogicalOr 7F0013E1 80000000 -> FF0013E1

BUT now, when I use Pluggy's LongToRef on a number with the 32nd bit set, the script engine breaks again. GAH!!!

Okay, looking at the OBSE

Okay, looking at the OBSE source-code: internally, all arguments passed to the LogicalOr command get converted to UInt32 and then the operation is performed using the regular C bitwise OR operator... there must be an internal conversion bug, when going from script type ref to C type UInt32, "FF000000" becomes "80000000". However, "7F000000" passes through unchanged... I bet it's some variation of the same signed to unsigned madness that I'm running into in other places.

FYI, I'm splitting a 32bit

FYI, I'm splitting a 32bit integer ReferenceID into two 16 bit words and storing them as separate float point actorvalues to bypass the issue with Oblivion not supporting the full 32bit integer range. I then retrieve the two halves and recreate the original ReferenceID. This appears to be working on reference IDs that use 31bits or less.

The best solution to this

The best solution to this would be exposing in OBSE some unsigned types. Or using some conversion function that "delete" the signed bit during the operation and readd it after controlling operation possible overflows. So one can perform this kind of calculation without worring the signedness of two bit complement notation.Otherwise you may expose the needed function in an OBSE plugin (you will need the raw power of a system programming language).

It seems like you are encountering an overflow case. Maybe what you are seeing derive from the way refs type are saw inside the engine, or you are triggering in some cases undefined beheviour (note that pluggy is not free of bugs either).

What you have to do with all these computations?

llde wrote:

llde
What you have to do with all these computations?

All these computations are actually just a work-around to store full 32bit unsigned integer ReferenceIDs with each actor. For More-Armor-Slots, each actor/NPC has 3 additional "slots" which store a reference ID to an equippable item.

1. I decided to use AddActorValues to create and store these "slots" for each actor reference.

2. Since Oblivion stores ActorValues and nearly everything else internally as floating point numbers, I made a workaround by splitting the 32bit reference ID into a 16bit HIWORD and LOWORD. I use a bit-wise AND operation to mask the HIWORD and LOWORD so I can separate and copy them into separate variables. Then I bit-shift the HIWORD so that it is a 16bit value. These functions seem to be working without problems.

3. Then I store each 16bit value into the custom ActorValue using SetActorValueModF so that it stays unique to each reference and does not over-write the base object's actor values -- one 16bit value goes into the "max" mod value and the other 16bit value goes into the "script" mod value.

4. When I need to retrieve the original referenceID, I use GetActorValueModF to put each 16bit value into a HIWORD and LOWORD variable.

5. Then I bit-shift the HIWORD back into a 32bit value: this is where the trouble initially starts with unsigned values. Bit-shifting a signed variable by 16bits will cause problems for variable type long if the 16th bit is set on. However, I am able to work-around those issues by storing the HIWORD in a variable of type ref and then applying the bitshift to the ref variable. Problem #1 solved.

6. The next problem happens when I try to combine the HIWORD and LOWORD back into one 32bit value. Even if both values are stored in variables of type ref, passing them to the LogicalOr command must going through an internal conversion from unsigned to signed and back again. If the 32nd bit is set, the output of a Logical Or such as:

FF000000 OR 000013E1 becomes 800013E1.

FYI, 80000000 is equal to 32nd bit ON and all other 31bits to OFF. Essentially, it represents negative zero in a 32bit signed integer. This incorrect result is probably significant in some way, but I don't know what. Anyway, I have also worked-around this problem by detecting if the 32nd bit is set to ON and then setting it to OFF using a bitwise AND operation. Then I am able to combine the now 31bit HIWORD with the 16bit LOWORD using the LogicalOr function:

7F000000 OR 000013E1 becomes 7F0013E1.

For some reason, once I have this 31bit reference ID generated from HIWORD OR LOWORD, I can apply a second LogicalOr step to turn on the 32nd bit again without problem. I have no clue why it works as a single bit operation but not on the full 32bit values:

7F0013E1 OR 80000000 becomes FF0013E1.

Problem #2 solved.

7. Finally, the 3rd problem is that once I have the original full 32bit unsigned reference value re-created, OBSE will not let me use it as a reference. Apparently, once I start using bitwise and math operations on a variable of type-ref, it internally becomes recast as a 32bit unsigned integer and loses it's ability to act as a referenceID. I can get around part of this problem by using Pluggy to convert integers into referenceIDs again. However, Pluggy's LongToRef appears to once again be limited by signed variables internally, so I can only properly convert 31bit integers into a reference ID. Problem #3 unsolved. :(

I've tried converting to a string to try to go from StringToRef, but this also seems to be limited to 31bits.

So, it looks like I will probably need to write my own OBSE plugin which is able to operate on a full 32bit unsigned integer. But if I'm doing that, I might as well forget about all of my other workarounds like splitting references into 16bit halves or using AAV to store my slot values...

If you need only need to

If you only need to "store" reference and enable to recall them you may create an OBSE plugin that store TESReference pointers , bascially having a master Array (or nested Master Array). What you want to do seems actually very similar of what saebel did for Sneaky Detection Recalibrated.

Bascially you may have a plugin that expose some function.

For example (High Level representation):

fn RegisterMod( String ModName OR Stirng ModID) -> bool

fn RegisterNPC( TESReference* npc OR TESActor* npc) -> bool

fn AddItem(TESActor* npc, TESReference* item) -> bool

fn RemoveItem( TESActor* npc, TESReference* item) -> bool

fn GetItemsForNPC(TesActor* npc) -> OBSEArray<TESReference*>

Otherwise why cannot you store directly the reference?

For example having a duplex nested array (ArrayMap<Array<ref>>)with the structure:

{ NPCId {item1 : ref , item2 : ref , item3 : ref}}

Note if there is a reason why you cannot do so I may have not understanded what you are trying to implement. I mean most mods that needed to only store and recall refernece ids used this (or a variant) of on of these approaches (the first is more performant in heavy workloads if the mod must manage many references)

P.S from the Construction Set Wiki:

Although you can declare a variable to be Short, Long, or Float, all three will be stored like a float.

Basically Oblivion need a real integer/long integer type badly.

Yes, that was basically my

Yes, that was basically my original design plan but was not able to do it last year because I couldn't get OBSE to build. That's why I developed the ESP only hack of storing references in ActorValues. Now that I'm able to make working OBSE plugins, I'll try to re-implement More-Armor-Slots to store references in an ArrayMap -- but probably not for another year. I'm trying to get rid of all distractions and focus on modifying openmw-cs to export to TES4 so I can complete the Tamriel Rebuilt Conversion project.

Update:
Actually, I suppose I could implement the OBSE ArrayMap(NPCRef, Array[ItemRef,...]) in an ESP. I had implemented this in the ESP but then re-wrote the ESP using ActorValues because I was afraid of doing the memory management for the thousands of NPCs which would eventually be added to the ArrayMap... However, I can just delete NPCs from the ArrayMap if they are unloaded from memory... the only problem is to figure out an accurate way to detect if an NPC is unloaded from memory -- right now I'm checking if NPC is in same cell as PC to remove them from my equipNPC queue. That will probably work.