At places like Caldera, the first 5-10 seconds after I have enterd this city(as in standing in the streets). Stutter appears.
This is the most noticeable if I am in a inside area in Caldera, and from there I go to the outside. Doing this procedure, I experience quite immense stutter in the first 1-3 seconds.
I could be death wrong, but I noticed when I enter the street/outside area of Caldera. The lamps are moving a bit, and the movement of this lamps stops after a while. All the lamps do this movement in the same exact sequel. When the lamps stops moving, the stutter disseapers.
Is there any way to delete this lamp movement? Because I think the stutter is related with the lamps movement.
Edit: A second thing I noticed, and yet again I could be death wrong. A area which contains a high ammount of grabeable torches, posseses a constant game stutter. I noticed this because in areas where the sound of burning torches is high(due the huge ammount of this torches), there is also a constant stutter. This edited paragraph could do more observation though, not quite sure about this one.

#1
Lamps and torches have HAVOK, oblivion's physics system. That means they can move like any other grabbable object in Oblivion (plates, bowls, books etc.).
What are the specs on your video card? The stutter tends to happen with low-end cards.
Also, you could try Oblivion Stutter Remover and see if that helps.