In my fortress, the FPS drops very low when I'm in a siege and I open my drawbridge. When the drawbridge is closed during a siege, or when it is open not during a siege, the FPS returns to normal.

In this save, an ettin is about to run through the front gate. This is unrelated, but you can use him as a control to see whether hostile monsters are affected at all. You can kill the ettin, and then go kill the 4 invaders with the battering ram on the edge of the map, and the siege should end.