This might be a good example of how a siege is announced, enters the map, and then immediately steps off the map, leaving only one or few invader on the map.
When I send out my military (squads abcdef) to anticipate the siege on the west edge of the map (outside the wall), they will kill the one invader and the Siege tag disappears.

The attached archive contains two savegames. One is from the automatic seasonal autosave in Spring, the other is a manual autosave just a few minutes later.

When you load the Autosave, siege will be announced by the end of Granite 3rd, first pack enters the map, but all of them leave immediately, leaving only four invaders on the map.
When you load the Manual save, siege will be announced by the end of Granite 3rd, first pack enters the map, but all of them leave immediately, leaving only one on the map.

(These saves are probably relevant to the Bugs tracked in Mantis under the following IDs: 9202, 9236, 10167)

The saves are in ASCII tileset.