I actually had a scout there...
My assesion of the situation when I saw the armies arrayed at the start was that he stood a rather decent chance to beat you (unless you pulled some nasty magical trick), partly, as you said, because of the darkness, but also because of those tough prot 22, unbreakable statues (and somewhat less tough, but equally unbreakable claymen) that would have you standing in the gateway forever, while the castle arrows and spells rained down upon you.
(note, btw, that those statues didn't have darkvision either - they should have imho, but anyways - so between your troups and them darkness didn't matter much. The couple of men Agartha had did benefit form it though, and probably lasted a bit longer and killed a bit more because of it.)