I've found spellcasting AI to be quite good overall. In fact, I think it might be the strongest part of the tactical AI in Dom II.
The AI is conservative when it comes to using gems against weaker opponents, but doesn't hesitate to bring out the big guns when threatened. It is also very good about casting appropriate defensive spells.
Sometimes I'll wonder for second why it chose a seemingly pointless spell, like Holy Avenger by/on a priest that'll probably not get attacked, but then I realize the priest in question really didn't have anything more useful to cast that round anyway.
I've noticed a tendency to lean on conjuration, but that might just be because that's what I have researched for them most often. Bit of a contaminated sample there, if you know what I mean.
It is also very good at targetting. The only problem I've seen is when I was playing the flame-retardent Abyssians and had one group of Militia along for the ride. (Hey, they volunteered!) In one particular battle, my most powerful mage used some fire spells that didn't act as any threat to my regular guys, but were a bit hard on those soft-skinned Militia. I think that decision may have been weighted by the overall desperation of the battle: we were badly outnumbered and the enemy Pretender God was present on the field of combat. (Our guys took it on the chin in the end; it wasn't pretty!)
Other than that incident, the AI seems to do a pretty good job of trying not to toast your guys while frying the enemy. The "arrow-in-the-back" syndrome has been greatly reduced, especially with one particular beta-patch, and is now rather rare. I only run into problems when either I've given orders that force the situation (oopsie) or things have gotten either truly desperate or utterly chaotic.
The large spells are used fairly rarely because of their cost. Sometimes, I'd like to see them used more often, just for the pure joy of watching them go, but I can't argue with the conservation of gems. It can be a pain to keep your mages stocked with them if they are pushed into constantly using them up.
The portion of the combat AI that I'd like to see improved is in the "mad rush towards the enemy" department. Keep it together, lads! If Illwinter were to turn their hand towards AI improvements some day, I'd suggest they start with that and wait until Last to see if there's anything to tweak with the spell-casting.
Oh, Last minute thought: I have seen the spell-casting AI conjure things in back that can't seem to make it up to the front line in time to be a part of the battle. That particular problem is the only thing I've found I don't like, but it
might be a part of the fact I lean too heavily on Conjuration research. They may just not have anything better to do at those points beause I haven't researched them the spell options, but I can't say for sure. (It'll take more playing to find out . . . yeah, that's a good excuse to play more . . .
)