Question about Talisman

[Ragnarok]

Sort of an odd question but it's one that should be known. If you're in a game where you have the Talisman, is there any point in taking empire attack /aggressiveness up any? Or even to leave it at 100? Why not take it clear down to 50% or whatever it will go down to. Only reason I can see for keeping it at 100% or higher is for units or some-such that will not have the Talisman and need some kind of attack % increase to hit anything. Your opinions on this are most appreciated.