It's only a theory right now, but I think that if your game is running like a dog in debug, maybe there's something wrong with the number of branches you're doing. Not just in debug, but in general.
Asserts are the usual cause of slow down (they break the cache and branch and cause memory wastage), but they are valuable as warnings when things are going wrong. They provide great protection, but they aren't necessary if you make sure that the condition that would normally trigger them cannot be achieved.
One of the most common uses I've seen for an assert is checking for NULL. This can be avoided by just making sure that your queries are either returning null objects instead (dummy objects), or by making the query inherently null proof (use a queued up todo list rather than a fetch from and check for work to do).
Another slowdown can be from templates that aren't being fully expanded or optimised in debug code. This is a lot simpler to fix. Stop using template meta programming. Your compile time will go down too. Template meta-programming is also a replacement for moving wholesale to scripted development. Consider the problem your meta-programming is solving in light of a scripted environment.
On the PS2, we never saw that big a difference in final build and the debug build. This was mostly down to the fact that the PS2 was spending all its time streaming data from one place to another (no nulls in the middle of a stream), and processing them with data driven transforms (no meta-programming for us, just plain coded transforms driven by data demand). The only code that ever got all that faster was the C++ class style math library (which was hilarious to watch as it went from being slower than hand coded vector unit code by a factor of four to being faster than the hand coded vector unit code by a factor of two.)