I looked at the E46 M3, but I wanted a Tip/SMG and the god-awful transmission on the M3 was not to be thought of. Bimmer does good SMG's, they just don't use them on the E46 M3. Besides that, the S4 is both more refined and more usable (4 doors anyone) than the M3. I didn't by the S4 to mod (that's what my E36 is for), but as an excellent daily driver with good performance and excellent handling.
That all said, when the new S/C comes out soon from APR is combined with a fullback, I think there is going to be alot of surprise about what the B6 S4 can do compared to FI'd M3's. Much like a CAI, fullback, and chipped S4 is a rude surprise to an M3 with the same mods. E46 M3's just don't pick up much power from those mods, S4's do.
Oh, and, yes, the M3 does get 333hp out of a 3.2L engine, but it does so with nowhere near the torque the S4 has, and has the same or worse gas mileage. Yes, they are faster, but only when driven really hard, and only on dry roads.
If you want a car to drive like its on the track on the time and want a manual, by an M3 by all means (if you also never plan on having more than 1 passanger above 5'4"). The SMG's are ok if you don't plan on using it as a daily driver, but they are ROUGH, even on their smoothest setting. I've had some funny discussions with guys from the Bimmer dealerships here about the number of M3 SMGs traded in or sold after 2-3 months because people hated the transmissions.
E46 M3: Better race/ track car, more mod-able, more fun in an agressive sort of way
B6 S4: More luxurious, better if you don't want a manual, more usable, still fast, still fun
B5 S4: the perfect tuner's car in many ways. Not as "nice" as the B6, but still quite comfortable.
E90 M3: 426bhp 300-320 lb-ft, 4L (-ish) V8. Yes, it'll be a monster, and it'll have a much better transmission on the SMG. But, still not AWD and still not a 4 door. I'm just curious as to the price, since the engine is going to be EXPENSIVE! I'm guessing $65K or so real price ($5K less stripped down), plus the inevitable mark up for those who want theirs the day it arrives in showrooms. Besides, it's been Bangled...
Still, if I want a real go-fast coupe or roadster, I'll by an MZ4 now that they are officially going to be released. The S54 motor (the E46 M3's) in a Z4 will be alot of fun, and combined with the fact that they are considering a coupe version, will be really interesting.
Ultimately, the car I keep contemplating trading my S4 in on is the S6 or RS6 in a year or so when they come out again...