Straight up the B7 RS4 is a better buy than a B6/7 S4 for the track. They have fewer issues and are better suited for the track from the factory hence the RS badge. You can make a B6/7 S4 better for the track than a B7 RS4 (like I have) but you will easily spend more than the current market difference in up front purchasing price.
After you get the car then you are going to have to do maintenance to get it ready for the track because all these cars are old now. Certain things are kind of a wash depending on what you go with like tires, wheels, brakes, suspension, clutch, and other consumable parts. The RS4 brakes are really nice by the way. So much so that I had to go to RS front brakes on my B6 S4 because they just work! Just so that you know. I am currently running 265/35/18 tires (will be 275/35/18s when I wear out my current PSS) on 18x9 ET 25 wheels, RS4 front brakes (LW rotor rings), bigger 330 mm rear brakes, K&W Clubsport coilovers, all the spherical control arm bushings and sway bar end links from 034 Motorsport, the bigger RS4 Hotchkis rear sway bar, all the 034 Motorsport solid drivetrain mounts (snub cage, engine, transmission, front/rear subframe, and rear diff carrier bushing inserts), and the Apikol mounts (snub and rear diff) so don't let people sway you into thinking that you have to get a RS4 to run better parts there.
Where you really start to see a difference between these cars is with the engine components. We are going to assume that you are smart enough to do a PPI with a compression test before buying regardless of S4 or RS4 to avoid any scored cylinder wall issues.
RS4s will need a carbon clean, injector servicing, and a PCV valve replacement couldn't hurt. This isn't that bad in the grand scheme of things.
S4s will need a timing service if it has never been done and maybe valve stem seals replaced depending on how bad it is burning oil. The quotes from a shop to do this will scare most people into walking away from the car but you can DIY for a lot cheaper if you have time and the patience to do it.
I strongly recommend that if you do get a S4 or RS4 that you try to modify it to keep as much heat out of the engine as possible. Do things like install intake manifold spacers, throttle body or oil separator heater bypass, catless or piggy exhaust, etc. I see that you live in Cali but most inspectors are not going to notice intake spacers or the heater bypasses. The exhaust might be harder to work around though so that is why I mentioned piggies. These V8s run extremely hot and that heat really kills not only the performance but also the engine life.
Finally after that point both are on the same chassis and will have the similar weaknesses like curb weight and understeer that has already been mentioned. You can try to correct that with a bigger rear sway bar and/or coilovers but I have found that a front LSD really helped with understeer for me. However these are still big heavy cars that can be a lot of fun. Nothing makes me smile more than blowing away an older Turbo, GT3, etc Porsche. Can't do that with a newer Turbo or GT2, yet.