I am firmly on the side of using a larger compressor for several reasons:
1. Large compressors tend to have a more predicable flow pattern than the small airbrush variety - no pulsing of airflow.
2. A good 150 PSI compressor with a 5-10 gallon tank has a large enough reserve for about 20 minutes of painting without having to recharge - silent running.
3. They can run a variety of tools from inflating balls and tires to an air ratchet or framing nailer. This will come in handy when working on cars, or building bench work.
4. There is not a large cost difference so why not get a tool that can do more?
As for noise... and they are noisy. I have worked on a few solutions. First, I always recharge the tanks when home alone. I can paint models while my wife is home with the reserve tank. I use an 8 gallon 150 PSI compressor and it can run about 30 minutes at 20 PSI before needing to be recharged. That's plenty of time to paint and clean.
I separate the compressor from my layout room.

Layout room air ports. Left is for airbrush and sand blasting work and features an addition flow valve and regulator. The right is what I call dirty air. It does not have a moisture trap and is meant for air tools as well as a pressure pot. Both outlets are mounted in a standard electrical box.

From inside the utility room. You can see the air lines with moisture trap.

I added a shut off valve in front of the moisture trap. I cut off the airbrush port when using the dirty air line at high PSI. This will reduce stress on the glass dome of the moisture trap. At the time of the photo, I had not yet attached the air line to the compressor. You can see the unfinished connection at the top.