Way more time was spent on the grounds, but also in the very large and nice visitor's center. I'd say it's definitely still worth it even if you can't go in the house. The house tours are timed (at least, they used to be) so you probably only spent 20 mins in the house vs hours on other things.