Earlier is definitely better. Young kids don't need much in the way of resources to properly learn. As they get older, having access to things like real science labs, computer labs (for those who don't have a mini data center at home like me), athletic coaches, shop classes, etc. is a real advantage.
My folks didn't home school but I went to small private schools (all we could afford) up until High School, then went public for this exact reason. I think homeschooling would be similiar.
