That’s definitely what provides this place with so much of its allure… which means you really need to go at a time when the grounds will look nice. I’m sure spring is gorgeous, as Japan has a reputation for that, and I know that summer will be verdant and wonderful.
But if you can swing it, try to get there in autumn. If you have anyone in Japan and have flexibility in your schedule, ask what weekend the locals think the momiji leaves will turn this year, and go then! But err on the side of caution… it’s better to see mostly-green trees flecked with red and gold than to see scratchy bare branches. Blinding red in every direction is the sense-numbing sweet spot you’re looking for, though!