I visited Victoria as a weekend trip out of Seattle, which was enough time to see the city, hit several museums, and bus out to the Butchart Gardens (which are incredible). I don’t know that I’d have found that much to do staying there longer, unless I had friends in the area to hook up with, but a weekend was about the right length of time.