I took the ferry there today and it was definitely a good place to spend an afternoon. The historic district on the island is run by the National Park Service. Park rangers give free tours of the island and are very knowledgeable about its history. Interestingly, no one is allow to live on the island on a permanent basis which means all the people on it are either tourists or work there.

I would recommend taking one of the park ranger led tours (about 45 minutes long) and then find a spot under a shady tree to have a picnic or take a nap. There is a summer concert stage too on the weekends with live music to accompany any frisbee tossing you might want to do.