Looks like you got some good answers to your good question. As far as I know The First Parish Church on Congress Street is the oldest church in Portland. It’s at 425 Congress Street. I’ve been there several times. It definitely has a nice feeling about it. Cool balcony seating looking down into the church and odd details.

Have a great trip!

(And yes, definitely visit Two Lights State Park in Cape Elizabeth and Fort Williams in Cape Elizabeth too. )