The "secret lies with Charlotte" refers to the clue that Thomas Gates found and passed down leading Benjamin Franklin Gates to find a Meerschaum Pipe on the Charlotte that is the key to the treasure.
History[]
The riddle was given to Thomas before Charles Carroll died. It can be seen in the album when FBI Agent Peter Sadusky pick up the album and open it.