In August, District Lines, the publication of the Historic District Council, published a fascinating summary of the history of Vinegar Hill, a neighborhood just north of Downtown Brooklyn and bordering Navy Yards. It touches on history relevant to Duffield Street in this paragraph:
No doubt at least partly because of its proximity to the East River, the neighborhood became a link in the Underground Railroad that provided safe transport to escaping slaves. There could have been another reason, too. The designation report cites 1820 United States Census records documenting 657 free Africans in Brooklyn, about nine percent of the population at the time, which could have made the neighborhood a link not only to the river but to destinations in Brooklyn. One nearby church became a station on the escape route, and a house on Hudson Avenue within the historic district still has a door in its subcellar leading to a tunnel to the waterfront, another link in the train.
The article is not online yet, but you can find other volumes at here. The Historic District Council has a brief description of Vinegar Hill here.