How often do you get to spend the night at a fascinating historical site? Usually, at the end of each day, tourists are boarded onto buses, the gates are locked tightly, and an eerie silence falls over the grounds. Only the cleaning crew and the occasional bird or squirrel gets to see the usually crowded spot after dark. But not on Île de Gorée.

Gorée Island, off Senegal’s Cap-Vert peninsula, was one of the first African places settled by Europeans. The port changed hands—between the Portuguese and the Dutch, the French and the British—numerous times. But one of its main exports—slaves—remained the same until France abolished slavery. The slaves houses and the government buildings still stand. They’re now part of the UNESCO-designated island.

From Dakar, Senegal’s vibrant capital, Gorée Island is just a 30-minute boat ride away. The ferry arrives on a small beach lined with seafood restaurants. Narrow, car-free alleys behind them lead to stone buildings with terracotta roofs and dangling bougainvillea. Colonial brick buildings feature wrought-iron balconies. Art workshops fill the fortified Castle on the island’s southern tip. While former slave houses, including the House of Slaves with its haunting Door of No Return, are now memorials and museums.

You spend the morning exploring the artifacts at the House of Slaves and the IFAN Historical Museum in the former Fort d’Estrées. But when the midday heat and crowds arrive, you decide to take a break. Villa Castel, a small hotel near the 19th-century Church of St. Charles Borromeo, offers a respite. After relaxing in the flower-filled garden and terrace, find Chevalier de Boufflers, a garden restaurant, for a cold Gazelle Bière and an early dinner. The return ferries start to fill up as you eat a plate of rice and freshly caught fish. By the time the sun starts to set, the island is quiet again. From the old ramparts, you stare out at the endless Atlantic Ocean and picture those who must have hopelessly done the same thing centuries ago. Though you might not get much sleep, this is sure to be a night you’ll never forget.