Between 1808 and 1811 a fort was constructed on the rocks off the tip of Manhattan Island. Originally named "The Southwest Battery," it was renamed Castle Clinton in 1817 in honor of DeWitt Clinton, Mayor of New York City.

The army vacated the fort in 1821 and the structure was deeded to New York City in 1823. In the summer of 1824, a new restaurant and entertainment center opened at the site, now called Castle Garden. A roof was added in the 1840s and Castle Garden served as an opera house and theater until 1854. On August 3, 1855, Castle Garden, now leased to New York State, opened as an immigrant landing depot.

During the next 34 years, over 8 million people entered the United States through Castle Garden, until it was closed on April 18, 1890. On April 19, 1890 a temporary center was set up in the old Barge Office near the Customhouse on the southeast foot of Manhattan and used until January 1, 1892 when Ellis Island opened.

After immigration functions had been transferred to the Barge Office and Ellis Island, Castle Garden was altered once again and reopened as the New York City Aquarium on December 10, 1896. It was one of the city's most popular attractions until it closed in 1941. It was reopened later as Castle Clinton National Monument.