The medieval castle on the Schlossberg hill is presumed to have been built in the 11th century by the Counts of Lenzburg. It was inherited first by the Counts of Kyburg and later by the Counts of Habsburg. Subsequently, it served as the official residence of the governors of the Habsburg Swabian territories. When the Aargau was conquered by the Swiss in 1415, Stein Castle was captured and destroyed after a fierce battle.

Old illustrations show that the castle once consisted of a residential tower with an adjoining palace, a triangular courtyard with a watchtower and a castle chapel. The watchtower and chapel survived the castle's destruction. Thanks to various conservation efforts, they can still be visited today.

Like the ruins of the castle, the silhouette of the still extant city wall and rounded wall is a symbolic feature of the cityscape. The wall was originally attached to the castle chapel and, following the ridge of the hill, stretched as far as the northern wall of the houses in the old town.

From the Schlossbergplatz, passing through the City Tower's archway, a steep staircase leads up to Stein Castle on the right-hand side.