The first lodge established on the present site of Cape Coast Castle was built by Hendrik Caerloff for the Swedish Africa Company. Caerloff was a former employee of the Dutch West India Company who had risen to the rank of fiscal before employing himself with the latter company established by Louis de Geer. As a former high-ranking officer of the Dutch, Caerloff had the friendly relations with the local chiefs necessary to establish a trading post. In 1650, Caerloff succeeded in getting the permission of the King of Fetu to establish a fort at Cabo Corso (meaning “short cape” in Portuguese, later corrupted to English Cape Coast). The first timber lodge was erected at the site in 1653 and named Carolusborg after King Charles X of Sweden.
Caerloff returned to Europe in 1655, leaving a Swede by the name of Johann Philipp von Krusenstjerna in charge of Carolusborg. Louis de Geer had however died in the meantime, and Caerloff got himself involved in a serious dispute with his heirs. In Amsterdam, he convinced merchants to give a financial injection to the Danish West India Company, for which he set sail to the Gold Coast in 1657, with the goal in mind to capture for Denmark the Swedish lodges and forts he had established himself. With the help of the Dutch, Caerloff succeeded in driving the Swedes out, leaving the Gold Coast on the captured ship Stockholm Slott, and with Von Krusenstjerna on board as a prisoner.
Caerloff had left Samuel Smit, also a former employee of the Dutch West India Company, in charge of Carolusborg. The Dutch were able to convince Smit in 1659 of the rumour that Denmark had been conquered by Sweden, upon which Smit rejoined the Dutch West India Company, handing over all Danish possessions to the Dutch. The King of Fetu was displeased with this, however, and prevented the Dutch from taking possession of the fort. A year later, the King decided to sell it to the Swedes. After the King died in 1663, the Dutch were finally able to occupy the fort.
The Danes had in the meantime established another fort, Fort Frederiksborg (1661), just a few hundred metres east from Carolusborg. Although situated perfectly to launch an attack on Carolusborg, the English capture of Carolusborg (1664) during the prelude to the Second Anglo-Dutch War, prevented the Danes from challenging them; the English had reinforced the fort, which they named Cape Coast Castle, to such an extent that even Dutch Admiral Michiel de Ruyter deemed it impossible to conquer. As the Dutch had captured the former English headquarters at Kormantin and had rebuilt it as Fort Amsterdam, Cape Coast became the new capital of the English possessions on the Gold Coast.