The first settlements in this area date back to approximately 5000 BC, possibly situated on the hill where the castle now stands. During this time Castro Marim was closer to the sea and most likely an island. For thousands of years it was a port for the ships sailing up the Guadiana River to the metal mines to the north.

Phoenicians and Romans where present here, and a Roman road connected this area with Lisbon, running in parallel to the Guadiana River and passing Alcoutim, Mértola and Beja.

During the Moorish occupation a fortified structure existed, that became the core of the present castle. Due to the strategic position of the town on border with the kingdom of Castile, Castro Marim became the headquarters of the Order of Christ after the Christian reconquest. When the order later was transferred to Tomar a long period of decreasing importance and declining population began.

The period of stagnation due to the cut off from the sea and its economic basis, fishing, salt production and boat building only ended recently by the new dynamics of the last decades.