The institution, commemorating its 125th anniversary, has had a total of four previous headquarters before the current one. These historic locations include streets such as Veedor, San Miguel, Benjumeda, and Cervantes. Since June 2024, the College of Physicians has been operating from the modern Glorieta building.

The mandatory professional association for physicians in Spain was established by Royal Decree on April 12, 1898, leading to the creation of Professional Colleges nationwide. The aim was to ensure the competence and ethical standards of medical professionals for the public. However, in the early days, many physicians, including those in the province of Cádiz, resisted this institutional formalization.

Subsequent orders, such as those from June and November 1900, emphasized the necessity of establishing Colleges of Physicians and outlined their statutes. In November 1900, the Official Gazette of the Province of Cádiz published the list of 214 physicians summoned for elections. In February 1901, the Civil Governor appointed an interim Board of Directors, chaired by Dr. Cayetano del Toro, tasked with establishing the College and organizing elections.

The first elections took place between March 1 and 4, 1901. The inaugural Board of Directors was sworn in on March 15, 1901, at Veedor street, number 7, the seat of the Royal Academy of Medicine. Dr. Cayetano del Toro offered his residence as the College's first headquarters, and the newspaper ‘Anales Médicos Gaditanos’ became its first publication.

In November 1919, under the leadership of Dr. Bartolomé Gómez Plana, efforts were made to secure a dedicated premises and a distinct publication. On December 16, 1936, the "National" Government dissolved the elected Board of Directors and imposed a "Gestora" (Management Committee) chaired by Dr. Victoriano Lenzano Meirás. This committee inaugurated a new headquarters on San Miguel street, number 7, on May 16, 1937.

The headquarters on San Miguel street became a hub of significant cultural activity, hosting lectures by prominent figures such as Gregorio Marañón, Carlos Jiménez Díaz, José María Pemán, and Nobel laureate Jacinto Benavente. The College remained there until 1961, when the new headquarters at Benjumeda street 7 was inaugurated, featuring a large auditorium and a residence for doctors and medical students.

In 2001, coinciding with its centenary, the headquarters were expanded by acquiring the house of Dr. Portela, accessed via Cervantes street. Finally, in June 2024, the College moved to its current headquarters in the Glorieta building, a modern, accessible, and eco-responsible facility. In addition to the central office, the College has expanded and improved delegations in Campo de Gibraltar and Jerez.