In every province the district (Amphoe, อำเภอ) which contains the province hall is named same as the province, with the word Mueang (เมือง) added - with the exception of Ayutthaya province, where both the province and the district are named Phra Nakhon Si Ayutthaya (พระนครศรีอยุธยา). And of course Bangkok, which is not a province in the normal sense anyway. On the other hand, there are four districts which have that work in their name without being the capital district.
Now, these weren't the only exceptions, before the provinces Thonburi and Phra Nakhon were merged in 1972, the two capital district had no Mueang in the name either. To find more, one has to go much further into the history of this naming scheme. It was in fact adopted in 1938, with an announcement in the Royal Gazette renaming several districts. To be exact, 43 district became Mueang districts, whereas in 11 cases the word Mueang was stripped from districts which were not capital district. For example, the district Ban Don (อำเภอบ้านดอน) became Mueang Surat Thani (อำเภอเมืองสุราษฎร์ธานี), whereas Mueang Chaiya became simply Chaiya (อำเภอไชยา). Before 1938, the naming scheme was different - only those districts which date back to the provinces before the thesaphiban reforms 1892-1915 had a name with Mueang, and all other districts were named after the central subdistrict (Tambon).
Since 1938, all newly established provinces had their central district renamed accordingly, also for those provinces temporarily abolished in 1943 the Mueang was removed. There are however two further exceptions, of the four provinces annexed during World War II both Lan Chang and Phibun Songkhram had no district named Mueang.
The four district which now have a name containing Mueang without being a capital district were all created relatively recently, when apparently the fixed naming scheme was not followed that strictly anymore. In detail these are Mueang Suang (1973), Mueang Pan (1981), Mueang Chan (1992) and Mueang Yang (1995).