Preheat the oven to 375°F (190°C). Melt ¼ cup of butter in an 8-inch square pan and sprinkle with packed light brown sugar and a dash of ground nutmeg.
Halve, peel, and pit the fresh peaches and arrange them cut-side down in the pan.
Cream together ½ cup softened butter and 1 cup white sugar until light and fluffy. Beat in the large egg.
In a separate bowl, combine 1 ½ cups all-purpose flour, 2 teaspoons baking powder, and a pinch of salt. Gradually add this dry mixture to the butter and sugar mixture, alternating with ½ cup of milk.
Spread the batter over the arranged peaches in the pan. Smooth the top and bake for 35-40 minutes until golden brown.
Let the cake stand for about 5 minutes, then run a knife around the edges and flip it onto a platter.