Preheat oven to 350°F (177°C).
Prepare the topping: Combine the butter and brown sugar in a small saucepan over medium heat. Whisk occasionally until the butter has melted.
Cook, whisking constantly, for 1 minute as the mixture thickens. Remove from the heat and whisk in the cinnamon and vanilla extract.
Pour the topping into an ungreased 9×2-inch pie dish or round cake pan. Arrange the apple slices neatly on top, overlapping where necessary.
Place the pan in the refrigerator for a few minutes as you prepare the cake batter.
Make the cake batter: Whisk the flour, baking powder, cinnamon, nutmeg, and salt together. Set aside.
Using a handheld or stand mixer fitted with a paddle or whisk attachment, beat the butter on high speed until smooth and creamy, about 1 minute.
Add both sugars and beat on high speed until creamed together, about 1 minute. Scrape down the sides and up the bottom of the bowl with a rubber spatula as needed.
On high speed, beat in the eggs and vanilla extract until combined. Scrape down the sides and up the bottom of the bowl as needed.
Pour the dry ingredients into the wet ingredients. Turn the mixer onto low speed and, as the mixer runs, slowly pour in the milk. Beat on low speed just until all ingredients are combined. Do not over-mix.
Whisk the batter by hand if needed to make sure there are no lumps at the bottom of the bowl.
Remove the topping from the refrigerator. Pour and spread the cake batter evenly over the topping.
Bake for 40-46 minutes, tenting foil on top of the cake halfway through bake time to prevent the top from over-browning before the center fully cooks.
Remove the cake from the oven and cool on a wire rack for 15 minutes. Invert the slightly cooled cake onto a cake stand or serving plate.
Cool the cake completely at room temperature before slicing and serving.
Cover leftover slices and store for up to 3 days in the refrigerator or 3 months in the freezer. Thaw at room temperature.