Lightly grease a 9-inch springform pan.
In a medium bowl, whisk together the flour, baking powder, baking soda, salt, apple pie spice, cinnamon, ginger, and nutmeg.
In a large mixing bowl fitted with the whisk attachment, beat the eggs and granulated sugar at high speed until pale and fluffy, about 5 minutes.
Lower the speed to medium-high, and drizzle in the olive oil slowly, beating until incorporated.
With the mixer on low, add half the flour mixture until incorporated.
Add the buttermilk and vanilla, then the remaining flour mixture, until just blended.
Pour batter into prepared pan.
Bake at 325 for 40-45 minutes, or until the top is golden and a toothpick inserted near the center comes out clean.
Let the cake cool in pan on a wire rack for 20 minutes, then carefully remove the pan sides and allow cake to cool completely.
Dust with powdered sugar, and top with caramelized apples.