Wash and slice the peaches into 1/2-inch pieces, then toss with 2 tablespoons granulated sugar and 1 tablespoon lemon juice; let sit 15–30 minutes at room temperature to macerate and develop syrup.

Beat the softened cream cheese with 1/3 cup powdered sugar and 1 teaspoon vanilla until smooth; in a chilled bowl whip 2 cups heavy cream with the remaining powdered sugar to medium peaks, then fold one-third of whipped cream into the cream cheese to loosen, followed by the remaining whipped cream until airy and uniform.

Cube the yellow cake into roughly 1-inch pieces and place half the cubes in the bottom of a 9×13-inch dish. Spread half the mousse over the cake, then scatter half the macerated peaches and a few tablespoons of reserved syrup. Repeat with remaining cake, mousse, and peaches, finishing with a light mousse layer on top.

Chill the assembled trifle at least 2 hours (up to 6 hours) before serving so layers settle and flavors meld.

Just before serving whip 1/2 cup heavy cream with 1 tablespoon powdered sugar to soft peaks and spoon or pipe on top, then garnish with peach slices, toasted almonds, or mint if using, and serve.