Preheat the oven to 350°F.
In a medium bowl, mix together graham crumbs, brown sugar, and melted butter. Press this mixture into a 9-inch deep dish pie pan.
Bake the graham crumb crust for 10 minutes.
While that is baking, prepare the key lime filling. Using an electric hand mixer, combine the Key lime juice, zest, sweetened condensed milk, sour cream, egg yolks and powdered sugar just until smooth and combined.
Pour the filling into the baked crust.
Bake at 350°F for about 15 minutes, until the edges are set and the center still jiggles slightly when the pie is gently moved. If needed, continue baking and check again after a few minutes.
Remove the pie from the oven and allow it to cool to room temperature. Then refrigerate the pie for at least another 2.5 to 3 hours.
Serve with whipped cream and lime zest or lime slices if desired.