Preheat the oven to 325°F. Butter and flour a 9 X 5 X 3-inch loaf pan.
Cream the butter and sugar until light. Add the egg yolks, sour cream, lime juice and zest, and almond extract. Continue beating until fluffy. Sift together the dry ingredients and stir into the sour cream mixture. Beat the egg whites to soft peaks and gently fold into the batter.
Fill the pan three fourths full with batter and bake for 50 to 55 minutes, or until a cake tester comes out clean. Remove from pan and cool on a rack.