It turns out having your framework written in c and loaded as a php extension makes it fast as truck.

Other than that, smaller frameworks tend to have faster load times and smaller memory footprints.

Stay away from Zend frameworks.