Writing the code in a natural and readable way should be number one.

I mean, even there it depends what you’re doing. A small matrix multiplication library should be fast even if it makes the code uglier. For most coders you’re right, though.