Bullet Weight
I am no expert in Black Powder weapons (or ballistics for that matter ) , but a low velocity doesn't necessarily mean low damage. You also have to factor in the weight of the bullet.
Also, realism is nice, but if you find that it would ruin your idea for a campaign, than a little exageration might be in order...