I completely agree with you. It makes no sense. The men must have felt a sense of entitlement to sleep with whomever they pleased. They would have none of the boundaries that they would have with their wives or other white women since the slave women couldn't object to anything. When there were children as a result, the men seemed to be able to disassociate themselves from their children.
I guess in pursuit of sex these men chose to ignore the health risks. In The Healing, the "master" even denied what killed his child because he considered it a "slave disease".
Another double standard that makes no sense to me is that white families let black slaves raise their children, but still considered the blacks to be dirty and otherwise undesirable. You would think they'd want to isolate their children from the slaves. This was also demonstrated in more contemporary books like The Help or the comparable Indian novel, The Space Between Us.