I have SP set to only rename mutual children upon marriage, yet one of my sims married a guy with a kid and a dead mother, and the kid ended up with my sims name. What gives? She is no relation to this kid until marrying his father so I don't get why this happened. I do have it set to rename kids upon marriage, but I thought if I had the other marked then only kids they have together will be renamed?

That only applies to inactive sims not to sims that end up being part of the active household or when the sim is the mother as children will end up getting whatever last name the father gets.
The male getting the females last name when she's the one that initiates the proposal and marriage as well as because she's the active sim is one of the games quirks.

Rename Children on Marriage
Whether to rename any inactive children living with the mother, when the mother gets married. When disabled, the last names for the children are unaffected.
Default: True

Rename Only Mutual Children On Marriage
When enabled, only children blood related to both parents will be renamed to the new "Marriage Name"
Default: False