It wasn't this component's fault. It was the 3.3V line connected to the AREF pin of the Arduino. I didn't have the AREF pin set to EXTERNAL in the sketch, and so it was at 5V by default and actually driving my 3.3V bus up to 5V. The problem didn't occur until I ran a sketch that required this level converter, thus implicating the level converter, but the level converter is innocent.
This is a great little level converter. I'd pay twice the cost to have MOSFETs on the RX channels too. There's are also plenty of level shifter components that can handle many more channels; I would definitely pay for one of those on a breakout!