Two words: Sample size.
Plus, do you really think he'd be pitched the same way if he was they looked at him as a legitimate threat? There's a reason nobody does this and a reason he's a pitcher and not a hitter.
Fuck, I bet guys like Ceasar Izturis or Jose Vidro would be more productive DH's and you don't see anybody (aside from the retard brigade up in Seattle) attempting that, do you?