It is too one sided. I would allow Ray Rice for Tom Brady straight up. In some leagues all 3 players have been drafted in the first round. If the guy trading for Brady is just a homer and wants him at all costs just so he can enjoy his fantasy season tell him to go back and try to balance out that trade and then you will allow it.
But why should other owners be in charge of a 1-sided trade? What if Ray Rice/Graham both have bad years/get hurt and Brady is the best qb by a mile? If Im the owner that was dealing for Brady, I would want my money back. People have to make their own decisions and take on risks.