Rivera gave up back-to-back homeruns to Tampa Bay Rays hitters Carl Crawford and Evan Longoria.
The Rays beat the Yankees 8-6 and the Yankees have their worst losing streak (five games) under manager Joe Girardi.
According to Girardi, Rivera, who had off-season shoulder surgery, has a "tired shoulder," which is affecting his velocity.
"I've been giving my best," Rivera said. "I haven't been 100% yet, but that's no excuse. I won't say that was the reason I lost the game. I felt great. The strength, it will be there. That's not my concern at all."The Yankees are 13-15 and 5 1/2 games behind division-leading Toronto.
"No one's going to feel sorry for us," Girardi said. "It's our job to turn it around. ...Right now, we're in a little rut and we have to get out of it."