The Black Cats have made a disappointing start to the 2014-15 campaign and are currently just a place above the Premier League relegation zone.
Gus Poyet's side have also been knocked out of the Capital One Cup after losing to Stoke City at the Stadium of Light on Tuesday.
Sunderland took the lead through Altidore in the 16th minute but conceded twice to suffer a 2-1 defeat in the third-round tie, leading the 24-year-old USA international to warn against any more slip-ups.
"I didn't think we deserved to lose," he told theSunderland Echo. "I don't think we played that bad.
"But this has haunted us the whole of last season and a bit this year. Games we should win, we're not winning.
"It's something that we need to address now or else it's inevitable that what happened last season - and the pressure - will come back.
"If it's close and you're going to lose the game, you want to look back and be applauding the other team for a great strike.
"We're still losing games where we're leaving ourselves short. So, we deserve better and it's not saying just 'this guy' is playing bad. We need to be collectively better.
"We understand that and we've got home games coming up where we've got the chance to do that."