The court ruling from Circuit Judges Barrington Daniels Parker Jr and Denny Chin said that NFL Commissioner Roger Goodell had 'properly exercised his broad discretion under the collective bargaining agreement' when he gave Brady the four-game ban.
It went on to say that Goodell's decision was 'properly grounded in that agreement and did not deprive Brady of fundamental fairness.'
Brady could appeal to the full Second Circuit or even the Supreme Court, but both would be unlikely to conclude in his favor, The New York Times said.
The suspension originally came in response to claims that the Patriots had deflated balls in an AFC Championship game against the Colts, making them easier to handle.
Since each team provides its own balls and teams rarely touch those of their opponents, this can give a team a big advantage.
The Patriots ended up winning the game 45-7 and going on to take the Super Bowl.
In May 2015 the League ruled that tampering had taken place by two Patriots employees and that it was 'more probable than not that Brady was at least generally aware of the inappropriate activities.'
Goodell then served Brady with the suspension and upheld his own decision after Brady appealed.
But Brady wasn't going down without a fight, and he and the Patriots took the case to court, getting the suspension nullified a week before the 2015-2016 season began by Manhattan Judge Richard Berman in September last year.
The NFL then appealed, resulting in Monday's Second Circuit court verdict.
The original NFL ruling also said that the Patriots must pay $1million fine and would lose its first round draft picks in 2016 and fourth-round draft picks in 2017.
The team ultimately accepted those punishments.