It comes a day after he and President Donald Trump spoke over the phone for the first time since the two countries missed a self-imposed deadline to reach a trade agreement.
Canada had placed a 25% levy on about C$30bn (£16bn; $21.7bn) worth of US goods on an array of products, including orange juice and washing machines.
The tax hike was in retaliation to US tariffs on Canada, which as of August are valued at 35% on all goods not compliant with the countries' existing free trade deal.
Carney said Canada will now match the US by ending its tariffs on goods compliant with the US-Mexico-Canada free trade agreement (USMCA). He said that would "re-establish free trade for the vast majority" of goods that move between the two countries.
The decision will go into effect on 1 September, Carney said.
In a statement to the BBC's US news partner CBS, the White House said it welcomes Canada's move, adding that it is "long overdue" and that the US looks forward to continuing discussions with its northern neighbour about trade and national security.
Trump later told reporters on Friday that he and Carney will speak again over the phone soon.
Canada is one of many countries tariffed by the US as part of Trump's global trade strategy, but it is one of only two countries - along with China - that have placed retaliatory levies on American goods in response.
Polling shows the majority of Canadians support retaliatory tariffs on the US.
Carney, who was elected in an April general election, campaigned on an aggressive "elbows up" approach to negotiating with Trump, referencing a popular ice hockey term.
Conservative leader Pierre Poilievre criticised Carney for dropping the counter-tariffs, telling reporters that the prime minister's "elbows have mysteriously gone missing".
"It is yet another capitulation and climb down by Mark Carney," Poilievre said.
Asked by reporters about whether Canada was softening its approach, Carney said it has a better tariff deal with the US than many other countries because of the free trade carve-out.
That puts the actual tariff rate on Canadian goods at about 5.6%, much lower than the average of around 16% for other countries, he said.
"As we work to address outstanding trade issues with the US, it's important we do everything we can to preserve this unique advantage for Canadian workers and businesses," he said.