However, his press secretary, Victor Harju, told CNN not to put too much stock in those figures. He called them vague, hypothetical estimates based on last year's numbers.
"It's that 163,00 people were seeking asylum in Sweden last year and, due to this increase, there will also be an increase in how many ... don't stay," he said.
On average, about 45% of those applications get rejected, Harju said, which adds up to about 80,000, the figure that Interior Minister Anders Ygeman cited.
That's a higher rate of acceptance than the European Union's in which 45% of those applying for asylum for the first time (so-called first instance asylum decisions) received "positive outcomes."
Each individual application would be considered on its merits, the press spokesman said.
"We are not saying how many people should be deported from Sweden because everyone that is going through is individualized," he told CNN.
Charter flights for deportations considered
Harju said Sweden was already considering charter flights to deport migrants whose bids for asylum were denied, saying the government was investigating how such expenses should be accounted for in the budget.
Sweden also was considering coordinating charter flights with Germany for this purpose, he said.
Merkel calls for migrant quotas for EU states to combat crisis
The Swedish Migration Agency told CNN in an email that the charter flights would carry migrants who accepted the decision to deport them. In cases where an individual did not accept the decision to be deported, the case would be referred to police, the agency said.
Sweden, home to about 9.5 million people, is one of the most desirable final destinations for migrants entering Europe, and has proven one of the most hospitable host nations, taking in the highest number of migrants per capita.
Migrant crisis: Danish police won't stop refugees heading to Sweden
But the ever-growing numbers of asylum-seekers, combined with high-profile crimes blamed on migrants in Sweden and elsewhere, have led to the introduction of more hard-line policies toward migrants.
In Sweden, an anti-immigration party, the Sweden Democrats, has soared in popularity, polling at about 20% in recent polls.
Markus Wiechel, a member of parliament who is the party's migration spokesman, told CNN it was impossible to integrate tens of thousands of newly arrived migrants into Swedish society in such a short time.
More fatalities on journey
Refugees are still heading toward Europe in large numbers, with fatalities commonplace on the perilous journey.
At least 24 people -- including 10 children -- drowned in the Mediterranean Sea between Turkey and Greece after their boat capsized Wednesday evening, a Greek coast guard representative told CNN.
Forty-five people were on the boat when it sank.
Meanwhile, anti-immigration sentiment is on the rise throughout Europe, spurred in part by reports of mass sex assaults that migrants may have carried out on New Year's Eve in Cologne and other German cities.
UK to take more unaccompanied children
British Prime Minister David Cameron came under fire Wednesday for his comments about "a bunch of migrants" during a session of Parliament.
On Thursday, the UK Home Office announced that it will take in more unaccompanied child refugees from Syria and other conflict regions, though it's not clear how many.
It will be up to the United Nations to identify "several hundred children" to be resettled in the UK in addition to the country's pre-existing commitment to settle 20,000 refugees from camps near Syria.
CNN's Lindsay Isaac and Tim Hume contributed to this report.
