The men, calling themselves working class patriots, heckled the Iranian-born senator at a Melbourne bar on Wednesday night with “You terrorist. You little monkey” and demanded to know if he was a Muslim.

“They are the sickening face of white nationalists in this country. What’s happening is our politics is heading into a very, very ugly place,” Dastyari told the Nine Network today.

“It makes me feel small, makes me feel horrible, it makes you feel kind of terrible and that’s what they are designed to do.”

However, One Nation rejected Dastyari’s portrayal of the incident.

“Sam Dastyari gets heckled because he’s a wanker,” Pauline Hanson’s chief of staff James Ashby told AAP on Thursday. “And not because he’s a Muslim.”

One Nation leader Pauline Hanson later reiterated Ashby’s comments, saying Dastyari, who blamed the incident on the rise of the “One Nation right”, was attacked because of his character, not his religion.

“It’s got nothing to do about him being a Muslim, and he’s even said he’s a non-practicing Muslim, so he’s played up this whole thing,” she said.

She said she didn’t condone the abuse, but noted it wasn’t the first time Dastyari had been attacked in public.

Dastyari is considering a legal response to the incident, which may have infringed race discrimination laws.

Islamophobia and racism were getting worse in Australia on both the left and right of politics, he said.

“I worry about all the people out there that have to put up with all this kind of abuse who don’t have the structures that someone like I’m lucky enough to have.”

Malcolm Turnbull said there was no place for racial vilification in Australia.

“That is because our society is built on a foundation of mutual respect. It should have zero tolerance for racist abuse like this,” the prime minister told Seven.

Opposition Leader Bill Shorten said he was disgusted by the “ugly” incident.

“What is this country coming to when you cannot go out for a meal without being abused by racist idiots,” he told Seven.

The men from the right wing group Patriot Blue filmed the incident and posted it on their Facebook page.

When he called them racists, they replied with: “What race is Islam?”

In a Q&A session at the pub with fellow Labor MP Tim Watts, Dastyari said he was regularly subjected to similar incidents because he was a Muslim.

“All of this is the rise of the radical right in this country, it is the rise of One Nation right,” Dastyari said.

“These are people who feel incredibly empowered because of what Pauline Hanson has done for them.

“You dance so far to the right that it gives those a little bit further out a sense of entitlement.”

Far right campaigner Neil Erikson was wearing a high-vis Toll shirt in the video, prompting Dastyari to complain he hadn’t heard from the company after going public about the incident.

“I am amazed that Toll Holdings, a major Australian company, have these people who are proudly wearing their colours, proudly making their own videos of racist and Islamophobic taunts and the company still hasn’t called,” he told Seven News.

A Toll spokesperson said the actions in the video “in no way reflect our beliefs and values as a company”.

“We are currently investigating the matter and whether any of the individuals involved are currently employed by Toll.

“At Toll, we celebrate our diverse workforce and we expect all of our people to behave in a manner that is consistent with our values and stated policies at all times.”

Erikson told Melbourne radio 3AW he wasn’t employed by Toll and wore the shirt to his “other job”.

But he also said he had the right to free speech.

“If he wants to go for Toll and try and hurt me financially he can go for it … I believe that everyone has a right to freedom of speech,” Erikson told ABC radio.

He said Dastyari gives as good as he gets.

“Sam’s a strong bloke he’s got a thick skin, he’s been in politics for a long time, ” Erikson said.

“He called us rednecks, which is a racist term in fact, so look, he gives as good as he gets.

“I think he’s playing the victim a little bit.”

– with AAP