Former federal treasurer and prominent Jewish Victorian Josh Frydenberg has voiced his displeasure with the rally, saying it is unthinkable that the government would not intervene.