A cabinet minister has said his family received death threats and abuse over his involvement in a proposal for a national Holocaust memorial at Westminster.

An inquiry into the plan, which opened on Tuesday, was warned that the site, next to the Houses of Parliament, would become a target for terrorists.

It also heard that a nearby monument marking the abolition of slavery would be “engulfed” by the memorial to the 6 million Jews and others killed by the Nazis.

Ahead of the inquiry, Robert Jenrick, the communities secretary, revealed he had been subjected to “antisemitic smears” and threats over his role in the proposal.

On Monday, the high court ruled Jenrick had acted properly in regard to the planning application. A legal challenge brought against the minister by the London Historic Parks and Gardens Trust argued there had been a conflict of interest in the government’s handling of the planning application for the memorial, in Victoria Park Gardens next to Parliament.

Jenrick – who is married to the daughter of Holocaust survivors and whose children are being brought up as Jewish – recused himself from any decisions relating to the memorial after publicly backing the plans.

After the high court ruling, Jenrick told the Jewish Chronicle: “The behaviour of some of the opponents to the memorial has been shocking and disgraceful. The fact that I have been subjected to these smears, and my family to antisemitic abuse and death threats, only shows the paramount importance of the memorial.”

The minister had been given protection from counter-terrorism police after threats to burn down his home, the paper reported.

Jenrick also wrote on Twitter: “That I was subject to antisemitic smears for supporting it only confirms its paramount importance … This critical project is a national symbol of our determination to #neverforget.”

The inquiry into the memorial’s location opened on Tuesday after the application was “called in” by Esther McVey, the then housing minister, last November. The final decision on the application will be taken by her successor, Christopher Pincher.

The Holocaust memorial proposal has been controversial since it was proposed by David Cameron five years ago. It has been backed by Boris Johnson and Keir Starmer, along with the former prime ministers Theresa May, Gordon Brown, Tony Blair and John Major, more than 170 MPs and peers, and many faith leaders.

Opposition has come from some senior Jewish figures who have challenged the location of the memorial, and the Royal Parks, which said it would have a “significant harmful impact” on the area. Some local residents have expressed concerns the memorial will eat up much-valued green space.

Barrister Meyric Lewis, speaking at the inquiry on behalf of the Thorney Island Society, the London Gardens Trust and the campaign group Save Victoria Tower Gardens, said: “The proposals would turn a calm, green garden space into a cluttered, visually and physically congested and urbanised landscape.”

The Grade II*-listed Buxton slavery memorial would be “engulfed by the proposals, to the substantial detriment of its setting and significance”, Lewis added.

In a written submission, Lord Carlile, a former independent reviewer of terror laws, warned that a Holocaust memorial next to parliament would create a target for terrorists.

Carlile said: “From my extensive experience of observing, analysing and discussing terrorism issues with frontline practitioners, I have absolutely no doubt that the proposed site raises a clear – indeed self-evident – terrorism risk.

“I give this warning with regret, but with total conviction. This would be a threat to the public, and also a potential threat to parliament.”

Carlile told the BBC he had a strong interest in the issue. “Many of my close relatives were exterminated in the Holocaust. My half-sister’s mother was murdered in Auschwitz. I am absolutely determined that this should be remembered properly. I just feel that this isn’t the right place for it,” he said.

“This site would also be, potentially, a target for rightwing extremists. It seems to be foolish for these two iconic places to be on the same broad site.”

More than 40 Holocaust academics have raised concerns that the centre would portray Britain as “the ultimate saviour of the Jews”.

In a joint letter to the inquiry, they said: “Situating the UK Holocaust memorial next to the Houses of Parliament is likely to create a celebratory narrative of the British government’s responses to the Jewish catastrophe during the Nazi era and beyond.”

The project features 23 large bronze fin structures and an underground learning centre. Architects working on the project, led by Sir David Adjaye, said they were aiming to ensure 90% of the original park was retained.

The inquiry is expected to last five weeks.