A UN report says more than 6,000 Pakistani insurgents are hiding in Afghanistan, most belonging to the outlawed Pakistani Taliban group responsible for attacking Pakistani military and civilian targets.

The report released this week said the group, known as the Tehreek-e-Taliban (TTP), has linked up with the Afghan-based affiliate of the Islamic State group. Some of TTP’s members have even joined the IS affiliate, which has its headquarters in eastern Afghanistan.

The Afghan government did not respond Sunday to requests by The Associated Press for comment.

The report said IS in Afghanistan, known as IS in Khorasan province, has been hit hard by Afghan security forces as well as U.S. and NATO forces, and even on occasion by the Afghan Taliban. The report was prepared by the U.N. analytical and sanctions monitoring team, which tracks terrorist groups around the world.

The report estimated the membership of IS in Afghanistan at 2,200, and while its leadership has been depleted, IS still counts among its leaders a Syrian national Abu Said Mohammad al-Khorasani. The report also said the monitoring team had received information that two senior Islamic State commanders, Abu Qutaibah and Abu Hajar al-Iraqi, had recently arrived in Afghanistan from the Middle East.

“Although in territorial retreat, (the Islamic State) remains capable of carrying out high-profile attacks in various parts of the country, including Kabul. It also aims to attract Taliban fighters who oppose the agreement with the United States,” the report said, referring to a U.S. peace deal signed with the Taliban in February.

That deal was struck to allow the U.S. to end its 19-year involvement in Afghanistan, and calls on the Taliban to guarantee its territory will not be used by terrorist groups. The deal is also expected to guarantee the Taliban’s all-out participation in the fight against IS.

The second and perhaps most critical part of the agreement calls for talks between the Taliban and Kabul’s political leadership.

Late Saturday, the U.S. State Department issued a statement saying its peace envoy Zalmay Khalilzad was again shuttling through the region seeking to jump start those negotiations, which have been repeatedly postponed as both sides squabble over a prisoner release program.

The U.S.-backed deal calls for the Afghan government to release 5,000 Taliban prisoners and the Taliban to free 1,000 government and military personnel as a so-called good will gesture ahead of talks. Until now the government is refusing to release nearly 600 Taliban prisoners it calls high-profile criminals and has offered to free alternatives. The Taliban have refused.

“The parties are closer than ever to the start of intra-Afghan negotiations, the key next step to ending Afghanistan’s 40-year long war,” said the U.S. State Department statement. “Although significant progress has been made on prisoner exchanges, the issue requires additional effort to fully resolve.”

The Taliban’s political spokesman earlier this week said it was ready to hold talks with Kabul’s political leaders after the Islamic holiday of Eid al-Adha at the end of the month, providing the prisoner release is completed.

A big worry for Pakistan is the presence in Afghanistan of militants, particularly linked to the TTP or Jamaat-ul-Ahrar or Lashkar-e-Islam, as well as those with the Baluchistan Liberation Army, which has taken responsibility for high-profile attacks this month in the southern Sindh province as well as in southwestern Baluchistan Province. Several Pakistan military personnel have been killed this month in southwestern Baluchistan province in battle with insurgents.

The TTP took responsibility for one of the most horrific attacks in Pakistan in 2014, when a Pakistani army school was attacked and 140 were killed. Most were students, and some were as young as 5.

“The total number of Pakistani foreign terrorist fighters in Afghanistan, posing a threat to both countries, is estimated at between 6,000 and 6,500, most of them with TTP,” the report said.