Most cavity wall insulation is perfectly safe – but it depends on the material used and the quality of the workmanship.
Modern retrofit cavity wall insulation methods pose no threat to those living in the home, but if your walls were insulated decades ago, you might want to check what was used.