FA Chairman: 'There are harder groups'