When notifying a collaborator you will see an Invalid Notification Group Setting error if the collaborator account you are trying to notify has no users in it with the collaboration roll added.
For example if you add an account as an uploader, the account being added must have at least one user in it with the uploader role.
If there is a user with the uploader role the user will be notified that this is a file to upload. If there is no user with the uploader role you will see the above error.
In case of this error notification please contact with Customer Success team who will be able to help.
For more information on Collaborator roles please see here.