It appears that the only way to deselect a reference is to click in the white space below the references. If you have too many references, that white space needs to be reattained by stretching out the dialog box. I propose that either a single line of white space is always present, thereby the dialog box may not always be the same size when loaded or a new button to deselect the references is added.