Suggestion, you might want to start with a cheap (non angular contact bearing in that location, yes you might have a few thousands displacement, then when everything is working install the good bearing to get rid of that slop. I don't see why the retainer would need to be so tight, you can...