The Referred By field on the Contact record displays a list of active & inactive Contacts by design because any contact can be selected. A column has been added to the lookup to display contact status.