This can be done either by disabling just-in-time provisioning (a user must then be invited before their SAML login will work) or, if JIT provisioning is enabled, by controlling access to the SAML application through your SAML provider through your own directory membership.