No worries, Sam knows his brother, and his weakneses, so he sent Garth or some other hunter to get them moved right away... and then they find Dean and exorcise Abaddon and Lisa and Ben come out without a scratch, promise...
(and maybe Dean finds out that Ben is really his.)