When my wife and I go to a social dance, it means we are going for pleasure and recreation, and quite likely would dance with other friends, etc., and spend little or no time worrying about tempo or whether we did a ronde or a shadow twinkles properly.
If we were going to a setting in which very good dancers congregated and the music was strict tempo, we would consider that more along the lines of ballroom dancing because we would be working on technique, studying other couples and trying to do the dances as well as possible.
As Terence eludes to, you could say we are doing ballroom dancing at both events, and you could also say both are really social dancing because we're not really competing or dancing for an audience. But that's basically how we view the difference.
For an historical definition, I think the onset of the term social dancing came when people started "slow dancing" with those of the opposite sex. Prior to that, most primitive dancing was done by groups of the same sex -- mostly tribal rituals and such.