Not going into the class issue (I'm well posh and never have had a problem at matches), certainly reading these boards for as long as I have, RL fans, despite claiming to watch [ithe greatest game[/i , seem almost determined not to enjoy it more than they have to.
If you work then you are more than welcome to attend games. However the unemployed shall be forever shunned and remain as outcasts on the terraces.
On a more serious note, most people attending games go with their mates/family and tend to just socialize within their own clique. I know of several people who go to rugby matches on their own or don't bother at all because they've got no one to stand with. More effort needs to be made to reach out to these lonely fans.
