Is it freedom or restriction of speech?

I have a dilemma! I believe in expressing your ideas and opinions (as if you could not tell!). I believe in humor and quick wise cracks that are NOT hurtful or intentional. I make jokes about my birth place, religion, looks, height, skills and heritage all the time. and so I suppose it should be, but where do we draw the line?

I believe that no one takes anything to heart unless there is some truth to it.
I believe that no matter what I say or do not say, I will continue to believe what I believe and one can not change my mind.

With that being said,

I also believe that insecure people make ugly comments with intent to harm or attract attention.
I also believe that those who make nasty comments about others, despite what they make think or feel make bigger fools of themselves than others.

And here is the dilemma, our constitutional rights dictate the “Freedom of Speech” and at the same time imposes the restriction:

According to the Freedom Forum Organization, legal systems, and society at large, recognize limits on the freedom of speech, particularly when freedom of speech conflicts with other values or rights.[18] Limitations to freedom of speech may follow the "harm principle" or the "offense principle…(Wikepedia)

Then how is it that even with these restrictions some people get away with appalling remarks, yet some get scrutinized. Why are some people taken more seriously than others?
How is it that we are sympathetic to one group and negligent toward the next? Do we have a selective freedom of speech right? Is hatred and jealousy interfering with our freedom of speech? Is it again all about me? What works for me is ok and the rest does not concern me?

The way I see it, every year we get more advanced, whatever that means, we lose a bit of our freedom as well and we do not even realize it. I love this age of technology and freedom from menial tasks, but I also loved when I could speak freely without thinking ten times before spitting out my words, because no one analyzed it and no one took it more than face value. If I can not speak out for what I think and what I believe, then the constitution and the basic human rights bill is very confusing to me.

I hope that someone will shed some light on these concepts for me!