Never lose yourself in someone else's anger and pain. Never give in to keep the peace in your life. At the end of the day you will have pandered to everyone else and lost yourself in the process.
Have the strength to realise your own worth and say "Stop". Don't feel guilty for standing up for yourself and saying what you think or feel or need.
You are not wrong - everyone has an opinion and there is nothing wrong with yours. Be confident in yourself and believe in yourself. And very often you will find that others will too.
"Trying to meet everyone's expectations is like trying to drain the ocean with a teaspoon" Joyce Meyer
So true....you simply can't please everyone. So set your own goals, your own limits and work towards those. Everything else will fall in to place.