It's a flight or fight response because you feel threatened by the audience. It's a bit like swimming though, it takes practice and a lot of tenancity if you're not naturally extroverted. I had similar issues like you but had some training on public speaking at work and learned to distract myself from negative emotion by intellectualising the problem.
Knowing your audience and feeling comfortable with them really makes a difference. I tend to arrive early to an event and scrutinise everyone as they enter, usually from afar. Those that make eye contact with me as they come in are top of my list as people to make eye contact with as I'm talking as they typically are alpha types and leaders. So if I win them over by acknowledging them and playing to them, there's a good chance I can win the room. Passion and energy also counts which usually isn't difficult to generate given the adrenaline. Preparation and knowing your subject is critical for confidence and, lastly, just accept that you will make mistakes. There's no avoiding it, just deal with them as graciously as you can.
Same applies to performing on stage but you have the luxury of not needing to speak as much (if at all). If all else fails, just fake it until you make it. Be the success you want to become.
Worst gig I ever had was depping for a blue band called Soul Kitchen in the place of Jim E Sims. I had two weeks prep time for 14 songs and one rehearsal while still holding down a day job. The blues songs weren't straight forward either due to sudden stops and starts, changes in time signatures and keys, extra sections like pre-choruses and I made a total hash of it. Literally a living hell on stage as I attempted to busk my way though the parts I was less familiar with. The band were very gracious about it but I swore blind that this would be the last time I ever depped and it was.
Best gig I ever had must have been with my function band. There was one night when the on stage mix was perfect. I knew all the songs and liked them, the audience was buzzing and we were on fire. Stuff happening like that is what we all live for as musicians but experiences like that won't come without taking risks.