Hey man, ive been in your shoes before and fought people with no skill and technique and in the end it all became very much like a brawl instead, lasting only as long as the opponent could stand. That was just silly.

The best way to get confidence for yourself is to join boxing, it did wonders for me and once i gained all the skills for offensive and defensive boxing, i wasn't scared to face those that had no prior knowledge in boxing.

Cheers