A lot of what goes on in RPG combat has to do with chance, sometimes in percentages but most often in dice results. How easy or hard it is to achieve something is determined by the kind of dice you roll versus the dice rolled against you, and/or the required result.

For example, if you want to measure something on a result of 10 or better out of 12, you have to figure if you want to use 1d12 (one 12-sided die) or 2d6 (two six-sided dice). On the surface, it seems the same cause they both go to 12. But a 1d12 has a range of 1-12 with the same percentage chance of any result. The chance of getting a 10 or better is 3/12, or 25%. By contrast, a 2d6 has a different spread by combining two numbers 1-6. So the range is 2-12, and the chances of rolling a total of 10 or better is 6/36, or just 17%. So the odds are better to roll 10 on a 12-sided die than two six-sided dice.

Building a combat system involves weighing the different odds of using different dice in different combinations, or getting different results. And what kind of bonus does an item give, a raw +x increase or a random increase by rolling a die. And then, based on the power of various items, they should be easier or harder to obtain, or in a more complicated system, they should wear out over time so that a strong advantage eventually goes away instead of being overpowered forever.

A different system can be found in Cutthroat Caverns, a game that advertises itself as a "diceless RPG". The monsters in the game are a card that has information about their powers and hit points, and the players play cards that have attack values or other effects. Each round, the players attack the monster for a certain amount, then, if he survives, the monster attacks the players in a way stated on the card.

Descent is a board game published by Fantasy Flight that can be played in RPG-like campaigns (particularly with the Road to Legend expansion). Also, now that Wizards of the Coast owns the D&D license, they published a board gamed called Castle Ravenloft that uses a stripped-down version of the system to allow players to have a D&D-style experience in a board game in about an hour or so.

The obligatory disclaimer: pretty much all these titles are fantasy-themed, and contain elements that include magic and dragons and dwarves and trolls and such. If this offends your sensibilities, do not consider it an endorsement of the titles. I am only listing examples of certain game design styles for research purposes.