Die mechanics and task resolution is the meat and drink of the amateur game designer. Go to any discussion board and check out the game design threads. The majority of them are going to sound like this:
"Should I use die pools in my game?"
"I'm thinking of going with a combination of percentile dice and use polyhedra for damage."
"Should I use a roll-over or a roll-under system?"
"How can I make my game more lethal?"
All this is fine and good and warrants some thought, but most people seem to start out with these questions in mind rather than what I consider to be the real issues at hand. Don't confuse Game Elements with Game Systems. And don't confuse Game Systems with Rules.
Game Elements: dice, miniatures, character sheets, character creation rules, spell lists...in short, everything in the game. Even GM's and players! Note that a Game System is just another element of the game.
Game Systems: a laid-out set of rules on how game elements interact with a complement one another. The System is not the The Rules. The Rules tell you how to play the game...how to use the various systems in the game.
The First Law of RPG Design (formerly known as "Ebert's First Law" as applied to games rather than film)
"A game is not about what it is about, but how it is about it."
The Big Three Questions
The Big Three Questions all pertain to the First Law and all contribute to the focus of the game before pen is even set to paper. If you can't answer these three questions, then your game is not going to turn out well.
WHAT IS YOUR GAME ABOUT?
If you write a D&D clone, your game is not about "adventuring in a medieval fantasy world." Your game is about characters advancing in efficacy in order to meet greater and greater challenges. Do not confuse the genre, setting or color details with what's most important: the premise and structure of the game.
HOW DOES IT GO ABOUT THAT?
If you're designing that D&D clone and you put in a lifepath system as part of character creation, what does that accomplish? In order to fufill the requirements set my the first question, you must "put your money where your mouth is" with the discrete game elements. If that lifepath is purely cosmetic and doesn't affect the character's abilities or the game mechanics, then why is it in there?
WHAT BEHAVIORS DOES IT REWARD AND/OR ENCOURAGE?
The obvious game element to focus on as a "reward" is some kind of character advancement system. But this can go the other way as well; what behaviors does the game punish and/or discourage? If the ultimate goal of Call of Cthulhu is to die or go insane, does the game encourage this? Do insane characters get special abilities? Or is running/fighting rewarded and encouraged (as it is in Dungeons & Dragons)?
The Rule of Jared (coined by Kirt "Loki" Dankmyer)
Only roll the dice when it's important.
The Mearls Paradox
A roleplaying game that is "complete" (meaning no further explanation, rules or interpretation is required) is not a roleplaying game at all.
Jared's Rule of Combat
Fight scenes have to be exciting. Combat doesn't have to be.
If you want to play a game that encourages interesting fight scenes, play a game that encourages interesting fight scenes. Either one emphasizing style over tactics (octaNe, Wushu) or one where "a fight" is interesting because the mechanics make you feel engaged (Riddle of Steel, Burning Wheel).
If physical conflict is just an obstacle to be overcome somehow (Dungeons & Dragons), then the emphasis is in overcoming that obstacle and finding out what lies beyond it -- be it temporal reward (treasure, XP's, magic items) or a story-related reward (you resuce the princess or vanquish the Lich Lord). Combat is seen as a challenge, a kind of visceral puzzle, one that rewards strategic thinking and problem solving ability. Play the game you want to play!