What impact does size have on the game?
Size impacts hit dice (see the top of page 27, first bullet) and number of creatures you can attack at a time (see page 18 under range and distance). The bigger you are the more HD you have, and the smaller the fewer. You can engage in melee with 4 creatures of the same size; I would subtract 2 for every category smaller, and add 2 for every category bigger; a wyvern is huge (2 categories larger than humans) so up to 8 humans can melee against a wyvern at once. A rat is small, so only 2 humans can attack a single rat at the same time with melee attacks.
Gnomes and stoutlings are medium (but at the very, very low end of medium). I toyed with having them be small and giving them D4 hit dice, but that made them much less playable. So I tweaked the sizes a little. Wood trolls are large (hence the D8 hit die). A few monsters have abilities that vary by size ('can swallow a creature medium-sized or smaller on a critical attack'), but that's about it.
If you want more variety in your smaller folk and their mechanics, it's an easy plug in:
New Talent: Little Folk. You have learned to leverage your smaller size in combat with larger foes. Any foe larger than medium size takes -1 to attack rolls against you.
What impact does armor have on spell casters? Are there penalties for wearing armor?
There is no relationship between armor and spell casting. As a magic user, you haven't spent any time or energy learning how to wear armor or carry a shield, because it's just not your priority - not because of rules. Specific bonuses/penalties for armor/spell casting rules were a little to crunchy for my taste for where the game is.