Star Gems, often referred to as just Gems in the game, are used to evolve weapons. Weapons can be evolved once they reach their max levels. Max levels are reached by Power-Ups. To use Star Gems, simply go to the Weaponsmith and click on any weapon that has reached its max level. You can click Evolve and then use the Star Gems to complete the evolution.
The number of stars on a Star Gem corresponds with what evolution it can perform. For example, a 3-Star Gem can evolve a 2-Star weapon into a 3-Star weapon, but it cannot evolve a 3-Star weapon into a 4-Star one. Similarly, a 4-Star Gem can only be used for 3-Star weapons to evolve to 4-Star weapons.
2-Star weapons require at least 20 3-Star Gems to evolve.
Some weapons do not evolve no matter how many Star Gems you have collected. You can view each weapons potential evolution by hovering over the "Next Evolution" in the Weaponsmith menu or by clicking on that weapon in the Weapons Collection menu.