The game allows you to grow or shrink when you get a gem. you just needed to shrink again after the 3rd gem.
