You problem, fixably and very commonly :) "Added along with everything else" is the confession - the setting agent never fully hydrated and dissolved, and undissolved specks stay grainy forever while contributing zero set.
The rules by agent: AGAR must be whisked into the COLD liquid first, then brought up and boiled a genuine 30-40 seconds - it only activates at a boil. Sprinkled into warm liquid late, it seizes into the sooji-specks you met. GELATIN is the opposite personality: bloom it in cold water first, then melt it into WARM (never boiling) liquid - boiling murders its setting power.
Either way: strain the glaze before pouring (insurance against any survivor specks), and check your agar isn't expired - it weakens with age and humidity.
Your layers were gorgeous, which is the hard part. The glaze is just chemistry with manners - respect its order of operations and attempt two will shine, literally 🙂
