You might think it’s a perfect combo, but sugar is ruining your brandy