A beautifully embroidered but wine-stained waistcoat that once belonged to Captain James Cook did not find a buyer when auctioned in Sydney on Sunday.