In 1301 an older tower castle was destroyed at the same place in the customs war. After the Archbishop of Trier, Balduin of Luxembourg, was elected Archbishop of Mainz in 1328 by the Mainz cathedral chapter and the simultaneous appointment of Heinrich von Virneburg by the Pope, Eltville became a bishop's residence due to this double occupation of the Mainz archchair. The construction of the castle under Archbishop Balduin began in 1330 and was completed in 1345. Further building measures followed up until 1419, such as installations in the residential tower.
The castle was destroyed by the Swedes in 1635 during the Thirty Years' War. Only the residential tower survived and was partially rebuilt in 1682. From Wikipedia