Despite its potential, optical spectrum prediction for crystalline materials has been largely overlooked in machine learning for materials science. Here we present a proof of concept by creating a ab initio database of 9,915 dielectric tensors of semiconductors and insulators calculated in the independent particle approximation, and subsequently training graph attention neural networks to predict the dielectric function and refractive index. Our study shows that accurate prediction of optical spectra is possible using only the crystal structure and a database of about 10^4 materials.