Degenerative retinal diseases such as retinitis pigmentosa and macular degeneration cause irreversible vision loss in more than 10 million people worldwide. Retinal prostheses, now implanted in more than 250 patients worldwide, electrically stimulate surviving cells in order to evoke neuronal responses that are interpreted by the brain as visual percepts (‘phosphenes’). However, instead of seeing focal spots of light, users of current epiretinal devices perceive highly distorted phosphenes, which vary in shape not just across subjects but also across electrodes, resulting in distorted percepts. We characterized these distortions by asking users of the Argus retinal prosthesis system (Second Sight Medical Products, Inc.) to draw percepts elicited by single-electrode stimulation on a touchscreen. Based on ophthalmic fundus photographs, we then developed a computational model of the topographic organization of optic nerve fiber bundles in each subject’s retina, and used this model to successfully simulate predicted patient percepts. Our model shows that activation of passing axon fibers contributes to the rich repertoire of phosphene shapes reported by patients in our psychophysical measurements, successfully replicating visual percepts ranging from ‘blobs’ to oriented ‘streaks’ and ‘wedges’ depending on the retinal location of the stimulating electrode. This model provides a first step towards future devices that incorporate stimulation strategies tailored to each individual patient’s retinal neurophysiology.