Dang. Are we gonna need a Stupid Sexy Star Swirl tag now? You know what? Here ya go!

Clearly his hat is simply a pocket dimension, like a hat of holding.

@Background Pony #3C3A
 
Wait what.

@MeshGearFox  
Then why do all the unicorns and alicorns walk around with them out?

@Ventig  
That would explain the hat’s size…

Sweet awesomeness! It’s Hohenheim of Light meets Gandalf The White!

Horns can retract if necessary, like a dong.

@equestria-prevails
 
Starswirl is a womanizer

From the source:
 
Star Swirl the Bearded, born Astera Nocturna I, was a powerful Wizard and is widely considered to be one of the most influential conjurers of the pre-classical era. Conceived out of an illegitimate liaison between a low-ranking Princess and a commoner Guard, his Earth-pony status and bastardry in the strict class-dominated society of his time should have destined him to a life of soldiery and obscurity. Nevertheless, his mother defied the scorn and ridicule of her peers and so ensured him an education in the magical arts.
 
Star Swirl is credited with the invention of more than 200 spells and excelled in conjuration and chronomancy. Although vindicated by his rise to prominence and respected by many, he continued to be obsessed by his low-born status throughout his life, often using spells to change his appearance (he was especially fond of making himself look old and bearded) in order to adopt the wizardly image that many expected.
 
Star Swirl served within the Crystal Queen’s inner circle during the centuries of her reign of the Empire and as an advisor for the successive councils that followed. Having grown close to the Princess sisters Celestia and Selena, he accompanied them to Equestria after the fall of the Empire. He devoted the last years of his known life to pursuing an elusive spell that he believed could transform him into an alicorn.

“Dammit, shadow self, not now.”

What if his horn is extremely tiny

@Psychopomf  
Magic hats (not hats for magic, but magic hats). Same magic that Sweetie’s helmet has.

After musing so long upon the nature of magic and time itself, Starswirl is saddened to find out he has but one conclusion.
 
He has spent too much of it alone and he fears his time be running out soon.