To accomplish this you will need to build your own editor plug-in. Here is an article describing how to make editor plug-ins:
EPiServer CMS development team
Senario: I have an XHTML property on my page, In Edit Mode - i want to have button in the menu bar of XHTML property through which i need to select a video file (.flv) and its first frame will be displayed in the XHTML property. The same way as we do it while selecting an Image. But here it will be Video File.
Please the best possible way to achieve it.