We have a custom property with tinyMCE and we are using a function that creates the editor for us, it looks like this:

require(['tinymce/tiny_mce_src'], function () {

    Window.prototype.CreateTinyMCEEditor = function(settingsJSON, containerForTinyMCE) {

        console.log("Window.prototype.CreateTinyMCEEditor");
        
        var settings = JSON.parse(settingsJSON);

        var plugins = "epilink,epiimageeditor,media,epidynamiccontent,epipersonalizedcontent,epiquote,paste,table,directionality,fullscreen,searchreplace,inserthtmlbutton,advimage,epifilebrowser,epifilemanagerdragdrop,epipageleavecheck,epistylematcher,epiimageresize,epiaccesskeysremove,epieditordisable,episearchreplace,epiexternaltoolbar,epitrailing,epiwindowmanager,epivisualaid";
        var buttons = settings.toolbars;

        $("#" + containerForTinyMCE).height(settings.height);
        $("#" + containerForTinyMCE).width(settings.width);

        tinymce.PluginManager.load('inserthtmlbutton', '/util/Editor/tinymce/plugins/inserthtmlbutton/editor_plugin.js');
        
        var ed = new tinymce.Editor(containerForTinyMCE, {
            theme: "advanced",
            skin: "epi-light",
            skin_variant: '',
            plugins: plugins,
            language: '',
            relative_urls: false,

            theme_advanced_buttons1: buttons.length > 0 ? buttons[0] : "",
            theme_advanced_buttons2: buttons.length > 1 ? buttons[1] : "",
            theme_advanced_buttons3: buttons.length > 2 ? buttons[2] : "",
            theme_advanced_buttons4: buttons.length > 3 ? buttons[3] : "",

            theme_advanced_toolbar_location: "top",
            theme_advanced_toolbar_align: "left",
            theme_advanced_statusbar_location: "bottom",
            theme_advanced_resizing: false,

            content_css: "",

            // Drop lists for link/image/media/template dialogs
            template_external_list_url: "",
            external_link_list_url: "",
            external_image_list_url: "",
            media_external_list_url: ""
        });

        return ed;
    }
});

Then we call it like this, here tinyMCEEditorContainerForTinyMCE is the textarea to add it to (<textarea id="tinyMCEEditorContainer"  style="display: inline-block;"></textarea>)

var ed = CreateTinyMCEEditor(tinySettings, tinyMCEEditorContainerForTinyMCE);
            ed.render();

Hope this helps 

I tried your solution Henrik but I still can't get the insert image function to work. I get a bunch of errors and almost empty popup (2 buttons have appeard).

It feels like there is a javascript error in the loading part or another loading error, I think if you looking at the first message "Resource interpreted as Script but transferred with MIME type text/html: " it could be the main source of the error. Can you publish the whole custom prop code here?

I found the problem with  weird error. It did'nt find the script files. But after some addition of virtualpath the insert image popup is loading.

I still have some problems left though.

• The texts on the image popup are not loaded - insted {#advimage_dlg.dialog_title}
• When I click the browse for file button I get a 404 page. Requested URL: /episerver/UI/CMS/edit/FileManagerBrowser.aspx

I found this http://world.episerver.com/Modules/Forum/Pages/Thread.aspx?id=81015&epslanguage=en and wonder if this problem include the browse for file from inside the editor as well.

Sorry for late answer.
The 404 on filemanagerbrowser sounds like there are wrong versions in modules or modulesbin (at least it was for me when I had that problem), that might explain the other problems as well