Haven't tested this, but you should be able to do something like this:

ContentIndexer.Instance.Conventions.ForInstancesOf<MediaData>().ShouldIndex(x => !x.IsWayTooBigFile());

       public static bool IsWayTooBigFile(this MediaData mediaData)
        {
           // check file size
        }

Also, you may not want to check the file size every time you try to index. Check out this forum post

Thanks, I had that in mind but thought that maybe there were a more elegant solution.. :) My problem is that when indexing files, the site restarts. Thought that maybe it has something to do with big files..

Erik,

were the big files related to the site restarting?

Well, it does still restart. Seems to index a little more before restarting though. This is on a shared server, so it's hard to debug. :(

EDIT:

Log file says - which gives pretty much nothing in my eyes:

2014-11-12 11:47:35,376 [72] WARN EPiServer.Framework.Initialization.ShutdownTracker: ASP.NET shutting down due to HostingEnvironment

2014-11-12 11:47:35,641 [20] ERROR dispose: Nexus.Web.Business.Initialization.RestartModule: Application restarted because:HostingEnvironment
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at System.Web.Hosting.HostingEnvironment.InitiateShutdownInternal()
at System.Web.Hosting.PipelineRuntime.StopProcessing()
HostingEnvironment initiated shutdown
HostingEnvironment caused shutdown

Have you checked memory/cpu usage when indexing? 

So, now it seems to be solved. Updated Find to the latest version yesterday and I've managed to index the site with files. Happy days. Don't know what the problem was though.