If the file inherits EPiServer.Web.Hosting.VirtualFileEx (which UnifiedFile and hence NativeFile and VersioningFile) then it should not be loaded into memory. TransmitFile is used in those cases, see http://msdn.microsoft.com/en-us/library/12s31dhy(v=vs.100).aspx
Thanks for the reply, think the issue was caused by some bad code rather than the file it self.. :)
if we have large files in the VPP (let's say 1GB). Will they be loaded in to the memory while being downloaded? Or will they be loaded into the memory in any other process?