If you have your own scheduled job then you need to provide a lot more information about the error. Have you got any logs?
How is your IIS apppool configured for recycling? Could it be that you either hit a configured recycle time or that the application exceed a memory threshold for recycling?
Recycling of app pool is an IIS feature and nothing special for EPiServer. You can configure it in IIS Manager if you select your app pool and choose Recycling from the Actions menu.
You may get NullReferenceException if you are trying to use current HTTP context in the job code when the job is triggered automatically. In this case there is no HTTP context and things like HttpContext.Current.User or HttpContext.Current.Server.MapPath() will throw exception.
The shutdown of the worker process of being idle can be a problem.
Since the job is running in a background thread IIS will not be able to detect that the site is actually working. So if there is no other "ordinary" request at the same time IIS may take down the process. Then the soultion is to remove the app cycle recycling for idle time.
The typical reason for having recycling configured is to get a new "fresh" process and release all hold memory. For an EPiServer site memory allocation should typically not be a problem since most things that are stored are not "hard" rooted (like e.g. in static dictionary or similar) but instead the ASP.NET cache is used. The ASP.NET cache works so that if the process needs more memory if can evict items from the cache.
If you still want to have recycling configured you could instead configure it to run at a specific time (where you should choose a time that is not conflicting with where you have scheduled your job to run).