I am currently architecting a framework that will depend on IIS to host batch oriented code. The design will use a disconnected model with a controller (that lives remotely) and checks back periodically via an ASMX layer. When invoking the actual batch execution code the ASMX call will spawn as many threads as it is configured to run and immediately respond (hence avoiding any long running HTTP connection).
I like the idea for a number of reasons, obviously I get to use a standard communication protocol to invoke the batch code across tiers and I can even incorporate standard load balancing routines. I'm also able to use the same communication end points to check back in on the threads as they are executing. There are also a number of other constraint benefits I get with application pools that make this a very durable option. What I'm not 100% confident on is that IIS won't configure the aspnet_wp to have timeouts somewhere that I'm not familiar with (I'm aware of the basic http and runtime timeouts but is there something else ... I'm afraid there's something I am just not aware of) .
So there's my main question for the group ... have any of you have tried to do this and can you tell me what execution timeouts I might be dealing with Additionally, is there anything technical that makes IIS only a OLTP style hosting application I initially didn't feel comfortable with it but I think that's because I've "pigeon-holed" IIS as a web server.
Of course if I can't 'use it I can always fall back to ASMX without IIS or worst case .NET remoting (I would love to use WCF but I'm limited to using .NET 1.1 for this framework). Any and all comments are welcomed ...call me crazy if you want ... just please tell me why I am :)
Hosting Long Running Batch Processes In IIS 6
Leaf.
Most important: it was not designed for that. While you can certainly spawn new threads to do background work, asp.net can unload your application's appdomain at any time (and will do so after a period of "inactivity" whether you are running anything else in there or not) which can wreak havoc on your long running processes.
You're better off putting your long running processes into a hosting environment designed to cope with the necessities of long running processes, such as a COM+ server application (yes, COM+ has no problem if you hang on to one of its MTA threads for long periods of time), or your own custom Windows NT Service. Now, this does not mean you need to abandon your idea of providing a IIS-friendly (such as ASMX) job submission and status query interface, you can still do so and have it call into an internal interface (trivial with COM+, or with .NET remoting for a Windows Service) your process hosting environment exposes.
Clemens Vasters wrote some time ago about choosing the appropriate hosting environment for your code, which might be useful for you.
Javier_Uy
This was the same article I had read that made it seem like I could take back control from IIS and just use the HTTP channel to launch my own process. This is exactly what HTTP.sys gives me ... I don't see where I need WSE though Is that just a way of abstracting away the complex underlying API to HTTP.sys It looks like .NET 2.0 comes with the HttpListener without WSE.
I am still running some tests and it looks like Win2K3 and IIS 6 will let me configure things to run as long as I want. I had to find where they were (for example the Idle Timeout in a custom app pool is by default 20 minutes) but after changing that timeout to infinite I'm running disconnected with no problem (40+ minutes and still rolling). I'm trying to execute a test that runs for 48 hours to see if there's some hidden timeout I am not aware of. If there is then my next step will be to host it without IIS and continue to use the ASMX programming model. This paragraph from Skonnard really gives me a lot of confidence that I can use ASMX to host these long running processes
From Run ASMX Without IIS by Aaron Skonnard
However, I should point out that when you take this approach, you're giving up the sophisticated process model provided by IIS through the w3wp.exe (or aspnet_wp.exe) worker process. This means you lose things like process management (startup, failure detection, recycling), thread pool management, and ISAPI support. When you host ASMX, you're providing the process, so you're re-sponsible for providing the process model and associated services.
Thanks again to everyone in this thread for voicing your opinions and I'll be sure to post back any other interesting things I find as I go through this proof of concept.
Latso
Yikes, a hard coded value inside aspnet_wp that can not be changed is exactly what I was afraid of. You also make a good point about the appdomain recycling after it sees a config file change or some other dependent assembly replaced. These items do make ASP.NET different enough to be a concern. I am currently working on a quick proof of concept to truly prove this out.
WCF isn't an option at this time so I may be back to using COM+. Doesn't IIS use COM+ under the covers to host most of this code already. That's what still confuses me ... I keep hoping I just need to set some timeout to unlimited to make this work. Thanks again for all your help in this matter and if I discover anything noteworthy I'll be sure to bring it back to the thread.
Tiarnan
Hi Guys;
I see long thread here and I'm not sure if you are getting into something. I agree with Tomas that hosting long running transactions/process in IIS has many problems...
* AppDomain is limited to 25 threads concurrent. This means that your site cannot handle more than 25 requests concurrently!! However this value can be changed based on your machine H/W configuration, but it is still limitted. refer to the following blog http://blogs.msdn.com/darrenj/archive/2005/03/07/386655.aspx. I'm not sure of the usage scenario of your application, but you need to consider this.
* IIS6.0 uses shared libraries on ASP .Net to optimize memory usage. So, if any other ASP .Net application fails everything fails and at this point, you do not know what you're handling. A work around for this is to configure IIS to work in isolation mode that's compatible with IIS5.0 which will load ASP .Net libraries for each hosted application!! but you will be avoiding other failures. http://msdn.microsoft.com/msdnmag/issues/03/06/Threading/
COM+ looks like a good choice but your components are exposed in RPC format. However you can configure COM+ to expose your libarray in web-service, which is out-of-the-box functionality that COM+ provides. However, you have to consider that this again will run in IIS environment which is getting us back to the first point.
Remoting is a great choice. You can freely design your application and expose it in the way you like.
I hope that this helps.
A. Nagy
ravikk
IIS did use COM+ in IIS5 (and in IIS6 in compatibility mode) to handle medium and high isolated applications, but that's not used in IIS6 generally, and even so it didn't really impact ASP.NET.
Under IIS5, ASP.NET requests were always finally processed inside aspnet_wp.exe anyway, so the isolation level in IIS didn't matter at all (in fact, there was little to no isolation for ASP.NET applications under Windows 2000). Under IIS 6, isolation is provided through AppDomains, and under the "new" request model the ASP.NET runtime runs in-process into w3wp.exe. COM+ doesn't enter the figure anymore because you can use AppDomains by themselves to provide whatever isolation you needed (as weak or as strong as you wanted it).
Notice that if you don't want COM+, you can certainly create your own Windows NT service and expose a .NET remoting service from it, so that's another option, certainly.
Bartosz
As far as I know, all the recycling, start up and shot down, thread pool managmenet etc. are all IIS tasks and not ASP.NET and when you use http.sys (which is a kernel mode driver) and host the ASMX yourself you have to handle the above tasks yourself. (see http://msdn.microsoft.com/msdnmag/issues/04/12/ServiceStation/default.aspx)
Thus, you should not have the problems when you host ASMX yourself (either using WSE or WCF)
Arnon
Lester LaForce
The other thing is that ASP.NET has a mechanism to unload what it considers "idle" AppDomains (i.e. appdomains that are not getting any requests for longer than X time, where X is around 20 minutes, I think). A big problem with this is that it is not very documented. I can't remember exactly where I found this, but it was after researching a lot of time for a problem we had where some ASMX services were taking a lot of time to respond after about an hour of inactivity, but I did remember reading that it was a hard-coded time that could not be configured or changed in any way.
The problem isn't so much IIS, a lot of it is the design of ASP.NET, so hosting ASMX yourself using HTTP.sys wouldn't get around the problem. I *think* (though I haven't checked too deeply into it), that self-hosted WCF services have no problems, though, so you might consider if that might be an option for you.
Fixxer
Thanks for the feedback, after looking at your response and reading Clemens post I have a couple of follow ups. First of all, the idea of predictability with your business logic is absolutely a key concern with any architecture (including this one I'm envisioning). What I'm not 100% convinced of based on the explanations is that I don't get that with application pools and process isolation via IIS 6. Is the concern that someone could tear down all of IIS because it's a "UI" host Is that just a perception issue or truly a technical implementation feature You mention that it "was not designed for that" ... could you offer up some technical background for that statement or maybe a link to some detailed explanations of why IIS should never host processes that run longer than X amount of seconds.
Is there some timeout or constantly running listener that will kill a running process if it is running over some specific amount of time. You seem to be suggesting that the way .NET works and the CLR is different when IIS kicks it off and hosts it. Is this within the implementation of aspnet_wp I agree that ASP.NET is different and it is configured differently when it is started up but I remain skeptical that I have to take my business logic out of process to ensure predictability.
I can be convinced, I just need to understand a little bit more what are the true technical limitations of hosted .NET code in aspnet_wp. The overhead of pushing all the business logic to the dllhost.exe process seems un-necessary but maybe that's just because i'm still a little biased based on the picture I had in my head before. Of course, I wouldn't be posting this if I wasn't already somewhat unsure of the direction in the first place. If you can give me a little more detail as to why it won't work I'll change it to run outside of ASP.NET in IIS (what about ASMX without IIS using the HTTP.sys listener to kick off a self-hosted .NET process ... this might be the best of both worlds ... I would like to avoid COM+ if I can).
Thanks again for the help!!!!
Jason56
The WSE is for the different WS extenstions (e.g. security) if you don't need those, indeed you can just use the HttpListener
Arnon