Architecture options for running a long process from an ASP.NET website

Architecture options for running a long process from an ASP.NET website

• Jesse Wolgamott

  Hi,
  
  Actuly processing is not database oriented but storage of data for central management.
  SQL Server is just for example, you can use any database.
  There are many free as well e.g. mySQL
  
  Catalog of free database systems!
  
  by the way link you have mentioned also use database.
  
  More database is an easy way to centeralize the data.
  
  Best of Luck!
  
  

• MunishGupta

  If you are doing asynchronous work in a database, you need to consider using Service Broker in SQL Server 2005 to manage the work. You can queue the work to be processed on a Service Broker queue. Service Broker will manage the processing and provide a mechanism for returning status asynchronously to the caller. Because it is built into the database, the database transactionally provides reliability and will ensure that the tasks are processed in spite of any failures. Combined with Database Mirroring, Service Broker can provide unprecidented levels of reliability in asynchronous processing.
  

• Steev

  Service broker & Mirroring are indeed a powerful combination - I do suggest you benchmark them together first to make sure the performance meets your need (on your target platform)

  Arnon

  
  
  

• Brian_Tucker

  Hi,

  Thanks all for the extra info, but I should clarify a few things:

  The processing reads the information from the file and inserts the records into the database using a simple (extensible) pipeline.

  => this is where the heavy part of the logic is. 70% of the processing time is related to the statements executed against the SQL Server.

  1) Mubshir, that is why I was looking for an alternative way to communicate the status updates. I do not want to add extra load on the SQL Server.

  2) Sarit, we did evaluate using BizTalk as the central piece for our solution. but it was decided that we were not to use BizTalk (extra licence cost).

  3) Roger, I wish I could go full out on using SQL 2005, but that won't be for the near future. We are tied to supporting SQL 2000. (most likely until all of our customers have moved on)

  The solution really needs to be 'old' skool, .NET v1.1 with SQL 2000. But I want to enable the architecture to port to new technology as smoothly as possible, if possible.

  Meanwhile, I have tuned the process more and found/fixed a great big bottleneck, but inherently the solution is not fit to scale, and sales have found this tool to be the next ETL solution. I strongly object, but sales will be sales.

  Thanks again for the feedback.

  
  

• RickGaribay.NET

  Have you considered a BizTalk orchestration exposed as webservice as one of the options It is something you can consider. BizTalk orchestrations provide all of the below capabilites.

  1. Handling file based inputs

  2. Communication with/as web services

  3. Synchronous/Asynchronous communications

  4. Reliable messaging services

  
  
  

• Bastiaan Molsbeck

  Hi,
  You have almost suggested the solution, what I would like to use assuming that i have a database as well is...

  
  Solution Vision:
  Create a web service to upload the file. Put it in some shared folder at server and log its path in the database, give it some status for example pending. Make a window service that get the file location from the database and start processing the file change status of file to processing during processing and after process set status to processed. Create a web service to get status of file from the database so that Intranet web site can any time and see status.
  
  

  Process:

  
  Step1: File Upload
  Create a web service for the file upload. Make a folder on the server in which you will upload it. After uploading the file rename it to unique id (GUI ID), now store both original name (if needed) and new name (i.e. GUI ID) in a database table and give it status Pending.
  Following could be the table fields, You can change it according to your preference.
  
  

  FileNameNew	New filename
  FileNameOrignal	Old file name
  Status	Current status
  StatusUpdated	Status last update time

   

  Step2: File Processing
  Create a window service that will be checking database for an new files. If new file record found then get file name from the database and you can get shared folder path from the app.config of the window service after adding key and setting it value (e.g. D:\SYS\UPLOADS). Now create a queue in window service  and implement threading. You should also set some max parallel threads count so that if more than 10 or 15 files found system not get stuck, you should process a fixed number of files at a time in parallel threads for the system performance concerns.
  
  

  Step3: Status Report on intranet website
  Create web service to get status of the file from database and show. More if status is processed the you can show files items as well or what ever your logic is.
  
  Step4: Security/Exception Handling
  You can use log4net (free logging component) at in web service and window service for exception logging.

  Check file for viruses after uploading.

  Use web.config of web service to allow authorize users.
  
  Hope this will help.

   

  
  
  

• John Portnov

  Thanks for the reply. Good stuff.

  I have also found an interesting article: http://msdn.microsoft.com/msdnmag/issues/05/03/SchedulingASPNETCode/

  Since the processing is very SQL Server intensive, I'm also wondering if there are other options to communicate the progress, other than the database.

  
  
• Architecture options for running a long process from an ASP.NET website