TF14043: An error occurred computing the delta. Not enough storage is available to process this command

TF14043: An error occurred computing the delta. Not enough storage is available to process this command

• Babak Farahani

  Hi Grant:

  I recommend that you contact Product Support so we can help you further with this problem since it seems that is an issue where we will need more information and data from you.

  thanks, mario

  
  
  

• Saroj K. Nanda

  Cold you migrate the Db in smaller pieces using the mappings and settings files

  
  
  

• dr.acv

  Thanks Mario. We have now raised a PSS call.

  I will let you know the outcome.

   

  
  
  

• M.A.T

  The reverse delta patches are created on the application tier by using mspatch and then stored in the data tier. There's a little bit of information on it at http://blogs.msdn.com/jimlamb/articles/tfsfaq.aspx.

  What happens when I check in an update to a file in version control

  TFS uses the reverse delta model as most version control systems do. However, there’s a few notable things that TFS does. First the creation of the reverse deltas is done asynchronously as a background task. Initially the new copy is uploaded and saved. Sometime later (usually not very long), a background job will run and convert the older version from a full copy to a reverse delta (assuming the delta is actually smaller than the original – usually, but not always the case).

  Because recreation of old versions can be time consuming (especially if you have to apply many reverse deltas), TFS stores the copies of the most recently used files in a cache on the application tier (AT) machine. This mostly eliminates the performance difference between accessing the most recent version vs. older versions.

  Note: The reverse delta mechanism works for text and binary.

  One thing that's not mentioned there is that we only compute deltas for files that are 16 MB or smaller.

  My best guess would be that the "$/TeamProject/Folder/Path.wsdl" file is triggering some bug in either the patch library itself or the way we invoke it.

  Does this error get reported repeatedly for the same file

  Does it happen on any other files or just this one

  Buck

  
  
  

• sigol

  The data tier is healthy. No errors or warnings in the event log.

  It's a Dual-Proc 3.4Ghz (hyper-threading disabled) with 3.5Gb RAM and SAN storage. Plenty of disk space available.

  
  
  

• Xefan

  Hi Buck,

  To answer your two questions:

  > Does this error get reported repeatedly for the same file
  > Does it happen on any other files or just this one

  No, the error is repeating. Each entry is a different file. Here is an excerpt from 22:46.

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/AssemblyInfo.cs version 81763

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/Application.GenManifest.csproj version 81763.

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/GenManifestMain.cs version 81763.

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Updater/Application.GenManifest/GenManifestMain.cs version 81780.

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Updater/Application.GenManifest/GenManifestMain.cs version 81803

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.Interceptor/AssemblyInfo.cs version 81763.

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.RunOnce/AssemblyInfo.cs version 81763

  Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.Service/AssemblyInfo.cs version 81763
  

  Thanks,

  Grant

  
  
  

• Paul Fuhrmann

  Thanks Buck - it was worth a try.

  CHECKDB found 0 allocation errors and 0 consistency errors in database 'TfsVersionControl'.

  
  
  

• DSimmon

  We've now incurred this error 3 times. However all three times the Application Event Log on the App-Tier has filled up and we've lost the original errors.

  There has never been anything of interest in the data-tier event log.

   However, last night we enabled perfmon on the application-tier while trying to migrate again.

  Between 22:23 and 22:40 the available megabytes dropped by 1Gb.

  At 22:50 VSSConverter.exe failed with the following errors in VSSConverter.log

  [VersionControl,  Error,    18, 2006/09/04 22:50:44.736] Exception: System.Web.Services.Protocols.SoapException
   Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
   Stack Trace:    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
     at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
     at Microsoft.TeamFoundation.VersionControl.Proxy.Repository.CheckIn(String workspaceName, String ownerName, String[] serverItems, Changeset info, CheckinNotificationInfo checkinNotificationInfo, CheckinOptions checkinOptions, Failure[]& failures)
     at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit)
   Help Link:
   BaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.

  [VersionControl,  Error,    18, 2006/09/04 22:50:44.736] SubCode.Code.Name: OutOfMemoryException
  [VersionControl,  Error,    18, 2006/09/04 22:50:44.736] Exception: Microsoft.TeamFoundation.Converters.VersionControl.Common.ConverterHatterasException
   Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
   Stack Trace:    at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit)
     at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.ActivityCheckIn(Object stateInfo)
   Help Link:
   BaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
   InnerException: Microsoft.TeamFoundation.Converters.VersionControl.Common.ConverterHatterasException
   InnerException Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
   InnerException Stack Trace:    at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit)
     at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.ActivityCheckIn(Object stateInfo)
   InnerException Help Link:
   InnerExceptionBaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.

  [VersionControl,  Error,    18, 2006/09/04 22:50:44.814] Exception: System.InvalidOperationException
   Message: Client found response content type of 'text/html; charset=utf-8', but expected 'application/soap+xml'.
  The request failed with the error message:
  --
  < xml version="1.0" >
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml" >
  <head>
     <title>Server Unavailable
  </title>
     </head>
     <body>
        <h1><span style="font-family:Verdana;color: #ff3300">Server Application Unavailable
  </span></h1>
        <p>
         <span style="font-family:Verdana;">
          The web application you are attempting to access on this web server is currently unavailable.A  Please hit the "Refresh" button in your web browser to retry your request.
  </span></p>
     <p>
     <b>Administrator Note:
  </b> An error message detailing the cause of this specific request failure can be found in the application event log of the web server.  Please review this log entry to discover what caused this error to occur.
   </p>
     </body>
  </html>

  --.
   Stack Trace:    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
     at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
     at Microsoft.TeamFoundation.VersionControl.Proxy.Repository.UndoPendingChanges(String workspaceName, String ownerName, ItemSpec[] items, Failure[]& failures)
     at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.UndoPendingChanges(ItemSpec[] undoItems, Workspace ws)
   Help Link:
   BaseExceptionMessage: Client found response content type of 'text/html; charset=utf-8', but expected 'application/soap+xml'.
  The request failed with the error message:
  --
  < xml version="1.0" >
  <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
  <html xmlns="http://www.w3.org/1999/xhtml" >
  <head>
     <title>Server Unavailable
  </title>
     </head>
     <body>
        <h1><span style="font-family:Verdana;color: #ff3300">Server Application Unavailable
  </span></h1>
        <p>
         <span style="font-family:Verdana;">
          The web application you are attempting to access on this web server is currently unavailable.A  Please hit the "Refresh" button in your web browser to retry your request.
  </span></p>
     <p>
     <b>Administrator Note:
  </b> An error message detailing the cause of this specific request failure can be found in the application event log of the web server.  Please review this log entry to discover what caused this error to occur.
   </p>
     </body>
  </html>

  --.

  [VersionControl,  Warning,  18, 2006/09/04 22:50:44.814] Repository is down. Retrying...
  [VSS,             Error,     3, 2006/09/04 22:50:47.814] Exception: System.Runtime.InteropServices.COMException
   Message: Error HRESULT E_FAIL has been returned from a call to a COM component.
   Stack Trace:    at Microsoft.VisualStudio.SourceSafe.Interop.IVSSItems.get_Item(Object sItem)
     at Microsoft.TeamFoundation.Converters.VersionControl.Vss.VssWrapper.GetItemFromAncestor(VSSItem item, ActionMetadata info)
   Help Link:
   BaseExceptionMessage: Error HRESULT E_FAIL has been returned from a call to a COM component.
  *** lots of these COMExceptions until 22:54 **
  [VersionControl,  Warning,   3, 2006/09/04 22:59:54.605] Reached max limit for queue: PendChangeQueue, thread: Main Thread
  [VersionControl,  Warning,  18, 2006/09/04 23:25:51.044] Repository is back
  ** this is probably when the app-pool restarted **
  

  We have confirmed that there's no system job that happens at 22:00. And in any case, we have seen this same failure (OutOfMemoryException) at other points in the day with our previous migrate attempts.

  Comparing the logs of all three attempts, I pieced together the last "Source changeset" and TFS changeset messages before the failure (hoping they were the same):

  1. Source: 3054, TFS: 40656, Time before failure: 8H 27M (* I can't explain why this one took longer, we rebooted after this attempt as it had been running for 3 weeks)
  2. Source: 3084, TFS: 46815, Time before failure: 5H 35M
  3. Source: 3081, TFS: 81943, Time before failure: 5H 3M

  Our next steps are:

  • Increase the size of the application event log to a large number
  • Turn on tracelevel 4 for VSSConverter
  • Empty the application-tier cache
  • Try the migration again.
  • Raise a support call with the extended logs.

  Can you think of anything else that would be worth looking in to

   

  
  
  

• Darrell Norton

  Just to update, we have tried another migrate of the same VSS, and then a migrate of a smaller VSS and we received this exception on the app-tier (for each migration) at a random point during the migration.

  TF53010: An unexpected condition has occurred in a Team Foundation component. The information contained here should be made available to your site administrative staff.
  Technical Information (for the administrative staff):
  Date (UTC): 12/09/2006 6:10:51 PM
  Machine: EPTFS001
  Application Domain: /LM/W3SVC/3/Root/VersionControl-3-128025267317385077
  Assembly: Microsoft.TeamFoundation.Common, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727 Process Details:
    Process Name: w3wp
    Process Id: 4808
    Thread Id: 5916
    Account name: DOMAIN\User

  Detailed Message: TF53017: Request processing is being shut down due to the exception shown below.
  Web Request Details
      Url: http://server:8080/VersionControl/v1.0/repository.asmx
  [method: POST]
      User Agent: Team Foundation (VSSConverter.exe, 8.0.50727.147)
      Headers:
  Content-Length=451049&Content-Type=application%2fsoap%2bxml%3b+charset%3
  dutf-8%3b+action%3d%22http%3a%2f%2fschemas.microsoft.com%2fTeamFoundatio
  n%2f2005%2f06%2fVersionControl%2fClientServices%2f03%2fQueryItems%22&Acc
  ept-Encoding=gzip&Accept-Language=en-AU&Expect=100-continue&Host=eptfs00
  1%3a8080&User-Agent=Team+Foundation+(VSSConverter.exe%2c+8.0.50727.147)&
  X-TFS-Version=1.0.0.0
      Path: /VersionControl/v1.0/repository.asmx
      Local Request: False
      Host Address: 10.X.X.X
      User: DOMAIN\USER [authentication type: NTLM]

  Exception Message: Exception of type 'System.OutOfMemoryException' was thrown. (type OutOfMemoryException)

  Exception Stack Trace:    at
  System.Data.SqlClient._SqlMetaDataSet..ctor(Int32 count)
     at System.Data.SqlClient.TdsParser.ProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj)
     at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj)
     at System.Data.SqlClient.SqlDataReader.ConsumeMetaData()
     at System.Data.SqlClient.SqlDataReader.NextResult()
     at
  Microsoft.TeamFoundation.VersionControl.Server.VersionedItemComponent.Qu
  eryItems(String serverItem, Int32 changesetId, Int32 deletionId, RecursionType recursive, DeletedState deleted, ItemType itemType, IList items, String& queryPath, String& filePattern)
     at
  Microsoft.TeamFoundation.VersionControl.Server.VersionSpec.QueryItems(It
  emSpec itemSpec, Workspace localWorkspace, VersionedItemComponent db, DeletedState deletedState, ItemType itemType, IList items, String& queryPath, String& filePattern)
     at
  Microsoft.TeamFoundation.VersionControl.Server.DeferredQuery.Execute(Wor
  kspace localWorkspace, VersionedItemComponent db)
     at
  Microsoft.TeamFoundation.VersionControl.Server.DeferredQuery.Execute(Lis
  t`1 queries, Workspace localWorkspace, VersionedItemComponent db)
     at
  Microsoft.TeamFoundation.VersionControl.Server.ItemSet.QueryItems(IPrinc
  ipal userPrincipal, Workspace localWorkspace, ItemSpec[] items, VersionSpec version, DeletedState deleted, ItemType itemType, Boolean
  generateDownloadUrls)
     at
  Microsoft.TeamFoundation.VersionControl.Server.Repository.QueryItems(Str
  ing workspaceName, String workspaceOwner, ItemSpec[] items, VersionSpec version, DeletedState deletedState, ItemType itemType, Boolean
  generateDownloadUrls)
  

  PSS advised us that "The server ran out of memory while running QueryItems()." and asked how many files and folders are involved in the migration.

  Our VSS database includes:
  91,986 files
  8,782 folders
  1.27Gb total.

  The SourceSafe\data folder itself is:
  353,467 files
  35 folders
  2.66Gb total.

  I also tried a migration of our smaller VSS database overnight, and it failed with the same exception.

  The size of that database is:
  38,242 files
  4,065 folders
  941Mb total.

  The SourceSafe\data folder is:
  120,826 files
  407 folders
  1.37Gb

  The output of http://yourtfsserver:8080/VersionControl/v1.0/administration.asmx/QueryRepositoryInformation is the following:
  UserCount="182"
  GroupCount="1536"
  WorkspaceCount="456"
  ShelvesetCount="0"
  FileCount="1288939"
  FolderCount="130117"
  MaxChangesetID="154659"
  PendingChangeCount="13"

  It would also be worth noting that we have tried this migration 3-4 times using the same Team Project name, and deleting it with TfsDeleteProject after each attempt. But of course, TFS still maintains all the old changesets.

  Grant

   

  
  
  

• AlexDav

  Is the data tier disk equally healthy looking That's where deltas are stored.
  
  

• UA Jeremy

  This is a bit of a shot in the dark, but given the number of files that are triggering the problem, you may want to check the version control db to make sure some disk error hasn't occurred. Here's the final bit from a thread where bizarre things were happening and were traced back to the db index being corrupted on disk due to a power failure.

  From the thread http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=550091&SiteID=1&PageID=1:

  I think we've gotten to the bottom of the issue. Here's what Chandru used to determine the problem.

  The TFSVersionControl database is corrupt. The table tbl_Version has a corrupt index (IX_tbl_VersionedItem_InitialVersionFrom) and is missing entries for some items – hence you are seeing null values when any query uses the particular index.

  Usually such corruption occurs due to a faulty disk or a power outage. Serge noted that a power outage had occurred not long before the problem started happening.

  The following command detects database corruption.

  run dbcc checkdb('TFSVersionControl')

  Scroll down and you should see allocation errors (in red).

  In this case, the corruption in the index was fixed using the following procedure.

  1. Backup your existing database

  2. Put the database in single user mode by running:

  alter database TFSVersionControl

  SET SINGLE_USER

  2. Repair the corruption, by running:

  dbcc checkdb('tfsversionControl_bkp', REPAIR_REBUILD)

  3. Restore the database to multi_user mode by running:

  alter database TFSVersionControl

  SET MULTI_USER

  4. Verify that version control works correctly.

  Buck

  
  
  
• TF14043: An error occurred computing the delta. Not enough storage is available to process this command