Inconsistent File System Enumeration with FindFirstFile / FineNextFile

Inconsistent File System Enumeration with FindFirstFile / FineNextFile

• djmikke

  Do you run studio and the debugger elevated. Do you think that might be the differance
  
  

• TJC2

  Documents and Settings is an actual folder and not shortcut file. It has special permissions on it if you look at the ACLs. Documents & Settings is virtualized and so are other directories. I assume this infinite recursion may be inspired by the virtualization. I don't know that for sure that's causing the problem, but executables with manifest do not participate in virtualization according to my documentation. I'm curious what your results would be with a manifest
  
  

• dimkaz

  I was running VS2005 as administrator. As a result of your asking, I unchecked that and now it works correctly - the FindFirstFile() now behaves as do the test programs I wrote that do not call into the dll. Thanks for the tip. This resolves the problem from my perspective.

  
  

• huabing78

  The exe that contains the code that recurses into oblivion had an XP style manifest. Since I just started looking into Vista issues, I had to research what you suggested in order to understand it, which took some time. I've now added the "trustInfo" section to the manifest with the requestedExecutionLevel set to "asInvoker". Unfortunately, the FindFirstFile / FineNextFile doesn't appear to have been affected by the is change.

  I assume this change, explained in the User Access Control white paper is what you were suggesting. The architecture of the app at the time of the failure is this - an exe with the manifest; a dll that contains the code that does the enumeration; the enumeration runs on a separate thread; debug mode build.

  When I build I get the warning:

  manifest authoring warning 81010002: Unrecognized Element "requestedPrivileges" in namespace "urn:schemas-microsoft-com:asm.v2"

  I researched this warning and found a note from Microsoft saying it was benign. I've coded the manifest section so it can work on XP and Vista. I'm not building with command line and mt.exe. I'm building in the VS2005 GUI.

  Here is a slightly edited version of the code that goes into oblivion. Note that I start the call with the szFolder as "c:". When it gets to the "Application Data" folder in "c:\Documents and Settings\All Users\Application Data" the call to FindNextFile always returns "Application Data" and the function recursively adds "Application Data" to the path until the stack overflows.

  void Enum(LPCTSTR szFolder)
  {
  WIN32_FIND_DATA wfd;
  TCHAR szWildcard[MAX_PATH];
  _stprintf(szWildcard, L"%s\\*.*", szFolder);
  HANDLE hf = FindFirstFile(szWildcard, &wfd);
  if (hf != INVALID_HANDLE_VALUE)
  {
  do
  {
  // skip the "." and ".." pseudofolders
  if (wfd.cFileName[0] == L'.')
  {
  if (wfd.cFileName[1] == 0) continue;
  if ((wfd.cFileName[1] == L'.') && (wfd.cFileName[2] == 0)) continue;
  }
  // we either have a folder or a file
  if (wfd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY)
  {
  // recurse into it
  TCHAR szSubfolder[MAX_PATH];
  _stprintf(szSubfolder, L"%s\\%s", szFolder, wfd.cFileName);
  Enum(szSubfolder);
  }
  else
  {
  // do some work on the file...
  }
  } while (FindNextFile(hf, &wfd));
  FindClose(hf);
  }
  }

  Any further suggestions would be helpful. Is there a way for me to programatically find out if the folder I'm dealing with is "virtualized" That would tell me if the manifest was working as expected.

  I looked at the dwFileAttributes returned from the "Application Data" object and they include FILE_ATTRIBUTE_REPARSE_POINT. Is this significant

  
  

• devdept

  I've discovered more about this problem. It appears to be related to running in the VS2005 debugger, and it requires a certain situation to occur. It does not seem to be directly related to manifests and virtualized folders because I can get or not get the problem whether the manifest is present or removed. And, I determined that when the manifest is present it is being used, which I speculated about in an earlier post.

  After writing several test apps that did enumerations with no problem, I tried running the enumeration from other apps within my solution. These all demonstrated the problem. So, I added a new test project to my solution. It ran ok. I then added code to include a dll I share among all the apps in my solution. Just linking it in didn't cause the problem, but calling _anything_ in this dll triggered the problem. Then I discovered that running within the debugger demonstrates the problem, but running the same executable (in either debug or release builds) outside the debugger does not cause the problem.

  So, at this point it appears there is some weird circumstance related to using dlls that causes the FindFirstFile() to fail when run from within the debugger. I assume this is some bug in either VS2005 sp1 or more likely the Vista update beta. If anyone at Microsoft on the team working on VS2005 wants to follow up, post an email address and I'll get in touch. I don't see what more can be done on my end. I'll just have to comment the enumeration code when I want to debug unless someone has some other suggestions for further tests.

  
  

• Moim Hossain

  I have had to resort to a terrible kludge to resolve the problem of the enumeration returning multiple locations for the same file. I've moved the enumeration out of the service, basically redesigning the application to accomodate vista's schizophrenic behavior. The kludge has drawbacks compared to the original design, so this changes as forced a weakness into the application.

  Frankly, any excuse microsoft makes for this terrible terrible behavior of their new os is just self-centered rationalization. File system enumeration is broken in vista from the perspective of applications.

  
  

• Duncan_

  YES OF COURSE YES

  THE ANSWER, OBVIOUSLY, IS THE REPARSE POINT FLAG.

  And this not new to Vista.

  This special folder, and many others, is a so-called reparse point. This not a hard link nor a soft link, but just a junction point. The same thing, for example, that appears when you mount a volume on an existing folder rather that on a letter.

  What's new in Vista is that Reparse Points can now point to a subfolder and not only to a volume root.

  And here, this special folder is a Reparse-Point (a junction point) that points to ... its father.

  So when an old program wants to write a file to ApplicationData, he will in fact writes the file in the parent's folder. This is because the whole profil organization has been changed in Vista.

  So when you recursely browse, of course you should test the attribute in THIS SPECIAL CASE.

  But, generally speaking, you should always check that you will not enter a more complex loop. For example, on a very low folder, you can 'jump' to C:\ and restart the while thing....

  This has absolutely nothing to deal with VS2005, debugging and so on.

  --SergeFs

  
  

• ShadowFil

  Although this bug is titled correctly, the description of the problem is incorrect. In fact, FindFirstFile() does not return the same folder until the stack overflows. Instead, it returns the same folder name until the buffer holding the name overflows because the buffer was declared MAX_PATH, which is too small.

  If you enumerate drive "C:" with a large enough buffer, the enumeration does eventually reach an end and FindFirstFile / FineNextFile will enumerate everything successfully. Some folders may look very odd with several dozen subfolders all named "Application Data" and some folder and file paths may be longer than MAX_PATH. I say "may" because the enumeration may or may not return these odd looking paths.

  The problem does not seem to be related to VS2005. Instead, it seems to be related to whether the application is running "requireAdministrator" or "Run as administrator" and to some other circumstance I haven't figured out. What I do know is that two nearly identical programs can enumerate the file system and return two different sets of results. Whether a manifest with Vista's "requestedExecutionLevel" is present or not may or may not affect the enumeration results - I have seen test programs work identically whether "requestedExecutionLevel" is present or absent, set to "asInvoker" or "requireAdministrator". I have also seen them work differently. It is very confusing.

  Since I have not found a consistent way to demonstrate one behavior vs. the other, I can only say "I've seen inconsistent behavior". At this point I assume the behavior is intended. So, this post has devolved into a simple warning that you may encounter this behavior.

  
  

• Warner Young

  Despite hours of trying I have not found a solution. This problem is in fact worse than described. Another behavior is the multiple reporting of the same file in different locations. For example, I have one file located in a subfolder under My Documents. A full enumeration returns 4 instances of this same file, each with a different path and each with a unique pidl. It is utterly infuriating.

  I guess if enough people run into it someone from Microsoft might actually explain what's going on. Since the pidls are different for each path that points to this same file, I assume switching from FindFirstFile to a shell based enumeration will also exhibit the problem. Since a shell based enumeration is much slower as well as being much more complicated to code, I have not bothered to test it.

  
  

• jdrawmer

  Just some additional information.

  This infinite return of "Application Data" occurs for other virtualized folders in addition to "Documents and Settings", e.g. "ProgramData".

  I used ResHacker to see what the manifest looked like in my exe. It looks as expected with the "requestedExecutionLevel" showing up.

  One thing leads me to believe that although the manifest changes appear to be in the exe, they may not be getting picked up. I change "asInvoked" to "requireAdministrator" and did not get prompted to boost authority.

  Here is the manifest as returned from ResHacker (slightly edited to preserve anonymity).

  < xml version="1.0" encoding="UTF-8" standalone="yes" >
  <assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">
  <assemblyIdentity
  version="11.0.0.10"
  processorArchitecture="X86"
  name="AppName"
  type="win32">
  </assemblyIdentity>
  <description>AppName Utility</description>
  <dependency>
  <dependentAssembly>
  <assemblyIdentity
  type="win32"
  name="Microsoft.VC80.DebugCRT"
  version="8.0.50727.762"
  processorArchitecture="x86"
  publicKeyToken="1fc8b3b9a1e18e3b">
  </assemblyIdentity>
  </dependentAssembly>
  </dependency>
  <dependency>
  <dependentAssembly>
  <assemblyIdentity
  type="win32"
  name="Microsoft.Windows.Common-Controls"
  version="6.0.0.0"
  processorArchitecture="X86"
  publicKeyToken="6595b64144ccf1df"
  language="*">
  </assemblyIdentity>
  </dependentAssembly>
  </dependency>
  <ms_asmv2:trustInfo
  xmlns:ms_asmv2="urn:schemas-microsoft-com:asm.v2">
  <ms_asmv2:security>
  <ms_asmv2:requestedPrivileges>
  <ms_asmv2:requestedExecutionLevel
  level="asInvoker"
  uiAccess="false">
  </ms_asmv2:requestedExecutionLevel>
  </ms_asmv2:requestedPrivileges>
  </ms_asmv2:security>
  </ms_asmv2:trustInfo></assembly>

  
  

• siaj

  We are also facing exactly the same problem in one of our commercial products. Has anyone found a solution to this.

  Regards

  Swapnil D.

  
  
• Inconsistent File System Enumeration with FindFirstFile / FineNextFile