limit on number of files indexed?

What is the limit on the # of files indexed

My count for # of items indexed (in wds v2.6.5) never exceeds about 11% of my actual file count. (164,832 indexed of 1,500,000 actual files) Yes, I'm selecting all file/drives to index against. Yes, I've rebuilt the index more than once.

I'm guessing that for folks here thinking of indexing their entire business' file sets, be they on servers or local, the index has to scale well beyond anything I've seen so far, and yet...

Thanks



Answer this question

limit on number of files indexed?

  • Bendermon

    Paul Nystrom - MSFT wrote:

    Jeff,

    Thanks for the email providing further information. At this point, I'll let you know that we have filed a bug and handed the issue over to the development team. I'll let you know when we have some additional information.

    Paul Nystrom - MSFT

    Paul,

    Did this bug's fix get included in the released version (non-beta) of WDS

    I haven't been willing to re-try WDS until I hear more.

    (I did see in another of my threads that you replied to my other bug found about not indexing .one files - Thank you).

    -Jeff


  • Alex MacFarlane

    Hi Paul,
    Thanks for the reply. I've been trying to get answers for weeks w/o luck, so any reply is a good start.

    I'd say 40% of the files are jpg's, 40% are flac, and there are significant %'s of .doc and .txt.

    Yes, the locations where the .jpg and .flac filetypes are stored are included in the sources to index, yet the problem behavior remains. I am not excluding any file types, except the defaults on my %SYSTEM% drive (C:\)

    While I do have substantial .pst files to index eventually, in fact these low index 'counts' are excluding my Outlook files.

    I upgraded to WDS v3.0 beta 2 today in an attempt to see if the newer code would help.

    I found that the new WDS model's to more closely using the Windows standard user interface with regards to file extensions. I see .flac in the file types list, but there is no option to associate the extension with a new filter (music file type).

    With WDS 3.0 the indexed items count went to 82,420 before claiming there are 0 left to index of the 1.5M files.

    So my review is +1 to the closer to standard GUI, -1 for no better results, and -10 for not handling known file types like .txt (see below).

    My install experience on XP sp2 with wds 3.0 was very problematic:
    a.) at first Search service wouldn't start, w/o any user feedback such was true
    b.) then Windows Defender identified some of the Search dll's as possible malware vectors (as seen in event viewer, but no where else),
    c.) indexing gave different 'count' results at the same time depending on where you looked at the UI for status, 8k in the status window, 36k in the options window
    - after a reboot, results now match
    d.) the Search service kept dying (20+ times) (via event viewer logs) w/o useful error codes.
    e.) entire drives that WDS indexed OK were 'grayed' out due to their "folder's" not having been set to allow indexing via the "folder properties" sub-window
    f.) my file types extension list doesn't go past the file type starting with .mda as seen from the Advanced options menu of WDS. - THIS is really surprising! .txt isn't even in the list! But of course I have known filetypes using letters starting with letters past M in the alphabet.

    Is there somewhere else to report WDS beta 2 problems to that I should seek

    Thanks again,
    Jeff

  • sureshsundar007

    Paul,

    I composed a multi-point, multi-paragraph relpy that somehow just disappeared as I 'posted,' so pardon this short version.

    Thanks for the info. I've been doing more testing and have posted further questions and observations that I think would be worth your time (and others) to look over.

    I believe there's a lot of value to had in making search extend much farther into people's data, but ONLY without it being shared to Google and the like. So I value what wds is about (and phlat if it comes back to working with newer code), but I will never value it's online counterparts for myself or my clients.

    Thus, I hope wds improves a LOT in the short term (pre-Vista) and definately until it fully handles not just what it is meant to do today, but manages any form of meta-data searching along with the file data itself.

    Good luck,

    Jeff


  • Thaina

    Hello SpeedNut,

    The WDS index should easily handle well over one million files (though the index will obviously grow fairly large). There could be several reasons that you're index is not going up over the apprx 170k items.

    1. What type are the majority of your files Are they on the default list of files to index If not, you might want to add the extension of these files to the items to index as text.

    2. Are many of your files stored in some sort of email system (example: Outlook or Lotus Notes) If this is the case, you'll want to make sure that for Outlook: you are running in cached exchange mode and that, for Lotus Notes, you have the Notes protocol handler installed.

    Please let me knwo if these suggestions help.

    Paul Nystrom - MSFT



  • vaishalli

    Jeff,

    Thanks for the email providing further information. At this point, I'll let you know that we have filed a bug and handed the issue over to the development team. I'll let you know when we have some additional information.

    Paul Nystrom - MSFT



  • perstam

    Jeff,

    This is the perfect place to report this feedback. I'll be getting your information over to our development team ASAP. It might help if we could contact you directly. Would you mind sending me an email a-paulny@microsoft.com

    Paul Nystrom - MSFT



  • limit on number of files indexed?