0 votes

I regularly need to search very large repositories, e.g. 5-10 TB. FileLocator Pro works fine most of the time but sometimes my search terms can pull back thousands of results and can cause Out of Memory issues. How can I avoid this?

by (715 points)

1 Answer

0 votes

When running unrestricted searches it's very easy for a single large file to take up a lot of memory, especially if the files are non-text based, e.g. EXE or DLLs, because they don't have standard End-Of-Line (EOL) identifiers.

FileLocator Pro will default to capturing the first 10,000 hits in a file but many times this is overkill when you only need to find just the first few.

How many lines do you need to capture?

Therefore unless you need ALL hits I'd recommend changing the Configuration->General->Display Settings to reduce the captured hits, something like

Max displayed lines per file: 20

Also, if you don't need the surrounding lines reduce those to zero, e.g.

Lines before: 0
Lines after:  0

How many characters on those lines do you need?

Another important setting is the maximum characters per line, especially for file types which don't have text-based data and therefore don't have EOL markers. The default max line size is around 20k but can be changed in the Configuration->General->Character processing Settings. If you're expecting the data to have very long lines reducing this value can have significant impact on memory consumption:

Max chars per line: 256

Will you be using ALL the results?

If it's possible that you won't actually need all the results, ie you'll just be browsing the result list and picking a few files, then switching on Just-In-Time (JIT) searching can have a dramatic effect on memory use. JIT searching only searches enough of a file to match the result, it doesn't actually collect any line/hit information until the user clicks on the file. Therefore each file only takes up a few bytes until the full results are required for display or export.

To switch ON JIT Searching go to the Configuration->General->Search Settings and set:

Calculate Hit Count: OFF
by (715 points)