You can't exclude files using the 'Look In' field and therefore you can't do it with the Multiple Folders dialog.
You should be able to do what you want using the option Include folder path in file name search (found in Configuration -> General -> Character Processing Settings).
With that option switched on you can use the File name field to match against the full path of the file and then switch to using Boolean expression mode to specify your criteria, e.g.
File name: *.pdf OR (*.msg NOT Folder1)
For example, to list all .txt and .xml files but not .xml files in the Agent Ransack folder:
