Agent Ransack doesn't support Boolean expressions on the file name but you can achieve what you want, depending on which version of Agent Ransack you're running. See below.
FYI, FileLocator Pro DOES support Boolean expressions on the file name so you could do exactly what you describe, ie cat NOT mat
Agent Ransack 2014 (or higher)
With the new DOS Expression functionality you can use the format:
cat;NOT:mat
More information on the new syntax here:
Agent Ransack Help: DOS Expressions
Agent Ransack 2010 (or lower)
You'd need to use a some regex magic to achieve what you want. Using negative look behinds and look aheads this would work:
^(.(?<!mat))*cat(.(?!mat))*$
Don't forget to switch 'Regular expressions' on in the Options tab.