Like many new file formats today .docx and .dotx Word formats are actually ZIP files with compressed XML (and other) files inside. It means that they are completely 'opaque' unless the data is extracted. However, the older .doc and .dot files are stored uncompressed, which means that a simple binary search over the data often produces adequate results.
Agent Ransack only applies special processing (ie IFilters or ZIP extraction) to .doc, .docx, .xls, .xlsx, .ppt, .pptx, .odt, .ods, .sxw, .sxc files. Other formats, such as .dotx, are only supported by the premium version of the product FileLocator Pro.
Agent Ransack will still try and search all file types but it's only those listed that have the 'special' processing applied. Therefore Agent Ransack may well produce adequate results on uncompressed formats, like .dot, but won't be able to do much with compressed formats like .dotx.