A comma is a valid file name character so try using a semi-colon to separate the files instead, e.g.
NOT:*.avi;*.mp4;*.mkv
This will show all files AND folders that do not match the expression. If you only want files you have two options:
Use attributes
Go to the attributes tab and set the Folder attributes to Off
OR,
Use the File name preset
You could change the expression type from DOS Expression to Files Only, but this also requires changing the actual expression. When you use a File name presets the expression type defaults to Boolean. So you would need to change the expression to something like
NOT (.avi OR .mp4 OR .mkv)
or you can even stick with your original DOS Expression using the DOSEX operator, e.g.
DOSEX "NOT:*.avi;*.mp4;*.mkv"