Generally you should launch scriptUI scripts from the "Window" menu in AE, not from the "File" menu.
The error you are seeing can be caused when the script is not fully installed (you must unzip everything in the zip to your ScriptUI folder). It can also be caused by using an unsupported OS.
What OS are you running?