The dialog for mail rules is a simple GUI for administrators to create mail rules. The mail rule itself is stored as a formula in the $FilterFormula in the MailRule form.
In fact the Rules dialog and the script library "Rules" is used only for creating or editing mail rules. These elements are not used for the mail rule to work on the server. ( you can change the rules manually with tools like ScanEZ or such )
As only Contains, Not Contains, Is, IsNot is used to build the rule, you are not able to use pattern like *.domain.*
You could easily add this functionality to the existing rules script lib.
My idea is to enhance the conditions part of the rules dialog to be able to type in a freehand formula. Doing this you could use @Like to search for pattern in fields. Once the formula evaluates to true, the according action will take place.