Currently I am creating an application where a modal dialog box opens when the NewForm.aspx page for a WSS list is opened, to search against an external data source and pass the results back to the form (I'll let you know if this proves to be a viable design pattern!).
The default behaviour of the modal dialog has been driving me a little crazy - when the form in the ASP.Net page in the modal dialog box is submitted by clicking on a button in that page, that same form was opening in a new browser window. The dialog was also remaining open.
Finally came across the solution tucked away in a forum answer by Ken Dopierala Jr. His tip:
Remember, when you post back to a Modal page you must put: <base target="_self"> in the <head> section of the HTML on that page.
And that is all it needed to take the nastiness away, just adding that simple tag into the PlaceHolderAdditionalPageHead placeholder of the WSS page being presented in the dialog. Thanks, Ken!