I don't think this is feasible. Even though the world is ultimately in a JPanel, with things like the hooks to capture keyboard and mouse input in the world, I reckon trying to display a GUI in the world isn't feasible without customising big chunks of Greenfoot. You'll have to live with the separate window, I'm afraid.

I looked into Greenfoot's source code. As @nccb said, the world is based on a JFrame. But, it use different classes when it run as a JAR file or in the Greenfoot program. For Jar packages, it's based on a class GreenfootScenarioViewer which extends from JApplet. When it runs in the Greenfoot program, it's based on a class called GreenfootFrame which extends from JFrame. I think it will help, if you look into Greenfoot's source