Overview
Game engines with a GUI editor are extremely intuitive, and lets you focus on iteration, experimenting, and prototyping quickly to get your game idea out there to see if it is any good. The number of developers opting to use an engine that has an editor are way higher then those engines that simply just contains a C++ library.
This old engine supported textures, debug drawing (lines, shapes), and a scripting language implementation all using The Forge API.
The Problem
Adding an editor is complicated on top of an engine. My first initial thoughts are creating a non-gui version of some engine first, and then using that as a .dll that the editor will query. The engine I made was completely data driven and read from binary files to check positioning for example. The editor layer will interact with the data alone, meaning the engine has no idea that an editor layer even exists.
GUI Framework Choice
I used Qt for the editor. I thought to choose imgui but I figured that I will have another project soon that I will use imgui for, so I will opt to use a familiar technology that I have used before. I never fully got how I can use the newer qtquick libraries to work with the engine, so I mainly used the older QtWidgets instead. The main technique for the editor to work was mainly getting the custom Qt widget and its hwnd paramter into the graphics swapchain. This way it is kinda like a Win32 way of working dealing with the raw hwnd.
Either way, Qt has a rich set of widgets already made and available, and it was just a matter of looking at examples and looking on existing projects to see how they styled and/or created. imgui has the same flexibility and modularity which is nice as well.
Integration Challenges
It look a fair amount of time to think in terms of an editor and how it behaves in contrast to the engine itself. For example, I knew I had to make the editor have a completely different camera system compared to the game. I also had to track the states of whether I am in "running game" mode or an "editor mode". In other words, everything changes functionality wise when you are in one of the other state.
Editor Features
I used the flatbuffers library to do a lot of the serialization for the scene editing. It took a bit of work since there was a build step during this process to create the needed header files from the flatbuffers build too, which you will then include in the actual engine to serialize and unserialize. This was my most recent feature I added, and perhaps the more difficult one I would say, though going back to do it a second time will probably be easier.
Reflection
I tried a bit using a reflections library to get some of the data from C++ onto my GUI, but this was also a big challenge to get work. I did not get that far but the reflect-cpp library is really nice! Also, looking forward to the reflection standard as part of the C++ update, though I heard that the reflect-cpp style of reflection is completely different from the one proposed in the standard.