Since the game loads in massive swathes of area all at once, it can't keep up and becomes incredibly unstable. The biggest downside to the engine is how it loads in cells, making things like fast moving airplanes or cars in Gamebryo titles impossible. You can alter the time between cell resets in the game files, but things get really unstable after a couple hundred hours. Kill something, it's body will stay until the cell resets. Accidentally cause the death of an NPC, they will stay dead. Knock over a lamp, it will stay knocked over until the cell resets no matter how far away you go. Anything you interact with anywhere in the entire world stays that way until the cells all reset. It's ability to track objects is unparalleled. Gamebryo has strengths no other game does. There are huge DLC expansion sized mods that play like actual level-based FPS games, massive improvements to gunplay, armor setups, how encumberment is handled, item sorting, etc. If mods can make massive changes that fix so many of the limitations of the engine, including huge gameplay fixes and upgrades, there is no reason Bethesda can't. The main issue with Gamebryo and it's evolutions isn't the engine itself, but limitations of hardware and the people making games on it not really breaking the mold since way back in Morrowind.

Gamebryo is very capable of looking really good. There's no reason a new enigine can't be built on the backbone of Gamebryo. Way too old and inept to to the job, but has someone pretty it up for public appearances to trick people into giving them their money.

Gamebryo engine vs creation engine