Zero hit points means temporary insanity

Here's a rules change that makes the game more of a realistic role-playing game and less of a mindless-enemy wargame:

Whenever any creature is reduced to zero hit points, that creature becomes incapable of fighting or taking any useful or strategic action. It suffers temporary insanity, which usually takes the form of wildly fleeing from combat. Most of the time, the GM can simply remove enemy minis from the table when this happens.

Players whose characters fall to zero hit points are not removed from the storytelling. When their turn comes up in the initiative order, they should narrate their character's panic attack or other emotional freakout. Allow and encourage them to chew the scenery and steal the show. Tell them that, due to the temporary insanity, they cannot be strategic or act rationally or have any self-control, so their character's most extreme personality defects should be exhibited without restraint.

Player characters or key NPCs who start their turn with 0 hit points make saving throws for prolonged insanity. These follow exactly the same rules as death saving throws, including suffering a failure when taking damage, but three failures results in prolonged insanity rather than death. Players make the saving throw before roleplaying their insanity, using its result to guide the narrative. 

Players can use their action to talk to a temporarily insane creature and attempt to stabilize it, which requires a successful DC 15 skill check. This is usually a Charisma (Persuasion) check, but other skills may apply. Give advantage on this check for good roleplaying. If the insane creature is fleeing from combat, and the person talking is also retreating alongside it, the talking can be done as part of a Dash action.
A stable creature doesn’t make insanity saving throws, even though it has 0 hit points, but it does remain unable to take any reactions or actions other than Dash, Disengage, Dodge, and Hide. The creature stops being stable, and must start making insanity saving throws again, if it takes any damage. A stable creature that isn’t healed regains 1 hit point after 1d4 hours.

A character suffering prolonged insanity becomes an NPC under the GM's control, although all of the 'revivify' and 'raise dead' type spells and effects can be used to cure this insanity and return control to the player. If the GM is feeling nice or does not want to derail the plot, they can return the character to the player's control after the episode of insanity causes at least 500 gp worth of their possessions to be lost or destroyed (or accumulates 500 gp worth of debt or obligation or legal troubles). Another option is to add a random type of Indefinite Madness.

Massive damage does not kill creatures instantly. If the remaining damage after dropping to 0 hp exceeds a creature's hit point minimum, it falls unconscious and starts to make death saving throws.

Discussion

One of the worst parts about RPGs is the fact that by default all fights end in death or incapacitation. This is completely inaccurate from a historical or scientific point of view.

Historically, almost all battles and skirmishes ended in a retreat. It was extremely rare for more than 10% of an army to actually be killed in combat. When things started to go badly, or when enough of your allies started to think that things were going badly, everyone ran away.

Similarly, in nature, it's extremely unusual for animals to fight to the death. As soon as one of them has clearly established dominance, the other submits or runs away. If a predator attacks a large prey animal and the prey animal fights back enough to be dangerous, the predator will usually give up rather than avoid risking serious injury.

I think that the 'fight to the death' trope started with Tolkien and other fantasy writers. Tolkien was enough of a historian to know the facts about ancient battles. He deliberately described his battles as unusually lethal in order to emphasize the rare nature of the events he was describing. But subsequent generations of readers didn't realize what normal was, and assumed that extremely high casualty encounters were typical.

There is a lot of discussion of this online, with many attempts at houserules to make retreating a more attractive option. But I think the cleanest way to fix this problem is to really lean into the idea that hit point loss is an erosion of willpower rather than a physical injury. Also see my Temptation and Willpower post.

With this rule change, by default, a combat loss leads to a retreat rather than everyone dying. As long as anyone capable of healing or talking is above zero hit points and chooses to retreat when they see their colleagues suffer temporary insanity, it is unlikely that anyone will suffer prolonged insanity. This allows the GM to introduce much more difficult encounters, either deliberately or by rolling at random, without risking derailing the plot or ruining everyone's day. 

Wounds

This rule change does require you to rethink what a hit point of wounding means, but I think that is an improvement. One of the weirdest things about the game has been getting a feel for what it means for a character to increase from 10 to 100 hit points. Video games illustrate this in an absurd way, by giving characters the supernatural ability to run around after having taken half a dozen lethal wounds. But with this approach, a sword strike for 12 hit points is assumed to be a glancing blow that causes a little bit of bleeding, enough to cause someone unfamiliar with combat to freak out and run away, but something that a veteran could continue to fight through.

This framing is much more historically accurate: "Marcus Servilius, a man of consular dignity ... parted his garment and displayed upon his breast an incredible number of wounds. ... and turning to Galba, said: ‘Thou laughest at these scars, but I glory in them before my fellow-citizens, in whose defence I got them," h/t Bret Devereaux, who explains, "The modern reader may be puzzled by the repeated framing in ancient texts of units in combat being ‘wearied by wounds’ since generally speaking a soldier wounded in combat with modern weapons is typically a lot more than ‘wearied’ by the experience. But pre-gunpowder weapons aren’t that lethal, especially with blows land against the limbs instead of the head or the chest (which might be better armored in any case)." 

Special Cases

Different types of enemies will react differently to dropping to zero hit points. Some might play dead or cower in fear. Constructs, mindless undead, and some extraplanar creatures should follow the normal rules, where zero hit points is physical incapacitation.

Enemies with good unit cohesion or a sense of martial honor will not flee individually. When reduced to zero hit points, they will either fall down and fake a debilitating injury, or stay in the line of battle and simply take the Dodge action, perhaps while pretending to act. The GM can remove them from the table for convenience, while explaining they they are actually still on the field. Then, when a critical mass of the enemy unit, or its leader, has been reduced to zero hit points, they will all flee at once.

Hunting zero hit point enemies

If the players need prisoners to interrogate, or they need a key item that an enemy is carrying, they may need to attack or hunt down fleeing enemies.

When a GM-controlled creature drops to zero hit points, it immediately drops whatever it is holding and, using its reaction, moves its speed in whatever direction it thinks is most safe. This action does not provoke opportunity attacks. If a player announces that they plan on hunting it down, the GM should, instead of removing it, track its negative hit points (the remaining damage from the attack) and keep it on the map. In future rounds, it takes the Dash action to move to safety if it can. In some situations, it may instead take the Hide action. If cornered, it takes the Dodge action. When it has negative hit points equal to its hit point maximum, it falls unconscious.

If the enemy the players want to hunt was the last one fighting, then the GM should run a chase as the players track it down. If there are other enemies fighting, the players may also choose to transition into a chase by taking opportunity attacks from any remaining enemies they are adjacent to or must run past.