Most diversity controllers use RSSI to choose the best video source. RSSI indicates signal strength, but not necessarily signal quality. 

But there is another way. One could do diversity by examining signal quality. How? Minimosd uses the MAX7456 chip for getting the osd into the video signal. This chip also has a pin (LOS) that indicaties if there is any valid video signal.

Not all video receivers have RSSI pins exposed so a diversity controller that examines the video signal would be compatible with all video receivers that output analog video.

So here is my question: Why aren't there more projects exploting this functionality?

The LOS pin can only detect complete loss of sync, so you could easily end up with extremely bad image that is still able to sync and therefore not switch to the other antenna. It's an on-off function, not a gradual one.

The Fatshark diversity module uses the sync pulse quality instead of RSSI to do its 'antenna diversity' switching - and I think it does a decent enough job.

And as a follow up, do you have to have 'diversity' at the receiver and transmitter?

Diversity is normally on the receiver side