I have FLYWOO GOKU GM8 MINI V2.0 GPS W/ COMPASS and magnetometer shows red in INAV5, gps looks fine. I picked QMC8553 in configuration tab

Make sure the compas isn't too close to other equipment and noise sources.

But it shouldn't show up in INAV as RED icon if it is detected right ?

How have you wired the Compass?

If you go into the cli and type “status” do you get any I2C errors?

IIRC the red logo in the INAV configurator indicates its detected the compass but you have probably not calibrated the compass.

I tired to calibrate few times, here is the status:

# status
System Uptime: 28 seconds
Current Time: 2022-07-10T00:38:31.610+00:00
Voltage: 0.00V (1S battery - NOT PRESENT)
CPU Clock=216MHz, GYRO=MPU6000, ACC=MPU6000, BARO=BMP280
STM32 system clocks:
SYSCLK = 216 MHz
HCLK = 216 MHz
PCLK1 = 54 MHz
PCLK2 = 108 MHz
Sensor status: GYRO=OK, ACC=OK, MAG=UNAVAILABLE, BARO=OK, RANGEFINDER=NONE, OPFLOW=NONE, GPS=OK, IMU2=NONE
Stack size: 6144, Stack address: 0x20010000, Heap available: 1820
I2C Errors: 60, config size: 9704, max available config: 32768
ADC channel usage:
BATTERY : configured = ADC 2, used = ADC 2
RSSI : configured = ADC 3, used = none
CURRENT : configured = ADC 1, used = none
AIRSPEED : configured = none, used = none
System load: 10, cycle time: 510, PID rate: 1960, RX rate: 49, System rate: 9
Arming disabled flags: HWFAIL RX CLI
VTX: not detected

How to did you connect the I2C from the Compass to the FC?

What FC are you using? Does it have a proper INAV target?

Yes, I am pretty sure target is correct , INAV/FLYWOOF745NANO , I can fly acro without a problem with INAV 5. I just replaced the stock GPS on my flywoo explorer (looks the same...), there was a plug there already.

Even if it had a plug, did you try confirming the pinout? There are no standards for jst connector pinouts in the FPV hobby and so we need to do our own due diligence.

Does the same GPS/Compass unit work on another FC?

The Flywoo Explorer LR4 has the GPS version without compass as stock.

Flywoo do have 2 versions of the GM8 Mini V2.0 GPS. With and without compass. The difference is that the QMC8553 chip is missing.
It looks by the pictures that they use the same connector/cable for both but you have to solder the SCL and SDA signal cables yourself between the GPS and the FC.

I'm not sure of this but i'm about to do the same "upgrade" myself and had a good look at the pictures at Flywoos site and i think this could be your issue if You just swapped the GPS unit using the original cable.