Hey guys, I've been learning about props lately and have a few questions

I have an armattan marmotte DJI that I use a gopro 10 on. The quad is 887g AUW

It is a heavy quad and I'm trying to figure out how I can get the motor heat down.

I was using a 5146 prop. Then I switched to a 5236. Now I'm trying a 5226.

Would using a 4.9" prop with a higher pitch be better than using a larger diameter prop with lower pitch for a heavy quad?

You would have to determine if the heat really comes from the motor load (from the props or how hard you push it) or from D-Term noise. I tend to believe it's noise/D-term
For heavier builds I would not reduce prop size. Pitch angle of the props also primarily helps with speed. It depends on your flying style and what you want to achieve. You can try if the 4.9 suits your flying style but if all the props you tried didn't reduce motor temp, prop size is not your problem....

what motors do you use? what kind of heat do you get?

as jasc pointed on it, prop choice doesnt need to be the source of the heat. prop choice should be done by handling/feeling or performance and efficiency. if motor load is the issue, another motor or decrease weight should be aimed than.
there is a nice article by oscar about the theme; https://oscarliang.com/mini-quad-motors-overheat/