Use 90%+ isopropyl alcohol or 100% acetone to wash it off using cotton balls / a fabric that doesn't fall apart easily. You could dunk your entire GPU into a vat of either of those, let it air dry, and run it within an hour.
Be wary that both of those substances are great for cleaning because of their high vapor pressure, however it is for this same reason that you can easily get sick from breathing in those fumes. I wasn't entirely aware of that so I got a pretty big headache when I worked with them when I applied a shunt mod to my GPU.
Rubber pencil eraser is the best way to clean these contacts, rubbing alcohol or other solvents can deteriorate the contacts. Rubber pencil erasers are naturally non-conductive, so you can't cause damage to your hardware by doing it.