G and mass M can be measured with less precision individually, but the product G*M = mu, the standard gravitational parameter, can be measured with higher precision, and is more useful anyway, since most applications require or can use GM, not (necessarily) G and/or M separately.

But unless the distance from the mass is great enough to treat it as a single point mass, then well below even 6 digits of precision, the actual force of gravity at a given location cannot be treated as simply GM/r^2 . So a more precise value of G, M, or even GM, will not, by themselves, give a more accurate answer. Planets and other "spherical" celestial bodies are not perfectly spherical, and have topography, and an internal mass distribution that is not radially symmetric. Depending on the location and reference frame, you also have to consider centrifugal acceleration.

Because of the equatorial bulge and centrifugal acceleration, the effective force felt on Earth's surface is on average 1% lower at Earth's equator than at the poles. The non-uniform gravity field of Earth, and even more so the Moon, must be considered for spacecraft and artificial satellites orbiting these bodies.

So what does your statement even mean then? Your answer is only good to however many decimal places your least precise measurement/constant is. But what do you mean that mass is only good to three decimal places? You can have as accurate a mass as you want but the answer, a force in Newtons, will only be good to 3 or 6 or however many decimal places in your least precise constant or measurement.