Height to Normal
A height map says how far each pixel sticks out. A normal map says which way the surface is facing. Getting from one to the other means measuring how fast the height changes: the steeper the slope, the further the normal tips over.
Strength is the setting that matters. Too low and the detail vanishes as soon as it is lit. Too high and flat areas turn to noise, because you are amplifying whatever compression mess is in the height map along with the real detail.
DirectX or OpenGL?
Unreal and 3ds Max want DirectX. Unity, Blender and glTF want OpenGL. If dents come out as bumps you picked the wrong one, and Flip Green will fix it without converting again.
The result came out noisy.
The height map is probably 8-bit and banded, and every band edge becomes a little ridge. Blur the height map slightly first, or lower the strength.
Can I do this from a photo?
Not directly, because brightness is not height. Material from Photo works out a height map first and goes from there.
Related: Flip Green · Normal Strength · Material from Photo