RBS said in a statement that it reached a preliminary deal with the US Justice Department "to resolve its investigation into RBS's issuance and underwriting of US residential mortgage-backed securities between 2005 and 2007".
It follows a separate fine of $5.5 billion agreed in July 2017 with the Federal Housing Finance Agency over the same matter.
Edinburgh-based RBS, which remains 71 percent owned by the British government after receiving the world's biggest banking bailout at the height of the subsequent 2008 global financial crisis, described the news as a "milestone".
"Today's announcement is a milestone moment for the bank," said chief executive Ross McEwan.
"Reaching this settlement in principle with the US Department of Justice will, when finalized, allow us to deal with this significant remaining legacy issue and is the price we have to pay for the global ambitions pursued by this bank before the crisis.
"Removing the uncertainty over the scale of this settlement means that the investment case for this bank is much clearer."
The lender added that $3.46 billion of the settlement will be covered by existing provisions, while it will take another $1.44billion charge for the second quarter of 2018.