The 10-year JGB futures price fell 0.11 point to 147.29 while the yield on the current 10-year cash JGB yield rose 1.5 basis point to 0.390 percent.
US Treasuries prices fell on Monday on profit-taking after gains last week on weak US data, with many investors still nervous about spikes in volatilities in major bond markets around the world.
The auction of 2.5 trillion yen ($20.8 billion) five-year JGBs drew tepid demand, with bid-to-cover ratio coming at 2.80, the lowest level in three and a half years.
The results showed few real-money investors are eager to buy the paper below the yield of 0.10 percent as banks can earn the same rate of interest by parking funds at their accounts in the Bank of Japan.
The 20-year yield rose 3.5 basis points to 1.170 percent while the 30-year JGB yield rose 2.5 basis points to 1.420 percent.