The yield on the 20-year JGBs dropped as much as 5.0 basis points to 0.400 percent, hitting its lowest level in nearly two weeks.
Long-dated JGB yields have been soaring since the BOJ's last policy meeting in late July on growing expectations that the central bank may reduce buying in long-dated bonds.
Such views strengthened after BOJ officials including Governor Haruhiko Kuroda and Deputy Governor Hiroshi Nakaso acknowledged concerns about the unfavourable impact of its negative rates on financial institutions.
The 20-year yield had risen to a six-month high of 0.495 percent last week from a record low of minus 0.005 percent touched in early July.
After the big rise in yields, profit-taking was seen in steepening trades ahead of the BOJ's policy announcement on Wednesday.
Some market players expect the BOJ to cut interest rates deeper into negative territory and to also tweak its bond-buying to help steepen the yield curve, possibly by reducing or even scrapping buying in long-dated bonds.
Others in the market believe the BOJ is more likely to spell out such a strategy only as options for the future while refraining from making any immediate policy shifts, given there have been limited changes in economic fundamentals.
Traders say the current levels of short-term bond yields and other instruments suggest markets are pricing in only about 50 percent chance of a rate cut on Wednesday.
The two-year JGB yield was flat at minus 0.270 percent , still above a record low of minus 0.370 percent touched just before the BOJ's previous meeting in July.
The benchmark 10-year JGB yield dipped 2.0 basis points to minus 0.060 percent.
The price of 10-year JGB futures rose 0.20 point to 151.89 .
The results of the BOJ's buying of 750 billion yen JGBs, including 12 billion yen in maturities of 25 years or longer, also showed limited selling pressure, traders also said.