The 20-year JGB auction produced the tail, or the gap between the lowest and average prices, of 0.41, much wider than 0.08 in the previous auction.
The bid-to-cover ratio also fell to 2.88 from 3.59.
The poor results hit the longest maturities hardest, with the 30-year bond yield rising 7.0 basis points to 1.490 percent.
The 20-year yield also rose 6.0 basis points to 1.300 percent while the 10-year yield rose 4.0 basis points to 0.510 percent.
The market was also pressured as Japanese share prices staged a strong recovery after sharp falls on Monday following surprisingly poor gross domestic product (GDP) data showing Japan slipped into a recession.
The Nikkei share average rose 2.2 percent by early afternoon, erasing almost three quarters of its losses on Monday.
The 10-year JGB futures prices fell 0.27 point to 146.00. But the shorter end of the curve was firm after the government's one-year discount bills sold at a negative yield for the first time at auction.
The one-year bills were sold at a high yield of minus 0.0009 percent. The issue was bought further in the secondary market later, trading at minus 0.010 percent.
That also brought down the three-month yield 1.2 basis points to minus 0.070 percent.