The survey of workers such as taxi drivers, hotel workers and restaurant staff - called "economy watchers" for their proximity to consumer and retail trends - showed their confidence about current economic conditions rose 2.7 points to 39.4 from 36.7 in October.
In October, Japan raised a twice-delayed sales tax to 10% from 8% while a powerful typhoon dented consumer sentiment and business activity, sending the sentiment index in the month to its lowest since May 2011.
The outlook index, which indicates the level of confidence in future conditions, rose 2.0 points to 45.7 in November from 43.7 in October.
A survey reading above 50 indicates optimism while a reading below 50 indictates pessimism.
The index showed "the pace of recovery has been slow recently and there were still some reactionary falls from rush demand ahead of the tax hike," the government said.
The Cabinet Office started compiling the data in comparative form in August 2001.
"The pace of declines in sales has slowed compared with three months ago. The impact from the sales tax hike is waning comparatively," said a respondent from a restaurant.
Another respondent from a department store said "About two months passed since the sales tax hike, but its impact still appears in sales of jewellery, cosmetic goods and clothing."
Slowing global demand has also affected business sentiment.
"Demand from overseas remains low, it is hard to expect sales to increase," said a general machinery firm.
"It's likely the tax hike has made consumers more frugal," said Yoshiki Shinke, chief economist at Dai-ichi Life Research Institute.
"The economy in the fourth quarter could shrink more than previously expected as consumer spending and factory output is weak," he said.
Japan's cabinet last week approved a $122 billion (95 billion pounds) fiscal package to support the economy amid offshore risks and as policymakers look to sustain activity beyond the 2020 Tokyo Olympics.
(Reporting by Kaori Kaneko; Editing by Chang-Ran Kim and Sam Holmes)
By Kaori Kaneko