Yamadaya Hotel probably has the best view of Fijuyama around. It is a small, family-run ryokan, where they spare no effort in making lovely food and providing what they can. However, the building is slightly run down, and the baths are basic (albeit with great view of Fuji).
What they lack in infrastructure, they more than make up for in their meals and the friendly service from the family. It's good value for money, if you like Japanese food (one of the other groups was American, and they left more than half their meal on the table, which was a real shame, given the effort and care taken in the preparation).
I would definitely go back again for the unbeatable view of Fuji from their baths. Access by public transport is not difficult, as there is a bus stop just outside, but the bus schedule may not be too frequent.