The poker game Lynch saw before arriving was actually Mr. Fox passing the time by deliberately "losing cards." Simply put, he was using losing money to gather some ordinary old men around him.
Some might think this actually shows loneliness and consider Mr. Fox a pitiful old man with no friends.
But thinking from another perspective, at least he still has money...
Lynch saw right through it. Someone who deals with money all day and isn't strictly a good person, being left with very few chips in front of him isn't normal.
He didn't think Mr. Fox was pitiful. Maybe in Mr. Fox's heart, those who play with him and take advantage of him are the pitiful ones. At least he wouldn't trade the most valuable time of his life for some insignificant money.
