Given the material that has worn or 'flaked' off the one lifter, there has to be some wear on the cam. You should really think about putting a new cam in it. Send a PM to PETE (on this thread) and send him a stock cam to have it reground. His work is second to none and he has a lot of different grinds to choose from. Your engine has a bunch of good parts in it - might as well make sure that the critical top-end portion of it is as good as it can be.
Many of us have learned, that one of the most important parts of a flathead build is --- the cam, the lifters and the valves . . . do it right now and you'll be happy for a long time.