Here's an interesting post from the November 25, 1885 edition of the Punxsutawney Spirit:
THE HEADLESS HORSEMAN
The Mystery Some Lawrence County People Are Trying To Solve
New Castle - November 20th: Smith Connor, Joseph Wolfe, Crawford Cunningham, George Ayres and John Stewart, all said to be reputable and reliable citizens of Chewton, Lawrence county, are the authors of the following narrative which they possessively assert to be true:
A few nights ago they were going toward Chewton in a small wagon, when there suddenly appeared before them a white horse. They could not see from whence it came nor could they account for its noiseless tread. Thinking it would get out of their road, they whipped up their own horse, which had become frightened and sped along on a hard run, yet the white horse kept an even distance ahead of them. Mr. Cunningham, about this point in the chase, observed, he says, a headless man leading the horse, and thinking some one was endeavoring to play a joke upon them, drew his revolver, and, after warning the supposed joker, he fired. This did not have any perceptible effect, and he then emptied the six remaining chambers of his revolver directly at the mystery. It didn't disturb either horse or man in the least.
By this time the men say they were thoroughly frightened. The phantom led them direct to Chewton, where, they say it stopped before the home of a well-known citizen and, pointing its finger at the door, disappeared as mysteriously as it had appeared.
This strange story is to a certain extent corroborated by another resident of that village, named Samuel Liebendorfer, a blacksmith, who says that a few nights ago he had occasion to go to his shop between 11 and 12 o'clock, when he observed about six feet from him the phantom horse and its ghastly rider. He says blood, or a substance resembling it, was plainly seen upon the shoulders and breast. The horse was headed direct for his neighbor's house. The whole town is busily engaged in endeavoring to solve the mystery.
An old resident named Parker declares that some 20 years ago this same strange occurrence happened, and a horse and rider similar to the one described by the observers of the present mystery, rode up to his neighbor's house, and in three days afterward that neighbor was killed - having fallen from his horse while going through the gate. This story has given considerable credence, and many claim that the visitation is a bad omen for somebody.
Astonishing Encounters: Pennsylvania's Unknown Creatures, Casebook 3
Weird Pennsylvania: Your Travel Guide to Pennsylvania's Local Legends and Best Kept Secrets
Pennsylvania Ghost Towns: Uncovering the Hidden Past
Silent Invasion: The Pennsylvania UFO-Bigfoot Casebook