Now that I think about it, the answer might be simpler than I thought. Many primitive societies did not have written languages, so there would be a few people who spent their time memorizing long stories and other things word for word. If someone were disabled and couldn't hunt or farm, then it was only natural that they would fill that role. To the extent that that person was perceived as more knowledgable than others, people would take their advice on spiritual matters as well.