I do this for audiobooks especially - i'll keep one of those tiny little notebooks in my back pocket, pause the book and jot a note, or look a character up. This even becomes fun with a murder mystery because I'll keep track of motivations and clues.

Every character can have a page (or half) with a note or two if you feel like it. Sometimes just writing their name down helps you remember.