If you're interested in learning more about time and time travel, here are a couple of amazing books on the subject:

From Eternity to Here: The Quest for the Ultimate Theory of Time (2010), by Sean Carroll (in my opinion, the best book on the physics of time)

A Brief History of the Philosophy of Time (2013), by Adrian Bardon (in my opinion, not only the best book on the philosophy of time, but the best on time ever written for the general public, and I've read quite a few)

EDIT: Fixed link.

>Heard a bunch of scientists talk that paradoxes make it impossible

There's nothing about time travel into the past that is logically impossible, as there are ways to get around things like the Grandfather Paradox.

The Novikov Self-consistency Principle, for instance, allows travel to the past by positing that no actions taken by time travelers will be able to alter the timeline in any way, as a matter of principle. A lot of people interpret this as some spooky godlike mechanism that's watching you and tweaking things to keep you from messing everything up, but that doesn't have to be the case. A better way of looking at it is that your time travel journey has always been part of the past from the get-go, as something that's always been inherently and deterministically baked into the fabric of the universe. In other words, your time travel actions will inherently be compatible with a logically coherent universe by virtue of you being able to time travel at all. This is easier to intuitively appreciate in the context of a block universe model of time, which is what arises from modern physics.

Some people don't like this because it implies that we have no free will, but there's no good reason to believe that we have libertarian free will in the first place. Try as you might to kill your grandfather, it's just part of the structure of the universe, which transcends your illusion of free will, that you will not succeed in doing so.

Another conceivable possibility deals with alternate timelines altogether, like with the Interacting Many Worlds theory of quantum mechanics, in which backwards time travel events force you into a quantum leap of sorts into another universe altogether, which you're free to "mess up" however you like.

All of this is assuming that time travel into the past is physically possible, of course, which is a real big assumption and currently an open question with grim prospects.