Latex can be harvested directly from trees, and still is, but most plastics used today are from petroleum.

Simply put, plastics are chains of carbon+other stuff. Petroleum is very rich in carbon+other stuff. Refineries break down big chains of petroleum into smaller parts. They then combine these smaller parts into other big chains, which are the various types of plastic we use today. The same can be done with natural latex, but that stuff isn't as abundant so we don't use it much anymore.

An analogy of this would be starting with a house. You break down the house into smaller parts: wood, nails, drywall, paint, and so on. You then use these parts to build a bunch of sheds.