The tracks would have to be replaced in order to operate passenger service over them. The current tracks can support up 80mph in some areas with 60mph bing the highest along most of the route. So to upgrade to 90-110mph wouldn't be that hard to do and add more double track to run more service and to reduce conflicts with freight trains which is possible since most of the ROW used to have 2 tracks with one removed. You can run 3 RTs between Bangor and Boston with an overnight service extending into Canada to Halifax. The current Downeaster service loses money and so does the Interstate highway / road network no one complains about that.