Being silent will never serve your marriage and this fantasy will keep happening…. Put a boundaries for this , this is just disrespectful for your privacy , a real man will never disrespect his wife privacy just to satisfy his fantasy , this is a reasons why most marriages ruins because of other significant fantasy .