You basically need some kind of value function that estimates how good one assignment of teams is. For example, if each player has score between 1 and 100 your value function could simply be to minimize the difference between the strongest and weakest team. Typically you design this by hand. Then you run a constraint optimization method that makes sure that each player gets assigned to exactly one team and probably also takes team size into account. Then you can optimize this. It's not really ML but more of an optimization problem. Though if you really want to you might try to learn a player score, although it might be hard to collect training data for that.