TeH_Venom t1_je21d7u wrote on March 28, 2023 at 8:59 PM Reply to Is it possible to merge transformers? [D] by seraphaplaca2 Not quite cross model architecture, but it's not impossible to merge different fine tunes of a model into one. I personally have a few scripts for a few strategies such as Average merge; Diff merge; Block merging. (link) I haven't tested diff merging or block merges too much (me and a friend finished adapting SD's block merge to LMs last week) but weighted average merges are a pretty safe way of mixing models. Permalink 1
TeH_Venom t1_je21d7u wrote
Reply to Is it possible to merge transformers? [D] by seraphaplaca2
Not quite cross model architecture, but it's not impossible to merge different fine tunes of a model into one.
I personally have a few scripts for a few strategies such as
I haven't tested diff merging or block merges too much (me and a friend finished adapting SD's block merge to LMs last week) but weighted average merges are a pretty safe way of mixing models.