Where the assumptions behind the two-tower model architecture break — and easy methods to transcend

Two-tower models are amongst probably the most common architectural design decisions in modern recommender systems — the important thing idea is to have one tower that learns relevance, and a second, shallow, tower that learns observational biases similar to position bias.

On this post, we’ll take a more in-depth have a look at two assumptions behind two-tower models, specifically:

• the factorization assumption, i.e. the hypothesis that we will simply multiply the chances computed by the 2 towers (or add their logits), and
• the positional independence assumption, i.e. the hypothesis that the one variable that determines position bias is the position of the item itself, and never the context through which it’s impressed.

We’ll see where each of those assumptions break, and easy methods to transcend these limitations with newer algorithms similar to the MixEM model, the Dot Product model, and XPA.

Let’s start with a really temporary reminder.

Two-tower models: the story up to now

The first learning objective for the rating models in recommender systems is relevance: we wish the model to predict the very best possible piece of content given the context. Here, context simply means all the things that we’ve learned in regards to the user, for instance from their previous engagement or search histories, depending on the appliance.

Nevertheless, rating models normally exhibit certain remark biases, that’s, the tendency for users to have interaction roughly with an impression depending on the way it was presented to them. Probably the most distinguished remark bias is position bias — the tendency of users to have interaction more with items which are shown first.

The important thing idea in two-tower models is to coach two “towers”, that’s, neural networks, in parallel, the major tower for learning relevance, and…