Two-Stage Architecture: 38% Precision Lift Over Rules

```html

A benchmark study on local commerce leads reveals a striking pattern: a two-stage recommender outperforms a rule-based baseline by a wide margin, but the advantage vanishes when only high-volume categories are considered. This suggests the lift is not from a smarter algorithm but from the architecture's ability to separate recall from precision, allowing the ranking stage to exploit features that rules ignore.

In the first stage, candidate generation prioritizes recall, casting a wide net over potential matches. The second stage, a ranking model, then applies a richer set of features—including contextual signals like company demographics and user interaction history—to refine the list. Rule-based systems, by contrast, typically apply a single set of heuristics that conflate these objectives, often overfitting to frequent patterns and missing long-tail opportunities.

The implication for practitioners is clear: the precision gain is a structural benefit, not a tuning artifact. When evaluating such systems, one must look beyond aggregate metrics and examine performance across category segments. The architecture's true value emerges in niche contexts where rules are too rigid to adapt, as documented in recent research on hybrid business recommender systems and corporate portal usage.

Two-Stage Architecture

The benchmark numbers are the clearest way to see why the two-stage architecture wins: the retrieval stage hit 92% recall, while the rule-based baseline—filtering by rating >4.5 and distance—managed only 78% recall. That 14-point gap at the retrieval layer is the entire ballgame. If the candidate set is missing relevant merchants, no ranking model can recover them. The rules are not slightly worse; they are structurally incapable of surfacing what they never see.

The retrieval stage runs alternating least squares (ALS) on implicit click data. This is a deliberate choice. Click-through is sparse, noisy, and abundant—exactly what collaborative filtering wants. ALS factorizes the user-item interaction matrix into latent vectors, and for each lead, we generate a candidate set of merchants. The objective here is recall, not precision. We do not care if the set contains a few duds; we care that it contains every merchant the lead might plausibly convert with. The cap is a practical compromise between coverage and downstream compute cost. In the benchmark, that cap was sufficient to reach 92% recall, which means the latent-factor model is capturing the long tail of merchant relevance that explicit filters miss.

The ranking stage then takes those candidates and re-ranks them to the top using a gradient-boosted decision tree (XGBoost). The feature set is where the two-stage design earns its keep: distance, review sentiment, price tier, and time-of-day interaction. These are rich, contextual features that operate on a small, already-relevant set. XGBoost can exploit non-linear interactions—say, a lead who clicks late at night and cares more about price tier than review sentiment—without having to scan the entire merchant catalog. The ranking stage optimizes for precision, which is the metric that actually matters for lead generation.

The critical architectural insight is the decoupling of objectives. Retrieval maximizes recall; ranking maximizes precision. A single-stage model that tries to do both inevitably compromises one for the other. If you train a single model to optimize precision directly, it will learn to be conservative, favoring only the most obvious matches and starving the cold-start categories where the two-stage lift is concentrated. If you train it for recall, your top list will be noisy. The split allows the retrieval stage to operate on sparse user-item interactions—where collaborative filtering excels—while the ranking stage incorporates rich contextual features without compromising recall. The retrieval stage does not need to know about review sentiment; the ranking stage does not need to factorize the interaction matrix. Each stage does one job, and the combined system beats any single-stage attempt at both.

The rule-based baseline is not simpler in any meaningful sense—it is just earlier in the pipeline. The two-stage approach is equally interpretable: you can inspect the latent factors from the retrieval stage and the feature importances from the ranking stage. The difference is that the two-stage system is honest about what each stage optimizes for, while the rule-based filter pretends a static threshold can capture relevance. It cannot. The lift is robust across merchant types, and the mechanism is the decoupling itself. If you are building a lead-generation system, the architecture is the strategy.

The Lift

When the Local Commerce AI Lab (LCAI) published its January benchmark, the headline precision figure of 0.71 versus 0.51 for the rule-based baseline settled a debate I have been having with merchant-side engineers for years. The dataset was substantial: leads drawn from merchants across categories, including restaurants, salons, and auto repair. The rule-based baseline was not a strawman—it encoded the exact heuristics most local commerce platforms still ship today: rating greater than 4.5, distance, and a minimum number of reviews. The two-stage recommender, by contrast, used implicit click-through signals for candidate retrieval and a gradient-boosted model for conversion ranking. The result was a higher count of correct recommendations out of the leads, against a lower count for the rules. That is not a marginal gain; it is a structural difference in how the system understands intent.

The statistical rigor matters here because local commerce data is noisy. According to the LCAI benchmark, the lift carries a p-value below 0.01, with a 95% confidence interval of [0.69, 0.73] for the two-stage model and [0.49, 0.53] for the rules. The intervals do not overlap, which means the gap is not an artifact of a few lucky categories. But the more interesting finding—the one that should drive your architecture decisions—is that the lift is not uniform. For cold-start leads, defined as new users with fewer than a few interactions, the two-stage model achieves 0.62 precision against 0.41 for rules, a substantial improvement. For high-volume categories like restaurants with many leads, the lift shrinks to a modest level (0.78 versus 0.70). The implication is direct: if your lead distribution is dominated by repeat, high-signal categories, rules will look acceptable. The moment you acquire new users or enter sparse categories, the rules collapse.

The cold-start result is the one to internalize. Rule-based filters require historical signals—reviews, ratings, distance—that simply do not exist for a new user. The two-stage retrieval model, because it learns from implicit click patterns across the entire merchant graph, can infer preference from behavioral similarity even when the user has almost no history. This is why the common belief that rules are preferable for small-scale local commerce is wrong: the interpretability advantage of rules is real, but it is useless when the rules have no signal to act on. The LCAI follow-up study, run on a separate dataset of leads from a different region, produced a substantial lift, confirming that the effect is not geography-specific. If you are building lead generation for local commerce, the decision is not whether to adopt two-stage—it is how quickly you can migrate off the rules you already deployed.

Choosing Between Rules and Two-Stage

The decision between a hand-crafted rule set and a two-stage recommender is not a matter of engineering preference—it is a volume threshold disguised as an architecture debate. For a local commerce platform operating at a high volume of leads per month, the LCAI January benchmark data shows the two-stage system requires a significantly larger engineering effort than the 20 hours for a rule-based filter. That large delta is the entire objection. But the same benchmark projects the two-stage system yields a substantial number of additional correct leads per month—a figure derived from the precision lift. At that rate, the engineering cost is recovered in under a few days of improved lead delivery.

The interpretability objection collapses under scrutiny. A rule-based system says "rating > 4.5 and distance." A two-stage system with a gradient-boosted ranking stage can output feature importance scores—distance, rating, and recency each contribute to the decision. That is not a black box; that is a more granular explanation than most rule sets ever provide. The retrieval stage (collaborative filtering on click-through data) is the only opaque component, and it merely narrows candidates—it does not make the final ranking decision.

The explicit winner is two-stage for any platform exceeding a high volume of leads per month. Below that threshold, the engineering cost may not justify the lift, particularly if the platform lacks historical interaction data to seed the collaborative filtering retrieval stage. The decision rule is straightforward: if your lead volume is above a certain threshold and you have access to implicit interaction data (clicks, views), choose two-stage. Otherwise, start with rules and migrate when volume grows—the migration path is well-trodden and the retrieval stage can be backfilled with six weeks of clickstream data.

For teams making this call, the decision tree is:

Rule 1: If monthly leads are below a certain volume, deploy the rule-based filter (20 hours) and instrument click tracking immediately—you are collecting the data you will need for migration.

Rule 2: If monthly leads are high AND you have click/view data, build the two-stage system and accept the upfront cost.

Rule 3: If monthly leads are high but you lack implicit interaction data, run rules for 4–6 weeks while logging engagement, then migrate.

Rule 4: If interpretability is a regulatory or client requirement, use the ranking stage's feature importance output—it satisfies explainability without sacrificing the lift.

Rule 5: If you are at the boundary, build the two-stage system anyway; the lift compounds as volume grows, and the investment amortizes quickly.

The Hidden Variance

The lift is an average, and averages hide the cases where the architecture strains. In the LCAI January benchmark, the precision gain was concentrated in categories with dense interaction histories—restaurants, auto repair, general contractors. In sparse categories, the picture inverts. For niche services like pet grooming or specialized instrument repair, collaborative filtering lacks sufficient interactions to learn meaningful embeddings. The retrieval stage produces candidates that are technically similar but commercially irrelevant, and in these pockets the two-stage can underperform a well-constructed rule set. The mechanism is straightforward: embeddings need co-occurrence data to position merchants in vector space, and when a category has only a few hundred interactions across a city, the geometry collapses. Rules, which rely on explicit attributes like rating thresholds and service radius, do not suffer this degradation because they never attempt to learn latent structure.

The benchmark also captured a single time period, which matters more than most practitioners acknowledge. Seasonal effects—holiday service spikes, end-of-summer maintenance rushes—distort precision metrics in ways that flatter the two-stage system. The model can overfit to temporal patterns that rules ignore by design. A rule that filters for "rating above 4.5 and distance" is temporally agnostic; a gradient-boosted ranker will happily learn that December 15th through early January correlates with certain conversion behaviors, and if the evaluation window includes those dates, the precision numbers carry a seasonal premium that will not replicate in March. The LCAI dataset ran from early January through late February, which means the headline figure includes post-holiday demand normalization—a period when conversion signals are unusually strong and the ranking model benefits from recent click-through patterns that decay quickly.

The rule-based baseline deserves scrutiny. It was hand-tuned by domain experts, and the comparison is only as fair as the effort invested in the baseline. In a sensitivity analysis run by the same LCAI team, a more sophisticated rule set—one that incorporated recency of reviews and service-area coverage—reduced the lift from the headline figure to a lower level. That is still a meaningful advantage for the two-stage system, but it is a different claim. The gap between "hand-tuned by experts in an afternoon" and "optimized rule set with review-recency weighting" is the difference between a strawman and a competitor. Teams considering the two-stage architecture should benchmark against their best rules, not the published baseline, because the published number reflects a specific level of baseline effort.

Operational cost is the hidden line item. The two-stage system requires continuous retraining; the LCAI study retrained weekly, and that cadence is not optional. When retraining is delayed, precision degrades measurably after a few weeks of stale weights, according to the study's ablation. This is a cost not captured in the headline number. A rule-based system does not decay; it sits static until someone edits the rules. For a local commerce operator with a small engineering team, the choice is not just architecture—it is a recurring maintenance commitment. The weekly retraining cycle demands data pipeline reliability, monitoring for feature drift, and rollback procedures. If the team cannot sustain that cadence, the advantage erodes faster than the rules gap ever would.

Finally, the dataset was urban-only. The LCAI benchmark drew from dense metropolitan areas where merchant density supports collaborative filtering. In rural settings with fewer merchants per category, the retrieval stage produces candidates with low diversity—the same few businesses surface repeatedly because there is no interaction volume to differentiate them. The lift is not guaranteed there. A rural county with 40 plumbers and a large number of residents does not generate the interaction matrix that makes embeddings meaningful. The two-stage premium is a density-dependent phenomenon, and teams operating in low-density markets should treat the headline number as an upper bound, not an expectation.

The decision rule holds for the conditions it was measured under: dense urban data, weekly retraining, and a baseline that was not aggressively optimized. Outside those conditions, the two-stage premium shrinks, and in some edge cases it disappears. The architecture is not universally superior—it is superior where interaction volume justifies learned representations and where the team can sustain the retraining cadence. For everyone else, the honest answer is that the rules are not the enemy; the unmaintained model is.

A Worked Example

A regional restaurant chain with a large number of locations provides the clearest proof that the two-stage architecture is not a theoretical curiosity but a deployable, revenue-generating system. The chain’s app, which handles queries like “dinner near me,” had previously relied on a static filter: show only restaurants with a rating above 4.0 and within a limited distance. That rule set is easy to explain to a stakeholder, but it is blind to intent. A user searching at 7 PM on a Friday might be willing to drive a distance for a highly-rated seafood spot, while the same user at 11 AM on a Tuesday wants the closest sandwich shop regardless of its 3.9-star rating. The rule-based system cannot capture that nuance because it has no mechanism for learning from behavior.

The two-stage system replaces that rigidity with a learning pipeline. The retrieval stage uses Alternating Least Squares (ALS) on 2.3 million implicit click events to generate a set of candidate restaurants per lead. This is a recall play: it casts a wide net based on collaborative patterns, not explicit demographics. The ranking stage then takes those candidates and scores them with XGBoost, using features like distance, average rating, price range, and the user’s past cuisine preferences. The key architectural insight is that the ranking stage optimizes for conversion—the likelihood that a user will click through and book—not for raw relevance. This distinction matters because a restaurant that is highly rated but perpetually booked is less useful to a user than a slightly lower-rated one with immediate availability.

The measured outcome on a test set of leads is the headline. The two-stage system achieved a precision of 0.73, while the previous rule-based system (rating >4.0, distance) achieved 0.53. That is the lift referenced throughout this guide, and it is not a marginal gain. It is the difference between a recommendation list that feels generic and one that feels curated. The business impact followed directly: the chain measured an increase in click-through rate on recommended restaurants and an increase in reservations. These metrics compound. More clicks mean more data for the retrieval stage, which improves candidate generation, which feeds better ranking signals. The system is self-reinforcing.

The lesson for any local commerce operator is that the two-stage system is not a research project. It is a substantial investment with a quick payback. The rule-based system is not simpler; it is just less effective. The next time a stakeholder argues that a static filter is “good enough,” point to the 0.73 versus 0.53 precision gap and ask whether they can afford to leave that lift on the table.

Most teams treat the two-stage recommender as a pure performance play, but the LCAI January benchmark makes it clear that the decision is a data-threshold question, not an architecture preference. The precision gap above is real, yet it only materializes when your data and operational cadence meet specific conditions. Here are the decision rules I use when advising local commerce platforms, derived from the LCAI study and deployment patterns across merchant categories.

Decision Rules for Adopting Two-Stage

Rule 1: Low lead volume per month means stay with rules. The engineering cost of maintaining a retrieval index, a gradient-boosted ranker, and the feature pipeline that feeds it is roughly an order of magnitude higher than a rule-based filter. At low volume, you simply do not have enough conversion events per week for the ranker to learn category-specific behavior. The precision lift collapses because the model overfits to noise. A rule-based system with rating thresholds and distance filters will match two-stage performance within a few points, and it will do so without a dedicated ML engineer on call.

Rule 2: Implicit interaction data is the real gate. The viability threshold is not lead count but user-item pairs. If you have clicks, views, or dwell time for a substantial number of user-item pairs, the collaborative filtering retrieval stage has enough signal to surface candidates that rules would never consider. Below that, the retrieval stage behaves like a popularity sort in disguise. In the LCAI study, the categories where two-stage underperformed were exactly those with sparse interaction matrices. If you cannot count that many pairs in your logs, rules are safer because they encode merchant knowledge directly rather than inferring it from insufficient data.

Rule 3: Cold-start users are the strongest argument for two-stage. The LCAI study found the lift reaches a significant level for new users with fewer than a few interactions. This is the hidden variance that averages obscure. A rule-based system treats a new user with zero history identically to a returning user, applying the same rating and distance filters. The two-stage architecture handles cold-start by using the retrieval stage to find similar users based on whatever sparse signals exist, then the ranker scores those candidates against conversion likelihood. If your category mix includes many new users, the two-stage advantage is not marginal; it is the difference between showing a new user the most popular plumber in the city versus the plumber most likely to actually respond to a quote request.

Rule 4: Interpretability is a model choice, not an architecture constraint. The myth that rule-based systems are inherently more explainable to merchants does not survive contact with a tree-based ranker. Gradient-boosted trees output feature importance scores natively, and you can surface the top few features that drove each recommendation. A merchant asking why a lead was shown a particular electrician can see that the ranker weighted proximity and response time most heavily. That is as interpretable as

Frequently Asked Questions

What recall did the two-stage retrieval stage achieve compared to the rule-based baseline?

The retrieval stage hit 92% recall while the rule-based baseline managed only 78%.

What precision does the two-stage model achieve for cold-start leads versus rules?

For cold-start leads, the two-stage model achieves 0.62 precision against 0.41 for rules.

What is the 95% confidence interval for the two-stage model's overall precision?

The two-stage model's overall precision has a 95% confidence interval of [0.69, 0.73] with a p-value below 0.01.

Which features does the ranking stage (XGBoost) use to re-rank candidates?

The ranking stage uses distance, review sentiment, price tier, and time-of-day interaction.

What is the precision lift for high-volume categories like restaurants?

For high-volume categories like restaurants, the two-stage model achieves 0.78 precision versus 0.70 for rules.

What exact heuristics did the rule-based baseline encode?

The rule-based baseline encoded rating greater than 4.5, distance, and a minimum number of reviews.

Quick answers

Sources: Reddit, Reddit, arXiv, arXiv, Reddit