[P] searchthearxiv.com: Semantic search across more than 250,000 ML papers on arXiv

Submitted by universal_explainer t3_106hxgk in MachineLearning

I just launched searchthearxiv.com, a simple semantic search engine over virtually all ML papers published on arXiv since 2012. The site uses OpenAI's `text-embedding-ada-002` model to match the embedding of your query against each of the paper embeddings, retrieving the ones with the highest cosine similarity. It also allows you to insert an arXiv link to find similar papers.

This was mostly meant as a fun side project. However, if people find it useful, I'm happy to maintain it and keep the database up-to-date. I'd love to know what you think! ❤️

Update: Thanks to u/ml-research for pointing out that some papers were excluded from search results regardless of the search query. This was due to a bug in the way the database was queried, and should now be fixed.

37

Comments

You must log in or register to comment.

coumineol t1_j3hm4t4 wrote

Is that meaningfully better than just googling?

5

jakderrida t1_j3hsusc wrote

I looked up clown porn and I didn't find anything useful.

8

universal_explainer OP t1_j3if5d5 wrote

Replying to coumineol (#1,309,955)

Might be in some cases, maybe not in others. Anecdotally, a query like "model using only attention mechanism site:arxiv.org" on Google doesn't bring up "Attention Is All You Need", while it does here. Aside from that, it might be a useful resource for finding similar papers based on an arXiv link.

3

universal_explainer OP t1_j3ij9jz wrote

Replying to [deleted] (#1,310,508)

Hey, thanks for trying it out!

First, do you mind sharing an example of different queries that return the same results? I have not been able to reproduce that (unless, of course, the queries are semantically similar, in which case that would be expected).

Also, of course exact search is far superior if you know the title of the paper you are looking for! In that regime, Google Scholar wins every time. However, semantic search might be better if you either a) can't remember the title but do remember some of the content or b) are simply looking to explore papers based on a handful of keywords.

Finally, the size of the database has no bearing on the quality of the embeddings, since I'm using the pretrained model by OpenAI. There is no notion of "popularity" except to rank the 10 papers with the highest cosine similarity to the query embedding according a citation score (if it's available).

3

ml-research t1_j3l36cj wrote

Does it omit some papers if it fails to parse them? Because I cannot find some arXiv papers.

1

universal_explainer OP t1_j3les1h wrote

Replying to ml-research (#1,314,571)

Are you talking about when inserting an arXiv link to find similar papers? In that case, it is important that the paper being referenced is already stored in the database. If it's a very recent paper (as in less than a week or two old), it won't work. This should be easy to fix, though, by simply scraping the abstract from arxiv.org and using it as the query.

If you're talking about searching for specific papers, I'd be interested to know the queries and the desired result. Feel free to post it here or in a DM 🙂

1