Search engine within a text document

Submitted by iacoposk8 t3_zvkgar in deeplearning

Hi everyone! I have a 300+ page odt file, which I can simply convert to a txt file. On the other hand, I have several dozen scattered notes that I have to insert into this document. I would like to know if there is a software, a library or a project on github (preferably in python) that can help me find the best, most coherent place to insert this note.

Alternatively, if you were to create the code from scratch, how would you go about it?

0

Comments

You must log in or register to comment.

knight1511 t1_j1qf7dz wrote

Is the text in image format or can it be directly extracted in digital format?

If it is digital format then you can extract the text directly by using pdfminer. It has packages available in Java and python.

If the the pdf has images inside it you need to ocr the text first. ocrmypdf is a very handy python psckage that uses Google's Tesseract OCR Engine to convert images into digital characters. It is not a perfect process but if the images are of good quality then it is almost perfect. Once you have the text in digital format, it can be indexed by a search engine.

To search through the text you can simply use a ready to use a search engine like elasticsearch. You just need to supply the extracted text to the engine to be indexed. Then you can query it easily.

One easy way to use elastic search is to use it via docker. It's easy to get started with provided you are already familiar with docker

Edit: Alternatively you can explore free and open source software called Papermerge. link

3

iacoposk8 OP t1_j1tz1yy wrote

I looked at elasticsearch but its quite expensive. Are there any free or cheaper alternatives? I also tried Papermerge but the search doesn't work, even if I try to search for exact phrases or the file name. Is there something I may be forgetting to do?The file is already text without images

1

knight1511 t1_j1u3lik wrote

Elastic search is free if you self host it. You are only charged if you want to rely on their infrastructure or some advanced features. From what you're saying, sounds like an inverted index is more than enough for your use case which is a free feature.

I have used elasticsearch for simple use cases via docker. https://hub.docker.com/_/elasticsearch

Just spin up the container and index the extracted text data. Then you can run queries against it.

For Papermerge I am not sure. I had heard about it some time back and it looks like the project is abandoned or updated infrequently

1

knight1511 t1_j1u3uv3 wrote

Also if you are adept at coding in Java, you can look at the Apache Lucene or Solr library. This is what most search engines use behind the scenes. It is low level but it allows you to configure your program to do exactly what you want it to do.

By the way, do you want to search the filenames as well or just the content inside the files?

1

iacoposk8 OP t1_j1udt2j wrote

I would like to search only in the content of a text, but in an intelligent way.

So if in the text file it says: "Hardware is the physical part of the computer and software is the logical part"

and to find it I wrote: "Hardware is the part of the computer that we can touch while software is the programs"

It should be able to find it for me anyway, right?

1

knight1511 t1_j1w32f7 wrote

What you are querying borders semantic search. That is you are not looking for the exact phrase but for something that semantically means the same thing. Unfortunately this is not something elasticsearch can do for you out of the box.

Traditional search engines work by matching the exact words of your query and looking for its occurrences in the document. They do this by creating an inverted index, which is nothing but a lookup table of all the words/tokens present in a document. They do this for all documents you want to index. Then when a query comes in they use some similarity algorithm to evaluate the contents of all the indexed documents against the words/tokens present in the query. They then return the documents in a ranked order from most similar to least based on the score. The semantics or the “meaning” of the text is not considered.

If your query has some overlapping words with the document you are looking for, then sure you will get some relevant documents back. But if there is NO words that are same then this will not work. For example you cant expect it to return a document with the phrase “the computer is not working “for the query “the pc is broken”

What you are looking for is semantic search. There are some pre-trained language models on HuggingFace whose embeddings can be used as search index. And there is an open-source FAISS library by Facebook that allows you to search it. But the specifics of this is highly dependant on your use case. Also the implementation is a bit more complex and will requires some coding expertise and understanding of ML. You will need professional help unless you already know the stuff or are willing to learn

I have experience doing this before. But to be honest this is a time consuming task and not something to be done for free ;)

1

iacoposk8 OP t1_j1uj6mu wrote

I installed it. There is no gui right?

Could you give me an example of command to index and search inside the file? Thank you so much

1