Recoll semantic searches – polygon.uploadbackup.com

The Recoll Python API gives easy access to the text contents of all the documents in the index,
which makes it well fitted as a language model input. In turn there are two directions for using
this: Retrieval Augmented Generation (RAG), where a generative LLM output is informed by the result
documents from another (probably keyword-based) search, and semantic queries, where we look for
documents matching the concepts in the query, and not necessarily the exact terms.

RAG operations are centered in the LLM interface and probably need little from Recoll apart from
getting data from the index.

There is a new branch in the Recoll source with slight modification to allow interfacing with a
language model for the purpose of running semantic searches.

In practise, all the specific language model work is performed by Python programs outside of the
main recoll source. One of the scripts is a worker for the Recoll GUI, receiving questions and
providing results. In consequence, the Recoll main code needed very minimal modifications, mostly
handling the additional search type in the GUI.

ollama is used to run the models, and chromadb to store the embeddings.

The general process is as follows:

The index is created or updated by recollindex, as always.
You then run the rclsem_embed.py script. This extracts the texts for new documents, splits it
into segments and asks a language model to generate embeddings. The model which is used by default
is a relatively small nomic-embed-text (137M parameters). Running this on a CPU is already
orders of magnitude slower than the original indexing. The embedding vectors are stored in
chromadb, along with the Recoll document identifier. See the sem_rclquery configuration
variable below for limiting the part of the index which will actually be processed.
At query time, an embedding is generated for the question, and chromadb is asked for
neighbours. The Recoll identifiers are used to retrieve the document and the relevant text
context.
An rclsem_query.py script allows running queries on the command line, and an
equivalent rclsem_talk.py script is used to communicate with the GUI, when it executes a simple
search in the new Semantic mode.

The Python part runs in a virtual environment. A simple shell script allows building it easily.

I only have a modest PC to run this, with no GPU. Running a reranking model on this has proven
impossible (see comments in rclsem_query.py), so that the current implementation is rather
primitive and mostly provides the scaffolding for experimentation.

The modification of the main Recoll GUI code is minimal, so it has been merged into the master
branch (it used to be on a semantic branch, the same code is now in master).
In practise, if you want to try this:

Clone the default branch of the source tree:

git clone https://framagit.org/medoc92/recoll.git

Then build and install as per
the usual method. You
need to add a -Dsemantic=true option to the meson setup command to enable the semantic query
option code.
Go to the src/semantic directory and run the initsemenv.sh script:
```
sh initsemenv.sh /path/to/where/I/want/the/scripdir
```

This will install ollama if it’s not already there, pull the nomic-embed-text model, create the
Python virtual environment, install ollama and chromadb in it, and copy the Recoll scripts and
Python module.

There are a number of configuration variables which must be set in the main configuration file for
the index you will be using:

sem_venv: the location of the Python virtualenv directory. This is for use by the main recoll code
and mandatory.
sem_rclquery: the query which will select the documents to process. By default, this is “mime:*”,
which will select all documents, but you may want to restrict this (e.g. with dir: clauses),
because the embedding operation is very slow on a CPU. You really want to begin experimenting
with a small set. There is no problem with running embedding multiple times on different sets, it
tests if a document is already there before doing actual work.
sem_chromadbdir: where the chromadb data will be stored. By default, this will be the chromadb
directory inside the Recoll configuration directory.
sem_embedmodel: nomic-embed-text by default, you may want to experiment if your configuration
allows it. Of course, you need to delete the chromadb directory when changing models.
sem_embedsegsize: the target character size of the segments created for embedding.

Voila. If you have a GPU, and can code a little Python, I think that the most interesting direction
would be experimenting with reranking.

As a vaguely interesting aside, and from my initial testing on the works of Jane Austen, it appears
that horseriding is a much more frequent subject in Mansfield Park than in the other
books… It’s all the more interesting that horseriding is of course not an English word, and a
normal Recoll search would yield no results at all. It remains to be seen if the feature can be
useful for anything beyond litterary ‘research’.

Source link

Leave a Reply Cancel reply