Skip to main content

Embeddings & Vector Storage Overview

This page provides a general overview of what embeddings are, how they are generated, and how to store them for use in scenarios such as semantic search or RAG.

1. What is an embedding?​

An embedding is a numerical representation (a vector of real numbers) of content:

  • text (sentences, paragraphs, documents)
  • sometimes images, audio, code, etc.

Objective: bring semantically similar content closer together in vector space.

  • Similar texts → close vectors
  • Very different texts → distant vectors

This property makes it possible to use simple mathematical operations (dot product, cosine similarity, Euclidean distance…) to perform meaning-based search rather than exact keyword matching.

2. Typical use cases​

Embeddings are used for:

  • Semantic search
    Finding documents that “talk about the same thing” as a user query, even if the exact words differ.

  • RAG (Retrieval-Augmented Generation)

    • Convert documents into embeddings
    • Store them in a vector index
    • For each user query, retrieve the most similar passages
    • Provide these passages to the LLM as context

  • Clustering & assisted classification

    • Automatically group documents by topic
    • Perform lightweight classification using KNN / cosine similarity

3. Generating embeddings​

Embeddings are generated by an embedding model (e.g., ClovisEmbeddings), via a dedicated API.

Best practices for generation (reminder)

  • Use the same model to:

    • index all documents
    • generate embeddings for user queries

  • Apply chunking to long documents:

    • chunks of 300–1,000 tokens
    • slight overlap if needed (10–15%)

  • Clean the text before vectorization:

    • remove menus, repetitive headers/footers, page numbers, boilerplate…

4. Storing embeddings (vector index)​

Once embeddings are generated, they must be stored to be efficiently queried. This is referred to as a vector index or vector store.

4.1. What to store

For each chunk/document, you typically store:

  • the embedding vector: [0.0123, -0.9821, 0.4421, ...]

  • an identifier: doc_id, chunk_id, etc.

  • metadata:

    • source (Confluence, PDF, SharePoint…)
    • title, author
    • date
    • tags (project, product, client…)

  • the raw text or a key to retrieve it

4.2. Types of possible stores

Depending on the technical context:

  • Specialized vector databases

    • (e.g., Pinecone, Qdrant, Weaviate, etc.) – optimized for similarity search

  • Hybrid search engines

    • (e.g., OpenSearch, Elasticsearch with vector fields) – combine full-text and vector search

  • Traditional databases + extension

    • PostgreSQL + vector extension
    • Other engines with a “vector” type

  • Ingestion

    • Collect sources (Confluence, files, tickets, etc.)
    • Extract text

  • Pre-processing

    • Cleaning (remove noise)
    • Normalization (encoding, special characters…)

  • Chunking

    • Split documents into usable segments
    • Add metadata (document_id, section, language, etc.)

  • Embedding generation

    • Call /v1/embeddings with model: "ClovisEmbeddings"
    • Retrieve the vector for each chunk

  • Storage in the vector index

    • Insert (vector + metadata) pairs into the chosen store

  • User query

    • The user asks a question in natural language
    • Generate the query embedding
    • Search for the nearest vectors in the index

  • Result usage

    • RAG: relevant passages are provided to the LLM
    • Semantic search: directly display matching documents
    • Analytics: clustering, duplicate detection, etc.

6. Key considerations & storage best practices​

  • Model consistency

    • Do not change the embedding model without re-indexing existing data

  • Data governance

    • Manage access rights (who can query what?)
    • Ensure traceability of indexed documents (source, update date)

  • Index updates

    • Refresh strategy (full vs incremental)
    • Handling deletions / archiving

  • Metadata quality

    • The richer the metadata, the better the filtering (by project, client, language…)

7. What’s next?​