env creation errors

[schorndorfer]

Hello!

I'm following the README instructions to install the repository codebase. Using RHEL9, running "conda env create --file conda_env.yaml" does not cleanly install. First getting matplotlib compilation errors. I moved the matplotlib installation to the conda install section (so not using pip), then getting version issues:

---

warning libmamba You are using 'pip' as an additional package manager.
Be aware that packages installed with 'pip' are managed independently from 'conda-forge' channel.

Installing pip packages: invoke, docker, transformers>=4.38, datasets, evaluate, huggingface_hub, rouge_score, sacrebleu, scipy, spacy, tqdm, pytest, pytest-async
io, langcodes[data], chainlite==0.2.2, fastapi, email-validator, git+https://github.com/laurentS/slowapi.git@a72bcc66597f620f04bf5be3676e40ed308d3a6a, async_lru,
gunicorn, uvicorn[standard], qdrant-client==1.11, onnxruntime, chainlit==1.1.202, pymongo, pygments, pyyaml, json_repair, markdownify, beautifulsoup4, cchardet, m
wparserfromhtml, orjson, orjsonl
WARNING: langcodes 3.5.0 does not provide the extra 'data'
WARNING: langcodes 3.5.0 does not provide the extra 'data'
WARNING: langcodes 3.5.0 does not provide the extra 'data'
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependenc
y conflicts.
torch 2.5.1 requires sympy==1.13.1, but you have sympy 1.13.3 which is incompatible.

[AIconicInsight]

The below conda_env.yaml version worked for me. I moved matplotlib to the conda section as you did and used the pypi version of slowapi instead of the given git version. I also updated conda before creating the environment (conda update -n base conda), maybe that helped.

Note: it seems that the default retriever endpoint does not work currently.

name: wikichat
channels:
  - pytorch
  - conda-forge
  - defaults
dependencies:
  - python=3.10
  - pytorch
  - pip
  # gcc, gxx and make are needed to build some of the pip packages
  - gcc
  - gxx
  - make
  - redis # Redis is used to persist FastAPI's rate limit data, and cache LLM outputs. This installs redis-server
  - matplotlib # Plotting library
  - pip:
      - invoke # used for running multi-part commands. A more Python-friendly alternative to Make
      - docker # Python bindings for docker. You should also install docker on your system separately.
      - transformers>=4.38 # 🤗 transformers library
      - datasets # 🤗 datasets library
      - evaluate # 🤗 evaluation library
      - huggingface_hub # Package to download datasets and models from 🤗 hub
      - rouge_score # Library for calculating ROUGE scores
      - sacrebleu # Library for calculating BLEU scores
      - scipy # Scientific computing library
      - spacy # NLP library
      - tqdm # Progress bar library
      - pytest # Testing framework
      - pytest-asyncio # pytest plugin that facilitates the testing of async functions
      - langcodes[data] # Language codes library with data extras
      - chainlite==0.2.2 # The interface to various LLM providers. A wrapper around LangChain and LiteLLM

      # For the retrieval server
      - fastapi # Web framework for building APIs based on standard Python type hints
      - email-validator # Used for Redoc
      - slowapi # Rate limiting library for FastAPI applications
      - async_lru # Async LRU cache decorator
      - gunicorn # Python WSGI HTTP Server for UNIX
      - uvicorn[standard] # standard extra packages improve uvicorn's performance
      - qdrant-client==1.11 # Used for nearest neighbor search
      - onnxruntime # To run embedding models faster on CPU. Optional.
      - chainlit==1.1.202 # used for the demo. Newer versions have too many breaking changes
      - pymongo # MongoDB python package
      - pygments # Used for syntax-highlighting conversation logs in the terminal
      - pyyaml # Used for loading the config file
      - json_repair # helps fix LLM json outputs

      # for processing Wikipedia HTML dumps:
      - markdownify # Converts HTML to Markdown
      - beautifulsoup4 # HTML parsing
      - cchardet # speeds up beautifulsoup parsing
      - mwparserfromhtml # used for some of its helper functions to process Wikipedia HTML
      - orjson # Faster alternative to Python's json package
      - orjsonl # Fast light-weight package to work with jsonline files. Uses orjson internally.

[AIconicInsight]

Another fix: pydantic==2.10.1 is required so that chainlit works fine. Newer versions break the installation.

Source: Update of pydantic to version 2.10.2 breaks chainlit