directLFQ is an open-source Python package for quantifying protein intensities based on peptide intensities or fragment-ion intensities measured with from Mass Spectrometry-based proteomics. It preserves peptide ratios, shows very accurate quantification and has a robust normalization approach. Furthermore, it allows fast processing also of very large sample cohorts, as runtime increases linearly with sample number. It is part of the AlphaPept ecosystem from the Mann Labs at the Max Planck Institute of Biochemistry and the University of Copenhagen.
You can process DIA and DDA data analyzed by AlphaPept, MaxQuant, FragPipe, Spectronaut and DIANN as well as generic formats, using a Graphical User Interface (GUI) or the python package.
Generating protein intensities from Mass Spectrometry proteomics data comes with a variety of challenges. Differing peptides that belong to the same protein can have strongly differing intensities, for example due to differing ionization efficiencies. Missing values (i.e. peptides that have been detected in one run but not in the other) make simple summarization of peptide intensities to protein intensities problematic. Differences in sample loading can introduce systematic biases into the analysis. With directLFQ, we provide a novel algorithm for addressing these challenges in an efficient and accurate manner. directLFQ retains peptide ratios and uses them to infer protein ratios and uses the concept of intensity traces for it's main processing steps. For futher details on the algorithm, please refer to the preprint.
Visit alphapept.org for other packages of AlphaPept ecosystem.
directLFQ can be installed and used on all major operating systems (Windows, macOS and Linux). There are different types of installation possible:
- One-click GUI installation: Choose this installation if you only want the GUI and/or keep things as simple as possible.
- Pip installation: Choose this installation if you want to use directlfq as a Python package in an existing Python 3.9 environment (e.g. a Jupyter notebook). If needed, the GUI and CLI can be installed with pip as well.
- Developer installation: Choose this installation if you are familiar with CLI tools, conda and Python. This installation allows access to all available features of directlfq and even allows to modify its source code directly. Generally, the developer version of directlfq outperforms the precompiled versions which makes this the installation of choice for high-throughput experiments.
- Docker installation: Choose this installation if you want to use directlfq without any changes to your system.
The GUI of directlfq is a completely stand-alone tool that requires no knowledge of Python or CLI tools.
You can download the latest release of directlfq here.
Download the latest directlfq-X.Y.Z-windows-amd64.exe build and double click it to install. If you receive a warning during installation click Run anyway.
Important note: always install directlfq into a new folder, as the installer will not properly overwrite existing installations.
Download the latest directlfq-X.Y.Z-linux-x64.deb build and install it via dpkg -i directlfq-X.Y.Z-linux-x64.deb.
Download the latest build suitable for your chip architecture
(can be looked up by clicking on the Apple Symbol > About this Mac > Chip ("M1", "M2", "M3" -> arm64, "Intel" -> x64),
directlfq-X.Y.Z-macos-darwin-arm64.pkg or directlfq-X.Y.Z-macos-darwin-x64.pkg. Open the parent folder of the downloaded file in Finder,
right-click and select open. If you receive a warning during installation click Open.
In newer MacOS versions, additional steps are required to enable installation of unverified software.
This is indicated by a dialog telling you “directlfq. ... .pkg” Not Opened.
- Close this dialog by clicking
Done. - Choose
Apple menu>System Settings, thenPrivacy & Securityin the sidebar. (You may need to scroll down.) - Go to
Security, locate the line "directlfq.pkg was blocked to protect your Mac" then clickOpen Anyway. - In the dialog windows, click
Open Anyway.
Older releases remain available on the release page, but no backwards compatibility is guaranteed.
directlfq can be installed in an existing Python environment with a
single bash command. This bash command can also be run directly
from within a Jupyter notebook by prepending it with a !:
pip install "directlfq[dask]"Installing directlfq like this avoids conflicts when integrating it in other tools, as this does not enforce strict versioning of dependencies. However, if new versions of dependencies are released, they are not guaranteed to be fully compatible with directlfq. This should only occur in rare cases where dependencies are not backwards compatible.
You can always force directlfq to use dependency versions which are known to be compatible with:
pip install "directlfq[stable,dask-stable]"It is also possible to directly install any branch (e.g. some-branch) from GitHub with
pip install "git+https://github.com/MannLabs/directlfq.git@some-branch#egg=directlfq[stable,development-stable]"The GUI version can be installed with
pip install "directlfq[gui,dask]"or
pip install "directlfq[stable,gui-stable,dask-stable]"Note: The 'dask' extra can be omitted for slimmer dependencies, but this comes with a performance penalty for large datasets.
directlfq can also be installed in "editable" mode. This allows to fully customize the software and even modify the source code to your specific needs.
First, clone the directlfq repository from GitHub to a new directory
mkdir -p ~/directlfq/project/folder && cd ~/directlfq/project/folder
git clone https://github.com/MannLabs/directlfq.git && cd directlfqNext, it is highly recommended to use a separate conda virtual environment, as otherwise dependency conflicts can occur with already existing packages
conda create --name directlfq python=3.9 -y
conda activate directlfqFinally, directlfq and all its dependencies need to be
installed. To take advantage of all features and allow development (with
the -e flag), this is best done by also installing the development
dependencies instead of only
the core dependencies:
pip install -e ".[dask,development]"By default this installs 'loose' dependencies (no pinned versions),
although it is also possible to use stable dependencies
(e.g. pip install -e ".[stable,dask-stable,development-stable]").
By using the editable flag -e, all modifications to the directlfq
source code folder are directly reflected when running
directlfq. Note that the directlfq folder cannot be moved and/or renamed
if an editable version is installed. In case of confusion, you can
always retrieve the location of any Python module with e.g. the command
import module followed by module.__file__.
The containerized version can be used to run directlfq without any installation to your system.
Install the latest version of docker (https://docs.docker.com/engine/install/).
Set up your data to match the expected folder structure: create a folder and store its name in a variable, and specify a port
DATA_FOLDER=/home/username/data; mkdir -p $DATA_FOLDER
PORT=5006
docker run -v $DATA_FOLDER:/app/data -p $PORT:5006 mannlabs/directlfq:latestAfter initial download of the container, directlfq will start running immediately, and can be accessed under localhost:$PORT.
Note: in the app, the local $DATA_FOLDER needs to be referred to as "/app/data".
If you want to build the image yourself, you can do so by
docker build -t directlfq .and run it with
docker run -p $PORT:5006 -v $DATA_FOLDER:/app/data -t directlfqThere are three ways to use directlfq:
NOTE: The first time you use a fresh installation of directlfq, it is often quite slow because some functions might still need compilation on your local operating system and architecture. Subsequent use should be a lot faster.
If you have installed directlfq with the one-click GUI installer, you can run the GUI simply by clicking the directLFQ icon on your desktop/applications folder.
If the GUI was not installed through a one-click GUI installer, it can be activate with the following bash command:
directlfq guiNote that this needs to be prepended with a ! when you want to run this from within a Jupyter notebook. When the command is run directly from the command-line, make sure you use the right environment (activate it with e.g. conda activate directlfq or set an alias to the binary executable (can be obtained with where directlfq or which directlfq)).
The CLI can be run with the following command (after activating the conda environment with conda activate directlfq or if an alias was set to the directlfq executable):
directlfq -hIt is possible to get help about each function and their (required) parameters by using the -h flag.
directLFQ can be imported as a Python package into any Python script or notebook with the command import directlfq.
Running the standard analysis (with plots) can be done via the command:
import directlfq.lfq_manager as lfq_manager
example_input_file_diann = "/path/to/example_input_file_diann.tsv"
lfq_manager.run_lfq(example_input_file_diann)Several use cases for applying directLFQ can be found as Jupyter Notebooks in the tests folder. See for example the quicktests notebook.
Note that the nbdev_nbs folder contains the source code as Jupyter notebooks. These notebooks are automatically converted to Python scripts using the nbdev package and stored in the directlfq folder. The notebooks contain additional documentation and comments as well as unit tests that can be executed directly from the notebooks themselves.
In case of issues, check out the following:
- Issues: Try a few different search terms to find out if a similar problem has been encountered before
- Discussions: Check if your problem or feature requests has been discussed before.
If you like this software, you can give us a star to boost our visibility! All direct contributions are also welcome. Feel free to post a new issue or clone the repository and create a pull request with a new branch. For an even more interactive participation, check out the discussions and the the Contributors License Agreement.
In order to have release notes automatically generated, changes need to be tagged with labels.
The following labels are used (should be safe-explanatory):
breaking-change, bug, enhancement.
This package uses a shared release process defined in the alphashared repository. Please see the instructions there
directLFQ is started internally via the directlfq.lfq_manager.run_lfq() command. In principle and for most use cases, you only need to provide the path to the AlphaPept/MaxQuant/DIA-NN etc. file of interest. However, there are several other options that can be used to customize the analysis. The following commands are available:
- input_file: The input file containing the ion intensities. Usually the output of a search engine.
- columns_to_add: Add the names of columns that are present in the output table and that you want to keep in the directLFQ output file. Separated by semicolons. Note that some basic additional columns such as gene names are always added to the output table by default. WARNING: Take care that columns you add are not ambigous. For example, adding the peptide sequence column will not work, because there are multiple peptide sequences per protein.
- selected_proteins_file: If you want to perform normalization only on a subset of proteins, you can pass a .txt file containing the protein IDs, separeted by line breaks. No header expected.
- mq_protein_groups_txt: In the case of using MaxQuant data, the proteinGroups.txt table is needed in order to map IDs analogous to MaxQuant. Adding this table improves protein mapping, but is not necessary.
- min_nonan: Min number of ion intensities necessary in order to derive a protein intensity. Increasing the number results in more reliable protein quantification at the cost of losing IDs.
- input_type_to_use: The type of input file to use. This is used to determine the column names of the input file. Only change this if you want to use non-default settings.
- maximum_number_of_quadratic_ions_to_use_per_protein: How many ions are used to create the anchor intensity trace (see paper). Increasing might marginally increase performance at the cost of runtime.
- number_of_quadratic_samples: How many samples are used to create the anchor intensity trace (see paper). Increasing might marginally increase performance at the cost of runtime
- num_cores: The number of cores to use (default is to use multiprocessing).
- filename_suffix: Suffix to append to the output files.
- deactivate_normalization: Set to true, if no between-sample normalization should be performed before processing.
- filter_dict: In case you want to define specific filters in addition to the standard filters, you can add a yaml file where the filters are defined (see example here). In the Python API you can also directly put in the dictionary instead of the .yaml file.
directLFQ writes three output files into the directory of the input file:
- The main output file ends with .protein_intensities.tsv and contains the estimated LFQ protein intensities.
- The second output file ends with .ion_intensities.tsv and contains the aligned intensity traces of all ions. This allows to compare profiles of different ions to each other. In particular, if you run directLFQ with peptide-level quantification, you can use this file to compare the intensity traces of different peptides of the same protein.
- The third output file ends with .aq_reformat.tsv and contains the reformatted input data in matrix format (ions are rows, samples are columns). The values are identical to the values of the original input file, just the format is different.
directLFQ takes a Spectronaut .tsv table as input. When exporting from Spectronaut, the correct columns need to be selected. These can be obtained by downloading one of the export schemes available below. We provide one export scheme for sprecursor quantification and one export scheme for fragment ion quantification. Fragment ion quantification shows slightly more accuracy, but the files are around 10 times larger.
An export scheme can then simply be loaded into Spectronaut as follows:
Go to the "Report" perspective in Spectronaut, click "Import Schema" and provide the file.
The data needs to be exported in the normal long format as .tsv file
Download Spectronaut export scheme for precursor quantification
Download Spectronaut export scheme for fragment ion quantification
Provide the path to the DIANN "report.tsv" output table.
Provide the path to the MaxQuant "peptides.txt" output table or the MaxQuant evidence.txt output table. Additionally and if possible, provide the path to the corresponding "proteinGroups.txt" file.
Provide the path to the "combined_ion.tsv" output table.
In the case that you working with a search engine that is not supported by directLFQ, you can use the generic input format. This format is a tab-separated quantity matrix file with the following columns: "protein", "ion", "run_id1", "run_id2", ..,"run_idN".
Each row contains therefore all the ion intensities that were measured for an ion in each run (see examples below). The ion identifier only needs to be unique for each ion and can be on the level you want (peptide, charged peptide, or fragment ion). After reformatting your file into this format, save the file with the ending ".aq_reformat.tsv". Then you can simply give this file as input to directLFQ and it will automatically detect the generic input format.
If you want to reproduce data analyses presented in the manuscript, you can first download the data by executing
python tests/download_testfiles.py all_testsThis will download the underlying datasets into the appropriate location. The notebooks carrying out the analyses themselves are located in the tests directory in the respective subfolders ratio_tests, normalization_tests, runtime_tests and organellar_maps.
In the case that directLFQ is useful to you, please consider supporting us by citing the paper
Accurate label-free quantification by directLFQ to compare unlimited numbers of proteomes. Ammar, C., Schessner, J.P., Willems, S., Michaelis, A.C., and Mann, M. (2023). Molecular & Cellular Proteomics, 100581. doi: 10.1016/j.mcpro.2023.100581
directLFQ was developed by the Mann Labs at the Max Planck Institute of Biochemistry and the University of Copenhagen and is freely available with an Apache License. External Python packages (available in the requirements folder) have their own licenses, which can be consulted on their respective websites.