Tools for Manipulating NEF [NMR Exchange Format] Files and Foreign File Access
Please note NEF-Pipelines continues to be developed so if you do find problems put in an issue on the issue tracker. Even better if you can help improve the software do get in touch, there is much to do!
NEF-Pipelines is a set of command line (currently... there maybe a gui later!) tools for manipulating [NEF])(https://github.com/NMRExchangeFormat/NEF) or NMR Exchange Format files which can be used to move NMR meta data [peaks, shifts, sequences etc] between NMR data processing programs.The programs provide basic tools for manipulating nef files:
- molecular chains: listing, renaming and cloning molecular chains
- save-frames [tables]: deleting, inserting, listing and pretty printing (tabulate)
- headers creation / updating NEF headers with correct UUIDs and history
- streaming: NEF files into a pipeline
- testing: self testing of NEF pipelines
- accessing online data: chemical shift predictions
- fitting data: basic relaxation data, j-modulated RDC experiments [in development]
It also provides tools for importing and exporting non NEF files from the following programs [transcoding / translators]
R = Read / import to NEF, W = write / export from NEF, 🐍= supported in python code, α = alpha quality code [few tests]
|
xplor dihedrals R distances R sequence R rdcs W |
csv peaks R 🐍 rdcs R 🐍 |
deep peaks R α 🐍 |
nmrpipe peaks R 🐍 sequence R 🐍 shifts R 🐍 |
nmrview peaks RW 🐍 sequence R shifts RW 🐍 sequences W 🐍 |
|
fasta sequence RW 🐍 |
echidna peaks R 🐍 |
sparky shifts R 🐍 peaks RW 🐍 sequence R 🐍 |
mars fixed W 🐍 fragments W 🐍 input W 🐍 sequence RW 🐍 shifts RW 🐍 peaks R 🐍 |
modelfree data W α 🐍 |
|
pales rdcs RW 🐍 template W |
rcsb sequence R |
rpf shifts W α 🐍 |
shifty shifts W 🐍 |
shiftx2 shifts R α 🐍 |
|
nmrstar project R α rdcs R α 🐍 sequence R α 🐍 shifts R α 🐍 |
talos order-parameters R 🐍 restraints R α 🐍 secondary-structure R 🐍 sequence R 🐍 shifts W α 🐍 |
xcamshift shifts W 🐍 |
xeasy peaks R 🐍 sequence R 🐍 shifts R 🐍 |
The easiest way to install NEF-Pipelines is using the tool uv provided by astral. You can do this either using the script installation_scripts/install.sh from the NEF-Pipelines distribution [on macos or a linux] as shown below or alternatively using uv manually [macos, windows and linux] as shown further down
curl -sL https://raw.githubusercontent.com/varioustoxins/NEF-Pipelines/refs/heads/master/install_scripts/install.sh | shcurl -LsSf https://astral.sh/uv/install.sh | sh
uv tool install nef-pipelines --with rich --with streamfitter --python 3.11note: currently it is recommended to install with a python version <= 3.11 on macOS as there is a bug in installing a dependency call pydantic on some version of the OS
# note: changing the execution policy is required for running a script from the internet.
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
uv tool install nef-pipelines --with richnote: not all of NEF-Pipelines is working on Windows currently, we are working on it!
after the installation has completed you can type
neftype
nefand should see output that starts
Usage: nef [OPTIONS] COMMAND [ARGS]... ✔ ╱ took 14s ╱ nef_pipelines ╱ at 23:29:36
Options:
...
Some older installation methods using a script in ccpn assign or pip are deprocated but still provided for completenes
You can use the same tools you used to in stall NEF-Pipelines to upgrade it
running the following script again
curl -sL https://raw.githubusercontent.com/varioustoxins/NEF-Pipelines/refs/heads/master/install_scripts/install.sh | shwill update NEF-Pipelines to the latest version
again now you have UV installed upgrading NEF-Piplines manually should be easy
uv tool upgrade nef-pipelines --python 3.11The entry point to NEF-Pipelines is the command nef used on its own as seen under installation
it will list a set of top level commands and gives there descriptions.
nef
Usage: nef [OPTIONS] COMMAND [ARGS]...
...
Commands:
chains - carry out operations on chains
csv - read [rdcs]
deep - read deep [peaks]
echidna - read echidna data [peaks]
entry - carry out operations on the nef file entry
fasta - read and write fasta sequences
fit - carry out fitting operations [alpha]
frames - carry out operations on frames in nef files
globals - add global options to the pipeline [use save as your last...
header - add a header to the stream
help - help on the nef pipelines tools and their usage
loops - carry out operations on loops in nef frames
mars - read and write mars [shifts and sequences]
...Some of these commands such as frames and entity deal with components of a NEF file. Then there are a series of top levels
that deal with common SaveFrames related to NMR data such as chains and shifts and finally there
are many commands named after file formats and programs such as talos, xplor nmrpipe and nmrview
which provide entry points to the import and export of data.
NEF-Piplines uses hierarchical commands. So foir example the the toip level command mars which deals
with the import and export of data from the mars auto assignment program has two sub-commands import ane export
which not suprising deal with the import and export of data for the program to and from NEF. Then below these commands
are further sub commands deaing with shifts, sequences etc. An easy way to see the sub commands for a format is to
use the nef help commands tool which provides a tree display of the available commands. Typing
nef help commands on its own produces output for all commands which can be overwhelming so lets
filter it down to just talos commands by adding a filter. To do this we type nef help commands mars
which gives the following output
nef
└── mars
├── export
│ ├── fixed [P]
│ ├── fragments [P]
│ ├── input [P]
│ ├── sequence
│ └── shifts [P]
└── import
├── peaks [P]
├── sequence [P]
└── shifts [P]
key: [X] has a python function [P]ipe / [C]md
[α] alpha featurehere we can see that we can import peaks, shifts and sequencdes from mars and that we export fixed assignments,
fragments, inputs [an input file], sequences and shifts to mars. Each individual file sub-command also has help text and
a series of options. So to see the options for exproting shift we can typenef mars export shifts --help
which will show
nef mars export shifts --help
Usage: nef mars export shifts [OPTIONS] [SHIFT_FRAME_SELECTORS]...
- write a mars chemical shift file
Arguments:
[SHIFT_FRAME_SELECTORS]... selector for the shift restraint frames to use,
can be called multiple times and include wild
cards
Options:
-i, --input |PIPE| input to read NEF data from [- is stdin]
[default: -]
-c, --chain <CHAIN-CODE> chain to export [# and @ and the 1st chain code
in the project if the is only one]
-o, --out <MARS_SHIFT_FILE> file name to output to [default
<entry_id>_shifts.tab] for stdout use -
-f, --force force overwrite of output file if it exists and
isn't empty
--help Show this message and exit.However, mostly you don't need to worry about many of the optiosn as the defaults are well chosen.
It should be noted also that many commands also take arguments in the case of mars export shifts
the default is the default chemical shift frame (again a well chosen default).
many commands also have common options, the most well used ones are
| option | description |
|---|---|
| -i / --input | the path to a file to read NEF data from which becomes the stream running through the pipeline |
| -h / --help | help for the particular sub command |
| -o / --out | a filename or a template file name to write output to [usually native files not NEF] |
| -c / --chain | a chain or list of chains to output, multiple invocations add more chains and the argument can also be comma separated list [don't add any spaces!] |
| -f / --force | force overwriting files if newly generated files would overwite old ones |
| -v / --verbose | provide verbose information on the STDERR stream, in some cases repeating the option gives more output, e.g -vv gives more output the -v' |
Note 1. It should be noted that many options that take multiple values can be repeated to add more values or can or can take comma separated lists of values [no spaces] so
command --chain A --chain B```selects chains A and B as does
command --chains A,BNote 2. Many arguments take wild cards, so when selecting frames you can use +'s and * when choosing the frame name or category to select multiple frames. Furthermore all frame names are typically treated as being surrounded by *s when it is reasonable to make selecting frames easier
Note 3. When selecting frames there is usually an explicit option to allow differentiation between frame names or categories. However, by default it is off and searches for frames are based on both names and categories the same time
Individual nef pipeline components can be used standalone with NEF files. To read in a nef fileo or fiel for conversion
into a command when using it standalone you can use the --in option
read a foreign file for translators or read a NEF file for commands that manipulate NEF files. So for example to lists
the frames in NEF frame you can type
nef frames list --in test_data/pales_test_1.nefwill list all the Saveframes in the file test_data/pales_test_1.nef and the entry name of the nef file as follows
entry pales_test
nef_molecular_system nef_rdc_restraint_list_test_1as a short cut the following command also work for nef frames list
nef frames list test_data/pales_test_1.nefThe real power in NEF-Pipelines comes from combining pipeline commands together or with command line tools. For example the following command
nef header \
| nef fasta import sequence tailin1.fasta \
| src/nef nmrview import shifts tailin1.out \
| src/nef nmrview import peaks tailin1.out \
> tailin1.nefwill create a valid NEF header followed by the tailin1 sequence as a molecular system and then import shifts and peaks
for talin1 from nmrview before writing a new NEF file tailin1.nef
As discussed above the commands provided by NEF-Pipelines are hierarchical in nature. All commands are accessed from the
root nef command but there many are sub commands so for example to import a sequence from a pdb file you would type
nef rcsb import sequence followed by the name of a PDB / MMCIF file and relevant options. This can look long
winded but NEF-Pipelines support command completion. This is is installed for your shell by typing
nef --install completion <SHELL-NAME> and the restarting your shell, so fro example for the bash shell
nef --install-completion bashThe after restarting you shell, typing a double tab will list all available sub commands.
For example
nef<tab><tab>shows
chains fasta frames header mars nmrpipe nmrview pales pdb stream test
and
nef nmrview import<tab<tab>shows
peaks sequence shifts
note: currently I find this works better with the bash shell rather than the zsh but your mileage may vary...
NEF-Pipelines maybe used as library of routines that are acessible in python using
in development...
The origional NEF paper is not a good description of the format more an outline of ideals and needs. The CCPN website has a good guide to nefand development takes place on github via the NEF consortium repository which includes an annotated NEF example and a dictionary defining the format. NEF is a STAR format which means it has similarities to NMRStar files used by the BMRB and PDBx / mmCIF used by the RCSB / PDB because its uses the same underling format. However, its is not directly interchangeable with these file formats as it has a different syntax and intention / underlying use easy and accurate NMR data interchange] as opposed to archiving for example.