What’s new in chemfp 4.0

Released 12 June 2022.

The main themes for chemfp 4.0 are “notebook usability” and “diversity selection.”

(Note: this has been edited on 24 April 2023 to reflect API changes in chemfp 4.1.)

High-level API

The chemfp API was primarily designed for application developers who want precise control over what happens. The API was used for command-line tools, like the ones that come with chemfp, and for web services.

The command-line tools were written for a wider audience. As a result, I saw that people who used chemfp in a Jupyter notebook, where they had access to the full chemfp API, still preferred “!shell”-ing out to the command-line tools. And rightly so, as it was much simpler.

Chemfp 4.0 includes new “high-level” functionality to remedy this problem. The table below shows which API functions correspond to which command-line tools:

Command-line

High-level API

Description

simsearch

chemfp.simsearch()

Similarity search

rdkit2fps

chemfp.rdkit2fps()

Use RDKit to generate fingerprints

ob2fps

chemfp.ob2fps()

Use Open Babel to generate fingerprints

oe2fps

chemfp.oe2fps()

Use OEChem/OEGraphSim to generate fingerprints

cdk2fps

chemfp.cdk2fps()

Use CDK to generate fingerprints

(no command-line tool)

chemfp.convert2fps()

Figure out which tool to use to generate fingerprints

fpcat

(no high-level API)

Convert between fingerprint file formats

chemfp maxmin

chemfp.maxmin()

MaxMin diversity picking

chemfp spherex

chemfp.spherex()

Sphere exclusion diversity picking

chemfp heapsweep

chemfp.heapsweep()

HeapSweep diversity picking

Here’s an example showing the distribution of fingerprint similarity to caffiene (CHEMBL113) for all fingerprints in ChEMBL 30 which are at least 0.4 similar:

import chemfp

chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", threshold=0.4
                ).to_pandas().hist("score", log=True)

array([[<Axes: title={'center': 'score'}>]], dtype=object)
_images/whatsnew_3_1.png

Pandas integration

As you saw in the previous example, chemfp 4.0 add new methods which export chemfp data to a Pandas dataframe. The following shows the nearst 5 neighbors to CHEMBL1113 (using k=6 as caffine is already present):

import chemfp

chemfp.simsearch(query_id="CHEMBL113", targets="chembl_30.fpb", k=6).to_pandas()
target_id score
0 CHEMBL113 1.000000
1 CHEMBL4591369 0.733333
2 CHEMBL1232048 0.709677
3 CHEMBL446784 0.677419
4 CHEMBL1738791 0.666667
5 CHEMBL2058173 0.666667

Multi-query search have three columns of output, with the query id in the first:

import chemfp

with chemfp.load_fingerprints("chembl_30.fpb") as arena:
    queries, targets = arena.train_test_split(train_size=10, test_size=100, rng=12345)

result = chemfp.simsearch(queries=queries, targets=targets, k=1, threshold=0.2)
result.to_pandas()

queries:   0%|                                                                         | 0/10 [00:00<?, ? fps/…
query_id target_id score
0 CHEMBL1316726 CHEMBL2261339 0.224490
1 CHEMBL4588846 CHEMBL495421 0.250000
2 CHEMBL3696763 CHEMBL185354 0.214286
3 CHEMBL4091061 CHEMBL1641956 0.278481
4 CHEMBL2349135 * NaN
5 CHEMBL1604960 CHEMBL343141 0.315789
6 CHEMBL2012902 * NaN
7 CHEMBL20406 CHEMBL2322823 0.230769
8 CHEMBL1796344 CHEMBL61106 0.207547
9 CHEMBL3400341 * NaN

The placeholder target id “*” and score of None (which pandas converts to a NaN) is because three of the compounds had no nearest neighbor with a similarity of at least 0.2.

The column names and placeholders can be changed. The following uses empty = None to not include those queries in the output:

result.to_pandas(columns=["FROM", "TO", "Tanimoto"], empty=None)
FROM TO Tanimoto
0 CHEMBL1316726 CHEMBL2261339 0.224490
1 CHEMBL4588846 CHEMBL495421 0.250000
2 CHEMBL3696763 CHEMBL185354 0.214286
3 CHEMBL4091061 CHEMBL1641956 0.278481
4 CHEMBL1604960 CHEMBL343141 0.315789
5 CHEMBL20406 CHEMBL2322823 0.230769
6 CHEMBL1796344 CHEMBL61106 0.207547

Progress bars

A you saw in the previous example, progress bars have been implemented, both on the command-line and through the high-level API.

Here’s an example using the command-line to generate RDKit fingerprints from a PubChem file:

!rdkit2fps Compound_099000001_099500000.sdf.gz -o Compound_099000001_099500000.fps

Compound_099000001_099500000.sdf.gz: 100%|█| 6.77M/6.77M [00:10<00:00, 626kbytes

and here is the equivalent using the high-level API:

import chemfp
chemfp.rdkit2fps("Compound_099000001_099500000.sdf.gz", "Compound_099000001_099500000.fps")

Compound_099000001_099500000.sdf.gz:   0%|                                      | 0.00/6.77M [00:00<?, ?bytes/…

ConversionInfo("Converted structures from 'Compound_099000001_099500000.sdf.gz'. #input_records=10740, #output_records=10740 (total: 3.67 s)")

The progress bars use tqdm, which (EDIT for chemfp 4.1) is a required chemfp install dependency.

The default progressbar can be disabled and re-enabled through the command-line:

chemfp.set_default_progressbar(False) # Disable
chemfp.set_default_progressbar(True) # Enable

If the environment variable CHEMFP_PROGRESSBAR is “0” then the default progressbar start in the disabled state.

Improved reprs

Many of the chemfp objects now implement a custom __repr__ which is more useful than the default. For examples:

import chemfp
arena = chemfp.load_fingerprints("chembl_30.fpb")
arena

FPBFingerprintArena(#fingerprints=2136187)

arena.knearest_tanimoto_search_fp(arena.fingerprints[12345], k=5)

SearchResult(#hits=5)

subarena = arena.sample(50, rng=54321)
picks = chemfp.heapsweep(subarena, num_picks=5)
picks

picks:   0%|                                                                                | 0/5 [00:00<?, ?/…

HeapSweepScoreSearch('picked 5 fps. similarity <= 1.0, #candidates=50, seed=-1 (pick: 7.87 ms, total: 8.17 ms)', picker=HeapSweepPicker(#candidates=45, #picks=5), result=PicksAndScores(#picks=5))

result = chemfp.simsearch(queries=subarena, targets=arena, k=3)
result

queries:   0%|                                                                         | 0/50 [00:00<?, ? fps/…

MultiQuerySimsearch('3-nearest Tanimoto search. #queries=50, #targets=2136187 (search: 373.85 ms total: 373.88 ms)', result=SearchResults(#queries=50, #targets=2136187))

“Shortcut” toolkit imports

Chemfp supports four different cheminformatics toolkits, which it uses for molecule I/O and fingerprint generation. Chemfp also implements a “toolkit” wrapper API, so chemfp-based programs can work with multiple toolkits in a consistent way.

The standard way to access these toolkits is by importing the toolkit wrapper subpackage, like:

from chemfp import openbabel_toolkit
#from chemfp import cdk_toolkit
#from chemfp import openeye_toolkit
#from chemfp import rdkit_toolkit

This adds as a bit of overhead which makes interactive use a bit less enjoyable.

Chemfp 4.0 adds a “shortcut” importer object, where the import occurs the first time it is accessed. For example, here’s a way to get the list of RDKit formats that chemfp supports:

import chemfp
chemfp.rdkit.get_formats()

[Format('rdkit/smi'),
 Format('rdkit/can'),
 Format('rdkit/usm'),
 Format('rdkit/cxsmi'),
 Format('rdkit/sdf'),
 Format('rdkit/sdf3k'),
 Format('rdkit/smistring'),
 Format('rdkit/canstring'),
 Format('rdkit/usmstring'),
 Format('rdkit/cxsmistring'),
 Format('rdkit/molfile'),
 Format('rdkit/rdbinmol'),
 Format('rdkit/fasta'),
 Format('rdkit/sequence'),
 Format('rdkit/helm'),
 Format('rdkit/mol2'),
 Format('rdkit/pdb'),
 Format('rdkit/xyz'),
 Format('rdkit/mae'),
 Format('rdkit/inchi'),
 Format('rdkit/inchikey'),
 Format('rdkit/inchistring'),
 Format('rdkit/inchikeystring')]

The special importer objects are chemfp.openbabel, chemfp.openeye, chemfp.cdk and chemfp.rdkit.

You should not import these objects as object (like from chemfp import openeye) because you will likely get confused with the real import openeye – I certainly do! Instead, alway use chemfp.openbabel and, if you want to import a specific toolkit use from chemfp import openbabel_toolkit as ob_toolkit or as T.

NOTE: it seems that Jupyter doesn’t understand how to get the properties of these importer objects as tab completion in the notebook doesn’t work, while it’s just fine in the Python shell.

FingerprintType improvements

A FingerprintType object is the interface to a given toolkit fingerprint type, including its parameters. Before chemfp 4.0 the only way to get a FingerprintType was to use a fingerprint type string, like:

import chemfp
fptype = chemfp.get_fingerprint_type("OpenBabel-FP2")
fptype

OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)

This was too generic, as it proved difficult to remember the correct string name and it’s parameters.

With chemfp 4.0, each of the toolkit wrapper modules includes a FingerprintType with the default values for each of the toolkit fingerprints families. For the Open Babel example:

from chemfp import openbabel_toolkit
openbabel_toolkit.fp2

OpenBabelFP2FingerprintType_v1(<OpenBabel-FP2/1>)

from chemfp import rdkit_toolkit
rdkit_toolkit.morgan

RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=2 fpSize=2048 useFeatures=0 useChirality=0 useBondTypes=1>)

The newly available FingerprintType objects are:

A FingerprintType is now callable, which lets you make a copy, with some arguments changed.

rdkit_toolkit.morgan(fpSize=128, radius=3)

RDKitMorganFingerprintType_v1(<RDKit-Morgan/1 radius=3 fpSize=128 useFeatures=0 useChirality=0 useBondTypes=1>)

There are new helper methods on the FingerprintType to simplify common cases of working with structure data. For example, the from_smiles method takes a SMILES string as input and generate the corresponding fingerprint.

rdkit_toolkit.morgan(fpSize=128, radius=3).from_smiles("CN1C=NC2=C1C(=O)N(C(=O)N2C)C")

b"x0bx06x01x08x93x10x19x04x00x84n@x10'x08x07"

Or, using the shortcut importer and using the InChI string as input:

import chemfp
chemfp.rdkit.morgan(fpSize=128, radius=3).from_inchi(
    "InChI=1S/C8H10N4O2/c1-10-4-9-6-5(10)7(13)12(3)8(14)11(6)2/h4H,1-3H3")

b"x0bx06x01x08x93x10x19x04x00x84n@x10'x08x07"

String and file I/O helper functions

There are several helper functions to make it easier to read and write from strings and files.

If you have an FPS file as a string, you can load it with load_fingerprints_from_string:

import chemfp
arena = chemfp.load_fingerprints_from_string("""\
aabbcc\tfirst
10d9b4\tsecond
""")
arena

FingerprintArena(#fingerprints=2)

There are helpers in the chemistry toolkit wrapper modules to read and write molecules:

from chemfp import openbabel_toolkit as ob_toolkit
ob_toolkit.parse_smiles("c1ccccc1O")

<openbabel.openbabel.OBMol; proxy of <Swig Object of type 'OpenBabel::OBMol *' at 0x115d2c180> >
ob_toolkit.create_inchi(ob_toolkit.parse_smiles("c1ccccc1O"))

'InChI=1S/C6H6O/c7-6-4-2-1-3-5-6/h1-5,7H n'
ob_toolkit.open_sdf3k_writer("example.sdf").write_molecule(ob_toolkit.parse_smiles("c1ccccc1O"))

==============================
*** Open Babel Warning  in WriteMolecule
  No 2D or 3D coordinates exist. Stereochemical information will be stored using an Open Babel extension. To generate 2D or 3D coordinates instead use --gen2D or --gen3D.
!head example.sdf

 OpenBabel04242317452D

  0  0  0     0  0            999 V3000
M  V30 BEGIN CTAB
M  V30 COUNTS 7 7 0 0 0
M  V30 BEGIN ATOM
M  V30 1 C 0 0 0 0
M  V30 2 C 0 0 0 0
M  V30 3 C 0 0 0 0

“chemfp” command

Chemfp 4.0 added several user-level commands. Rather than create program names which might collide with other name, I decided to add a new “chemfp” command, which implements subcommands. Many of the subcommands are the same as the top-level commands. There are also new commands for diversity search, and a couple related to licensing.

!chemfp

Usage: chemfp [OPTIONS] COMMAND [ARGS]...

Options:
  --version                Show the version and exit.
  --traceback              Print the traceback on KeyboardInterrupt
  --license-file FILENAME  Specify a chemfp license file
  --help                   Show this message and exit.

Generation commands:
  cdk2fps    Generate fingerprints using CDK.
  ob2fps     Generate fingerprints using Open Babel.
  oe2fps     Generate fingerprints using OEChem and OEGraphSim.
  rdkit2fps  Generate fingerprints using RDKit.
  sdf2fps    Extract fingerprints from an SDF file.

Algorithms:
  simsearch  Search an FPS or FPB file for similar fingerprints.
  spherex    Diversity selection using the sphere exclusion algorithm.
  maxmin     Diversity selection using the MaxMin algorithm.
  heapsweep  Diversity selection using the heapsweep algorithm.

Fingerprint file commands:
  fpcat     Combine multiple fingerprint files into a single file.
  fpb_text  Show the TEXT sections of an FPB file.

Other commands:
  license   Show the chemfp license status.
  report    Report chemfp similarity search implementation details.
  toolkits  Show underlying cheminformatics toolkit availability.

Diversity selection

Chemfp 4.0 added several forms of diversity picking.

MaxMin (see Ashton et al., https://doi.org/10.1002/qsar.200290002) is an approximate but fast method to select maximally diverse fingerprints from a candidate data set. Chemfp’s version of MaxMin also supports reference-based MaxMin, which selects diverse fingerprints from the candidates which are also diverse from a set of references. (For example, select 1,000 fingerprints from a vendor catalog which most enrich the diversity of a corporate collection.)

The following example uses MaxMin to select 5 diverse compounds from ChEMBL 30 which are also diverse from ChEMBL 29 (this takes over a minute):

!chemfp maxmin -n 5 --references chembl_29.fpb chembl_30.fpb --times

#Diversity/1
#num_bits=2048
#type=maxmin threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=1397588774
#software=chemfp/4.0
#candidates=chembl_30.fpb
#references=chembl_29.fpb
#date=2023-04-24T15:45:11
i   pick_id score
1   CHEMBL4778247   0.2500000
2   CHEMBL4797319   0.2553191
3   CHEMBL4764617   0.2577320
4   CHEMBL4761572   0.2592593
5   CHEMBL4754099   0.2638889
T_init: 0.02 T_pick: 75.65 #picks: 5 picks/s: 0.07 T_total: 75.68

Heapsweep is an exact but slow method to select maximally diverse fingerprints. It is used to seed the MaxMin algorithm when no initial fingerprint is specified. The 5 globally most diverse fingerprints in ChEMBL 30 are:

!chemfp heapsweep -n 5 chembl_30.fpb --times

#Diversity/1
#num_bits=2048
#type=heapsweep threshold=1.0 num-picks=5 all-equal=0 randomize=1 seed=2006307975
#software=chemfp/4.0
#candidates=chembl_30.fpb
#date=2023-04-24T15:46:27
i   pick_id score
1   CHEMBL1796997   0.0769231
2   CHEMBL4297424   0.0769231
3   CHEMBL2105487   0.0769231
4   CHEMBL1201290   0.0833333
5   CHEMBL4300465   0.0833333
T_init: 0.03 T_pick: 6.92 #picks: 5 picks/s: 0.72 T_total: 6.97

Sphere exclusion (see Hudson et al. https://doi.org/10.1002/qsar.19960150402) is used to reduced some of the clumpiness of random sampling, by avoiding picking fingerprints which are close to previous picks. Chemfp’s version of sphere exclusion also have a reference-based version. Chemfp also implements directed sphere exclusion (DISE) (see Gobbi et al., https://doi.org/10.1021/ci025554v) where a candidate fingerprint where the lowest-rank is chosen, rather than picking from all remaining candidates. The ranks can be assigned by the method of Gobbi et al. or with user-defined ranks.

Here are 10 fingerprints picked from ChEMBL such that no picks are with 0.2 similarity of each other, with the output in csv format, and with a fixed seed:

!chemfp spherex -n 10 --threshold 0.2 --out csv chembl_30.fpb --seed 12345

pick_id,count
CHEMBL1199645,157334
CHEMBL1544208,22982
CHEMBL72725,64693
CHEMBL600500,16792
CHEMBL533667,12671
CHEMBL4073454,28297
CHEMBL3827378,43489
CHEMBL3114290,29501
CHEMBL1184175,25290
CHEMBL566963,45206

CSV and TSV output

The simsearch, maxmin, heapsweep, and spherex commands support alternative output formats, using --out. The two alternatives are “csv” and “tsv”, for comma-separated and tab-separated, and are appropriately quoted for use by Excel. These do not contain a metadata header, and the diversity outputs do not include the pick index. The default output format is “chemfp”.

Here are two examples using simsearch:

!simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out csv

query_id,target_id,score
Query1,CHEMBL113,1.0000000
Query1,CHEMBL4591369,0.7333333
Query1,CHEMBL1232048,0.7096774

!simsearch --query "CN1C=NC2=C1C(=O)N(C(=O)N2C)C" chembl_30.fpb --out tsv

query_id    target_id       score
Query1      CHEMBL113       1.0000000
Query1      CHEMBL4591369   0.7333333
Query1      CHEMBL1232048   0.7096774

The csv and tsv formats use fixed number of columns, even when the default chemfp format uses a variable number of columns. Instead of having N extra columns for each line, there are N lines, one for each output column.

This can cause problems if N = 0, as when a simsearch query has no matches. In this case there is a synthetic output line with a target id of * and a score of NaN. The simsearch option --empty-target-id and the spherex option --empty-hit-id change the default id. The option --empty-score changes the default score.

Use --no-include-empty to not include a synthetic line when N = 0.

Miscellaneous

  • Use the output format “sdf3k” to always write SD files in V3000 format.

  • The low-level k-nearest arena query search API accepts an initial array of minimum thresholds, one per query. This may be useful when searching multiple target arenas, as the scores from earlier results may help reduce the search space.

  • Fingerprint arenas have a new get_bit_counts() method which for each bit counts the number of fingerprints where that bit is on.

  • Added open_from_string() and load_fingerprints_from_string() to make it easier to work with FPS or FPB content as a string.

  • Added support for Python 3.10 and CDK 2.7.

  • The SearchResult has a query_id, if known.

  • The NxN and NxM arena similarity search functions in chemfp.search now support optional batch_size and batch_callback parameters. These are used to implement progress bars.

  • The Location object now also supports position (the approximate location in the input file), end_position (the expected end position, or None), and position_units (currently only “bytes”). These are used for the progress bar.

  • The FPB writers have a location, which the high-level conversion functions used to get the number of output records.

  • Added byte_xor_popcount, hex_xor_popcount, byte_union_popcount, and hex_union_popcount to the bitops module.

  • Changed CDK-ShortestPath version to “2.7” because the PRNG change in 2.7 results in new bit patterns.

  • Added CDK ExtendedFingerprinter support for CDK 2.5 and later.

I also split the C extension into several extensions, including some based on Cython, and I dropped most of the slowest popcount implementations on the assumption that __builtin_popcountll is always good enough.

Breaking changes

Chemfp 4.0 drops support for Python 2.7, for Python 3.7 or earlier, and for toolkits versions before 2020.

Nearly everything else is backwards compatible with older versions of chemfp. There are three changes which I hope breaks no existing code, and if it does, they should be easy to fix.

  1. At the API level, the default similarity search had k=3 nearest neighbors with threshold=0.7 as Python default parameters. This kept tripping me up when I because when I wanted (say) all k=10 nearest neigbhors I would forget to change the threshold to 0.0. The default now is that if neither k nor threshold are specified then do a k=3 and threshold=0.7 search. If k is specified and threshold is not, use threshold=0.0. This matches the simsearch command-line behavior.

  2. The SearchResults.reorder_all() order parameter has been renamed ‘ordering’ be consistent with SearchResult.reorder() and with internal use.

  3. The “difference” functions in chemfp.bitops function have been renamed to xor because even I couldn’t remember what “difference” meant. The xor fuction names are byte_xor and hex_xor.

Deprecation warning

The FingeprintType method compute_fingerprint and compute_fingerprints are deprecated in favor of the have been deprecated in favor of the names from_mol and from_mols. The old methods will be removed in a a future version of chemfp.