chemfp.count_converters module

This module contains converters from count fingerprints to binary

The API is unlikely to change but may not be fully stable. The next release may have minor breaking changes, without the migration policy which applies to the stable portions of the chemfp API.

A CountConverter converts count fingerprints in one of several different forms into a byte fingerprint. The CountConverter.create_bytes() method, for example, starts with a chemfp CountFingerprint.

There are also methods to parse an FPC-encoded fingerprint string and an RDKit sparse fingerprint. By default theese work by converting the input to a CountFingerprint then using create_bytes, though some converters may have a specialized and faster interface.

The available convert converters are:

  • FoldedConverter_v1 for index folding

  • BinaryConverter_v1 for direct index mapping with no folding

  • RDKitCountSimConverter_v1 for RDKit count simulation

  • SuperimposedConverter_v1 for superimposed encoding

  • ScaledConverter_v1 for scaled superimposed encoding

  • SeqConverter_v1 for dense sequence encoding

  • ScaledSeqConverter_v1 for dense scaled sequence encoding

These can be instanciated directly, or through the get_count_converter() function, which takes a converter type string and optional keywoard arguments. Processing error will raise a CountConverterError subclass.

The underlying implementation and exception class structure is similar to get_fingerprint_type().

For example:

>>> from chemfp import count_converters
>>> conv = count_converters.get_count_converter("folded", {"num_bits": 64})
>>> conv
FoldedConverter_v1(num_bits=64, hash=0)
>>> conv.get_type()
'folded/1 num_bits=64'
>>> conv.parse_fpc("1,70,127")
b'B
class chemfp.count_converters.CountConverter

Bases: object

Base class for converters from count fingerprint to byte fingerprints

Every converter has the attribute:

num_bits: positive integer

The number of bits in the output fingerprint.

combine_with_metadata(metadata: Metadata) → tuple[Metadata, CountConverter]

Return a new metadata and a converter for the entire conversion pipeline.

For most processing, the new Metadata is the result of CountConverter.update_metadata() and the converter is left unchanged.

In some cases the converter may need information from the metadata, most likely its num_bits, to describe the full conversion. In that case, this method will first create a new CountConverter instance with the needed details, then use the new count converter to create the new metadata, and return both new objects.

This method may raise a ValueError if the combined metadata and converter do not have enough information or are incompatible.

Parameters:

metadata (a Metadata) – metadata for a count fingerprint dataset

Returns:

a 2-element tuple with the Metadata for the output binary fingerprints and a :class.:.CountConverter for the full conversion.

create_bytes(fp: CountFingerprint)

Convert a chemfp count fingerprint to a byte string

Calling the method on the base class will raise a NotImplementedError. It must be implemented in the subclass.

Parameters:

fp (a CountFingerprint) – a chemfp count fingerprint

Returns:

a binary fingerprint as bytes

classmethod from_text_settings(settings: Dict[str, str] | None = None) → CountConverter

Return a CountConverter instance based on the text settings

Calling the method on the base class will raise a NotImplementedError. It must be implemented in the subclass.

classmethod get_kwargs_from_text_settings(settings: Dict[str, str] | None = None) → Dict[str, Any]
get_type(complete: bool = False) → str

Return a canonical description of the converter as a string

Some parameters may not be included in the default type string, for example, if the value is not used or if a rarely used value equals the default. If complete is True then show all of the parameters.

The default implementation uses API hooks which are not yet part of the public API. If you derive your own subclass you will need to implement your own version.

Parameters:

complete (bool) – If True, show all of the type parameters

Returns:

a type string

num_bits
parse_fpc(s: str | bytes, stop_at_tab=True)

Parse an FPC-encoded count fingerprint to get a byte fingerprint

The input can be a text or byte string. By default parsing will stop at the end of string or the first tab. If stop_at_tab is False then the entire string must an FPC-encoded fingerprint.

This is equivalent to using countops.parse_fpc() to parse the input FPC string to get a CountFingerprint then converting that fingerprint to bytes with CountConverter.create_bytes(), which is how the default implementation works. A subclass may override this with a more optimized version.

Parameters:
  • s (str or bytes) – A string containing an FPC-encoded fingerprint

  • stop_at_tab (bool) – If True, stop parsing at the first tab

Returns:

bytes

parse_id_and_fpc(s: str | bytes, require_newline=False)

Parse a line from an FPC file to get the id and converted byte fingerprint

The input can be a text or byte string. By default the terminal newline may be present but is not required. If require_newline is True then the terminal newline must be present.

This is equivalent to using countops.parse_id_and_fpc() to parse the input FPC string to get the id and a CountFingerprint then converting that fingerprint to bytes with CountConverter.create_bytes(), which is how the default implementation works. A subclass may override this with a more optimized version.

Parameters:
  • s (str or bytes) – A string containing a fingerprint line from an FPC file.

  • require_newline (bool) – If True, the string must end with a newline.

Returns:

a tuple of id string and bytes

parse_rdkit_countfp(rdkit_countfp: _typing.RDKitCountFingerprint)

Parse an RDKit fingerprint binary string and convert to a byte fingerprint.

The rdkit_countfp may be a byte string generated by any of these RDKit’s DataStruct methods:

  • IntSparseIntVect().ToBinary()

  • UIntSparseIntVect().ToBinary()

  • LongSparseIntVect().ToBinary()

  • ULongSparseIntVect().ToBinary()

or any object with a ToBinary() method returning bytes in one of those formats.

The default implementation first converts the RDKit count fingerprint to a chemfp CountFingerprint, then converts that to the bye string. A subclass may decide use a more optimized implementation.

Parameters:

rdkit_countfp (bytes, or its ToBinary() returns bytes) – An RDKit count fingerprint

Returns:

bytes

update_metadata(metadata: Metadata) → Metadata

update the metadata to include details about this converter

The input Metadata should contains details about the input count fingerprint dataset. This method returns a new metadata instance which is a copy of the original metadata but with several fields changed so the result contains details about the output binary fingerprint dataset.

o The new ‘type’ combines the old ‘type’ and the converter’s

type string.

o The new ‘num_bits’ is the number of converter output bits.

o The new ‘num_bytes’ reflects the new ‘num_bits’

Parameters:

metadata (a Metadata) – metadata for a count fingerprint dataset

Returns:

a Metadata for the output binary fingerprints

exception chemfp.count_converters.CountConverterError

Bases: ValueError, ChemFPError

Base exception for many types of count converter errors

get_help() → str

Additional help with possible hints on how to fix the error.

Not currently used. This exists to match the API for get_fingerprint_type.

get_reason() → str

A one-line description of the reason for the error

exception chemfp.count_converters.CountConverterTypeSyntaxError(type_str: str, reason: str)

Bases: CountConverterError

The count converter type string does not follow the correct syntax.

type_str: str

The invalid count converter type string

reason: str

A short description of why the type string is invalid

get_reason()

A one-line description of the reason for the error

reason: str
type_str: str
exception chemfp.count_converters.CountConverterTypeUnknownError(base_name: str)

Bases: CountConverterError

The requested count converter family is unknown.

The available attribute is:

base_name: str

The convert family name, which is the first term in the type string, without version information.

get_reason() → str

A one-line description of the reason for the error

class chemfp.count_converters.FoldedConverter_v1

Bases: CountConverter

Convert count fingerprint to binary using folding.

This converter implements modulo folding based on each feature index. The count is ignored. The basic method as pseudocode is:

output_fp = BinaryFingerprint(num_bits)
for index, count in count_fingerprint:
    output_fp.SetBitOn(index % num_bits)

If hash is True then the hash of the index is used for the modulo folding:

output_fp = BinaryFingerprint(num_bits)
for index, count in count_fingerprint:
    output_fp.SetBitOn(hash_function(index) % num_bits)

return output_fp

Use hashing if the index values are not not well distributed after folding.

For example, suppose each index is a 15-bit atom triplet value where the first 5 bits come from the properties of atom 1, the next 5 bits come from the properties of atom 2, and the last 5 bits come from the properties of atom 3.

Now, the modulo operation is simply the remainder after division. To generate a 1024-bit fingerprint, the modulus is 1024, which means the bit positions with value 1024, 2048, 4096, etc. are all ignored (they are all multiple of 1024), leaving only the bit positions with values 1, 2, 4, … 512. These are the first 10 bits.

The result is a masking operation, where only the first 10 bits are used, and the remaining bits are ignored. In the above atom triplet, this means only the properties from atoms 1 and 2 will be used - atom 3 (which sets the 11th - 15th bits) will be ignored.

The hash method converts the input indices into new values which are not well-aligned with a fixed fingerprint length. This folded values incorporate more information from the input index, giving a better bit distribution.

num_bits: positive integer

The number of bits in the output fingerprint.

hash: bool

If True, use the index to seed a PRNG and generate the value too fold, instead of folding directly on the index.

create_bytes(fp: CountFingerprint)

Fold a chemfp count fingerprint to a binary fingerprint

Parameters:

fp (a CountFingerprint) – a chemfp count fingerprint

Returns:

a binary fingerprint as bytes

hash
name = 'folded/1'
parse_fpc(s: str | bytes, stop_at_tab=True)

Parse an FPC-encoded count fingerprint as folded bytes.

The input can be a text or byte string. By default parsing will stop at the end of string or the first tab. If stop_at_tab is False then the entire string must an FPC-encoded fingerprint.

The FPC-encoded count fingerprint is folded directly to byte fingerprint.

Parameters:
  • s (str or bytes) – A string containing an FPC-encoded fingerprint

  • stop_at_tab (bool) – If True, stop parsing at the first tab

Returns:

bytes

parse_id_and_fpc(s: str | bytes, require_newline=False)

Parse a line from an FPC file as the id and folded byte fingerprint

The input can be a text or byte string. By default the terminal newline may be present but is not required. If require_newline is True then the terminal newline must be present.

The implementation parses the input FPC string directly to extract the id and create the folded fingerprint bytes.

Parameters:
  • s (str or bytes) – A string containing a fingerprint line from an FPC file.

  • require_newline (bool) – If True, the string must end with a newline.

Returns:

a tuple of id string and bytes

parse_rdkit_countfp(rdkit_countfp: _typing.RDKitCountFingerprint)

Parse an RDKit fingerprint binary string and convert to a folded fingerprint.

The rdkit_countfp may be a byte string generated by any of these RDKit’s DataStruct methods:

  • IntSparseIntVect().ToBinary()

  • UIntSparseIntVect().ToBinary()

  • LongSparseIntVect().ToBinary()

  • ULongSparseIntVect().ToBinary()

or any object with a ToBinary() method returning bytes in one of those formats.

The RDKit count fingerprint is folded directly to a byte fingerprint.

Parameters:

rdkit_countfp (bytes, or its ToBinary() returns bytes) – An RDKit count fingerprint

Returns:

bytes

class chemfp.count_converters.RDKitCountSimConverter_v1

Bases: CountConverter

Convert a count fingerprint to a binary fingerprint using RDKit count simulation.

RDKit count simulation is based around a list of countBounds of length w. The countBounds are used to turn a count into a set of w bits, where the first bit is set if the count is greater than or equal to the first countBound, and so on.

Generation is a two-step process. The first step uses modulo folding to convert the input count fingerprint to a reduced count fingerprint. The reduced count fingerprint has a length of num_bits/w counts, where the count for each position is the sum of the input counts for all input indices which fold to that position.

The second step uses the countBounds to convert the total count at each reduced position to the corresponding bit pattern.

As pseudocode:

num_bins = len(countBounds)
reduced_num_bits = num_bits // num_bins
counts = [0] * reduced_num_bits
for idx, count in count_fingerprint:
    counts[idx % reduced_num_bits] += count

output_fp = BinaryFingerprint(num_bits)
for i, count in enumerate(counts):
    if count == 0:
        continue
    for j, min_count in enumerate(countBounds):
        if count >= min_count:
            output_fp.SetOnBit(i * num_bins + j)

return output_fp

The public attributes are:

num_bits: positive integer

The number of bits in the output fingerprint.

countBounds: a Python array.array of "I" (unsigned integers)

The countBounds used to set the bit pattern. The input countBounds are converted to a new Python array instance.

The default countBounds are [1, 2, 4, 8], which approximates a int(log2(n+1)) scaling for counts up to 15.

A countBounds of None means to use the default.

The countBounds in a type string is represented as a comma-separated list of values. The default generates the string “1,2,4,8”.

countBounds
create_bytes(fp: CountFingerprint)

Use RDKit count simulation to convert a chemfp count fingerprint to a binary fingerprint

Parameters:

fp (a CountFingerprint) – a chemfp count fingerprint

Returns:

a binary fingerprint as bytes

name = 'rdkit-countsim/1'
parse_fpc(s: str | bytes, stop_at_tab=True)

Parse an FPC-encoded count fingerprint as RDKit count simulation bytes.

The input can be a text or byte string. By default parsing will stop at the end of string or the first tab. If stop_at_tab is False then the entire string must an FPC-encoded fingerprint.

Parameters:
  • s (str or bytes) – An FPC-encoded fingerprint

  • stop_at_tab (bool) – If True, stop parsing at the first tab

Returns:

bytes

parse_id_and_fpc(s: str | bytes, require_newline=False)

Parse a line from an FPC file as the id and RDKit count simulation byte fingerprint

The input can be a text or byte string. By default the terminal newline may be present but is not required. If require_newline is True then the terminal newline must be present.

The implementation parses the input FPC string directly to extract the id and create the RDKit count simulation fingerprint bytes.

Parameters:
  • s (str or bytes) – A string containing a fingerprint line from an FPC file.

  • require_newline (bool) – If True, the string must end with a newline.

Returns:

a tuple of id string and bytes

class chemfp.count_converters.SuperimposedConverter_v1

Bases: CountConverter

Convert a count fingerprint to a binary fingerprint using random superimposed coding.

Each feature has an index and a count. For each feature, the index is used to seed a pseudo-random number generator to generates min(count, max_count)*bits_per_count random values from 0 to num_bits-1 (duplicates are allowed).

Using bits_per_count values > 1 appears to make the Tanimoto scores less reliable.

If the on-bit density of the output binary fingerprint is low (~5% or less) then the binary Tanimoto between two output fingerprints is a good approximation to the count Tanimoto between the two input count fingerprints.

num_bits: positive integer

The number of bits in the output fingerprint.

bits_per_count: positive integer

The number of bits to set for each count of each feature. The default is 1. It does not seem useful to set more than 1 bit per count.

max_count: positive integer

If the feature count is larger than max_count then use max_count instead. The largest possible value is 2**32-1. The default is 1000, which prevent features like 0:1000000000 from generating 1 billion random values. NOTE: a future of this generator may place a hard upper-bound (likely 1000) on max_count.

bits_per_count
create_bytes(fp: CountFingerprint)

Superimpose a chemfp count fingerprint to a binary fingerprint

Parameters:

fp (a CountFingerprint) – a chemfp count fingerprint

Returns:

a binary fingerprint as bytes

max_count
name = 'superimposed/1'
parse_fpc(s: str | bytes, stop_at_tab=True)

Parse an FPC-encoded count fingerprint as superimposed bytes.

The input can be a text or byte string. By default parsing will stop at the end of string or the first tab. If stop_at_tab is False then the entire string must an FPC-encoded fingerprint.

Parameters:
  • s (str or bytes) – An FPC-encoded fingerprint

  • stop_at_tab (bool) – If True, stop parsing at the first tab

Returns:

bytes

parse_id_and_fpc(s: str | bytes, require_newline=False)

Parse a line from an FPC file as the id and superimposed byte fingerprint

The input can be a text or byte string. By default the terminal newline may be present but is not required. If require_newline is True then the terminal newline must be present.

The implementation parses the input FPC string directly to extract the id and create the superimposed fingerprint bytes.

Parameters:
  • s (str or bytes) – A string containing a fingerprint line from an FPC file.

  • require_newline (bool) – If True, the string must end with a newline.

Returns:

a tuple of id string and bytes

parse_rdkit_countfp(rdkit_countfp: _typing.RDKitCountFingerprint)

Parse an RDKit fingerprint binary string to get the superimposed binary fingerprint

The rdkit_countfp may be a byte string generated by any of these RDKit’s DataStruct methods:

  • IntSparseIntVect().ToBinary()

  • UIntSparseIntVect().ToBinary()

  • LongSparseIntVect().ToBinary()

  • ULongSparseIntVect().ToBinary()

or any object with a ToBinary() method returning bytes in one of those formats.

The RDKit count fingerprint binary is superimposed directly to a byte fingerprint.

Parameters:

rdkit_countfp (bytes, or its ToBinary() returns bytes) – An RDKit count fingerprint

Returns:

bytes