ChEMBL Resources

The SARfaris: GPCR, Kinase, ADME

Tuesday, 28 July 2015

Paper: Activity, assay and target data curation and quality in the ChEMBL database

We've just published an Open Access paper in the Journal of Computer-Aided Molecular Design on the curation of bioactivity, assay and target data in ChEMBL, including current practices and future plans. 

Here is the abstract:

The emergence of a number of publicly available bioactivity databases, such as ChEMBL, PubChem BioAssay and BindingDB, has raised awareness about the topics of data curation, quality and integrity. Here we provide an overview and discussion of the current and future approaches to activity, assay and target data curation of the ChEMBL database. This curation process involves several manual and automated steps and aims to: (1) maximise data accessibility and comparability; (2) improve data integrity and flag outliers, ambiguities and potential errors; and (3) add further curated annotations and mappings thus increasing the usefulness and accuracy of the ChEMBL data for all users and modellers in particular. Issues related to activity, assay and target data curation and integrity along with their potential impact for users of the data are discussed, alongside robust selection and filter strategies in order to avoid or minimise these, depending on the desired application.

%T Activity, assay and target data curation and quality in the ChEMBL database
%A G. Papadatos
%A A. Gaulton
%A A. Hersey
%A J.P. Overington
%J Journal of Computer-Aided Molecular Design
%D 2015
%O doi:10.1007/s10822-015-9860-5

Thursday, 23 July 2015

ChEMBL python client update

Along with updating ChEMBL web services to the new 2.x version, we've also updated the python client library (chembl_webresource_client). The change was backwards compatible so it's possible that existing users haven't even noticed the change.

As we've already provided examples of using new web services via cURL or using live docs, now it's good time to explain the changes made to the python client.

First of all, if you haven't installed (or updated) it yet, you can do it using Python Package Index:

Now you can access new functionality using the following import statement:

Just as a mild warning, in 0.8.x versions of the client the new part will be called new_client. In 0.9.x it will change the name to client and the old part will be renamed to old_client and deprecated. In 1.0.x the old functionality will be removed completely.

OK, so since we know how to import our new_client object, we can try to do something useful. Let's retrieve some activities. We know, that the new web services introduces filtering, so we can try to get only activities with standard type equal to 'Ki' and with standard value greater or equal 5:

We've already applied two filters, so you may expect that the client made two requests to server as well (one to get all activities with specified standard type and another one to filter on the standard value). In reality no interaction with server has been performed so far. This is because our client is lazy - but in the good sense. It waits until you are actually trying to use some of the data requested and then tries to perform the most sensible query to the server so you don't have to care.

So If we would like to access the fifth element:

the client will interact with server and retrieve a chunk of data that is large enough to prevent from hammering the server too frequently. The data is then cached locally, so if you restart your computer and rerun your script all the data will be fetched from the cache (if available).

A careful reader with IT background will notice that our client presents an interface that is very similar to Django QuerySet. This is true, we designed our client to mimic Django ORM behavior by implementing chaining filters and lazy evaluation. You don't have to be a Django expert to see advantages of such approach, but if you do have some experience with Django, you will feel like home.

To cover most important use cases we've created an IPython notebook, comparing the client with the plain requests library approach. The notebook is loaded into myChEMBL so if you have myChEMBL running locally, you can give it a try immediately. If not, just install the client and follow the tutorial below:

Monday, 20 July 2015

Biological annotations in SureChEMBL

Termite annotation in action. (Termite not to scale)

SureChEMBL is perhaps the only freely available, large-scale, comprehensive and live resource of chemistry extracted from the patent literature. SureChEMBL automatically annotates, normalises and indexes chemistry found in the full text, images and attachments (i.e. mol files) of patent documents. The next logical step for us, was to complement the chemical annotations with biological ones, such as mentions of gene names and classifications, protein classes and disease indications.

As the first step towards this direction, we used Termite provided by SciBite (via funding from OpenPHACTS) to integrate these annotations dynamically into the full text patent view of the SureChEMBL user interface; in other words, you can now view biological annotations on-the-fly.

How do I add the annotations and navigate through them?

There is now an additional checkbox underneath the 'Highlight additional recognised chemical terms' checkbox:

Simply check the box and let the magic happen! Once the loading process is complete (the annotation process can take quite some time on large patents, please be patient!), you should be greeted with an annotation type tree that allows you to navigate through the highlighted annotations.

Expanding a node will show the terms, sorted by frequency of occurrence in the patent document. Hovering over the term entry will yield navigation buttons to traverse forwards or backwards through all instances of a particular annotation.

The text is also highlighted in the corresponding colour, with the currently selected term instance bouncing for your attention.

Clicking on the annotations also allows you to view cross-references and links to external resources, in a similar fashion to the chemical annotations.

As part of our contribution to the OpenPHACTS project, all chemical and biological annotations found in our huge patent backfile (6 million documents and counting) will be semantically integrated in the OpenPHACTS system and will be part of the freely available OpenPHACTS API. This will be available later this year.

As always, comments and feedback are very welcome.

The SureChEMBL Team

Thursday, 16 July 2015

myChEMBL + docker

In addition to the myChEMBL 20 VM images released earlier, today we are very happy to release myChEMBL Docker images.

What's docker?


Docker is a new open-source project that automates the deployment of distributed applications. It takes advantage of some new cool features of modern Linux kernel in order to run virtual containers, avoiding the overhead of starting and maintaining virtual machines [from Wikipedia].

In contrast to virtual machines, which emulate virtual hardware, docker containers employ the kernel of the host machine so they don't require or include the whole operating system. While still separated from the host, they only add a very thin level of abstraction [ZDNet article].

Why docker?


Docker is an emerging technology; it has become extremely popular over the last year and been adopted and used by the largest IT companies, such as RedHat, Canonical and Microsoft. Basically, using this platform you can do three things:
  1. Build
  2. Ship
  3. Run
an arbitrary complex piece of software in a very convenient way.

But why docker and myChEMBL?


For myChEMBL in particular, using docker means three things:

  1. Less data to download so less time to wait. VM-based Ubuntu and Centos images available from our FTP have 8.5 and 6.5 GB of size respectively (after compression). Compressed myChEMBL docker containers are both about 5.7 GB.
  2. No waiting to boot up. Normally, when you attach a disk image containing myChEMBL to your virtual machine or use vagrant to do it for you, you still have to wait a few minutes for the machine to start. With docker containers there is no booting so everything runs immediately.
  3. This is a future plan - layers. The whole concept of docker images is based on the fact, that they are build on top of other containers. Since this is our first release of myChEMBL which supports docker, we put everything into a single layer that is build on top of Ubuntu or CentOS, depending on the flavour you choose. However, in future we plan to divide our software into several layers so you can mix and match layers, for example download an image containing database without webservices or webservices layer but no interface and so on.

How do I install docker?


This is the tricky part. The easiest way is to have a clean 64-bit Linux machine (yes, a physical machine) with modern Linux kernel, such as Ubuntu (versions 14.04, 12.04 and 13.10 are officially supported). Many other Linux distributions can work as well, but the most important thing is that the minimal version of the Linux kernel is 3.10 and it has to be a 64-bit machine. In Ubuntu case, all you have to do is to run:

wget -qO- | sh
Installation on OS X is much more complicated. This is because the standard OS X installation downloads and configures VirtualBox and runs a very lightweight 64-bit Linux with docker installed. Now the problem is, that it won't work in case of myChEMBL. This is because this Virtual Machine has only 20GB of available disk space and our myChEMBL container is 23GB after decompressing. So in order to use it, you first have to resize the volume, which is explained here: The same instructions apply to Windows.

How to get myChEMBL running on docker?


The steps are very simple:
  1. Download the image from the FTP.
  2. Uncompress
  3. Load image into docker
  4. Run it
You may wonder why you have to download the image from ChEMBL FTP website when docker is providing Docker Hub - a place where you can upload and share docker images? Well, this is due to a bug in docker, which prevents some images from pushing. This may be related to the big size of ChEMBL container but there is no easy way to reduce it given that the ChEMBL database itself has a size of 8GB.

So instead of pulling from the Docker Hub we retrieve the images from ChEMBL FTP. This is how it should look like:

  1. Download:
  2. Uncompress:
    gunzip mychembl_20_ubuntu_docker.tar.gz
    gunzip mychembl_20_centos_docker.tar.gz
  3. Load:
    docker load < mychembl_20_ubuntu_docker.tar

    docker load < mychembl_20_centos_docker.tar
  4. Run:
    docker run -p 2222:22 -p 80:80 -p 9612:9612 -t -i chembl/mychembl_20_ubuntu /usr/local/bin/supervisord
    docker run -p 2222:22 -p 80:80 -p 9612:9612 -t -i chembl/mychembl_20_centos /usr/bin/supervisord

After successful completion of the steps above, you can open you browser and go to if you are running docker locally or http://some_other_host/ if you are running docker on some other host. You should then be able to see myChEMBL launchpad page.

As you can see we expose three ports: 80 for myChEMBL launchad web page, 9612 for IPython notebook server and 22 (which is mapped to 2222 in our example) for ssh. This means that you can easily ssh into the container by executing:

ssh chembl@some_other_host -p 2222

The password is always chemblvm. As you see we've decided to remap the standard ssh 22 port into 22 because it can happen, that your machine is running sshd as well in which case there could be a clash. The same applies to ipython notebook, we've chosen 9612 port avoid clash if you are running Ipyhton on your machine already. The only exception here is port 80, which we kept as it is, so if you have some other server running on this port you have to remap as well, for example:

 docker run -p 2222:22 -p 8080:80 -p 9612:9612 -t -i chembl/mychembl_20_ubuntu /usr/local/bin/supervisord

If you would like to expose postgres as well, then enabling standard postgres 5432 port is required: 

docker run -p 5432:5432 -p 2222:22 -p 80:80 -p 9612:9612 -t -i chembl/mychembl_20_ubuntu /usr/local/bin/supervisord 

Future work


Working on docker support for myCHEMBL was great fun and an important lesson of DevOps. There are a number of things we would like to improve in future in myChEMBL itself and the docker related parts to make it even better:

  1. Use supervisor as a default process manager for myChEMBL.
    Docker is designed to run one (typically foreground) process per container. Moreover, Docker ignores initialization of OS-specific daemons such as upstart, systemd etc. This is where supervisor comes handy and it can be used as a default process manager instead of any OS-specific mechanisms even if docker is not used. Same as `pip` tool is better for installing python packages, instead of using 'apt-get' or 'yum', in the similar way using supervisor for managing python web applications is better than systemd or upstart.
  2. Provide a Dockerfile.
    Dockerfile is like a recipe, describing all steps, dependencies and settings necessary to repetitively build and use docker image. Having a dockerfile (or a set of dockerfiles) for myChEMBL would make a process of preparing a container more transparent and robust. We already have a script called used to build our VM images but because of subtle differences between docker containers and VMs (look point 1) it can't just be used as is.
  3. Implement layers.
    As mentioned above, having defined some logical components of myChEMBL and distributing them into separate layers would encourage combining different aspects of the system, leading to less time spent on downloading and installing the whole thing.

We're recruiting!

Want to join the ChEMBL team?

We are seeking to recruit an experienced Web Application Developer to join the Chemogenomics Team at the European Bioinformatics Institute (EMBL-EBI).

You will develop a series of web-based applications and interfaces for the ChEMBL chemogenomic resources. In collaboration with senior team members you will also have a role in determining and advising on the web development strategy for the chemogenomic resources. In addition you will be involved with the development, maintenance and documentation of these tools and supporting their usage within EMBL-EBI and externally. The position will also involve some requirement gathering and use-case development.

For more information or to apply for this position follow this link:

The ChEMBL team

Tuesday, 14 July 2015

myChEMBL 20 has landed

We are very pleased to announce that the latest myChEMBL release, based on the ChEMBL 20 database, is now available to download. In addition to the ChEMBL upgrade, you will also find a number of changes and new features:

  • Updates in system and Python libraries, including the iPython notebook server
  • Upgrade in the web services (data and utils) to match the new functionality provided by the main ChEMBL ones
  • Current stable version of RDKit (2015.03)
  • Two brand new notebooks, namely an RDKit tutorial and a tutorial on SureChEMBL data mining, increasing the total number of notebooks to 14
  • Updates in several other iPython notebooks and the KNIME workflow, in order to take advantage of the new data, models and web services functionality
  • Several bug fixes
  • A CentOS 7 VM version, in addition to the existing Ubuntu 14.04 one
  • New virtualisation technologies, as explained in the section below

Lots of flavours

dipping case3.JPG

This new myChEMBL release is technical feature-rich, as we’ve decided to focus on providing a  variety of myChEMBL boxes and image formats via different distribution channels:

  1. New CentOS-based distribution - as requested by many users, we now provide a CentOS-based image, along with the existing Ubuntu one. CentOS is a Linux distribution that is focused on security and enterprise-class computing. It’s free and widely used in industry so no further introduction is needed. Our box is based on the latest stable version 7. One thing worth noting is that CentOS-based images are significantly smaller than Ubuntu ones.
  2. Different image formats - in addition to the standard vmdk images now available for Ubuntu and CentOS, we also provide other image formats. Although VMDK is an open format, it’s mainly used by proprietary software, such as vSphere. We decided to support free and open-source hypevisors as well, so this is why we are now publishing QEMU compatible qcow2 format. To help even more, we are providing a generic raw disk image dumps in img format which can then be converted to any other specific format to provide support for other virtualisation platforms. In fact, we used img files to generate qqow2 by running qemu-img convert -f raw -O qcow2 ubuntu.img ubuntu.qcow2
  3. Distribution channels - the traditional way to get myChEMBL image is to visit our FTP page. You can find there compressed images of our Ubuntu and CentOS myChEMBL distributions in different image formats. If you want to save time creating and configuring your Virtual Machine from scratch, you can use Vagrant instead of FTP. If you have Vagrant already installed, all you need to do is to open the terminal and type:
    vagrant init chembl/mychembl_20_ubuntu && vagrant up or:
    vagrant init chembl/mychembl_20_centos && vagrant up
    depending on the version you would like to use.
  4. An additional cool new feature is docker support but, since docker is a quite new technology, we would like to dedicate a separate blog post to this topic - so come back soon for exciting details.


There are now several different ways for installing myChEMBL:

  1. Follow the instructions in the INSTALL file on the ftpsite. This will import the myChEMBL VM into VirtualBox.
  2. Use Vagrant to install myChEMBL. See  point 3 in the section above.
  3. Bare metal - if you have a clean Ubuntu or CentOS box with root access and want to install myChEMBL software directly, then you may run:
    wget && chmod +x && bash
    for Ubuntu or:
    wget && chmod +x && bash
    for CentOS.
  4. Instructions for Docker will be released #soon in a coming blog post.

As usual, the full codebase lives on GitHub

Publications and webinars

myChEMBL is reported and documented in two Open Access publications, namely here and here. In case you're new to myChEMBL, there is also a recorded webinar and its associated slides here

Future plans

The myChEMBL resource is an evolving system and we are always interested in new open source projects, tools and notebooks. Please get in touch if you have any suggestions or questions.

The myChEMBL team

Tuesday, 7 July 2015

ChEMBL @ Boston this August

A couple of us will be visiting Boston, MA for the ACS Meeting between 16 and 20 August. We'll be talking about SureChEMBL and ChEMBL. If you'd like to arrange a meeting/seminar or just go out for drinks and clams, just let me know