Plot calibration curve with scikit-learn 1.0 #chemoinformatics #scikit-learn #memo

Recently scikit-learn ver. 1.0(nightly build) is released. I often use sklearn for my ask. So I would like to use new version ;)

Current stable version is 0.24 so I installed 1.0.rc2 via pip.

$ pip install scikit-learn==1.0rc2

Here is a release Highlights and notes.

https://scikit-learn.org/dev/auto_examples/release_highlights/plot_release_highlights_1_0_0.html

https://scikit-learn.org/dev/whats_new/v1.0.html#changes-1-0

Ver 1.0 CalibrationDisplay method which can make calibration-curve plot easily. So I tested it with solubility data. As you know calibration is applied against classification model such as SVC. To use calibration, classifier object should have predict_proba method.

Calibration curve shows reliability of the model. To make the curve, binned data is used and X-axis should be average probability of test_dataset and Y-axis should be fraction of positive. If the model work perfectly, the curve will be straight line with gradient 1.

OK let’s test it. Most of code is same as previous my post. And I used RF and SVC for test. I uploaded my code on my gist.

The curve before calibration is below. RF model seems work better than SVC.

Then I calibrate these models and try to make calibration curve again. To perform model calibration, I used CalibratedClassifierCV which is implemented scikit-learn. SVC seems slightly improved after calibration. The curve is shown below.

Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

Data visualization is important and useful for understanding my model/analysis.

Scikit-learn is really useful package for chemoinformatics. I’ll read documentation deeply and test them.

Advertisement

Published by iwatobipen

I'm medicinal chemist in mid size of pharmaceutical company. I love chemoinfo, cording, organic synthesis, my family.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: