Build regression model in Keras

I introduced Keras in mishimasyk#9. And my presentation was how to build classification model in Keras.
A participant asked me that how to build regression model in Keras. I could not answer his question.
After syk#9, I searched Keras API and found good method.
Keras has Scikit-learn API. The API can build regression model. ;-)
Example code is following.
The code is used for build QSAR model.

import numpy as np
import pandas as pd
import sys

from rdkit import Chem
from rdkit.Chem import AllChem
from rdkit.Chem import DataStructs

from sklearn.cross_validation import train_test_split
from sklearn.cross_validation import cross_val_score
from sklearn.cross_validation import KFold
from sklearn.metrics import mean_squared_error
from sklearn.metrics import r2_score

from keras.models import Sequential
from keras.layers import Activation, Dense, Dropout
from keras.wrappers.scikit_learn import KerasRegressor

def getFpArr( mols, nBits = 1024 ):
    fps = [ AllChem.GetMorganFingerprintAsBitVect( mol, 2, nBits=nBits ) for mol in mols ]
    X = []
    for fp in fps:
        arr = np.zeros( (1,) )
        DataStructs.ConvertToNumpyArray( fp, arr )
        X.append( arr )
    return X

def getResponse( mols, prop="ACTIVITY" ):
    Y = []
    for mol in mols:
        act = mol.GetProp( prop )
        act = 9. - np.log10( float( act ) )
        Y.append( act )
    return Y

def base_model():
    model = Sequential()
    model.add( Dense( input_dim=1024, output_dim = 100 ) )
    model.add( Activation( "relu" ) )
    model.add( Dense( 100 ) )
    model.add( Activation( "relu" ) )
    model.add( Dense( 1 ) )
    #model.add( Activation( 'relu' ) )
    model.compile( loss="mean_squared_error",  optimizer="adam" )
    return model

if __name__ == '__main__':
    filename = sys.argv[1]
    sdf = [ mol for mol in Chem.SDMolSupplier( filename ) ]
    X = getFpArr( sdf )
    Y = getResponse( sdf )

    trainx, testx, trainy, testy = train_test_split( X, Y, test_size=0.2, random_state=0 )
    trainx, testx, trainy, testy = np.asarray( trainx ), np.asarray( testx ), np.asarray( trainy ), np.asarray( testy )
    estimator = KerasRegressor( build_fn = base_model,
                                 ) trainx, trainy )
    pred_y = estimator.predict( testx )
    r2 = r2_score( testy, pred_y )
    rmse = mean_squared_error( testy, pred_y )
    print( "KERAS: R2 : {0:f}, RMSE : {1:f}".format( r2, rmse ) )

Run the code.
I used CHEMBLdatast.

mishimasyk9 iwatobipen$ python sdf/CHEMBL952131_EGFR.sdf 
Using Theano backend.
Epoch 1/100
102/102 [==============================] - 0s - loss: 62.2934     
Epoch 100/100
102/102 [==============================] - 0s - loss: 0.0123     
KERAS: R2 : 0.641975, RMSE : 0.578806

R2 is 0.64. It’s not so bad. ;-)
I pushed the script to the syk 9 repository.

RemoteMonitor in keras

There are several packages to perform deep learning in python. And my favorite one is keras.
Today, I found new function in keras.callbacks named RemoteMonitor.
The function provide real time visualization of learning.
So, I wrote very simple example using IRIS dataset.
At first to use RemoteMonitor, I need clone api from following URL.
And run
Now, I can access localhost:9000 via web browser.

iwatobipen$ git clone
iwatobipen$ cd hualos
iwatobipen$ python

Then, perform machine learning.

from sklearn.datasets import load_iris
from sklearn.cross_validation import train_test_split
from keras.utils.np_utils import to_categorical
from keras.callbacks import RemoteMonitor
# prepare dataset
irisdata = load_iris()
X,Y = irisdata["data"], irisdata["target"]
train_X, test_X, train_Y, test_Y = train_test_split( X, Y, test_size=0.2, random_state=123 )
# make monitor with default settings.
monitor = RemoteMonitor()

from keras.models import Sequential
# build model
model = Sequential()
from keras.layers import Dense, Activation
model.add( Dense( output_dim=12, input_dim=4 ) )
model.add( Activation( "relu" ) )
model.add( Dense( output_dim=3 ) )
model.add( Activation( "softmax" ) )
model.compile( optimizer="sgd",
               metrics = ["accuracy"] )

OK. Let’s learn!
To monitor the progress, I set [ monitor ] in callbacks argument.

hist = train_X, train_Y, nb_epoch=50, batch_size=20, callbacks=[ monitor ] )

During the learning, localhost:9000 is automatically updated and I can get following image.
This page made by Flask and d3.js. Cool !!!
“” will not work on python3.x.
If you want to run the code in python3.x, I recommend replace following line because of iteritems is renamed to items in python3.

        lines = ["%s: %s" % (v, k)
--                 for k, v in self.desc_map.iteritems() if k]
++                 for k, v in self.desc_map.items() if k]

9th Mishima.syk will be held 10th Dec. Topic is TensorFlow and Keras hans on.

Don’t miss it. ;-)