Simple Neural Network get 95.83% accuracy with RMSprop optimizer


Simple Neural Network by using RMSprop optimzer with only 5 layers.

 Here's the coding.

from __future__ import print_function

import keras
from keras.models import Sequential
from keras.layers import Dense, Dropout
from keras.optimizers import RMSprop
from keras.utils import np_utils
import os
from six.moves import cPickle as pickle

batch_size = 128
num_classes = 10
epochs = 20

def load_data():
    test_filename = "notMNIST.pickle"
    if os.path.exists(test_filename):
        with open(test_filename, 'rb') as f:
            letter_dataset = pickle.load(f)
    return (letter_dataset)
     
lt_dt = load_data()
 
train_dataset = lt_dt['train_dataset']
train_labels = lt_dt['train_labels']
valid_dataset = lt_dt['valid_dataset']
valid_labels = lt_dt['valid_labels']
test_dataset = lt_dt['test_dataset']
test_labels = lt_dt['test_labels']

x_train = train_dataset.reshape(train_dataset.shape[0], 28*28)
y_train = np_utils.to_categorical(train_labels, 10)

x_valid = valid_dataset.reshape(valid_dataset.shape[0], 28*28)
y_valid = np_utils.to_categorical(valid_labels, 10)

x_test = test_dataset.reshape(test_dataset.shape[0],28*28)
y_test = np_utils.to_categorical(test_labels, 10)

x_train = x_train.astype('float32') / 255
x_valid = x_valid.astype('float32') /255
x_test = x_test.astype('float32') / 255

print(x_train.shape[0], 'train samples')
print(x_valid.shape[0], 'valid samples')
print(x_test.shape[0], 'test samples')

model = Sequential()
model.add(Dense(512, activation='relu', input_shape=(784,)))
model.add(Dropout(0.2))
model.add(Dense(512, activation='relu'))
model.add(Dropout(0.2))
model.add(Dense(num_classes, activation='softmax'))

model.summary()

model.compile(loss='categorical_crossentropy',
              optimizer=RMSprop(),
              metrics=['accuracy'])

history = model.fit(x_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1,
                    validation_data=(x_valid, y_valid))

score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss:', score[0])
print('Test accuracy:', score[1])

Outputs:
Using Theano backend.

200000 train samples
10000 valid samples
10000 test samples
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 512)               401920    
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 512)               262656    
_________________________________________________________________
dropout_2 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 10)                5130      
=================================================================
Total params: 669,706
Trainable params: 669,706
Non-trainable params: 0
_________________________________________________________________
Train on 200000 samples, validate on 10000 samples
Epoch 1/20
200000/200000 [==============================] - 25s 127us/step - loss: 0.6390 - acc: 0.8158 - val_loss: 0.4971 - val_acc: 0.8476
Epoch 2/20
200000/200000 [==============================] - 27s 134us/step - loss: 0.4582 - acc: 0.8614 - val_loss: 0.4366 - val_acc: 0.8634
Epoch 3/20
200000/200000 [==============================] - 26s 128us/step - loss: 0.4092 - acc: 0.8757 - val_loss: 0.3952 - val_acc: 0.8774
Epoch 4/20
200000/200000 [==============================] - 26s 129us/step - loss: 0.3776 - acc: 0.8854 - val_loss: 0.3747 - val_acc: 0.8840
Epoch 5/20
200000/200000 [==============================] - 26s 128us/step - loss: 0.3560 - acc: 0.8918 - val_loss: 0.3678 - val_acc: 0.8871
Epoch 6/20
200000/200000 [==============================] - 26s 128us/step - loss: 0.3397 - acc: 0.8956 - val_loss: 0.3568 - val_acc: 0.8915
Epoch 7/20
200000/200000 [==============================] - 26s 131us/step - loss: 0.3261 - acc: 0.9003 - val_loss: 0.3476 - val_acc: 0.8954
Epoch 8/20
200000/200000 [==============================] - 27s 134us/step - loss: 0.3163 - acc: 0.9028 - val_loss: 0.3446 - val_acc: 0.8966
Epoch 9/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.3056 - acc: 0.9062 - val_loss: 0.3339 - val_acc: 0.8999
Epoch 10/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2966 - acc: 0.9086 - val_loss: 0.3363 - val_acc: 0.9009
Epoch 11/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2899 - acc: 0.9108 - val_loss: 0.3439 - val_acc: 0.8981
Epoch 12/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2845 - acc: 0.9123 - val_loss: 0.3378 - val_acc: 0.9040
Epoch 13/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2769 - acc: 0.9150 - val_loss: 0.3379 - val_acc: 0.9018
Epoch 14/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2724 - acc: 0.9158 - val_loss: 0.3412 - val_acc: 0.8999
Epoch 15/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2661 - acc: 0.9176 - val_loss: 0.3547 - val_acc: 0.9020
Epoch 16/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2610 - acc: 0.9191 - val_loss: 0.3435 - val_acc: 0.9036
Epoch 17/20
200000/200000 [==============================] - 26s 132us/step - loss: 0.2560 - acc: 0.9197 - val_loss: 0.3500 - val_acc: 0.9034
Epoch 18/20
200000/200000 [==============================] - 26s 132us/step - loss: 0.2524 - acc: 0.9219 - val_loss: 0.3387 - val_acc: 0.9038
Epoch 19/20
200000/200000 [==============================] - 26s 132us/step - loss: 0.2477 - acc: 0.9232 - val_loss: 0.3428 - val_acc: 0.9056
Epoch 20/20
200000/200000 [==============================] - 27s 133us/step - loss: 0.2430 - acc: 0.9247 - val_loss: 0.3492 - val_acc: 0.9065
Test loss: 0.144602749768
Test accuracy: 0.9583

Popular posts from this blog

Accuracy for Logistic Regression model is always lower than using Convolutional neural network

Image
Here's the coding for a logistic regression model with 100k samples. The accuracy is only 86.7% compared than 96% for ConvNet. ROC Curve for logistic is 0.97 compared to 0.99 for ConvNet. from __future__ import print_function import numpy as np import matplotlib.pyplot as plt import os from sklearn.metrics import roc_curve, auc from sklearn.linear_model import LogisticRegression from six.moves import cPickle as pickle from sklearn.metrics import confusion_matrix def load_data(): test_filename = "notMNIST.pickle" if os.path.exists(test_filename): with open(test_filename, 'rb') as f: letter_dataset = pickle.load(f) return (letter_dataset) sample_size = 10000 print('Sample size: ', sample_size) lt_dt = load_data() x_train = lt_dt['train_dataset'][:sample_size] y_train = lt_dt['train_labels'][:sample_size] x_test = lt_dt['test_dataset'] y_test = lt_dt['test_labels'] regr = LogisticRegressi...
Read more

Using mini batch SGD Neural Network in Alphabet recognition!

Here we are using mini-batch stochastic gradient descent neural network method in Alphabet recognition with previous data set mentioned in  nonMNIST data set . First, we are going to build the neural network part. Here we have adjustable number of hidden neurons, layers, epochs, mini batch sizes that allow us to tune the accuracy later. #network part import random import numpy as np class Network(object): def __init__(self, sizes): #sizes: number of neurons (eg: (2,3,1)) #biases and weights are initialized randomly self.num_layers = len(sizes) self.sizes = sizes self.biases = [np.random.randn(y, 1) for y in sizes[1:]] self.weights = [np.random.randn(y, x) for x, y in zip(sizes[:-1], sizes[1:])] def feedforward(self, a): #return output of network f a is input for b, w in zip(self.biases, self.weights): a = sigmoid(np.dot(w, a) + b) return a ...
Read more

96.24% accuracy with higher epoch numbers for Convolutional Neural Network

Image
For Convolutional Neural Network , there are a lots of factors affect the model accuracy. Factors included the structure of Convolutional Neural Network , hyper parameters values, over-fitting and etc. Only correct or suitable network structure can produce high model accuracy. If wrong network structure is used, whatever you tune the hyper parameters, the accuracy will still worse. So, after we found the right network structure, only we can start to tune the hyper parameters. However, over-fitting may happened on your model accuracy. There are several methods to prevent that. One of the over-fitting prevention is by using the dropout function. Dropout give the model a way to find alternative way to train the model by removed some characteristic according to the ratio. Another way to prevent over-fitting is by increasing the number of samples. Besides, we can also simplify the network structure. Some complicated network structure sometimes can cause over-fitting too.  Fi...
Read more

Tower Defense (Endless TD Beta)

Image
Hi All, this is my third game written by myself. This game is built with Visual Studio 2005 in C#. ** If game cannot be opened, please run as administrator. Screenshot: Software Link:  https://drive.google.com/open?id=0B7_726VmtxyoUmk3aTdEZFBXT0k Software coding Link:  https://drive.google.com/open?id=0B7_726VmtxyoLWJpdm9pVzRLZ2M
Read more

Simple data curation practices using Numpy (Udacity Deep Learning Assignment 1)

First, download the compressed dataset "notMNIST_large.tar.gz" and "notMNIST_small.tar.gz" from  http://yaroslavvb.com/upload/notMNIST/ OR download from others site using full code provided below. url = 'https://commondatastorage.googleapis.com/books1000/' last_percent_reported = None data_root = os.getcwd() #path to save file def download_progress_hook ( count , blockSize , totalSize ): """A hook to report the progress of a download. This is mostly intended for users with slow internet connections. Reports every 5% change in download progress. """ global last_percent_reported percent = int ( count * blockSize * 100 / totalSize ) if last_percent_reported != percent : if percent % 5 == 0 : sys . stdout . write ( " %s%% " % percent ) sys . stdout . flush () else : sys . stdout . write ( "." ) sys . stdout . flush () ...
Read more

Accuracy 96% with Simple Deep NN using Keras Theano backend for nonMNist alphabet recognition

Image
Here's first try to build a simple simple deep neurons network by using Keras with Theano backend. By using the previous data pickles sample   nonMNist dataset we have saved, here's the completed coding. The summarized deep filter layers can be described as combination of common dense layer with 512 units, relu activation, dropout ratio of 0.2, softmax activation, and RMSprop optimizer. from __future__ import print_function import keras from keras.models import Sequential from keras.layers import Dense, Dropout from keras.optimizers import RMSprop from keras.utils import np_utils import os from six.moves import cPickle as pickle batch_size = 128 num_classes = 10 epochs = 20 def load_data(): test_filename = "notMNIST.pickle" if os.path.exists(test_filename): with open(test_filename, 'rb') as f: letter_dataset = pickle.load(f) return (letter_dataset) lt_dt = load_data() train_dataset = lt_dt['train_...
Read more

Python to read and record SQM-LE data, search saved data and find IP address..

Image
Unihedron Sky Quality Meter with lens and Ethernet (SQM-LE) is used to monitor sky brightness for light pollution analysis. Here's some codes used to read and record SQM-LE data, search saved data and find IP address. Google Drive: https://drive.google.com/open?id=1Dfc1mvKKRsWB12AAYpjOnvCekrGP85xh Below is the example of recorded format in txt files. 15-01-17-Time:22:40 r, 15.54m,0000000052Hz,0000008592c,0000000.019s, 030.9C 15-01-17-Time:22:41 r, 16.21m,0000000050Hz,0000015911c,0000000.035s, 031.5C 15-01-17-Time:22:42 r, 16.41m,0000000025Hz,0000019128c,0000000.042s, 031.9C 15-01-17-Time:22:43 r, 16.30m,0000000027Hz,0000017304c,0000000.038s, 032.2C 15-01-17-Time:22:44 r, 16.37m,0000000025Hz,0000018388c,0000000.040s, 032.5C 15-01-17-Time:22:45 r, 16.43m,0000000024Hz,0000019481c,0000000.042s, 032.8C  Sample picture for Unihedron SQM-LE
Read more

Alphabet recognition by using LogisticRegression model from sklearn.linear_model. (Udacity Assignment 1)

Image
Continued from previous assignment, we saved pickle data from some sample alphabet images. Now we are going to extract the data and train the data, so that we can recognize the alphabet. from __future__ import print_function import matplotlib.pyplot as plt import numpy as np import os import sys import tarfile import random import hashlib from IPython.display import display, Image from sklearn.linear_model import LogisticRegression from six.moves.urllib.request import urlretrieve from six.moves import cPickle as pickle test_filename = "notMNIST.pickle" def read_data(picklename,data_name): if os.path.exists(picklename): with open(picklename, 'rb') as f: letter_set = pickle.load(f) return letter_set[data_name] train_dataset = read_data(test_filename,'train_dataset') train_labels = read_data(test_filename,'train_labels') valid_dataset = read_data(test_filename,'valid_dataset') valid_labels = read_data(test_filename,'valid_labels...
Read more

First try to solve Polynomial equations by Tensorflow

First try to solve polynomial equations y=3x^2+2x-5 using Tensorflow Output with 100000 looptimes: A: [ 2.99184537] B: [ 2.0156281] C: [-5.00763798] D: [ 2.00114322] loss: 4.62821e-07 from __future__ import print_function import tensorflow as tf sess = tf.Session(); #Model Parameters A = tf.Variable([ 1. ], dtype =tf.float32) B = tf.Variable([ 1. ], dtype =tf.float32) C = tf.Variable([- 1. ], dtype =tf.float32) D = tf.Variable([ 1. ], dtype =tf.float32) #Model Input and Output x = tf.placeholder(tf.float32) y = tf.placeholder(tf.float32) unknown_model = A * x ** D + B * x + C #Loss loss = tf.reduce_sum(tf.square(unknown_model - y)) #Optimizer optimizer = tf.train.AdamOptimizer( 0.01 ) train = optimizer.minimize(loss) #Training Data x_train = [ 1 , 2 , 3 , 4 ] y_train = [ 0 , 11 , 28 , 51 ] #Training Loop init = tf.global_variables_initializer() sess.run(init) for i in range ( 100000 ): sess.run(train, {x:x_train, y:y_train}) #evaluate training accuracy curr_A, curr_B, c...
Read more

Endless Falling

Image
Hi all, please get this mini game called Endless Falling. ** If game cannot be opened, please run as administrator. Screenshot: Game Link:  https://drive.google.com/open?id=0B7_726VmtxyoZ3dTREE3c3Q4Tnc Software coding Link:  https://drive.google.com/open?id=0B7_726VmtxyodXlVdVdtY0I5R3c
Read more