Commit f7113aae authored by Alexander Fuchs's avatar Alexander Fuchs
Browse files

Merged changes from different Repo and implemented minimal example for MNIST

parent a60d2e96
# Birdcall Identification challenge
# Tensorflow 2 Trainer
The SPSC Intelligent Systems group git-project for the Cornell Birdcall Identification challenge on kaggle.
Link to challenge: https://www.kaggle.com/c/birdsong-recognition/overview
This repository implements a non-Keras trainer for Tensorflow 2 models
## Contents
......@@ -13,7 +12,7 @@ Link to challenge: https://www.kaggle.com/c/birdsong-recognition/overview
## Getting Started
These instructions should help you start working on the Cornell Birdcall Identification challenge.
These instructions should help you start working with this trainer.
### Prerequisites
......@@ -27,7 +26,7 @@ We need [Miniconda](https://docs.conda.io/en/latest/miniconda.html) for managing
Clone this repository
git clone git@git.spsc.tugraz.at:fuchs/kaggle_birdcall.git
git clone git@git.spsc.tugraz.at:fuchs/tensorflow2_trainer_template.git
Create a virtual environment from the included environment.yml
......@@ -41,27 +40,14 @@ Set the python path
export PYTHONPATH="/path/to/this/project/src"
### Data
A minimal dataset including 3 classes is located at:
/afs/spsc.tugraz.at/shared/user/fuchs/cornell_birdcall_recognition_mini/
### Training
For training an minimal example run src/scripts/mnist_main.py
For training mnist run src/scripts/mnist_main.py
python /path/to/this/project/src/scripts/birdsong_simple_main.py --data_dir path/to/data/ --model_dir /path/to/model
### Evaluation of pretrained models
TODO
python /path/to/this/project/src/scripts/birdsong_simple_main.py --model_dir /path/to/model
## Authors
- **Alexander Fuchs** - Created the initial version of this project.
## Acknowledgments
- Nguyen, Thi Kim Truc who provided code and ideas
<p><small>Template folder structure based on the <a target="_blank" href="https://drivendata.github.io/cookiecutter-data-science/">cookiecutter data science project template</a>. #cookiecutterdatascience</small><br>
<small>Readme based on <a target="_blank" href="https://github.com/PurpleBooth/a-good-readme-template">purple booths readme template</a>.</small></p>
name: tensorflow_2_2
channels:
- defaults
dependencies:
- _libgcc_mutex=0.1=main
- _tflow_select=2.1.0=gpu
- absl-py=0.9.0=py36_0
- alabaster=0.7.12=py36_0
- astunparse=1.6.3=py_0
- attrs=19.3.0=py_0
- babel=2.8.0=py_0
- backcall=0.2.0=py_0
- blas=1.0=mkl
- bleach=3.1.5=py_0
- blinker=1.4=py36_0
- brotlipy=0.7.0=py36h7b6447c_1000
- c-ares=1.15.0=h7b6447c_1001
- ca-certificates=2020.6.24=0
- cachetools=4.1.0=py_1
- certifi=2020.6.20=py36_0
- cffi=1.14.0=py36he30daa8_1
- chardet=3.0.4=py36_1003
- click=7.1.2=py_0
- cryptography=2.9.2=py36h1ba5d50_0
- cudatoolkit=10.1.243=h6bb024c_0
- cudnn=7.6.5=cuda10.1_0
- cupti=10.1.168=0
- cycler=0.10.0=py36_0
- cython=0.29.21=py36he6710b0_0
- dbus=1.13.16=hb2f20db_0
- decorator=4.4.2=py_0
- defusedxml=0.6.0=py_0
- docutils=0.16=py36_1
- entrypoints=0.3=py36_0
- expat=2.2.9=he6710b0_2
- fontconfig=2.13.0=h9420a91_0
- freetype=2.10.2=h5ab3b9f_0
- gast=0.3.3=py_0
- glib=2.65.0=h3eb4bd4_0
- google-auth=1.17.2=py_0
- google-auth-oauthlib=0.4.1=py_2
- google-pasta=0.2.0=py_0
- grpcio=1.27.2=py36hf8bcb03_0
- gst-plugins-base=1.14.0=hbbd80ab_1
- gstreamer=1.14.0=hb31296c_0
- h5py=2.10.0=py36hd6299e0_1
- hdf5=1.10.6=hb1b8bf9_0
- icu=58.2=he6710b0_3
- idna=2.10=py_0
- imagesize=1.2.0=py_0
- importlib-metadata=1.7.0=py36_0
- importlib_metadata=1.7.0=0
- intel-openmp=2020.1=217
- ipykernel=5.3.3=py36h5ca1d4c_0
- ipython=7.16.1=py36h5ca1d4c_0
- ipython_genutils=0.2.0=py36_0
- ipywidgets=7.5.1=py_0
- jedi=0.17.1=py36_0
- jinja2=2.11.2=py_0
- jpeg=9b=h024ee3a_2
- jsonschema=3.2.0=py36_0
- jupyter=1.0.0=py36_7
- jupyter_client=6.1.6=py_0
- jupyter_console=6.1.0=py_0
- jupyter_core=4.6.3=py36_0
- keras-preprocessing=1.1.0=py_1
- kiwisolver=1.2.0=py36hfd86e86_0
- ld_impl_linux-64=2.33.1=h53a641e_7
- libedit=3.1.20191231=h14c3975_1
- libffi=3.3=he6710b0_2
- libgcc-ng=9.1.0=hdf63c60_0
- libgfortran-ng=7.3.0=hdf63c60_0
- libgpuarray=0.7.6=h14c3975_0
- libpng=1.6.37=hbc83047_0
- libprotobuf=3.12.3=hd408876_0
- libsodium=1.0.18=h7b6447c_0
- libstdcxx-ng=9.1.0=hdf63c60_0
- libuuid=1.0.3=h1bed415_2
- libxcb=1.14=h7b6447c_0
- libxml2=2.9.10=he19cac6_1
- mako=1.1.3=py_0
- markdown=3.1.1=py36_0
- markupsafe=1.1.1=py36h7b6447c_0
- matplotlib=3.2.2=0
- matplotlib-base=3.2.2=py36hef1b27d_0
- mistune=0.8.4=py36h7b6447c_0
- mkl=2020.1=217
- mkl-service=2.3.0=py36he904b0f_0
- mkl_fft=1.1.0=py36h23d657b_0
- mkl_random=1.1.1=py36h0573a6f_0
- nbconvert=5.6.1=py36_0
- nbformat=5.0.7=py_0
- ncurses=6.2=he6710b0_1
- nose=1.3.7=py36_2
- notebook=6.0.3=py36_0
- numpy=1.18.5=py36ha1c710e_0
- numpy-base=1.18.5=py36hde5b4d6_0
- oauthlib=3.1.0=py_0
- openssl=1.1.1g=h7b6447c_0
- opt_einsum=3.1.0=py_0
- packaging=20.4=py_0
- pandoc=2.10=0
- pandocfilters=1.4.2=py36_1
- parso=0.7.0=py_0
- pcre=8.44=he6710b0_0
- pexpect=4.8.0=py36_0
- pickleshare=0.7.5=py36_0
- pip=20.1.1=py36_1
- prometheus_client=0.8.0=py_0
- prompt-toolkit=3.0.5=py_0
- prompt_toolkit=3.0.5=0
- protobuf=3.12.3=py36he6710b0_0
- ptyprocess=0.6.0=py36_0
- pyasn1=0.4.8=py_0
- pyasn1-modules=0.2.7=py_0
- pycparser=2.20=py_2
- pygments=2.6.1=py_0
- pygpu=0.7.6=py36heb32a55_0
- pyjwt=1.7.1=py36_0
- pyopenssl=19.1.0=py_1
- pyparsing=2.4.7=py_0
- pyqt=5.9.2=py36h05f1152_2
- pyrsistent=0.16.0=py36h7b6447c_0
- pysocks=1.7.1=py36_0
- python=3.6.10=h7579374_2
- python-dateutil=2.8.1=py_0
- pytz=2020.1=py_0
- pyzmq=19.0.1=py36he6710b0_1
- qt=5.9.7=h5867ecd_1
- qtconsole=4.7.5=py_0
- qtpy=1.9.0=py_0
- readline=8.0=h7b6447c_0
- requests=2.24.0=py_0
- requests-oauthlib=1.3.0=py_0
- rsa=4.0=py_0
- scipy=1.5.0=py36h0b6359f_0
- send2trash=1.5.0=py36_0
- setuptools=49.2.0=py36_0
- sip=4.19.8=py36hf484d3e_0
- six=1.15.0=py_0
- snowballstemmer=2.0.0=py_0
- sphinx=3.1.2=py_0
- sphinxcontrib-applehelp=1.0.2=py_0
- sphinxcontrib-devhelp=1.0.2=py_0
- sphinxcontrib-htmlhelp=1.0.3=py_0
- sphinxcontrib-jsmath=1.0.1=py_0
- sphinxcontrib-qthelp=1.0.3=py_0
- sphinxcontrib-serializinghtml=1.1.4=py_0
- sqlite=3.32.3=h62c20be_0
- tensorboard=2.2.1=pyh532a8cf_0
- tensorboard-plugin-wit=1.6.0=py_0
- tensorflow=2.2.0=gpu_py36hf933387_0
- tensorflow-base=2.2.0=gpu_py36h8a81be8_0
- tensorflow-estimator=2.2.0=pyh208ff02_0
- tensorflow-gpu=2.2.0=h0d30ee6_0
- termcolor=1.1.0=py36_1
- terminado=0.8.3=py36_0
- testpath=0.4.4=py_0
- tk=8.6.10=hbc83047_0
- tornado=6.0.4=py36h7b6447c_1
- tqdm=4.47.0=py_0
- traitlets=4.3.3=py36_0
- urllib3=1.25.9=py_0
- wcwidth=0.2.5=py_0
- webencodings=0.5.1=py36_1
- werkzeug=1.0.1=py_0
- wheel=0.34.2=py36_0
- widgetsnbextension=3.5.1=py36_0
- wrapt=1.12.1=py36h7b6447c_1
- xz=5.2.5=h7b6447c_0
- zeromq=4.3.2=he6710b0_2
- zipp=3.1.0=py_0
- zlib=1.2.11=h7b6447c_3
- pip:
- parameterized==0.7.4
from utils import trainer
from utils import data_loader
from utils import evaluation
from utils import summary_utils
from models.layers.dense_moe import DenseMoE
import tensorflow as tf
def basic_dmoe_cnn_mnist():
model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.MaxPooling2D((2, 2)))
model.add(tf.keras.layers.Conv2D(64, (3, 3), activation='relu'))
model.add(tf.keras.layers.Flatten())
model.add(DenseMoE(64, n_experts=10, expert_activation='relu', gating_activation='softmax'))
model.add(tf.keras.layers.Dense(10))
return model
from models.layers.dense_moe import DenseMoE
import tensorflow as tf
......
import tensorflow as tf
from models.layers.spectral_normalization import SpectralNormalization
class Discriminator(tf.keras.Model):
def __init__(self,
n_layers,
n_channels,
strides,
name = "discriminator",
kernel_regularizer = None,
kernel_initializer = tf.keras.initializers.he_normal()):
super(Discriminator, self).__init__()
self.n_layers = n_layers
self.strides = strides
self.n_channels = n_channels
self.model_name = name
self.kernel_regularizer = None
self.kernel_initializer = kernel_initializer
def build(self,input_shape):
self.model_layers = []
for i in range(self.n_layers):
self.model_layers.append(SpectralNormalization(tf.keras.layers.Conv2D(self.n_channels[i],
kernel_size=self.strides[i]+1,
strides=self.strides[i],
padding="same",
use_bias=False,
name=self.model_name+'_conv_' + str(i),
kernel_regularizer = self.kernel_regularizer,
kernel_initializer = self.kernel_initializer,
activation='relu')))
self.dense = SpectralNormalization(tf.keras.layers.Dense(1,
kernel_initializer = self.kernel_initializer,
use_bias = False,
name = self.model_name +"_dense",
activation = None))
def call(self,input,training=False):
x = tf.image.per_image_standardization(input)
for layer in self.model_layers:
x = layer(x)
x = self.dense(x)
return x
from models.layers.dense_moe import DenseMoE
import tensorflow as tf
def model_base(x, wd, name=''):
x = tf.keras.layers.Convolution2D(16, kernel_size=3, strides=1, data_format='channels_last',
kernel_initializer='glorot_uniform',
kernel_regularizer=tf.keras.regularizers.l2(wd))(x)
x = tf.keras.layers.Activation('relu')(x)
x = tf.keras.layers.BatchNormalization(axis=-1)(x)
# next layer has 32 convolution filters,
x = tf.keras.layers.Convolution2D(32, kernel_size=3, strides=2,
kernel_regularizer=tf.keras.regularizers.l2(wd))(x)
x = tf.keras.layers.Activation('relu')(x)
x = tf.keras.layers.BatchNormalization(axis=-1)(x)
x = tf.keras.layers.Convolution2D(32, kernel_size=3, strides=1,
kernel_regularizer=tf.keras.regularizers.l2(wd))(x)
x = tf.keras.layers.Activation('relu')(x)
x = tf.keras.layers.BatchNormalization(axis=-1)(x)
# next layer has 64 convolution filters
x = tf.keras.layers.Convolution2D(64, kernel_size=3, strides=2,
kernel_regularizer=tf.keras.regularizers.l2(wd))(x)
x = tf.keras.layers.Activation('relu')(x)
x = tf.keras.layers.BatchNormalization(axis=-1)(x)
x = tf.keras.layers.Convolution2D(64, kernel_size=3, strides=1,
kernel_regularizer=tf.keras.regularizers.l2(wd))(x)
x = tf.keras.layers.Activation('relu')(x)
x = tf.keras.layers.BatchNormalization(axis=-1)(x)
# Global average pooling
x = tf.keras.layers.GlobalAveragePooling2D(data_format='channels_last')(x)
return x
def ensemble_cnn_mnist(x,n_models,wd,n_classes):
x_tensor = tf.keras.layers.Input(x.shape[1:], name='Input')
outputs = []
for i_model in range(n_models):
output_i = model_base(x_tensor,wd)
outputs.append(output_i)
x = tf.keras.layers.Concatenate(axis=-1)(outputs)
# Softmax
x = DenseMoE(64, n_experts=10, expert_activation='relu', gating_activation='softmax')(x)
output = tf.keras.layers.Dense(n_classes, activation='softmax')(x)
model = tf.keras.Model(inputs=x_tensor, outputs=output)
return model
\ No newline at end of file
......@@ -20,7 +20,7 @@ class EvalFunctions(object):
def compute_loss(self, x, y, training=True):
"""Has to at least return a dict containing the total loss and a prediction dict e.g.{'total_loss':total_loss},{'predictions':predictions}"""
logits_classifier = self.classifier(x[0], training=training)
predictions = tf.nn.softmax(logits_classifier,axis=-1)
# Cross entropy losses
class_loss = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(
......@@ -35,5 +35,14 @@ class EvalFunctions(object):
weight_decay_loss = 0.0
total_loss = class_loss
total_loss += weight_decay_loss
return {'class_loss':class_loss,'weight_decay_loss':weight_decay_loss,'total_loss':total_loss}, {'predictions':predictions}
accuracy = self.accuracy(logits_classifier,y)
scalars = {'class_loss':class_loss,'weight_decay_loss':weight_decay_loss,'total_loss':total_loss,"accuracy":accuracy}
predictions = {'predictions':predictions,"logits":logits_classifier}
return scalars, predictions
def post_train_step(self,args):
return
import numpy as np
from copy import deepcopy
from models.layers.ops import *
import tensorflow as tf
class EvalFunctions(object):
"""This class implements specialized operation used in the training framework"""
def __init__(self,models):
self.generator = models[0]
self.discriminator = models[1]
@tf.function
def predict(self, x,training=True):
"""Returns a dict containing predictions e.g.{'predictions':predictions}"""
if len(x)>1:
states = x[1]
else:
states = None
logits,states = self.model(x[0],states=states)
return {'predictions':tf.nn.softmax(logits,axis=-1),'states':states}
@tf.function
def compute_loss(self,x,y,training=True,eval_training_steps=False):
"""Example GAN loss (untested)"""
batches = tf.shape(x[0])[0]
weight_decay_loss = 0
image = x[0]
z = tf.random.normal(shape=[batches, self.generator.latent_dim])
image_fake = self.generator(z)
real_logit = self.discriminator(image)
fake_logit = self.discriminator(image_fake)
d_loss = tf.reduce_mean(real_logit)-tf.reduce_mean(fake_logit)
g_loss = tf.reduce_mean(fake_logit)
total_loss = d_loss + g_loss
if len(self.generator.losses)>0:
weight_decay_loss += tf.add_n(self.generator.losses)
if len(self.discriminator.losses)>0:
weight_decay_loss += tf.add_n(self.discriminator.losses)
total_loss += weight_decay_loss
losses = {"d_loss":d_loss,
"g_loss":g_loss,
"total_loss":total_loss,
"weight_decay_loss":weight_decay_loss}
predictions = {"image_fake":image_fake}
return losses,predictions
def post_train_step(self,args):
return
import tensorflow as tf
class EvalFunctions(object):
"""This class implements specialized operation used in the training framework"""
def __init__(self,models):
self.classifier = models[0]
self.generator = models[1]
self.discriminator = models[2]
@tf.function
def predict(self, x,training=True):
"""Returns a dict containing predictions e.g.{'predictions':predictions}"""
logits_classifier = self.classifier(x[0], training=training)
return {'predictions':tf.nn.softmax(logits_classifier,axis=-1)}
@tf.function
def generate(self, x,training=True):
"""Returns a dict containing fake samples e.g.{'fake_features':fake_features}"""
fake_features = self.generator(x[1], training)
return {'fake_features':fake_features}
@tf.function
def discriminate(self, x,training=True):
"""Returns a dict containing scores e.g.{'scores':score}"""
scores = self.discriminator(x[0], training)
return {'scores':scores}
def accuracy(self,pred,y):
correct_predictions = tf.cast(tf.equal(tf.argmax(pred,axis=-1),
tf.argmax(y[0],axis=-1)),tf.float32)
return tf.reduce_mean(correct_predictions)
@tf.function
def compute_loss(self, x, y, training=True):
"""Has to at least return a dict containing the total loss and a prediction dict e.g.{'total_loss':total_loss},{'predictions':predictions}"""
logits_classifier = self.classifier(x[0], training=training)
fake_features = self.generator(x[1], training)
true = self.discriminator(x[0], training)
false = self.discriminator(fake_features, training)
# Cross entropy losses
class_loss = tf.reduce_mean(
tf.nn.softmax_cross_entropy_with_logits(
y[0],
logits_classifier,
axis=-1,
))
gen_loss = 0#tf.reduce_mean((fake_features - x[1]) ** 2)
# Wasserstein losses
gen_loss += tf.reduce_mean(false)
discr_loss = tf.reduce_mean(true) - tf.reduce_mean(false)
if len(self.classifier.losses) > 0:
weight_decay_loss = tf.add_n(self.classifier.losses)
else:
weight_decay_loss = 0.0
if len(self.generator.losses) > 0:
weight_decay_loss += tf.add_n(self.generator.losses)
predictions = tf.nn.softmax(logits_classifier, axis=-1)
total_loss = class_loss
total_loss += gen_loss
total_loss += discr_loss
total_loss += weight_decay_loss
return {'class_loss':class_loss, 'generator_loss':gen_loss, 'discriminator_loss':discr_loss, 'weight_decay_loss':weight_decay_loss,'total_loss':total_loss}, {'predictions':predictions, 'fake_features':fake_features}
import tensorflow as tf
class Generator(tf.keras.Model):
def __init__(self,
n_layers,
n_channels,
name = "generator",
kernel_regularizer = tf.keras.regularizers.l2(2e-4),
kernel_initializer = tf.keras.initializers.he_normal()):
super(Generator, self).__init__()
self.n_layers = n_layers
self.n_channels = n_channels
self.model_name = name
self.kernel_regularizer = None
self.kernel_initializer = kernel_initializer
def build(self,input_shape):
self.model_layers = []
#Compression layers
for i in range(self.n_layers):
self.model_layers.append((tf.keras.layers.Conv2D(self.n_channels[i],
kernel_size=(3, 3),
strides=(2,2),
padding="same",
use_bias=False,
name=self.model_name+'_conv_' + str(i),
kernel_regularizer = self.kernel_regularizer,
kernel_initializer = self.kernel_initializer,
activation=None),0))
self.model_layers.append((tf.keras.layers.BatchNormalization(axis=-1),1))
self.model_layers.append((tf.keras.layers.Activation("relu"),0))
#Decompression layers
for i in range(self.n_layers+1):
if i == self.n_layers:
self.model_layers.append((tf.keras.layers.Conv2DTranspose(input_shape[-1],
kernel_size=(1, 1),
strides=(1, 1),
padding="same",
use_bias=False,
name=self.model_name + '_conv_' + str(i),
kernel_regularizer=None,
kernel_initializer=self.kernel_initializer,
activation=None), 0))
else:
self.model_layers.append((tf.keras.layers.Conv2DTranspose(self.n_channels[self.n_layers-1-i],
kernel_size=(3, 3),
strides=(2, 2),
padding="same",
use_bias=False,
name=self.model_name + '_conv_tp_' + str(i),
kernel_regularizer=self.kernel_regularizer,
kernel_initializer=self.kernel_initializer,
activation=None), 0))
self.model_layers.append((tf.keras.layers.BatchNormalization(axis=-1), 1))
self.model_layers.append((tf.keras.layers.Activation("relu"), 0))
def call(self,input,training=False):
x = input
for layer in self.model_layers:
if layer[1]==0:
x = layer[0](x)
elif layer[1]==1:
x = layer[0](x,training)
x = x[:,:input.shape[1],:input.shape[2],:]
return x#+input