\( \DeclareMathOperator*\argmax{argmax} \)

Due at 3pm on Monday, Apr. 3. Please submit to Blackboard and follow the general guidelines regarding homework assignments.

This assignment is based on the CIFAR-10 dataset. This consists of 32x32 RGB images in 10 classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, truck. The images are rather low resolution, but still generally recognizable.

We assume that you have set up a Jupyter notebook that is communicating with an EC2 instance, following our AWS instructions. (Or you can do the whole thing locally; no instructions provided but you can find them online elsewhere.) If you are using Jupyter to communicate with an EC2 instance, select New > Terminal if you want to interact with a shell. (You can also ssh to the instance using your private key, as explained in the other blog post.) Type git clone https://github.com/cos495/code.git. From the Jupyter dashboard, navigate to code and then open the CIFAR10 or CIFAR10_Python notebook.

Google supports a Python API for use of TensorFlow. The Julia API is similar, and is a community effort documented here.

  1. Understanding the architecture.

    • How many kernels are in the first convolutional layer? TensorFlow uses a single four-dimensional “filter” tensor for each convolutional layer: however for this and the following questions, please calculate the number of two-dimensional kernels, i.e., there is one kernel for every input channel - output channel pair.

    • How many kernels are in the second convolutional layer?

    • How many kernels are in the third convolutional layer?

    • After the third convolutional layer, how many feature maps are there? How large is each feature map?

    • How many fully connected layers are there, and how large are the weight matrices?

  2. Visualization. You can convert the first layer kernels into a Julia array using run(session, weights["wc1"]) or into a Numpy array using sess.run(weights['wc1']). Display the kernels as RGB images.

  3. Data augmentation. If you run for at least 2000 minibatches, you should see evidence of overfitting in the loss on the validation set. Write code that left-right reflects a random subset of images in the minibatch. Is overfitting reduced? Does the validation error improve?

  4. Architecture modifications. Submit code and learning curves for two modified architectures of your choice. For example, you could vary the number of

    • feature maps in a convolutional layer

    • convolutional layers

    • maxpooling layers

    • fully connected layers

  5. Open competition (extra credit). Can you beat the state-of-the-art or at least your classmates? Try any trick you’ve learned from class or papers, or invent your own. For example, you could try other architectures, dropout, or other data augmentation techniques such as random translations (horizontal and vertical) and color jitter. This must be your own work, i.e., you can look up both papers and code to learn more about different tricks, but you should implement everything yourself rather than cloning or copying directly from someone else’s implementation. You’re honor-bound to not train or do model selection based on the test set. Once you’ve selected your model based on the validation set, quantify its error on the test set and then submit. The three submissions with the best test error will receive modest prizes.

For the competition, please submit by doing the following:

  • Submit entries separately from your problem set by sending an email to cos495.automated@gmail.com from your @princeton email.

  • Use the subject “CIFAR10 $SCORE”, where $SCORE is the accuracy percent (higher is better). E.g. “CIFAR10 97.1”.

  • In the body of the email, write a very brief description of what you did. One sentence is fine (e.g. “Dropout 0.5 at the final layer plus fractional max pooling after all convolutions.”)

  • Upload the output files from saving a fully trained model to a folder in Google Drive. Set the sharing settings to “anyone with the link can view”, and add a link to the folder to the body of the email.

  • Upload to the same folder in Google Drive a notebook which can either reload and test the saved model, or train a new model from scratch.

  • Put a line at the top of the notebook, “RELOAD = True”, and write some code so that RELOAD determines whether the model is reloaded or trained. If we change RELOAD to True and run the notebook, the model should be reloaded and then tested (no training). If we change RELOAD to False and run the notebook, the model should be trained from scratch.

  • Please ensure that we can reload the model or train the model from scratch just by downloading the folder from Google Drive, setting “RELOAD” to the correct value, and running all the cells in the notebook.

  • For info on how to save and reload models, see the TensorFlow Saver documentation and/or look at how savers are used in some code online. It’s well worth understanding how to save / reload models.

  • If you want to see what’s possible and what tricks might help you improve your accuracy, check out the papers mentioned under CIFAR10 here. Some tricks would be hard to implement, but some simple ones could help a lot. It could also help with choosing hyperparameters.