SIT744: Deep Learning with Progressive Levels of Challenges Assignment

Assignment Task

Assignment objective

This assignment is for you to demonstrate the knowledge in deep learning that you have acquired from the lectures and practical lab materials. Most tasks in this assignment are straightforward applications of the practical materials in weeks 1-5. Going through these materials before attempting this assignment is highly recommended.

This assignment consists of four group of tasks with progressive level of challenges.

Group 1 (P Tasks)

Group 2 (C Tasks)

Group 3 (D Tasks)

Group 4 (HD Tasks)

Group 1 (P Tasks) Construct a deep forward neural network

With this group of tasks, you are going to build a neural network for the image classification task. You will train the model on the Fashion MNIST dataset.

Task 1.1 Understanding the data

1. Describe the target classes for the classification task. Display 10 images from each target class.
2. How many training images and how many test images are in the dataset?
3. Describe the data type and shape of the images. What preprocessing steps are required? Why?

Task 1.2 Setting up a model for training

Construct a deep feedforward neural network. In other words, you can use only fully connected (dense) layers. You need to decide and report the following configurations:

• Output layer:

1. How many output nodes?
2. Which activation function?

• Hidden layers:

1. How many hidden layers?
2. How many nodes in each layer?

• Which activation function for each layer?
• Input layer

1. What is the input size?
2. Do you need to reshape the input? Why?

Justify your model design decisions.

Plot the model structure using keras.utils.plot_model or similar tools.

Task 1.3 Fitting the model

1.Decide and report the following settings:

• The loss function
• The metrics for model evaluation (which may be different from the loss function)

2.Explain their roles in model fitting.

• Decide the optimiser that you will use. Also report the following settings:

1. The training batch size
2. The number of training epochs
3. The learning rate. If you used momentum or a learning rate schedule, please report the configuration as well.

Justify your decisions.

Now fit the model. Show how the training loss changes. How did you decide when to stop training?

Group 2 (C Tasks) Analyse the model

Task 2.1 Model size

• What is the number of trainable parameters in the model? Explain how the total number can be estimated from the model configurations.
• How much memory is used for training and for inference, respectively?

Task 2.2 Visualise the parameter values

Think about what initialisation method have you chosen for training the model? If you did not specify the initialisation method, find out what is the default one.

Reinitialise the model parameters. Choose a layer and visualise its initial weights. (Hint: You may use a heat map to visualise a matrix.)

After fitting the model, visualise the model weights again. How did the weights change? Why?

Group 3 (D Tasks) Use Tensor Flow tools

Task 3.1 Check the training using TensorBoard

Use TensorBoard to visualise the training process. Show screenshots of your TensorBoard output.

Do you see overfitting or under fitting? Why? If you see overfitting, at which epoch did it happen?

Task 3.2 Apply regularisation

Improve the training process by applying regularisation. Below are some options:

1. Dropout
2. Batch normalisation

Compare the effect of different regularisation techniques to the model training. You may also try other techniques for improving training such as learning rate scheduling

Group 4 (HD Tasks) Research on deep learning methods

As a deep learning practitioner, you need to keep yourself updated on the latest models. Therefore it is important that you are able to understand research papers in key deep learning conferences. In this task, you will analyse a research paper from the Tenth International Conference on Learning Representations (ICLR 2022), following the steps below:

1. Browse the list of papers published in ICLR 2022. Select one that interests you most. Why do you choose that paper?
2. What problem does the paper address? How is it related to what you learn in SIT744 so far?
3. Are there any existing methods for the problem? Why aren't they good enough?
4. What contributions are made by the authors? For example, have they proposed a new method or have they raised more questions?
5. How do the authors validate their proposed method or hypothesis?
6. Overall, what connections do you discover between the paper and what you have learnt in SIT744.

In addition to short answers to the above questions, submit a short (less than 5 minutes) video presentation for your analysis and main conclusions. You need to show your face in the video.