# Essentials

This “PyTorch essentials” page has notes from most of the other sections of the Deep Learning with PyTorch section of the notes page. The other pages are modified versions of scripts provided by Udacity and can be verbose. This page strips out the commentary from those pages leaving just the essential information.

#### Background

PyTorch is an open source library for building deep learning models. Its been developed by the Facebook AI Research team. It was first released in early 2017 and has made a big impact on the deep learning community.

The following topics are covered below and in the linked pages:

**Tensors**, the main data structure of PyTorch. The notes will cover how to create tensors, use them to do simple operations, and how tensors and NumPy interact.**Autograd**, a module PyTorch uses to calculate gradients for training neural networks. It does all the work of backpropagation.- Network
**Validation**, a process to test that the network is able to generalize to new data. **Transfer Learning**, a way to use pre-trained networks to improve the performance of a classifier on more complex images.**GPU accelerated network computations**, specifically, PyTorch uses a library called CUDA.

On Linux, PyTorch and related libraries can be installed with Anaconda:

`conda install pytorch torchvision cudatoolkit=10.2 -c pytorch`

## 1. Tensors

#### Tensors

The **inputs** to a neural network and the **weights** the network contains can be conceptualized as __vectors__. Then, the output of a vector is the __dot product__ of those vectors, plus the bias term.

**Tensors** are a generalization of **vectors** and **matrices**. Equivalently, vectors and matrices are specific types of tensor. Vectors are generally 1D, for example length 6. A matrix is generally 2D, for example, 6 by 4 or [6,4]. A 3D tensor (for example, 6 by 4 by 3 or [6,4,3]) could be used to model RGB color images.

**Tensors** are the base data structure used in PyTorch and other neural network frameworks. Tensorflow, for example, is named after tensors.

#### PyTorch Basics

- The
`sigmoid`

activation function squeezes its input to a value between 0 and 1, and is useful for calculating probabilities. `torch.randn((1, 5))`

returns a vector of random, normal values with the specified shape.`torch.randn_like(features)`

returns a tensor of random, normal values with the shape of the tensor provided as the input.

In order to calculate the output of a simple neural network, we could use a line like the following:

`y = activation(torch.sum(features * weights) + bias)`

`y = activation((features * weights).sum() + bias)`

The lines above used **element-wise multiplication** followed by taking the sum of those values. An alternative approach would be to use **matrix multiplication** instead. In general, this is the preferred way to perform multiplication because it is more efficient. The linear algebra that is being performed has been accelerated using modern libraries such as **CUDA** that run on GPUs.

There are two methods key methods for **matrix multiplication**:

`torch.mm()`

- the preferred method, because it strictly enforces matrix size rules. Recall that in matrix multiplication the first matrix must have a number of columns that’s equal to the number of rows in the second matrix. Remember that`tensor.shape`

returns the number of rows and columns a given tensor contains.`torch.matmul()`

- more complex. Supports*broadcasting*of matrix sizes, so it will return a result even if the input matrices are not correctly sized according to the rules of matrix multiplication.

In order to calculate the output of a multi-layer network using weights, `W1`

and `W2`

, and biases, `B1`

and `B2`

. Note that the output of the hidden layer is stored as `h`

and I assume the availability of `features`

tensor as input.

```
h = activation(torch.mm(features, W1) + B1)
output = activation(torch.mm(h, W2) + B2)
```

There are three key methods for **reshaping tensors**:

`tensor.reshape(a, b)`

- sometimes returns a clone of the original tensor, which means copying the original tensor to another part of memory.`tensor.resize_(a, b)`

- returns the same tensor (not a clone) with a different shape. The underscore (_) denotes that the method is performed**in-place**. If the new shape results in fewer or more elements than the original, then some elements will be removed or added to the tensor. If they are added, the new elements will be uninitialized in memory.`tensor.view(a, b)`

- returns a new tensor with the same data as the input tensor. This is the**preferred method**.

Here’s an example of calculating the output of a network that uses the `view`

method:

`y = activation(torch.mm(features, weights.view(5,1)) + bias)`

PyTorch has a great feature for converting between Numpy arrays and Torch tensors. To create a tensor from a Numpy array, use `torch.from_numpy()`

. To convert a tensor to a Numpy array, use `.numpy()`

.

## 2. Neural Network Architecture

#### Torchvision and MNIST

- MNIST is a famous dataset that has greyscale handwritten digits, each of which is 28x28 pixels.
- MNIST is one of many datasets provided as part of the
`torchvision`

package. This package sits alongside PyTorch and provides utilities, datasets, and models for doing computer vision problems.

`from torchvision import datasets`

The following downloads and loads training data for MNIST.

```
trainset = datasets.MNIST('MNIST_data/', download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
```

The images and labels can be extracted by converting `trainloader`

to an iterator, as follows. Note that the batch_size in the previous initialization has been set to 64. This is the number of images we get in one iteration from the dataloader. This means that each image is a tensor with size `(64, 1, 28, 28)`

: 64 images per batch, 1 color channel, 28x28 pixel images.

```
for image, label in trainloader:
# Do things with images, labels
```

An important application of the `view`

method is shown below. This converts a 28x28 pixel image to a 784-element vector. Note that `images.shape[0]`

returns the number of batches, and the `-1`

means that `view`

chooses on its own what the second dimension needs to be. So, this is a quick way of “flattening” a tensor.

`inputs = images.view(images.shape[0], -1)`

#### Softmax

We want the output of our network to be a probability distribution that the image belongs to any one of our classes. For this, we use the `softmax`

function, which converts the output of the network to a value in the range (0,1). It also normalizes all the values so the sum of all the probabilities it returns is one.

The function to return the softmax is defined below.

```
def softmax(x):
return torch.exp(x)/torch.sum(torch.exp(x, dim=1).view(-1,1))
```

#### nn

PyTorch has a module, `nn`

, that has built in classes, methods, and functions to build large neural networks in a very efficient way. An example is shown below.

```
from torch import nn
class Network(nn.Module):
def __init__(self):
super().__init__()
# Inputs to hidden layer linear transformation
self.hidden = nn.Linear(784, 256)
# Output layer, 10 units - one for each digit
self.output = nn.Linear(256, 10)
# Define sigmoid activation and softmax output
self.sigmoid = nn.Sigmoid()
self.softmax = nn.Softmax(dim=1)
def forward(self, x):
# Pass the input tensor through each of our operations
x = self.hidden(x)
x = self.sigmoid(x)
x = self.output(x)
x = self.softmax(x)
return x
```

An alternative way of defining the network is to use the `torch.nn.functional`

module, commonly imported as `F`

as shown below.

```
import torch.nn.functional as F
class Network(nn.Module):
def __init__(self):
super().__init__()
# Inputs to hidden layer linear transformation
self.hidden = nn.Linear(784, 256)
# Output layer, 10 units - one for each digit
self.output = nn.Linear(256, 10)
def forward(self, x):
# Hidden layer with sigmoid activation
x = F.sigmoid(self.hidden(x))
# Output layer with softmax activation
x = F.softmax(self.output(x), dim=1)
return x
```

In general any function can be used as an activation function. The only requirement is that for a network to approximate a non-linear function, the activation functions my be non-linear. A few examples of common activation functions are `Sigmoid`

, `TanH`

(hyperbolic tangent), and `ReLU`

(rectified linear unit).

In practice, the ** ReLU function is used almost exclusively** as the activation function for hidden layers.

`ReLU`

is the simplest nonlinear function that can be used and networks tend to train much more quickly when using it compared to sigmoid and hyperbolic tangent.## 3. Training Neural Networks

Training a neural network requires a **loss function**, sometimes also referred to as a “cost.” The loss function is a measure of how far the network’s actual prediction is from the correct prediction. An example of a loss function is the **mean square error**, which is commonly used in regression.

The network’s weights are iteratively adjusted to reduce the network’s error via a method called **gradient descent**. The gradient is the slope of the loss function with respect to our parameters. We use **backpropagation** to accomplish this in mulitlayered neural networks. Backpropagation is an application of the chain rule from calculus that enables us to make adjustments throughout the entire network. The amount of change during each update step is controlled by a **learning rate**.

In practice, the loss function used when doing classification is usually the cross-entropy loss: `nn.CrossEntropyLoss`

. It is defined via the **criterion** parameter.

```
from torch import nn
# Build a feed-forward network
model = nn.Sequential(nn.Linear(784, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10))
# Define the loss
criterion = nn.CrossEntropyLoss()
# Get our data
images, labels = next(iter(trainloader))
# Flatten images
images = images.view(images.shape[0], -1)
# Forward pass, get our logits
logits = model(images)
# Calculate the loss with the logits and the labels
loss = criterion(logits, labels)
```

In the instructor’s experience, it is more convenient to build the model with a log-softmax output using `nn.LogSoftmax`

or `F.log_softmax`

. The actual probabilities can be calculated by taking the exponential `torch.exp(output)`

. With a log-softmax output, use the negative log likelihood loss, `nn.NLLLoss`

.

An example of this setup is shown below. The parameter passed in `nn.LogSoftmax(dim=1)`

means to calculate across each of our examples, not across each individual feature in the batches.

```
#### Build a feed-forward network using **LogSoftmax**
model = nn.Sequential(nn.Linear(784, 128),
nn.ReLU(),
nn.Linear(128, 64),
nn.ReLU(),
nn.Linear(64, 10),
nn.LogSoftmax(dim=1))
#### Define the loss
criterion = nn.NLLLoss()
# Get our data
images, labels = next(iter(trainloader))
# Flatten images
images = images.view(images.shape[0], -1)
# Forward pass, get our logits
logps = model(images)
# Calculate the loss with the logits and the labels
loss = criterion(logps, labels)
```

#### Autograd

PyTorch has a model called `autograd`

that is used for automatically calculating the gradients of tensors. To use autograd, pass the `requires_grad=True`

parameter during tensor initialization. This tells PyTorch to keep track of operations on a tensor so that the the gradient can be efficiently calculated.

`x = torch.zeros(1, requires_grad=True)`

Gradients can also be turned off using the `torch.no_grad()`

context:

```
with torch.no_grad():
# Something with gradients turned off
```

The loss is calculated when we call the backward pass with the following command. Once calculated, we can use these gradients in gradient descent to train the network.

`loss.backward()`

The next necessary piece before training is to see how to use those gradients to actually update our weights. For that, we use optimizers. These come from PyTorch’s `optim`

package.

```
from torch import optim
# Optimizers require the parameters to optimize and a learning rate
optimizer = optim.SGD(model.parameters(), lr=0.01)
```

Note that PyTorch, by default, accumulates gradients. This means if you do multiple passes forward and backward, it will keep summing up those gradients. They must be cleared with `optimizer.zero_grad()`

The general process for a single learning step with PyTorch:

- Make a forward pass through the network
- Use the network output to calculate the loss
- Perform a backward pass through the network with
`loss.backward()`

to calculate the gradients - Take a step with the optimizer to update the weights

The code itself follows.

```
# Clear the gradients, do this because gradients are accumulated
optimizer.zero_grad()
# Forward pass, then backward pass, then update weights
output = model(images)
loss = criterion(output, labels)
loss.backward()
# Take an update step
optimizer.step()
```

Training the network entails looping through the training data, pulling a new batch, and then performing the steps outlined above. Each pass through the dataset is referred to as an `epoch`

. Training that is working correctly will have a steadily decreasing running loss.

```
epochs = 5
for e in range(epochs):
running_loss = 0
for images, labels in trainloader:
# Flatten MNIST images into a 784 long vector
images = images.view(images.shape[0], -1)
optimizer.zero_grad()
output = model(images)
loss = criterion(output, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
else:
print('Training Loss: {1.3f}'.format(running_loss/len(trainloader)))
```

## 4. Classifying Fashion MNIST

The MNIST dataset of handwritten digits is a fairly trivial dataset these days. It is easy to get really high accuracy on it with a neural network. A different dataset that is more complex is called the Fashion-MNIST dataset. It contains 28x28 greyscale images of clothes, so it is a drop in replacement for MNIST.

The dataset is loaded through torchvision:

```
import torch
from torchvision import datasets, transforms
import helper
# Define a transform to normalize the data
transform = transforms.Compose([transforms.ToTensor(),
transforms.Normalize((0.5,), (0.5,))])
# Download and load the training data
trainset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/',
download=True, train=True, transform=transform)
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64, shuffle=True)
# Download and load the test data
testset = datasets.FashionMNIST('~/.pytorch/F_MNIST_data/',
download=True, train=False, transform=transform)
testloader = torch.utils.data.DataLoader(testset, batch_size=64, shuffle=True)
```

A possible class architecture is shown below: three hidden layers and one output layer. Note that the input tensor is flattened in the forward pass, so it is not necessary to flatten it in the forward pass itself.

```
from torch import nn, optim
import torch.nn.functional as F
class Classifier(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.fc2 = nn.Linear(256, 128)
self.fc3 = nn.Linear(128, 64)
self.fc4 = nn.Linear(64, 10)
def forward(self, x):
# make sure input tensor is flattened
x = x.view(x.shape[0], -1)
x = F.relu(self.fc1(x))
x = F.relu(self.fc2(x))
x = F.relu(self.fc3(x))
x = F.log_softmax(self.fc4(x), dim=1)
return x
```

Now, create the network and train it.

- The criterion is the
`NLLLoss`

function since the output is`log_softmax`

. - The
`Adam`

optimizer is used, which is like stochastic gradient descent except:- it has some nice properties like momentum which speeds up the actual fitting process and
- it adjust the learning rate for each of the individual parameters in your model.

Note that if you pass the images into `model`

as shown below, it will run the `forward`

method.

```
model = Classifier()
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)
epochs = 5
for e in range(epochs):
running_loss = 0
for images, labels in trainloader:
log_ps = model(images)
loss = criterion(log_ps, labels)
optimizer.zero_grad()
loss.backward()
optimizer.step()
running_loss += loss.item()
else:
print("Training loss: {1.3f}".format(running_loss/len(trainloader))
```

## 5. Inference and Validation

### Inference

**Inference** is a term used to describe using a model to make predictions about new data. Neural Networks, like most machine learning algorithms, have a tendency to **overfit** the data they have been trained on. This means the network finds correlations and patterns that are in the training set but not in the overall dataset.

To test for overfitting, we measure the performance of the network on data that isn’t in the dataset. This data is called the validation set or **test set**.

- To get the
**training**set, pass the parameter`train=True`

when downloading the dataset from torchvision. - To get the
**test**set, pass the parameter`train=False`

when downloading the dataset from torchvision.

The actual definition of performance on the validation set is up to the developer. Examples include:

**Accuracy**: percent predicted correctly.**Precision****Recall****Top-5 Error Rate**

To get the predictions, we use the model to perform a forward pass on the test set. Then, convert to probabilities using `torch.exp`

and find the most likely class using `ps.topk(1)`

. The prediction is whichever class has the highest probability. This one highest class is the value the network is predicting, and it is contained in `top_class`

in the snippet below.

```
images, labels = next(iter(testloader))
# Flatten MNIST images into a 784 long vector
images = images.view(images.shape[0], -1)
# Get the class probabilities
ps = torch.exp(model(images))
# Find the most likely class
top_p, top_class = ps.topk(1, dim=1)
```

### Validation

Now, we can compare this `top_class`

against the true `labels`

using an equality. The `view`

method below is used to ensure that the `labels`

have the same shape as the `top_class`

.

```
equals = top_class == labels.view(*top_class.shape)
accuracy = torch.mean(equals.type(torch.FloatTensor))
```

`equals`

is now a 1D byte tensor containing a `1`

for correct predictions and `0`

for incorrect predictions. The accuracy, then, is just the `mean`

of the values in `equals`

. The `.type(torch.FloatTensor)`

converts the byte tensor to a float tensor.

The following is the actual validation loop. In the validation pass, we are not updating the network parameters, so we can speed up the code by turning off gradients with `torch.no_grad()`

.

```
epochs, steps = 30, 0
for e in range(epochs):
running_loss = 0
for images, labels in trainloader:
optimizer.zero_grad()
log_ps = model(images)
loss = criterion(log_ps, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
else:
test_loss, accuracy = 0, 0
# Turn off gradients for validation, saves memory and computations
with torch.no_grad():
for images, labels in testloader:
log_ps = model(images)
test_loss += criterion(log_ps, labels)
ps = torch.exp(log_ps)
top_p, top_class = ps.topk(1, dim=1)
equals = top_class == labels.view(*top_class.shape)
accuracy += torch.mean(equals.type(torch.FloatTensor))
print("\t{:5}\t{:10.3}\t{:9.3}\t{:8.3}".format(epoch+1,
running_loss/len(trainloader),
test_loss/len(testloader),
accuracy/len(testloader)))
```

### Regularization

We also use **regularization techniques** such as **dropout** to prevent overfitting in the first place. Dropout involves randomly dropping input units between the layers. This forces the network to share information between the weights and spread the learning more evenly throughout the network. The effect of this is to help the network to generalize to new data. The following snippet shows how to add dropout to a model.

```
class Classifier(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 256)
self.fc2 = nn.Linear(256, 128)
self.fc3 = nn.Linear(128, 64)
self.fc4 = nn.Linear(64, 10)
# Dropout module with 0.2 drop probability
self.dropout = nn.Dropout(p=0.2)
def forward(self, x):
# make sure input tensor is flattened
x = x.view(x.shape[0], -1)
# Now with dropout
x = self.dropout(F.relu(self.fc1(x)))
x = self.dropout(F.relu(self.fc2(x)))
x = self.dropout(F.relu(self.fc3(x)))
# output so no dropout here
x = F.log_softmax(self.fc4(x), dim=1)
return x
```

Note that when doing making predictions with the network (“doing inference”), we need to have all of the nodes available, so we need to turn off dropout. This is accomplished with `model.eval()`

:

```
# turn off gradients
with torch.no_grad():
# set model to evaluation mode
model.eval()
for images, labels in testloader:
# validation
# set model back to train mode
model.train()
```

The validation code looks similar except that the `model.eval()`

and `model.train()`

lines are added.

```
epochs, steps = 30, 0
for e in range(epochs):
running_loss = 0
for images, labels in trainloader:
optimizer.zero_grad()
log_ps = model(images)
loss = criterion(log_ps, labels)
loss.backward()
optimizer.step()
running_loss += loss.item()
else:
test_loss, accuracy = 0, 0
# Turn off gradients for validation, saves memory and computations
with torch.no_grad():
model.eval()
for images, labels in testloader:
log_ps = model(images)
test_loss += criterion(log_ps, labels)
ps = torch.exp(log_ps)
top_p, top_class = ps.topk(1, dim=1)
equals = top_class == labels.view(*top_class.shape)
accuracy += torch.mean(equals.type(torch.FloatTensor))
model.train()
print("\t{:5}\t{:10.3}\t{:9.3}\t{:8.3}".format(epoch+1,
running_loss/len(trainloader),
test_loss/len(testloader),
accuracy/len(testloader)))
```

## 6. Saving and Loading Models

Saving and loading models is useful in situations where you want to train a model and later come back and use it to make predictions or continue training it later.

The way we save a model is by saving the `state_dict`

. This is a dictionary that contains all of the parameters for the model (weights, bias tensors, etc). In the example below, the state dict keys are printed out.

```
import torch
model = Network(784, 10, [512, 256, 128])
criterion = nn.NLLLoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)
# Assume training completed
print(model)
for key in list(model.state_dict().keys()):
print(key)
```

```
Network(
(hidden_layers): ModuleList(
(0): Linear(in_features=784, out_features=512, bias=True)
(1): Linear(in_features=512, out_features=256, bias=True)
(2): Linear(in_features=256, out_features=128, bias=True)
)
(output): Linear(in_features=128, out_features=10, bias=True)
(dropout): Dropout(p=0.5, inplace=False)
)
hidden_layers.0.weight
hidden_layers.0.bias
hidden_layers.1.weight
hidden_layers.1.bias
hidden_layers.2.weight
hidden_layers.2.bias
output.weight
output.bias
```

The simplest way to save the model is to save the state dict with `torch.save`

. For example, we can save it to a file `checkpoint.pth`

. `.pth`

is the typical extension for PyTorch checkpoint files.

`torch.save(model.state_dict(), 'checkpoint.pth')`

Load the state dict with `torch.load`

. Load the model with `model.load_state_dict`

:

```
state_dict = torch.load('checkpoint.pth')
model.load_state_dict(state_dict)
```

The model is now ready to be used.

Note that in order to use `model.load_state_dict`

, the model must be set up with the exact parameter setup contained in `checkpoint.pth`

. We need to rebuild the model exactly as it was when it was initially trained. A workaround is to save information about the model architecture in the checkpoint along with the state dict. The following snippets show an example of how to accomplish this.

```
checkpoint = {'input_size': 784,
'output_size': 10,
'hidden_layers': [each.out_features for each in model.hidden_layers],
'state_dict': model.state_dict()}
torch.save(checkpoint, 'saving-models/checkpoint.pth')
```

Note that this `load_checkpoint`

function is going to be based on the architecture of the model that you write. So, the function below will not work for every model that we create. This function may need to be customized.

```
def load_checkpoint(filepath):
checkpoint = torch.load(filepath)
model = Network(checkpoint['input_size'],
checkpoint['output_size'],
checkpoint['hidden_layers'])
model.load_state_dict(checkpoint['state_dict'])
return model
```

`model = load_checkpoint('saving-models/checkpoint.pth')`

## 7. Loading Image Data

The techniques in this section are useful for working with image data that comes from the real world, unlike the carefully preprocessed MNIST and fashion-MNIST datasets. A dataset of real-world cat and dog photos from Kaggle will be used.

The `datasets.ImageFolder`

method is the best method to use. It takes a path to a directory of photos that it expects to be organized with each class (in this case, `cat`

and `dog`

) having its own directory, as shown below. it also takes a list of transforms that are applied to the data.

```
root/dog/xxx.png
root/dog/xxy.png
root/dog/xxz.png
root/cat/123.png
root/cat/nsdf3.png
root/cat/asd932_.png
```

The transforms must be combined into a pipeline of transforms with `transforms.Compose()`

. There are many transforms available. Read through the documentation.

**Resize**: converts image to a square.**CenterCrop**: trims the outside of the image.**ToTensor**: converts the image to a tensor.

Once the images are opened and transformed, they can be passed to dataloader. Setting `shuffle`

to True is the recommended practice. Note that the dataloader is a generator, so to get data out of it you need to either:

- loop through it with a
`for`

loop or - convert it to an iterator:
`next(iter(dataloader))`

```
# Looping through it, get a batch on each loop
for images, labels in dataloader:
pass
# Get one batch
images, labels = next(iter(dataloader))
```

A common strategy is to use also use data augmentation on the **training** set. **Data Augmentation** a means of introducing randomness into the data itself. This helps the network generalize because its seeing the same images but in different locations, sizes, orientations, etc. This ultimately leads to better accuracy on the validation tests.

Note that data augmentation should not be applied to the **test** set. The test set should just be **resized** and **center cropped**. This is so that the validation images will be similar to the images that are expected when the model is in production.

**RandomRotation**: randomly rotates the image up to X degrees.**RandomResizedCrop**: randomly resizes and then crops from the center X pixels square.**RandomHorizontalFlip**: randomly flips the image horizontally.

```
data_dir = 'path/to/data/Cat_Dog_data'
train_transforms = transforms.Compose([transforms.RandomRotation(30),
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize([0.5, 0.5, 0.5],
[0.5, 0.5, 0.5])])
test_transforms = transforms.Compose([transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.5, 0.5, 0.5],
[0.5, 0.5, 0.5])])
train_data = datasets.ImageFolder(data_dir + '/train',
transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test',
transform=test_transforms)
trainloader = torch.utils.data.DataLoader(train_data,
batch_size=32)
testloader = torch.utils.data.DataLoader(test_data,
batch_size=32)
```

## 8. Transfer Learning

Transfer learning involves adapting a pre-trained neural network to a new, different data set.

Udacity provides a set of heuristics to help determine the best way to approach transfer learning based on the **size** and how **similar** it is to the original.

The models that will be used for transfer learning in this section are trained on ImageNet. The models themselves are available from torchvision. ImageNet is a massive dataset with over 1 million labeled images in 1000 categories.

The networks trained on ImageNet are deep neural networks that have an architecture that includes convolutional layers. Convolutional networks exploit patterns and regularities in images.

The list of available models are linked here and usually have the number of layers they contain included as part of their name. Deeper models tend to be both more accurate and slower to both predict and to train, so the tradeoffs between accuracy and speed must be carefully considered. These models’ depth also means they were very well as feature detectors on models they were not trained on.

To include these models, use the following import.

`from torchvision import models`

Most of these pretrained models require the input to be 224x224 images. Additionally, we need to match the color normalization used when the models were trained. Each color channel was normalized separately, the means are `[0.485, 0.456, 0.406]`

and the standard deviations are `[0.229, 0.224, 0.225]`

.

```
data_dir = 'path/to/data/Cat_Dog_data'
train_transforms = transforms.Compose([transforms.RandomRotation(30),
transforms.RandomResizedCrop(224),
transforms.RandomHorizontalFlip(),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],
[0.229, 0.224, 0.225])])
test_transforms = transforms.Compose([transforms.Resize(255),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406],
[0.229, 0.224, 0.225])])
# Pass transforms in here, then run the next cell to see how the transforms look
train_data = datasets.ImageFolder(data_dir + '/train', transform=train_transforms)
test_data = datasets.ImageFolder(data_dir + '/test', transform=test_transforms)
trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)
testloader = torch.utils.data.DataLoader(test_data, batch_size=64)
```

For this example, DensNet121 will be used:

`model = models.densenet121(pretrained=True)`

The model is made from two main parts: the `features`

and the `classifier`

. The `features`

will be frozen and not touched, and the `classifier`

will be replaced.

```
# Freeze parameters so we don't backprop through them
for param in model.parameters():
param.requires_grad = False
from collections import OrderedDict
classifier = nn.Sequential(OrderedDict([
('fc1', nn.Linear(1024, 500)),
('relu', nn.ReLU()),
('fc2', nn.Linear(500, 2)),
('output', nn.LogSoftmax(dim=1))
]))
model.classifier = classifier
```

Now, the network consists of the features from DenseNet121 and an untrained classifier. Since the model is so deep, training on it will take a very long time if run on the CPU. Training the network on a GPU, instead, can lead to 100x increased training speeds.

In PyTorch, it is easy to move a model to the GPU using `model.cuda()`

. It is important to make sure that the images are also on the GPU if the models are. This can be done with `images.cuda()`

. Move them to local memory and the CPU with `model.cpu()`

and `images.cpu()`

.

The following is device agnositic code that determines if the computer being used has a GPU, and if one is available, it uses it.

```
# at beginning of the script
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# then whenever you get a new Tensor or Module
# this won't copy if they are already on the desired device
input = data.to(device)
model = MyModule(...).to(device)
```

For a complete example of training the deep model, see my detailed Transfer Learning notes.

## Deep Learning Tips and Tricks from Udacity

#### Watch those shapes

In general, you’ll want to check that the tensors going through your model and other code are the correct shapes. Make use of the `.shape`

method during development and debugging.

#### A few things to check if your network isn’t training appropriately

Check that you’re clearing the gradients in the training loop with `optimizer.zero_grad()`

. When running a validation loop, be sure to set the network to evaluation mode with `model.eval()`

, then back to training mode with `model.train()`

.

#### CUDA errors

Sometimes you’ll see this error:

`RuntimeError: Expected object of type torch.FloatTensor but found type torch.cuda.FloatTensor for argument #1 ‘mat1’`

The second type is `torch.cuda.FloatTensor`

, this means it’s a tensor that has been moved to the GPU. It’s expecting a tensor with type `torch.FloatTensor`

, no `.cuda`

there, which means the tensor is on the CPU. PyTorch can only perform operations on tensors that are on the same device. If you’re trying to run your network on the GPU, check to make sure you’ve moved the model and all necessary tensors to the GPU with `.to(device)`

where `device`

is either `"cuda"`

or `"cpu"`

.