torchgan.losses

This losses subpackage is a collection of popular loss functions used in the training of GANs. Currently the following losses are supported:

These losses are tested with the current available trainers. So if you need to implement you custom loss for using with the trainer it is recommended that you subclass the GeneratorLoss and DiscriminatorLoss.

Warning

The override_train_ops gets only the arguments that were received by the default train_ops. Hence it might not be a wise to use this very often. If this is used make sure to take into account the arguments and their order. A better alternative is to subclass the Loss and define a custom train_ops.

Warning

train_ops are designed to be used internally through the Trainer. Hence it is highly recommended that this function is not directly used by external sources, i.e. no call to this function is made outside the Trainer.

Loss

GeneratorLoss

class torchgan.losses.GeneratorLoss(reduction='mean', override_train_ops=None)[source]

Base class for all generator losses.

Note

All Losses meant to be minimized for optimizing the Generator must subclass this.

Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
set_arg_map(value)[source]

Updates the arg_map for passing a different value to the train_ops.

Parameters:value (dict) – A mapping of the argument name in the method signature and the variable name in the Trainer it corresponds to.

Note

If the train_ops signature is train_ops(self, gen, disc, optimizer_generator, device, batch_size, labels=None) then we need to map gen to generator and disc to discriminator. In this case we make the following function call loss.set_arg_map({"gen": "generator", "disc": "discriminator"}).

train_ops(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]

Defines the standard train_ops used by most losses. Losses which have a different training procedure can either subclass it (recommended approach) or make use of override_train_ops argument.

The standard optimization algorithm for the generator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value = discriminator(fake)\)
  3. \(loss = loss\_function(value)\)
  4. Backpropagate by computing \(\nabla loss\)
  5. Run a step of the optimizer for generator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

DiscriminatorLoss

class torchgan.losses.DiscriminatorLoss(reduction='mean', override_train_ops=None)[source]

Base class for all discriminator losses.

Note

All Losses meant to be minimized for optimizing the Discriminator must subclass this.

Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
set_arg_map(value)[source]

Updates the arg_map for passing a different value to the train_ops.

Parameters:value (dict) – A mapping of the argument name in the method signature and the variable name in the Trainer it corresponds to.

Note

If the train_ops signature is train_ops(self, gen, disc, optimizer_discriminator, device, batch_size, labels=None) then we need to map gen to generator and disc to discriminator. In this case we make the following function call loss.set_arg_map({"gen": "generator", "disc": "discriminator"}).

train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by most losses. Losses which have a different training procedure can either subclass it (recommended approach) or make use of override_train_ops argument.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value_1 = discriminator(fake)\)
  3. \(value_2 = discriminator(real)\)
  4. \(loss = loss\_function(value_1, value_2)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Least Squares Loss

LeastSquaresGeneratorLoss

class torchgan.losses.LeastSquaresGeneratorLoss(reduction='mean', c=1.0, override_train_ops=None)[source]

Least Squares GAN generator loss from “Least Squares Generative Adversarial Networks by Mao et. al.” paper

The loss can be described as

\[L(G) = \frac{(D(G(z)) - c)^2}{2}\]

where

  • \(G\) : Generator
  • \(D\) : Disrciminator
  • \(c\) : target generator label
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • c (float, optional) – Target generator label.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(dgz)[source]

Computes the loss for the given input.

Parameters:dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:scalar if reduction is applied else Tensor with dimensions (N, *).

LeastSquaresDiscriminatorLoss

class torchgan.losses.LeastSquaresDiscriminatorLoss(reduction='mean', a=0.0, b=1.0, override_train_ops=None)[source]

Least Squares GAN discriminator loss from “Least Squares Generative Adversarial Networks by Mao et. al.” paper.

The loss can be described as:

\[L(D) = \frac{(D(x) - b)^2 + (D(G(z)) - a)^2}{2}\]

where

  • \(G\) : Generator
  • \(D\) : Disrciminator
  • \(a\) : Target discriminator label for generated image
  • \(b\) : Target discriminator label for real image
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • a (float, optional) – Target discriminator label for generated image.
  • b (float, optional) – Target discriminator label for real image.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(dx, dgz)[source]

Computes the loss for the given input.

Parameters:
  • dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

Minimax Loss

MinimaxGeneratorLoss

class torchgan.losses.MinimaxGeneratorLoss(reduction='mean', nonsaturating=True, override_train_ops=None)[source]

Minimax game generator loss from the original GAN paper “Generative Adversarial Networks by Goodfellow et. al.”

The loss can be described as:

\[L(G) = log(1 - D(G(z)))\]

The nonsaturating heuristic is also supported:

\[L(G) = -log(D(G(z)))\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
  • nonsaturating (bool, optional) – Specifies whether to use the nonsaturating heuristic loss for the generator.
forward(dgz)[source]

Computes the loss for the given input.

Parameters:dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:scalar if reduction is applied else Tensor with dimensions (N, *).

MinimaxDiscriminatorLoss

class torchgan.losses.MinimaxDiscriminatorLoss(label_smoothing=0.0, reduction='mean', override_train_ops=None)[source]

Minimax game discriminator loss from the original GAN paper “Generative Adversarial Networks by Goodfellow et. al.”

The loss can be described as:

\[L(D) = -[log(D(x)) + log(1 - D(G(z)))]\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
  • \(x\) : A sample from the data distribution
  • \(z\) : A sample from the noise prior
Parameters:
  • label_smoothing (float, optional) – The factor by which the labels (1 in this case) needs to be smoothened. For example, label_smoothing = 0.2 changes the value of the real labels to 0.8.
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the mean of the output. If sum the elements of the output will be summed.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(dx, dgz)[source]

Computes the loss for the given input.

Parameters:
  • dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

Boundary Equilibrium Loss

BoundaryEquilibriumGeneratorLoss

class torchgan.losses.BoundaryEquilibriumGeneratorLoss(reduction='mean', override_train_ops=None)[source]

Boundary Equilibrium GAN generator loss from “BEGAN : Boundary Equilibrium Generative Adversarial Networks by Berthelot et. al.” paper

The loss can be described as

\[L(G) = D(G(z))\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(dgz)[source]

Computes the loss for the given input.

Parameters:dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:scalar if reduction is applied else Tensor with dimensions (N, *).

BoundaryEquilibriumDiscriminatorLoss

class torchgan.losses.BoundaryEquilibriumDiscriminatorLoss(reduction='mean', override_train_ops=None, init_k=0.0, lambd=0.001, gamma=0.75)[source]

Boundary Equilibrium GAN discriminator loss from “BEGAN : Boundary Equilibrium Generative Adversarial Networks by Berthelot et. al.” paper

The loss can be described as

\[L(D) = D(x) - k_t \times D(G(z))\]
\[k_{t+1} = k_t + \lambda \times (\gamma \times D(x) - D(G(z)))\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
  • \(k_t\) : Running average of the balance point of G and D
  • \(\lambda\) : Learning rate of the running average
  • \(\gamma\) : Goal bias hyperparameter
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place ofthe default train_ops
  • init_k (float, optional) – Initial value of the balance point k.
  • lambd (float, optional) – Learning rate of the running average.
  • gamma (float, optional) – Goal bias hyperparameter.
forward(dx, dgz)[source]

Computes the loss for the given input.

Parameters:
  • dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

A tuple of 3 loss values, namely the total loss, loss due to real data and loss due to fake data.

set_k(k=0.0)[source]

Change the default value of k

Parameters:k (float, optional) – New value to be set.
train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by boundary equilibrium loss.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value_1 = discriminator(fake)\)
  3. \(value_2 = discriminator(real)\)
  4. \(loss = loss\_function(value_1, value_2)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
  7. Update the value of :math: k.
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

update_k(loss_real, loss_fake)[source]

Update the running mean of k for each forward pass.

The update takes place as

\[k_{t+1} = k_t + \lambda \times (\gamma \times D(x) - D(G(z)))\]
Parameters:
  • loss_real (float) – \(D(x)\)
  • loss_fake (float) – \(D(G(z))\)

Energy Based Loss

EnergyBasedGeneratorLoss

class torchgan.losses.EnergyBasedGeneratorLoss(reduction='mean', override_train_ops=None)[source]

Energy Based GAN generator loss from “Energy Based Generative Adversarial Network by Zhao et. al.” paper.

The loss can be described as:

\[L(G) = D(G(z))\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(dgz)[source]

Computes the loss for the given input.

Parameters:dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:scalar if reduction is applied else Tensor with dimensions (N, *).
train_ops(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]

This function sets the embeddings attribute of the AutoEncodingDiscriminator to False and calls the train_ops of the GeneratorLoss. After the call the attribute is again set to True.

Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

EnergyBasedDiscriminatorLoss

class torchgan.losses.EnergyBasedDiscriminatorLoss(reduction='mean', margin=80.0, override_train_ops=None)[source]

Energy Based GAN generator loss from “Energy Based Generative Adversarial Network by Zhao et. al.” paper

The loss can be described as:

\[L(D) = D(x) + max(0, m - D(G(z)))\]

where

  • \(G\) : Generator
  • \(D\) : Discriminator
  • \(m\) : Margin Hyperparameter
  • \(z\) : A sample from the noise prior

Note

The convergence of EBGAN is highly sensitive to hyperparameters. The margin hyperparameter as per the paper was taken as follows:

Dataset Margin
MNIST 10.0
LSUN 80.0
CELEB A 20.0
Imagenet (128 x 128) 40.0
Imagenet (256 x 256) 80.0
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • margin (float, optional) – The margin hyperparameter.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(dx, dgz)[source]

Computes the loss for the given input.

Parameters:
  • dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, batch_size, labels=None)[source]

This function sets the embeddings attribute of the AutoEncodingDiscriminator to False and calls the train_ops of the DiscriminatorLoss. After the call the attribute is again set to True.

Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

EnergyBasedPullingAwayTerm

class torchgan.losses.EnergyBasedPullingAwayTerm(pt_ratio=0.1, override_train_ops=None)[source]

Energy Based Pulling Away Term from “Energy Based Generative Adversarial Network by Zhao et. al.” paper.

The loss can be described as:

\[f_{PT}(S) = \frac{1}{N(N-1)}\sum_i\sum_{j \neq i}\bigg(\frac{S_i^T S_j}{||S_i||\ ||S_j||}\bigg)^2\]

where

  • \(S\) : The feature output from the encoder for generated images
  • \(N\) : Batch Size of the Input
Parameters:
  • pt_ratio (float, optional) – The weight given to the pulling away term.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(dgz, d_hid)[source]

Computes the loss for the given input.

Parameters:
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • d_hid (torch.Tensor) – The embeddings generated by the discriminator.
Returns:

scalar.

train_ops(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]

This function extracts the hidden embeddings of the discriminator network. The furthur computation is same as the standard train_ops.

Note

For the loss to work properly, the discriminator must be a AutoEncodingDiscriminator and it must have a embeddings attribute which should be set to True. Also the generator label_type must be none. As a result of these constraints it advisable not to use custom models with this loss. This will be improved in future.

Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Wasserstein Loss

WassersteinGeneratorLoss

class torchgan.losses.WassersteinGeneratorLoss(reduction='mean', override_train_ops=None)[source]

Wasserstein GAN generator loss from “Wasserstein GAN by Arjovsky et. al.” paper

The loss can be described as:

\[L(G) = -f(G(z))\]

where

  • \(G\) : Generator
  • \(f\) : Critic/Discriminator
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the mean of the output. If sum the elements of the output will be summed.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(fgz)[source]

Computes the loss for the given input.

Parameters:dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:scalar if reduction is applied else Tensor with dimensions (N, *).

WassersteinDiscriminatorLoss

class torchgan.losses.WassersteinDiscriminatorLoss(reduction='mean', clip=None, override_train_ops=None)[source]

Wasserstein GAN generator loss from “Wasserstein GAN by Arjovsky et. al.” paper

The loss can be described as:

\[L(D) = f(G(z)) - f(x)\]

where

  • \(G\) : Generator
  • \(f\) : Critic/Discriminator
  • \(x\) : A sample from the data distribution
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the mean of the output. If sum the elements of the output will be summed.
  • clip (tuple, optional) – Tuple that specifies the maximum and minimum parameter clamping to be applied, as per the original version of the Wasserstein loss without Gradient Penalty.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(fx, fgz)[source]

Computes the loss for the given input.

Parameters:
  • fx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • fgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by wasserstein discriminator loss.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. Clamp the discriminator parameters to satisfy \(lipschitz\ condition\)
  2. \(fake = generator(noise)\)
  3. \(value_1 = discriminator(fake)\)
  4. \(value_2 = discriminator(real)\)
  5. \(loss = loss\_function(value_1, value_2)\)
  6. Backpropagate by computing \(\nabla loss\)
  7. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

WassersteinGradientPenalty

class torchgan.losses.WassersteinGradientPenalty(reduction='mean', lambd=10.0, override_train_ops=None)[source]

Gradient Penalty for the Improved Wasserstein GAN discriminator from “Improved Training of Wasserstein GANs by Gulrajani et. al.” paper

The gradient penalty is calculated as:

The gradient being taken with respect to x

where

  • \(G\) : Generator
  • \(D\) : Disrciminator/Critic
  • \(\lambda\) : Scaling hyperparameter
  • \(x\) : Interpolation term for the gradient penalty
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the mean of the output. If sum the elements of the output will be summed.
  • lambd (float,optional) – Hyperparameter lambda for scaling the gradient penalty.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(interpolate, d_interpolate)[source]

Computes the loss for the given input.

Parameters:
  • interpolate (torch.Tensor) – It must have the dimensions (N, *) where * means any number of additional dimensions.
  • d_interpolate (torch.Tensor) – Output of the discriminator with interpolate as the input. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by the Wasserstein Gradient Penalty.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(interpolate = \epsilon \times real + (1 - \epsilon) \times fake\)
  3. \(d\_interpolate = discriminator(interpolate)\)
  4. \(loss = \lambda loss\_function(interpolate, d\_interpolate)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Mutual Information Penalty

class torchgan.losses.MutualInformationPenalty(lambd=1.0, reduction='mean', override_train_ops=None)[source]

Mutual Information Penalty as defined in “InfoGAN : Interpretable Representation Learning by Information Maximising Generative Adversarial Nets by Chen et. al.” paper

The loss is the variational lower bound of the mutual information between the latent codes and the generator distribution and is defined as

\[L(G,Q) = log(Q|x)\]

where

  • \(x\) is drawn from the generator distribution G(z,c)
  • \(c\) drawn from the latent code prior \(P(c)\)
Parameters:
  • lambd (float, optional) – The scaling factor for the loss.
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the mean of the output. If sum the elements of the output will be summed.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
forward(c_dis, c_cont, dist_dis, dist_cont)[source]

Computes the loss for the given input.

Parameters:
  • c_dis (int) – The discrete latent code sampled from the prior.
  • c_cont (int) – The continuous latent code sampled from the prior.
  • dist_dis (torch.distributions.Distribution) – The auxilliary distribution \(Q(c|x)\) over the discrete latent code output by the discriminator.
  • dist_cont (torch.distributions.Distribution) – The auxilliary distribution \(Q(c|x)\) over the continuous latent code output by the discriminator.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_generator, optimizer_discriminator, dis_code, cont_code, device, batch_size)[source]

Defines the standard train_ops used by most losses. Losses which have a different training procedure can either subclass it (recommended approach) or make use of override_train_ops argument.

The standard optimization algorithm for the generator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value = discriminator(fake)\)
  3. \(loss = loss\_function(value)\)
  4. Backpropagate by computing \(\nabla loss\)
  5. Run a step of the optimizer for generator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Dragan Loss

DraganGradientPenalty

class torchgan.losses.DraganGradientPenalty(reduction='mean', lambd=10.0, k=1.0, override_train_ops=None)[source]

Gradient Penalty for the DRAGAN discriminator from “On Convergence and Stability of GANs by Kodali et. al.” paper

The gradient penalty is calculated as:

\[\lambda \times (||grad(D(x))||_2 - k)^2\]

The gradient being taken with respect to x

where

  • \(G\) : Generator
  • \(D\) : Disrciminator
  • \(\lambda\) : Scaling hyperparameter
  • \(x\) : Interpolation term for the gradient penalty
  • \(k\) : Constant
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • lambd (float,optional) – Hyperparameter \(\lambda\) for scaling the gradient penalty.
  • k (float, optional) – Constant.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(interpolate, d_interpolate)[source]

Computes the loss for the given input.

Parameters:
  • interpolate (torch.Tensor) – It must have the dimensions (N, *) where * means any number of additional dimensions.
  • d_interpolate (torch.Tensor) – Output of the discriminator with interpolate as the input. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by the DRAGAN Gradient Penalty.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. \(interpolate = real + \frac{1}{2} \times (1 - \alpha) \times std(real) \times \beta\)
  2. \(d\_interpolate = discriminator(interpolate)\)
  3. \(loss = loss\_function(interpolate, d\_interpolate)\)
  4. Backpropagate by computing \(\nabla loss\)
  5. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Auxillary Classifier Loss

AuxiliaryClassifierGeneratorLoss

class torchgan.losses.AuxiliaryClassifierGeneratorLoss(reduction='mean', override_train_ops=None)[source]

Auxiliary Classifier GAN (ACGAN) loss based on a from “Conditional Image Synthesis With Auxiliary Classifier GANs by Odena et. al. “ paper

Parameters:reduction (str, optional) –
Specifies the reduction to apply to the output.
If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
override_train_ops (function, optional): A function is passed to this argument,
if the default train_ops is not to be used.
train_ops(generator, discriminator, optimizer_generator, device, batch_size, labels=None)[source]

Defines the standard train_ops used by the Auxiliary Classifier generator loss.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows (label_g and label_d both could be either real labels or generated labels):

  1. \(fake = generator(noise, label_g)\)
  2. \(value_1 = classifier(fake, label_g)\)
  3. \(value_2 = classifier(real, label_d)\)
  4. \(loss = loss\_function(value_1, label_g) + loss\_function(value_2, label_d)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized. For ACGAN, it must require labels for training
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

AuxiliaryClassifierDiscriminatorLoss

class torchgan.losses.AuxiliaryClassifierDiscriminatorLoss(reduction='mean', override_train_ops=None)[source]

Auxiliary Classifier GAN (ACGAN) loss based on a from “Conditional Image Synthesis With Auxiliary Classifier GANs by Odena et. al. “ paper

Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – A function is passed to this argument, if the default train_ops is not to be used.
train_ops(generator, discriminator, optimizer_discriminator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used by the Auxiliary Classifier discriminator loss.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows (label_g and label_d both could be either real labels or generated labels):

  1. \(fake = generator(noise, label_g)\)
  2. \(value_1 = classifier(fake, label_g)\)
  3. \(value_2 = classifier(real, label_d)\)
  4. \(loss = loss\_function(value_1, label_g) + loss\_function(value_2, label_d)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized. For ACGAN, it must require labels for training
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Feature Matching Loss

FeatureMatchingGeneratorLoss

class torchgan.losses.FeatureMatchingGeneratorLoss(reduction='mean', override_train_ops=None)[source]

Feature Matching Generator loss from “Improved Training of GANs by Salimans et. al.” paper

The loss can be described as:

\[L(G) = ||f(x)-f(G(z))||_2\]

where

  • \(G\) : Generator
  • \(f\) : An intermediate activation from the discriminator
  • \(z\) : A sample from the noise prior
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
forward(fx, fgz)[source]

Computes the loss for the given input.

Parameters:
  • dx (torch.Tensor) – Output of the Discriminator with real data. It must have the dimensions (N, *) where * means any number of additional dimensions.
  • dgz (torch.Tensor) – Output of the Discriminator with generated data. It must have the dimensions (N, *) where * means any number of additional dimensions.
Returns:

scalar if reduction is applied else Tensor with dimensions (N, *).

train_ops(generator, discriminator, optimizer_generator, real_inputs, device, labels=None)[source]

Defines the standard train_ops used for feature matching.

The standard optimization algorithm for the generator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value_1 = discriminator(fake)\) where \(value_1\) is an activation of an intermediate
    discriminator layer
  3. \(value_2 = discriminator(real)\) where \(value_2\) is an activation of the same intermediate
    discriminator layer
  4. \(loss = loss\_function(value_1, value_2)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for generator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

Historical Averaging

HistoricalAverageGeneratorLoss

class torchgan.losses.HistoricalAverageGeneratorLoss(reduction='elementwise_mean', override_train_ops=None, lambd=1.0)[source]

Historical Average Generator Loss from “Improved Techniques for Training GANs by Salimans et. al.” paper

The loss can be described as

\[|| \vtheta - \frac{1}{t} \sum_{i=1}^t \vtheta[i] ||^2\]

where

  • \(G\) : Generator
  • math:vtheta[i] : Generator Parameters at Past Timestep :math: i
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
  • lambd (float, optional) – Hyperparameter lambda for scaling the Historical Average Penalty
train_ops(generator, optimizer_generator)[source]

Defines the standard train_ops used by most losses. Losses which have a different training procedure can either subclass it (recommended approach) or make use of override_train_ops argument.

The standard optimization algorithm for the generator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value = discriminator(fake)\)
  3. \(loss = loss\_function(value)\)
  4. Backpropagate by computing \(\nabla loss\)
  5. Run a step of the optimizer for generator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_generator (torch.optim.Optimizer) – Optimizer which updates the parameters of the generator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.

HistoricalAverageDiscriminatorLoss

class torchgan.losses.HistoricalAverageDiscriminatorLoss(reduction='elementwise_mean', override_train_ops=None, lambd=1.0)[source]

Historical Average Discriminator Loss from “Improved Techniques for Training GANs by Salimans et. al.” paper

The loss can be described as

\[|| \vtheta - \frac{1}{t} \sum_{i=1}^t \vtheta[i] ||^2\]

where

  • \(G\) : Discriminator
  • math:vtheta[i] : Discriminator Parameters at Past Timestep :math: i
Parameters:
  • reduction (str, optional) – Specifies the reduction to apply to the output. If none no reduction will be applied. If mean the outputs are averaged over batch size. If sum the elements of the output are summed.
  • override_train_ops (function, optional) – Function to be used in place of the default train_ops
  • lambd (float, optional) – Hyperparameter lambda for scaling the Historical Average Penalty
train_ops(discriminator, optimizer_discriminator)[source]

Defines the standard train_ops used by most losses. Losses which have a different training procedure can either subclass it (recommended approach) or make use of override_train_ops argument.

The standard optimization algorithm for the discriminator defined in this train_ops is as follows:

  1. \(fake = generator(noise)\)
  2. \(value_1 = discriminator(fake)\)
  3. \(value_2 = discriminator(real)\)
  4. \(loss = loss\_function(value_1, value_2)\)
  5. Backpropagate by computing \(\nabla loss\)
  6. Run a step of the optimizer for discriminator
Parameters:
  • generator (torchgan.models.Generator) – The model to be optimized.
  • discriminator (torchgan.models.Discriminator) – The discriminator which judges the performance of the generator.
  • optimizer_discriminator (torch.optim.Optimizer) – Optimizer which updates the parameters of the discriminator.
  • real_inputs (torch.Tensor) – The real data to be fed to the discriminator.
  • device (torch.device) – Device on which the generator and discriminator is present.
  • batch_size (int) – Batch Size of the data infered from the DataLoader by the Trainer.
  • labels (torch.Tensor, optional) – Labels for the data.
Returns:

Scalar value of the loss.