Conformer

Conformer Model

class openspeech.models.conformer.model.ConformerLSTMModel(configs: omegaconf.dictconfig.DictConfig, tokenizer: openspeech.tokenizers.tokenizer.Tokenizer)

Conformer encoder + LSTM decoder.

Parameters

• configs (DictConfig) – configuraion set

• tokenizer (Tokenizer) – tokenizer is in charge of preparing the inputs for a model.

Inputs:

inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size (batch, seq_length, dimension). input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions that contains y_hats, logits,

encoder_outputs, encoder_logits, encoder_output_lengths.

Return type

outputs (dict)

set_beam_decoder(beam_size: int = 3)

Setting beam search decoder

class openspeech.models.conformer.model.ConformerModel(configs: omegaconf.dictconfig.DictConfig, tokenizer: openspeech.tokenizers.tokenizer.Tokenizer)

Conformer Encoder Only Model.

Parameters

• configs (DictConfig) – configuration set.

• tokenizer (Tokenizer) – tokenizer is in charge of preparing the inputs for a model.

Inputs:

inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size (batch, seq_length, dimension). input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions that contains y_hats, logits, output_lengths

Return type

outputs (dict)

forward(inputs: torch.Tensor, input_lengths: torch.Tensor) → Dict[str, torch.Tensor]

Forward propagate a inputs and targets pair for inference.

Inputs:

inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size (batch, seq_length, dimension). input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions that contains y_hats, logits, output_lengths

Return type

outputs (dict)

test_step(batch: tuple, batch_idx: int) → collections.OrderedDict

Forward propagate a inputs and targets pair for test.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

training_step(batch: tuple, batch_idx: int) → collections.OrderedDict

Forward propagate a inputs and targets pair for training.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

validation_step(batch: tuple, batch_idx: int) → collections.OrderedDict

Forward propagate a inputs and targets pair for validation.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

class openspeech.models.conformer.model.ConformerTransducerModel(configs: omegaconf.dictconfig.DictConfig, tokenizer: openspeech.tokenizers.tokenizer.Tokenizer)

Conformer: Convolution-augmented Transformer for Speech Recognition Paper: https://arxiv.org/abs/2005.08100

Parameters

• configs (DictConfig) – configuraion set

• tokenizer (Tokenizer) – tokenizer is in charge of preparing the inputs for a model.

Inputs:

inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size (batch, seq_length, dimension). input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions.

Return type

outputs (dict)

class openspeech.models.conformer.model.JointCTCConformerLSTMModel(configs: omegaconf.dictconfig.DictConfig, tokenizer: openspeech.tokenizers.tokenizer.Tokenizer)

Conformer encoder + LSTM decoder.

Parameters

• configs (DictConfig) – configuraion set

• tokenizer (Tokeizer) – tokenizer is in charge of preparing the inputs for a model.

Inputs:

inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size (batch, seq_length, dimension). input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions.

Return type

outputs (torch.FloatTensor)

set_beam_decoder(beam_size: int = 3)

Setting beam search decoder

Conformer Model Configuration

class openspeech.models.conformer.configurations.ConformerConfigs(model_name: str = 'conformer', encoder_dim: int = 512, num_encoder_layers: int = 17, num_attention_heads: int = 8, feed_forward_expansion_factor: int = 4, conv_expansion_factor: int = 2, input_dropout_p: float = 0.1, feed_forward_dropout_p: float = 0.1, attention_dropout_p: float = 0.1, conv_dropout_p: float = 0.1, conv_kernel_size: int = 31, half_step_residual: bool = True, optimizer: str = 'adam')

This is the configuration class to store the configuration of a Conformer.

It is used to initiated an Conformer model.

Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.

Parameters

• model_name (str) – Model name (default: conformer)

• encoder_dim (int) – Dimension of encoder. (default: 512)

• num_encoder_layers (int) – The number of encoder layers. (default: 17)

• num_attention_heads (int) – The number of attention heads. (default: 8)

• feed_forward_expansion_factor (int) – The expansion factor of feed forward module. (default: 4)

• conv_expansion_factor (int) – The expansion factor of convolution module. (default: 2)

• input_dropout_p (float) – The dropout probability of inputs. (default: 0.1)

• feed_forward_dropout_p (float) – The dropout probability of feed forward module. (default: 0.1)

• attention_dropout_p (float) – The dropout probability of attention module. (default: 0.1)

• conv_dropout_p (float) – The dropout probability of convolution module. (default: 0.1)

• conv_kernel_size (int) – The kernel size of convolution. (default: eq)

• half_step_residual (bool) – Flag indication whether to use half step residual or not (default: True)

• optimizer (str) – Optimizer for training. (default: adam)

class openspeech.models.conformer.configurations.ConformerLSTMConfigs(model_name: str = 'conformer_lstm', encoder_dim: int = 512, num_encoder_layers: int = 17, num_attention_heads: int = 8, feed_forward_expansion_factor: int = 4, conv_expansion_factor: int = 2, input_dropout_p: float = 0.1, feed_forward_dropout_p: float = 0.1, attention_dropout_p: float = 0.1, conv_dropout_p: float = 0.1, conv_kernel_size: int = 31, half_step_residual: bool = True, num_decoder_layers: int = 2, decoder_dropout_p: float = 0.1, max_length: int = 128, teacher_forcing_ratio: float = 1.0, rnn_type: str = 'lstm', decoder_attn_mechanism: str = 'loc', optimizer: str = 'adam')

This is the configuration class to store the configuration of a ConformerLSTM.

It is used to initiated an ConformerLSTM model.

Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.

Parameters

• model_name (str) – Model name (default: conformer_lstm)

• encoder_dim (int) – Dimension of encoder. (default: 512)

• num_encoder_layers (int) – The number of encoder layers. (default: 17)

• num_attention_heads (int) – The number of attention heads. (default: 8)

• feed_forward_expansion_factor (int) – The expansion factor of feed forward module. (default: 4)

• conv_expansion_factor (int) – The expansion factor of convolution module. (default: 2)

• input_dropout_p (float) – The dropout probability of inputs. (default: 0.1)

• feed_forward_dropout_p (float) – The dropout probability of feed forward module. (default: 0.1)

• attention_dropout_p (float) – The dropout probability of attention module. (default: 0.1)

• conv_dropout_p (float) – The dropout probability of convolution module. (default: 0.1)

• conv_kernel_size (int) – The kernel size of convolution. (default: eq)

• half_step_residual (bool) – Flag indication whether to use half step residual or not (default: True)

• num_decoder_layers (int) – The number of decoder layers. (default: 2)

• decoder_dropout_p (float) – The dropout probability of decoder. (default: 0.1)

• max_length (int) – Max decoding length. (default: 128)

• teacher_forcing_ratio (float) – The ratio of teacher forcing. (default: 1.0)

• rnn_type (str) – Type of rnn cell (rnn, lstm, gru) (default: lstm)

• decoder_attn_mechanism (str) – The attention mechanism for decoder. (default: loc)

• optimizer (str) – Optimizer for training. (default: adam)

class openspeech.models.conformer.configurations.ConformerTransducerConfigs(model_name: str = 'conformer_transducer', encoder_dim: int = 512, num_encoder_layers: int = 17, num_attention_heads: int = 8, feed_forward_expansion_factor: int = 4, conv_expansion_factor: int = 2, input_dropout_p: float = 0.1, feed_forward_dropout_p: float = 0.1, attention_dropout_p: float = 0.1, conv_dropout_p: float = 0.1, conv_kernel_size: int = 31, half_step_residual: bool = True, num_decoder_layers: int = 1, decoder_dropout_p: float = 0.1, max_length: int = 128, teacher_forcing_ratio: float = 1.0, rnn_type: str = 'lstm', decoder_hidden_state_dim: int = 640, decoder_output_dim: int = 640, optimizer: str = 'adam')

This is the configuration class to store the configuration of a ConformerTransducer.

It is used to initiated an ConformerTransducer model.

Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.

Parameters

• model_name (str) – Model name (default: conformer_transducer)

• encoder_dim (int) – Dimension of encoder. (default: 512)

• num_encoder_layers (int) – The number of encoder layers. (default: 17)

• num_attention_heads (int) – The number of attention heads. (default: 8)

• feed_forward_expansion_factor (int) – The expansion factor of feed forward module. (default: 4)

• conv_expansion_factor (int) – The expansion factor of convolution module. (default: 2)

• input_dropout_p (float) – The dropout probability of inputs. (default: 0.1)

• feed_forward_dropout_p (float) – The dropout probability of feed forward module. (default: 0.1)

• attention_dropout_p (float) – The dropout probability of attention module. (default: 0.1)

• conv_dropout_p (float) – The dropout probability of convolution module. (default: 0.1)

• conv_kernel_size (int) – The kernel size of convolution. (default: eq)

• half_step_residual (bool) – Flag indication whether to use half step residual or not (default: True)

• num_decoder_layers (int) – The number of decoder layers. (default: 1)

• decoder_dropout_p (float) – The dropout probability of decoder. (default: 0.1)

• max_length (int) – Max decoding length. (default: 128)

• teacher_forcing_ratio (float) – The ratio of teacher forcing. (default: 1.0)

• rnn_type (str) – Type of rnn cell (rnn, lstm, gru) (default: lstm)

• decoder_hidden_state_dim (int) – Hidden state dimension of decoder. (default: 640)

• decoder_output_dim (int) – Output dimension of decoder. (default: 640)

• optimizer (str) – Optimizer for training. (default: adam)

class openspeech.models.conformer.configurations.JointCTCConformerLSTMConfigs(model_name: str = 'joint_ctc_conformer_lstm', encoder_dim: int = 512, num_encoder_layers: int = 17, num_attention_heads: int = 8, feed_forward_expansion_factor: int = 4, conv_expansion_factor: int = 2, input_dropout_p: float = 0.1, feed_forward_dropout_p: float = 0.1, attention_dropout_p: float = 0.1, conv_dropout_p: float = 0.1, conv_kernel_size: int = 31, half_step_residual: bool = True, num_decoder_layers: int = 2, decoder_dropout_p: float = 0.1, num_decoder_attention_heads: int = 1, max_length: int = 128, teacher_forcing_ratio: float = 1.0, rnn_type: str = 'lstm', decoder_attn_mechanism: str = 'loc', optimizer: str = 'adam')

This is the configuration class to store the configuration of a JointCTCConformerLSTM.

It is used to initiated an JointCTCConformerLSTM model.

Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.

Parameters

• model_name (str) – Model name (default: joint_ctc_conformer_lstm)

• encoder_dim (int) – Dimension of encoder. (default: 512)

• num_encoder_layers (int) – The number of encoder layers. (default: 17)

• num_attention_heads (int) – The number of attention heads. (default: 8)

• feed_forward_expansion_factor (int) – The expansion factor of feed forward module. (default: 4)

• conv_expansion_factor (int) – The expansion factor of convolution module. (default: 2)

• input_dropout_p (float) – The dropout probability of inputs. (default: 0.1)

• feed_forward_dropout_p (float) – The dropout probability of feed forward module. (default: 0.1)

• attention_dropout_p (float) – The dropout probability of attention module. (default: 0.1)

• conv_dropout_p (float) – The dropout probability of convolution module. (default: 0.1)

• conv_kernel_size (int) – The kernel size of convolution. (default: eq)

• half_step_residual (bool) – Flag indication whether to use half step residual or not (default: True)

• num_decoder_layers (int) – The number of decoder layers. (default: 2)

• decoder_dropout_p (float) – The dropout probability of decoder. (default: 0.1)

• max_length (int) – Max decoding length. (default: 128)

• teacher_forcing_ratio (float) – The ratio of teacher forcing. (default: 1.0)

• rnn_type (str) – Type of rnn cell (rnn, lstm, gru) (default: lstm)

• decoder_attn_mechanism (str) – The attention mechanism for decoder. (default: loc)

• optimizer (str) – Optimizer for training. (default: adam)