Transformer Language Model¶
Transformer Language Model¶
-
class
openspeech.models.transformer_lm.model.
TransformerLanguageModel
(configs: omegaconf.dictconfig.DictConfig, tokenizer: openspeech.tokenizers.tokenizer.Tokenizer)[source]¶ Transformer language model. Paper: https://arxiv.org/abs/1904.09408
- Parameters
configs (DictConfig) – configuration set.
tokenizer (Tokenizer) – tokenizer is in charge of preparing the inputs for a model.
- Inputs:
inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded FloatTensor of size
(batch, seq_length, dimension)
.input_lengths (torch.LongTensor): The length of input tensor.
(batch)
- Returns
Result of model predictions.
- Return type
outputs (dict)
Transformer Language Model Configuration¶
-
class
openspeech.models.transformer_lm.configurations.
TransformerLanguageModelConfigs
(model_name: str = 'transformer_lm', num_layers: int = 6, d_model: int = 768, d_ff: int = 1536, num_attention_heads: int = 8, dropout_p: float = 0.3, max_length: int = 128, optimizer: str = 'adam')[source]¶ This is the configuration class to store the configuration of a
TransformerLanguageModel
.It is used to initiated an TransformerLanguageModel model.
Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.
- Parameters
model_name (str) – Model name (default: transformer_lm)
num_layers (int) – The number of lstm layers. (default: 6)
d_model (int) – The dimension of model. (default: 768)
dropout_p (float) – The dropout probability of encoder. (default: 0.3)
d_ff (int) – Dimenstion of feed forward network. (default: 2048)
num_attention_heads (int) – The number of attention heads. (default: 8)
max_length (int) – Max decoding length. (default: 128)
optimizer (str) – Optimizer for training. (default: adam)