Understanding of OpenSeq2Seq – GeeksforGeeks

by admin

On this article, we will probably be discussing a deep studying toolkit used to enhance the coaching time of the present Speech Recognition fashions amongst different issues like Pure Language Translation, Speech Synthesis and Language Modeling. Fashions constructed utilizing this toolkit give a state-of-the-art efficiency at 1.5-3x quicker coaching time. 

OpenSeq2Seq is an open-source TensorFlow primarily based toolkit that includes multi-GPU and mixed-precision coaching which considerably reduces the coaching time of varied NLP fashions. For instance,

1. Pure Language Translation: GNMT, Transformer, ConvS2S
2. Speech Recognition: Wave2Letter, DeepSpeech2
3. Speech Synthesis: Tacotron 2

It makes use of Sequence to Sequence paradigm to assemble and practice fashions to carry out a wide range of duties reminiscent of machine translation, textual content summarization. 

Sequence to Sequence mannequin 

The mannequin consists of three components: encoder, encoder vector and decoder.

Fig 1 : Encoder-Decoder Sequence to Sequence Mannequin

  • Encoder
    • On this, a number of recurrent models like LTSM (Lengthy Brief Time period Reminiscence) and GRU (Gated Recurrent Unit) are used for enhanced efficiency.
    • Every of those recurrent models accepts a single factor of the enter sequence, gathers the data for that factor and propagates it ahead.  
    • The enter sequence is a group of all of the phrases from the query.
    • The hidden states (h1, h2…, hn) are calculated utilizing the next components. [Eq 1]

Eq 1

the place, 
ht = hidden state
ht-1 = earlier hidden state
W(hh) = weights hooked up to the earlier hidden state. (ht-1) 
xt = enter vector 
W(hx) = weights hooked up to the enter vector.

  • Encoder Vector
    • The ultimate hidden state is calculated utilizing Eqn 1 from the encoder a part of the mannequin.
    • The encoder vector collects the data for all enter parts with a purpose to assist the decoder make correct predictions.
    • It serves because the preliminary hidden state of the decoder a part of the mannequin.
  • Decoder
    • On this, a number of recurrent models are current the place every one predicts an output yt at a time step t.
    • Every recurrent unit accepts a hidden state from the earlier unit and produces an output in addition to its personal hidden state.
    • The hidden states (h1, h2…, hn) are calculated utilizing the next components. [Eqn 2]

Eqn 2

For instance, Fig 1. Exhibits sequence to sequence mannequin for a dialogue system. 

Fig 2 : Sequence to Sequence mannequin for a Dialog System

Each Sequence to Sequence mannequin has an encoder and a decoder. For instance,

S-no. Activity Encoder Decoder
1. Sentiment Evaluation RNN Linear SoftMax
2. Picture Classification CNN Linear SoftMax

Design and Structure 

The OpenSeq2Seq toolkit supplies numerous lessons from the consumer can inherit their very own modules. The mannequin is split into 5 completely different components :

  1. Information Layer
  2. Encoder
  3. Decoder
  4. Loss Perform
  5. Hyperparameters
    • Optimizer
    • Studying Charge
    • Dropout
    • Regularization
    • Batch_Size and many others.
For instance, an OpenSeq2Seq mannequin for Machine Translation would appear to be :

Encoder - GNMTLikeEncoderWithEmbedding
Decoder - RNNDecoderWithAttention
Loss Perform - BasicSequenceLoss
Hyperparameters - 
    Studying Charge = 0.0008
    Regularization = 'weight decay' 
    Batch_Size = 32

Combined-Precision Coaching: 

When utilizing float16 to coach massive neural community fashions, it’s typically crucial to use sure algorithmic strategies and hold some outputs in float32. (therefore the identify, combined precision). 

Combined-Precision Assist [using Algorithm]

The mannequin makes use of TensorFlow as its base, thus have tensor-cores which delivers the required efficiency to coach massive neural networks. They permit matrix-matrix multiplication to be accomplished in 2 methods:

  • Single-Precision Floating-Level (FP-32)
    • A  single-precision floating-point format is a pc quantity format that occupies 32 bits (4 bytes in trendy computer systems)  in pc reminiscence.
    • In a 32-bit floating-point, eight bits are reserved for the exponent (“magnitude”) and 23 bits for the mantissa (“precision”).
  • Half-Precision Floating Level (FP-16)
    • A half precision is a binary floating-point format is a pc quantity format that occupies 16 bits (two bytes in trendy computer systems) in pc reminiscence.

Earlier, when coaching a neural community, FP-32 (as proven in Fig 2) had been used to signify the weights within the community due to numerous causes reminiscent of:

  • Greater Precision — 32-bit floats have sufficient precision such that we are able to distinguish numbers of various magnitudes from each other.
  • In depth Vary — 32-bit floating factors have sufficient vary to signify numbers of magnitude each smaller (10^-45) and bigger (10^38) than what’s required for many purposes. 
  • Supportable — All {hardware} (GPUs, CPUs) and APIs assist 32-bit floating-point directions fairly effectively.

Fig 3: FP-32 illustration

However, afterward, it was discovered that for optimum deep studying fashions, a lot magnitude and precision shouldn’t be required. So, NVIDIA created {hardware} that supported 16-bit floating-point directions and noticed that almost all weights and gradients are likely to fall effectively throughout the 16-bit representable vary. 

Due to this fact, in OpenSeq2Seq mannequin, FP-16 has been used.  Utilizing this, we successfully forestall losing all these additional bits. With FP-16, we cut back the variety of bits in half, lowering the exponent from eight bits to five, and the mantissa from 23 bits to 10. (As proven in Fig 3)

Fig 4: FP-16 illustration 

Dangers of utilizing FP-16

1. Underflow : trying to signify numbers so small they clamp to zero.
2. Overflow : numbers so massive (exterior FP-16 vary) that they turn out to be NaN, not a quantity. 

  • With underflow, our community by no means learns something.
  • With overflow, it learns rubbish.
  • For Combined-Precision Coaching, we observe an algorithm that includes the next 2 steps:
Step 1 - Keep float32 grasp copy of weights for weights replace whereas utilizing the float16 
     weights for ahead and again propagation.     
Step 2 - Apply loss scaling whereas computing gradients to stop underflow throughout backpropagation.

Fig 5: Arithmetic Operations in FP16 and gathered in FP32

The Combined Precision Coaching of the OpenSeq2Seq mannequin includes three issues:

  1. Combined Precision Optimizer
  2. Combined Precision Regularizer
  3. Computerized Loss Scaling

1. Combined Precision Optimizer

The mannequin has all variables and gradients as FP-16 by default, as proven in Fig 6. The next steps happen on this course of:

Fig 6:  Combined-precision wrapper round TensorFlow optimizers 

Working of Combined Precision Wrapper (Step by Step)

Every Iteration
    Step 1 - The wrapper robotically converts FP-16 gradients and FP-32 and feed them 
          to the tensorflow optimizer. 
    Step 2 - The tensorflow optimizer then updates the copy of weights in FP-32.
    Step 3 - The up to date FP-32 weights are then transformed again to FP-16.
    Step 4 - The FP-16 weights are then utilized by the mannequin for the subsequent iteration.   

2. Combined Precision Regularization 

As mentioned earlier the dangers concerned with utilizing F-16 like numerical overflow/underflow. The combined precision regularization ensures that such circumstances don’t happen through the coaching. So, to beat such issues, we do the next steps:

Step 1 - All regularizers needs to be outlined throughout variable creation.

Step 2 - The regularizer perform needs to be wrapped with the 'Combined Precision Wrapper'. This takes care of two issues:
    2.1 - Provides the regularized variables to a tensorflow assortment.
    2.2 - Disables the underlying regularization perform for FP-16 copy.

Step 3 - This assortment is then retrieved by Combined Precision Optimizer Wrapper.
Step 4 - The corresponding features obtained from the MPO wrapper will probably be utilized to the FP-32 copy 
of the weights guaranteeing that their gradients at all times keep in full precision. 

3. Computerized Loss Scaling

The OpenSeq2Seq mannequin includes automated loss scaling. So, the consumer doesn’t have to pick the loss worth manually. The optimizer analyzes the gradients after every iteration and updates the loss worth for the subsequent iteration. 

Fashions Concerned

OpenSeq2Seq presently gives a full implementation of a wide range of fashions:

OpenSeq2Seq options a wide range of fashions for language modelling, machine translation, speech synthesis, speech recognition, sentiment evaluation, and extra to come back. It goals to supply a wealthy library of generally used encoders and decoders. This was a fundamental overview of the OpenSeq2Seq mannequin overlaying the instinct, structure and ideas concerned. For any doubts/queries, remark beneath. 

For those who like GeeksforGeeks and wish to contribute, you can too write an article utilizing contribute.geeksforgeeks.org or mail your article to [email protected] See your article showing on the GeeksforGeeks foremost web page and assist different Geeks.

Please Enhance this text if you happen to discover something incorrect by clicking on the “Enhance Article” button beneath.

Related Posts

Leave a Comment