Skip to content

implement GLU activation function #44

Description

@ibeltagy

Pick and implement one of the variants of the GLU activation.

Check here for comparison and equations
https://arxiv.org/pdf/2002.05202.pdf
https://arxiv.org/pdf/2102.11972.pdf

and here for implementation of one of them
https://github.com/EleutherAI/gpt-neox/blob/main/megatron/model/activations.py

Metadata

Metadata

Assignees

Labels

arch&scaleArchitecture and Scaling Modeling Group

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions