Open-AI GPT 2 Implementation in Tensorflow

This Repository provides Tensorflow.keras implementation for Open-AI GPT-2. The implementation is modular using tf.keras subclassing method. It is compatible with both Graph Mode and Eager Mode execution.

This repository is built under Tensorflow 1.13.1, but is compatible with Tensorflow 2.0.

Example

The config file for model is same as original model.

import json

with open("117M/hparams.json") as f:
    config = json.load(f)

For creating model, it is enough to pass config as a dictionary

from gpt2 import GPT2

model = GPT2(config=config, name="gpt2")

Using the model is simple just as any other tf.keras.Model.

import tensorflow as tf

x = tf.placeholder(dtype=tf.int32, shape=[None, None])
y = model(x)

You can use more options in calling the model in both Graph and Eager mode.

In order to load pre-trained weights from original checkpoint, use builder.