Character, ngram or word based transformer model for text generation. Uses ml-indie-tools to run locally on M1-Mac, Nvidia or remotely on Colab with single code-base.
You can find the transformer attention implementation at the ml-indie-project, it's minimal and well documented and can server as base for further experimentations with transformer-like architecture.
- 2022-12-13: ml-indie-tools 0.4.0 removed all recurrence and gated memory, since it didn't improve things. Work-around for M1 tensorflow 2.11 problems with ADAM and XLA (crash on train), fixed using
legacy.ADAM. - 2022-12-11: ml-indie-tools 0.3.17 has new RecurrentSelfAttention layer that introduces a state similar to RNNs into key matrix of the Attention.
- 2022-11-21: ml-indie-tools ngram support.
- 2022-06-16: Tests with autoencoder-like bottlenecks in multi-head attention: in the middle of the layer-stack, decrease attention-units and increase attention-heads.
- 2022-01-13: Project split from LSTM-version tensor-poet. Further project-siblings are torch-poet, implementing char-based text generation with pytorch and syncognite rnnreader, implementing char-based text generation completely from scratch with c++.