MinimaLabs/min-spark-1.1
Text Generation • 5.76M • Updated • 1.17k • 20
🤏 quecto mode
appvoid
appvoid
AI & ML interests
singularity through byte-level tokens and small language models - creating applications and ideas out of the void
Recent Activity
liked a model about 4 hours ago
MinimaLabs/min-spark-1.1 updated a model about 12 hours ago
appvoid/void.0 updated a Space about 12 hours ago
thedotcompany/READMEOrganizations
liked a
model about 4 hours ago
updated a
model about 12 hours ago
updated a
Space about 12 hours ago
upvoted an article about 17 hours ago
Article
Extreme Overtraining in Tiny Language Models
Banaxi-Tech
• • 16 updated a
model 1 day ago
updated a
collection 1 day ago
Policy on finetuning on benchmark train splits
6
#140 opened 2 days ago
by 
sz14
congrats!
liked a
model 1 day ago
liked a
Space 2 days ago
liked a
model 2 days ago
Bruh this is crazy
❤️ 1
13
#1 opened 3 days ago
by 
Datdanboi25
posted an update 3 days ago
Post
87
We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
updated a
collection 3 days ago
published a
model 3 days ago
reacted to KlondikeDev's post with ❤️ 5 days ago
Post
2123
The SLM Consortium has begun work on a safety dataset for Small Language Models, with the creation of the dataset being headed by @wayneworkman2012
The dataset will focus on refusals and redirects surrounding dangerous or extreme sexual content, designed to be shaped sized appropriately for SLMs, without significantly lowering benchmark performance.
More info will be out soon!
slmconsortium
The dataset will focus on refusals and redirects surrounding dangerous or extreme sexual content, designed to be shaped sized appropriately for SLMs, without significantly lowering benchmark performance.
More info will be out soon!
- 3 replies
New License request
👀 2
37
#131 opened 6 days ago
by 
KlondikeDev
updated a
Space 5 days ago
Good
❤️ 1
1
#3 opened 5 days ago
by appvoid
liked a
model 5 days ago
