DataComp-LM: In Search of the Next Generation of Training Sets for Language Models
DataComp-LM establishes a controlled benchmark for dataset research and finds that aggressive model-based quality filtering is more effective than conventional source mixing.