Want Better Synthetic Data? Steer It: Activation Steering for Low Resource Language Generation

Cegin, J., Gurgurov, D.¹, Al Ghussin, Y.¹, Ostermann, S.¹

¹ German Research Institute for Artificial Intelligence (DFKI), Saarbrücken, Germany

Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages.

Cite: Cegin, J., Gurgurov, D., Ghussin, Y. A., & Ostermann, S. (2026). Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation. arXiv preprint arXiv:2606.18389.