Data Engineer for LLMs
Job Summary
The Data Engineer will design and maintain scalable ETL/ELT pipelines specifically for LLM pre-training, fine-tuning, and evaluation. This role focuses on high-quality dataset creation, including cleaning, preprocessing, and data curation of large text corpora. Additionally, the engineer will manage data infrastructure, versioning, and orchestration workflows in both on-premise and cloud environments.
Responsibilities
- Develop scalable ETL/ELT pipelines for LLM data
- Perform data cleaning, PII masking, and tokenization
- Manage data versioning, lineage, and reproducibility
- Maintain data infrastructure and orchestration workflows
Required Skills
- Python and SQL expertise
- LLM data engineering experience
- Experience with DVC, MLflow, or similar
- Knowledge of PyTorch, TensorFlow, or JAX
- Data warehousing and schema design
Job Details
Position Details:
Data Pipeline Development for LLMs: Design, develop, and maintain highly scalable, reliable, and efficient data pipelines (ETL/ELT) for ingesting, transforming, and loading diverse datasets critical for LLM pre-training, fine-tuning, and evaluation. This includes structured, semi-structured, and unstructured text data.
High-Quality Dataset Creation & Curation: Implement advanced techniques for data cleaning and preprocessing, including deduplication, noise reduction, PII masking, tokenization, and formatting of large text corpora. Explore and implement methods for expanding and enriching datasets for LLM training, such as data augmentation and synthesis. Establish and enforce rigorous data quality standards, implement automated data validation checks, and ensure data privacy and security compliance.
Data Job Management: Establish robust systems for data versioning, lineage tracking, and reproducibility of datasets used across the LLM development lifecycle. Identify and resolve data-related performance bottlenecks within data pipelines, optimizing data storage, retrieval, and processing for efficiency and cost-effectiveness.
Data Infrastructure & Orchestration: Build and maintain scalable data warehouses and data lakes specifically designed for LLM data on both on-premise and public cloud environments. Implement and manage data orchestration tools (e.g., Apache Airflow, Prefect, Dagster) to automate and manage complex data workflows for LLM dataset preparation.
Mandatory Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Science, Engineering, or a related quantitative field.
- 3+ years of professional experience in Data Engineering, with a focus on large-scale machine learning initiatives involving large text/image/voice datasets.
- Direct experience with data engineering for Large Language Models (LLMs), including pre-training and fine-tuning datasets.
- Familiarity with preprocessing massive text corpora (noise handling, deduplication, PII detection/masking, tokenization).
- Experience with data versioning and lineage tools (e.g., DVC, Pachyderm, LakeFS, MLflow).
- Familiarity with deep learning frameworks (PyTorch, TensorFlow, JAX) for data loading and preparation.
- Experience designing data annotation workflows.
- Strong proficiency in Python and its data ecosystem.
- Proficiency in SQL, data warehousing concepts, data modeling, and schema design.
Other Details:
- English Level: Business Conversation Level (TOEIC 735-860).
- Japanese Level: None required.
- Benefits: Social Insurance, Commuting/Transportation Allowance, Relaxation Facilities, Five-Day Workweek, Summer/Winter Holidays, Refresh Holidays, Sick Leave, Paid Holidays, Child-care Leave.