Data Engineer
Job Summary
The Data Engineer will design, develop, and deploy streaming and batch applications using big data technologies. You will be responsible for optimizing Spark application performance and performing data quality assurance. This role requires extensive hands-on experience in enterprise-level data environments.
Responsibilities
- Design and deploy streaming/batch data applications
- Develop and maintain Spark applications
- Optimize big data application performance
- Perform data quality assurance and troubleshooting
Required Skills
- 5+ years of Data Engineering experience
- Proficiency in Spark, Kafka, and BigQuery
- SQL and Cloud technology expertise
- Business level English proficiency
Job Details
Job Description:
- 5+ years of hands-on Data Engineering experience in enterprise products (e.g., e-commerce, financial services, telecommunications).
- Proven experience in designing, developing, and deploying streaming and/or batch applications using big data technologies such as Spark, Flink, Kafka, Cloud Pub/Sub, Dataproc, SQL, BigQuery, Cloud Dataflow, Cloud Storage, and Cloud Composer.
- Proficiency in developing Spark applications from the ground up and deploying them to production.
- Ability to understand and modify existing Spark applications to implement new features and resolve bugs.
- Experience in troubleshooting and optimizing Spark application performance through configuration adjustments, JVM tuning, and code refactoring.
- Ability to perform data quality assurance (QA) and investigate data discrepancies using SQL or Spark.
Requirements:
- English Level: Business Conversation Level (TOEIC 735-860)
- Japanese Level: None
Benefits:
- Social Insurance
- Commuting/Transportation Allowance
- Relaxation Facilities
- Employee Stock Purchase System
- Education/Training
- Five-Day Workweek
- Summer Holidays
- Winter Holidays
- Refresh Holidays
- Sick Leave
- Paid Holidays
- Child-care Leave