Senior Data Software Engineer, Spark, Java, Scala

We are seeking a Senior Software Data Engineer to join our team in a software engineering capacity. This is not a data-science or analytics position centered on ad-hoc data exploration; instead, the role focuses on building software, data processing jobs, and data pipelines consumed by internal and external partners. Data is our main product and first-class citizen, and we value correct, high-quality data as much as clean and maintainable code.

Responsibilities

• Write new data pipelines and jobs to produce new outputs (datasets) in scope of new features development
• Adopt existing data pipelines to integrate with new org-wide platforms, tools, services, and languages
• Fix bugs in code and correct data caused by incorrect logic or implementation
• Perform ad-hoc data exploration, validation, and investigation to help select the right tech design and support Product Management team decisions
• Monitor and troubleshoot production issues with pipelines owned by the team
• Develop and adopt data quality checks to monitor data issues in the systems
• Scope and plan new development, including assessing level of effort and providing timelines
• Maintain tickets hygiene in Radar (ticketing system)
• Evolve jobs, apps, and systems to a better state across all aspects: code quality, complexity, maintainability, and documentation
• Communicate with other data engineers in the team, peer teams (QA, UAT, Platform, etc), project managers, and engineering managers on status, blockers, estimates, and timelines

Requirements

• 3+ years of hands-on experience in the big-data field, including Hadoop (HDFS, YARN or Mesos) and Spark
• Excellent knowledge and hands-on experience of SQL in context of Big Data: Spark SQL, HiveQL
• Excellent knowledge of Spark, including ability to understand and optimize Spark execution plans via Spark UI, with upcoming migration to Spark 3
• Excellent knowledge of Scala or Java
• Understanding of batch processing and ETL principles in Data Warehouses
• Familiarity with data completeness signals and orchestration
• Knowledge of approaches for historical reprocessing and data correction
• Skills in handling bad data and late data in inputs and outputs
• Understanding of schema migrations and datasets evolution
• Strong speaking English, with ability to rely on information heard verbally in meetings and to explain own ideas clearly to native speakers
• Capability to learn fast new set of tools and technology used internally at the company: platform services, telemetry providers, Spark-as-a-Service, build system, and more

Nice to have

• Understanding of functional programming ideas and principles
• Experience in building and using web services
• Familiarity with any of Teradata, Vertica, Oracle, Tableau
• Skills in Spark Streaming and Kafka
• Knowledge of Apache Iceberg, Trino (Presto), Druid, Cassandra, or Blob storage like AWS
• Experience with Splunk
• Experience with Snowflake

Benefits

We connect like-minded people

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Five trust days per year (sick leave without a medical certificate)
• Benefits package (sports activities, a variety of stores and services)

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers
• Discounts in local language schools, including online courses for the Kazakh language

We cover it all:

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Six trust days per year (sick leave without a medical certificate)
• Coverage of psychology sessions of your choice
• Benefits package (sports activities, a variety of stores and services)
• Housing support program: preferential mortgage access via Otbasy Bank partnership

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all:

• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Six trust days per year (sick leave without a medical certificate)
• Coverage of psychology sessions of your choice
• Discounts for fitness clubs and sports programs
• Benefits package (sports activities, a variety of stores and services)

We connect like-minded people:

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your own home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth:

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

• Discounts in local language schools, including offline courses for the Uzbek language

We cover it all:

• Monetary bonuses for engaging in the referral program
• Medical & family care package
• Four trust days per year (sick leave without a medical certificate)
• Discounts for fitness clubs, dance schools and sports programs
• Benefits package (sports activities, a variety of stores and services)

We connect like-minded people

• Delivering innovative solutions to industry leaders, making a global impact
• Enjoyable working environment, whether it is the vibrant office or the comfort of your home
• Opportunity to work abroad for up to two months per year
• Relocation opportunities within our offices in 55+ countries
• Corporate and social events

We invest in your growth

• Leadership development, career advising, soft skills and well-being programs
• Certifications, including GCP, Azure and AWS
• Unlimited access to EPAM's internal learning database
• Free English classes with certified teachers

We cover it all

• Participation in the Employee Stock Purchase Plan
• Monetary bonuses for engaging in the referral program
• Comprehensive medical & family care package
• Four trust days per year for personal needs
• Discounts for fitness clubs
• Benefits package (hotels, restaurants, stores and services)