Jobgether

Compensation

Full Description

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Data Engineer in San Francisco, California. We are seeking a Senior Data Engineer to design, build, and optimize large-scale data solutions in a fast-paced, innovative environment. This role involves working on cloud-native architectures, modernizing on-premise systems, and delivering scalable, high-performance data pipelines that support critical business processes. You will collaborate with cross-functional teams to translate business requirements into robust technical solutions, ensuring data integrity, accessibility, and security. The position offers the opportunity to influence enterprise data strategy while working with cutting-edge technologies like AWS, Spark, and Hadoop. Your work will directly impact the efficiency and effectiveness of analytics and operational systems, enabling data-driven decision-making across the organization. Accountabilities Design, develop, and operationalize large-scale enterprise data solutions using AWS services such as S3, Glue, Athena, Lambda, RedShift, EMR, Spark, and DynamoDB. Analyze, redesign, and migrate on-premise Cloudera Hadoop data solutions to cloud-native architectures. Build, deploy, and maintain data pipelines from ingestion to consumption using Java, Python, or Scala. Participate in architecture and design discussions to translate business requirements into technical solutions. Take ownership of user stories through analysis, development, testing, and deployment. Develop and maintain scripts to automate batch jobs and support existing applications. Ensure compliance with technical best practices, internal processes, and information security standards. Maintain Hadoop-based data solutions (HDFS, Oozie, Impala, Hive) until full cloud migration. Bachelor’s Degree in Computer Science, Information Technology, or a related field. US Citizen or Green Card holder for over 3 years with intent to obtain US Citizenship. 7+ years of professional experience in data engineering or related software development roles. Strong proficiency in Java, Python, or Scala and complex SQL query development. Hands-on experience with AWS services including S3, Glue, Athena, Lambda, RedShift, EC2, EMR, Spark, and DynamoDB. Experience deploying solutions to cloud platforms via CI/CD in a DevOps model. Familiarity with application and data security concepts (AWS IAM, CloudTrail, CloudWatch, Config, Secrets Manager, KMS). Hands-on experience with Hadoop technologies (HDFS, Oozie, Impala, Hive). Experience with Infrastructure as Code tools such as Ansible, Terraform, or CloudFormation. Experience in Agile development environments with tools like Jira or Rally. Proficiency in UNIX commands and shell scripting. Strong collaboration, problem-solving, and communication skills. Preferred: Experience with RDBMS and data warehouses. Familiarity with machine learning technologies or data visualization tools. Competitive salary and comprehensive medical, dental, and vision coverage. 401(k) match and flexible spending accounts. Up to 27 days off per year, including your birthday, plus parental leave. Remote work opportunities and flexible working arrangements. Wellness benefits and other perks supporting work-life balance. Professional development opportunities including training, certifications, and Communities of Practice. Inclusive and diverse work environment in a woman-owned organization. Jobgether is a Talent Matching Platform that partners with companies worldwide to efficiently connect top talent with the right opportunities through AI-driven job matching. When you apply, your profile goes through our AI-powered screening process designed to identify top talent efficiently and fairly: 🔍 Our AI evaluates your CV and LinkedIn profile thoroughly, analyzing your skills, experience, and achievements. 📊 It compares your profile to the job’s core requirements and past success factors to determine your match score. 🎯 Based on this analysis, we automatically shortlist the 3 candidates with the highest match to the role. 🧠 When necessary, our human team may perform an additional manual review to ensure no strong profile is missed. The process is transparent, skills-based, and free of bias — focusing solely on your fit for the role. Once the shortlist is completed, we share it directly with the company that owns the job opening. The final decision and next steps (such as interviews or additional assessments) are then made by their internal hiring team. Thank you for your interest! #LI-CL1