Director, Software Engineer
Job Summary
DataPelago is at the forefront of revolutionizing data processing for traditional analytics and cutting-edge GenAI preprocessing. We are building an innovative data processing engine that is transforming how Apache Spark, Apache Flink, Ray and others operate on diverse, large-scale data. Our team of engineers drive and adopt advances in hardware-accelerated computing, parallel processing of large-scale data, query optimization, distributed systems, compilers, machine learning, and cloud-native computing. We are looking for specialists to join our engineering team and shape the future of accelerated data processing.
DataPelago Nucleus is a universal data processing engine that is designed to accelerate the processing of diverse data – structured through unstructured – with any parallel processing framework – e.g., Apache Spark – on any infrastructure – including vectorized CPU and GPU. DataPelago Accelerator for Spark (DPA-S), our product based on DataPelago Nucleus, is running large-scale production applications of many globally renowned customers.
The Opportunity:
As senior development leader of DataPelago data processing engine orchestration, you will spearhead the development and delivery of innovative capabilities in the planning, optimization, and coordination of complex workloads on large-scale distributed systems. The orchestration layer drives distributed execution plans adapted to the capabilities of the processing elements in the infrastructure, towards minimizing execution duration and cost. You will be responsible for growing and advancing the development team, development methodologies, and product capabilities.
a key individual contributor in adopting and advancing capabilities of open-source software (OSS) platforms such as Apache Gluten, Velox, Apache Spark, and Apache Flink in the context of DataPelago’s data processing engine. You will enhance functional breadth, performance, scale, and reliability of the DataPelago engine through downstream and upstream contributions. You will have the opportunity to engage with community working on these platforms. This is a unique opportunity to make a significant impact on a category-defining product and work with a talented team of engineers.
What You'll Do:
- Lead and grow a talented team of engineers with expertise in query planning, query optimization, plan execution orchestration for massively parallel data processing engines.
- Drive development with technical leads to implement and deliver industry-leading capabilities in data processing engine orchestration.
- Establish metrics including for product quality, performance, scale, and reliability and lead the team in ensuring these metrics are continuously improved and achieved in customer deployments.
- Continuously improve software development lifecycle practices leveraging the latest AI advances for higher code quality, release velocity, and productivity.
- Execute efficiently to fulfill product roadmap aligned with organization goals and priorities.
- Lead by example and foster a high-performance, high-integrity, collaborative work culture.
What You'll Bring:
- Bachelor’s degree in computer science or a related field. Master’s or PhD preferred.
- 12+ years of production software development experience with 7+ years of experience managing software development teams.
- 10+ years of experience in planning, developing, launching, and supporting core components of massively parallel data processing platforms (e.g., query engines, cloud data warehouses, lakehouse engines) for production deployment.
- Technical and process rigor in meeting and exceeding data accuracy, high performance, reliability, and security requirements of enterprise data processing products.
- Ability to attract, retain, and grow exceptional talent in data processing engine orchestration.
- Deep technical experience in one or more of database systems, data warehouses, query optimization, query engines. Experience with one or more of Apache Spark, Apache Flink, Apache Gluten, Velox, Apache DataFusion, Spark RAPIDS preferred.
- Solid understanding of the architecture, deployment, and operations of data processing platforms in applications such as data engineering, data preparation, and analytics.
- Excellent judgment on technology, software development, AI adoption, and people management. Strong leadership skills in all these areas.
- Good skills engaging with customers, product management, sales, and executive management.
All internal movements within the Product Group via requisition will be lateral, offering valuable growth opportunities to extend your skills in a new area. Opportunities for a promotion will be reviewed in the normal course of business, aligned with our promotion process.
Equal Opportunity Employer:
NetApp is firmly committed to Equal Employment Opportunity (EEO) and to compliance with all federal, state and local laws that prohibit employment discrimination based on age, race, color, gender, sexual orientation, gender identity, national origin, religion, disability or genetic information, pregnancy, protected veteran status, and any other protected classification.
Did you know...
Statistics show women apply to jobs only when they're 100% qualified. But no one is 100% qualified. We encourage you to shift the trend and apply anyway! We look forward to hearing from you.
Why NetApp?
Why You'll Thrive at NetApp
At NetApp, you won't wait for the perfect moment—you'll make it. The early planning, the extra thought, the bold idea that turns good into great: That's how our people operate and how we continue to push the boundaries of data infrastructure.
NetApp is the trusted partner for organizations transforming data into opportunity. As the only enterprise-grade storage service natively embedded in Google Cloud, AWS, and Microsoft Azure, we empower customers to run everything from traditional workloads to enterprise AI with unmatched performance, resilience, and security.
Our culture
We celebrate mold breakers, bold thinkers, and problem solvers. We reward initiative, impact, and ownership. We provide flexibility so you can balance professional ambition with your personal life. Here, differences are not just welcomed—they drive everything we do.
If you're ready to innovate, rise to the challenge, and own every moment - make your next move your best one. Apply now.
Apply nowRelated content
Diversity, equity, inclusion, and belonging
We are a collection of diverse people from around the world with our own individual stories. We share the drive to create innovative, world-class products. We're One NetApp, and we find strength in celebrating our individuality.
Learn more
NetApp Early in Career Program
The NetApp Early in Career Program is designed to help you grow your career, sharpen your skills, and foster meaningful relationships. When you join the Program, you join a diverse, global community where innovative ideas flourish.
Learn more
Find your ideal job
In a world consistently defined by new expectations, can you quickly adapt for the success of our customers? We reward innovation and problem-solving.
Learn moreNetApp is firmly committed to Equal Employment Opportunity (EEO) and to compliance with all federal, state and local laws that prohibit employment discrimination based on age, race, color, gender, sexual orientation, gender identity, national origin, religion, disability or genetic information, pregnancy, protected veteran status, and any other protected classification. We pledge to take every reasonable step to ensure that our applicants and employees are respected, treated fairly, and with dignity. See the EEO poster. NetApp makes reasonable accommodations, consistent with applicable laws, for religious purposes and for the known physical or mental limitations of an otherwise qualified applicant or employee with a disability, who can perform the essential job functions unless undue hardship would result.
Reasonable accommodation
If you are an applicant with a physical or mental disability that requires reasonable accommodation for any part of our application process, please email accessibility@netapp.com. Each request for reasonable accommodation will be considered on a case-by-case basis, consistent with applicable laws and regulations. Please note, this email address is only for accommodation requests; we do not accept unsolicited resumes.
Data privacy
We care about your privacy and therefore ask that you read our Applicant Privacy Policy before you submit any personal information to us.
Note to agencies
We’re sorry, but we cannot accept unsolicited resumes that are sent to NetApp employees or contractors. We will not compensate for a referral without a current contract on file with our Talent Acquisition team. If you’re interested in helping us with a particular role, please call your partner in Talent Acquisition to discuss.