Reports To: Director of Technology
The Senior Data Engineer will play a critical role in building, optimizing, and maintaining the data architecture, database management, data processing, and data mining activities. They will ensure that the data pipelines and data systems are scalable, repeatable, and secure, and can serve as a foundation for data transformations and insights.
- Design, construct, install, test, and maintain highly scalable data pipelines and architectures.
- Collaborate with data scientists, analysts, and other stakeholders to ensure data-related needs are met.
- Develop and maintain data ingestion and processing systems.
- Optimize data delivery for improved reusability, scalability, and operations.
- Analyze, organize, and provide insights on big data while ensuring data privacy and integrity.
- Ensure architecture supports the company’s business requirements and goals.
- Identify, design, and implement process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Collaborate with data architects, modelers, and IT team members on project goals.
- Develop set processes for data mining, data modeling, and data production.
- Ensure data quality and consistency within the data pipelines and promote data best practices across the company.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field with a focus on data or a comparable field.
- A minimum of 5-7 years of experience in a Data Engineering role.
- Expertise in big data technologies like Hadoop, Spark, AWS services, or similar technologies.
- Proficient in SQL, data modeling, and data warehousing.
- Familiarity with programming languages such as Java, Python, etc.
- Experience with orchestration tools like Airflow, Kubernetes, etc.
Other Key Attributes:
- Strong analytic skills related to working with unstructured datasets.
- Strong project management and organizational skills.
- Ability to work in a dynamic, fast-paced environment.
- Excellent communication skills to collaborate effectively across various teams.