The project, a platform for creating and publishing content on social media using artificial intelligence tools, is looking for a Lead Data Engineer.
Responsibilities:
- Design, develop, and maintain robust and scalable data pipelines for collecting, processing, and storing data from diverse social media sources and user interactions.
- Design of data warehouse.
- Implement rigorous data quality checks and validation processes to uphold the integrity.
accuracy, and reliability of social media data used by our AI models.
- Automate Extract, Transform, Load (ETL) processes to streamline data ingestion and transformation, reducing manual intervention and enhancing efficiency.
- Continuously monitor and optimize data pipelines to improve speed, reliability, and scalability, ensuring seamless operation of our AI Assistant.
- Collaborate closely with Data Scientists, ML Engineers, and cross-functional teams to understand data requirements and provide the necessary data infrastructure for model development and training.
- Enforce data governance practices, guaranteeing data privacy, security, and compliance with relevant regulations, including GDPR, in the context of social media data.
- Establish performance benchmarks and implement monitoring solutions to identify and address bottlenecks or anomalies in the data pipeline.
- Collaborate with data analysts and business teams to design interactive dashboards that enable data-driven decision-making.
- Develop and support data marts and dashboards that provide real-time insights into social media data.
- Stay updated with emerging data technologies, tools, and frameworks, evaluating their potential to improve data engineering processes.
Qualifications:
- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.
- Proven experience in data engineering, focusing on ETL processes, data pipeline development, and data quality assurance.
- Strong proficiency in programming languages such as Python, SQL and knowledge of data engineering libraries and frameworks.
- Experience with cloud-based data storage and processing solutions, such as AWS, Azure, or Google Cloud.
- Familiarity with DataOps principles and Agile methodologies.
- Excellent problem-solving skills and the ability to work collaboratively in a cross-functional team.
- Strong communication skills to convey technical concepts to non-technical stakeholders.
- Knowledge of data governance and data privacy regulations is a plus.
