Lead Data Engineer

The project, a platform for creating and publishing content on social media using artificial intelligence tools, is looking for a Lead Data Engineer.

Responsibilities:

- Design, develop, and maintain robust and scalable data pipelines for collecting, processing, and storing data from diverse social media sources and user interactions.

- Design of data warehouse.

- Implement rigorous data quality checks and validation processes to uphold the integrity.

accuracy, and reliability of social media data used by our AI models.

- Automate Extract, Transform, Load (ETL) processes to streamline data ingestion and transformation, reducing manual intervention and enhancing efficiency.

- Continuously monitor and optimize data pipelines to improve speed, reliability, and scalability, ensuring seamless operation of our AI Assistant.

- Collaborate closely with Data Scientists, ML Engineers, and cross-functional teams to understand data requirements and provide the necessary data infrastructure for model development and training.

- Enforce data governance practices, guaranteeing data privacy, security, and compliance with relevant regulations, including GDPR, in the context of social media data.

- Establish performance benchmarks and implement monitoring solutions to identify and address bottlenecks or anomalies in the data pipeline.

- Collaborate with data analysts and business teams to design interactive dashboards that enable data-driven decision-making.

- Develop and support data marts and dashboards that provide real-time insights into social media data.

- Stay updated with emerging data technologies, tools, and frameworks, evaluating their potential to improve data engineering processes.

Qualifications:

- Bachelor's or Master's degree in Computer Science, Data Engineering, or a related field.

- Proven experience in data engineering, focusing on ETL processes, data pipeline development, and data quality assurance.

- Strong proficiency in programming languages such as Python, SQL and knowledge of data engineering libraries and frameworks.

- Experience with cloud-based data storage and processing solutions, such as AWS, Azure, or Google Cloud.

- Familiarity with DataOps principles and Agile methodologies.

- Excellent problem-solving skills and the ability to work collaboratively in a cross-functional team.

- Strong communication skills to convey technical concepts to non-technical stakeholders.

- Knowledge of data governance and data privacy regulations is a plus.