Lead/Senior Data Engineer

Job Description

Job Brief:We?re looking for a Senior Data Engineer to join our team, where you?ll be instrumental in developing and managing data workflows that support large-scale analytics and business intelligence. Your responsibilities will include designing and optimizing ETL processes, ensuring data quality, and maintaining robust data infrastructure.The ideal candidate will possess a robust computer science background, advanced proficiency in Python, and hands-on experience with cloud services, SQL/NoSQL databases, and containerization tools like Docker and Kubernetes.You?ll collaborate closely with engineering and product teams to drive innovation and generate valuable insights.VentureDive Overview:Founded in 2012 by veteran technology entrepreneurs from MIT and Stanford, VentureDive is the fastest growing technology company in the region that develops and invests in products and solutions that simplify and improve the lives of people world-wide. We aspire to create a technology organization and an entrepreneurial ecosystem in the region that is recognized as second to none in the world.Key Responsibilities:Design and Develop Data Pipelines:

1. Architect, develop, and maintain scalable and efficient data pipelines to process large volumes of data from diverse sources.
2. Implement ETL processes to integrate data from various internal and external sources into the data warehouse.

Data Modeling and Architecture:

1. Design and optimize data models to support analytics and business intelligence initiatives.
2. Ensure data quality and integrity across the data lifecycle.

Collaboration and Leadership:

1. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
2. Mentor and provide guidance to junior data engineers, fostering a culture of continuous learning and improvement.

Performance Optimization:

1. Monitor and improve the performance of data systems, identifying and resolving bottlenecks.
2. Implement best practices for data governance, security, and compliance.

Technology Evaluation and Implementation:

1. Stay current with emerging technologies and industry trends, evaluating and recommending new tools and technologies to enhance our data capabilities.
2. Lead the implementation and integration of new data technologies and frameworks.

Qualification Requirements:

• Bachelor?s or Master?s degree in Computer Science, Engineering, Information Technology, or and related field.
• 5+ years of experience in data engineering, with a proven track record in designing and implementing large-scale data solutions.
• Experience with cloud data platforms such as AWS, Azure, or Google Cloud.
• Proficiency in programming languages such as Python, Java, or Scala.
• Strong experience with SQL and NoSQL databases.
• Expertise in ETL tools and frameworks (e.g., Apache Spark, Apache Kafka, Talend).
• Familiarity with data warehousing solutions (e.g., Snowflake, Redshift, BigQuery).
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.
• Ability to work independently and as part of a team in a fast-paced environment.

Technical Skills:ÿ

• Proficiency in programming languages such as Python, Java, or Scala. ? Strong experience with SQL and NoSQL databases.
• Expertise in ETL tools and frameworks (e.g., Apache Spark, Apache Kafka,ÿ Talend).
• Familiarity with data warehousing solutions (e.g., Snowflake, Redshift,ÿ BigQuery).
• Mandatory expertise in the following Azure services:
• Azure Active Directory
• Azure App Service
• Azure Blob Storage
• Azure Application Insights
• Azure Key Vault
• Azure Cosmos DB
• Azure Data Factory
• Azure Open AI
• Azure Functions

Soft Skills:ÿ? Excellent problem-solving and analytical skills.ÿ? Strong communication and collaboration skills.ÿPreferred Qualifications:

1. Experience with big data technologies (e.g., Hadoop, Hive, Presto).
2. Experience in developing data pipelines utilizing tools such as Fluentd, Fluent Bit, Vector, and Logstash.ÿ
3. Knowledge of data visualization tools (e.g., Tableau, Power BI).
4. Certifications in relevant technologies or methodologies.

In order to thrive at VentureDive, you?are intellectually smart and curious?have the passion for and take pride in your work?deeply believe in VentureDive?s mission, vision, and values?have a no-frills attitude?are a collaborative team player?are ethical and honestAre you ready to put your ideas into products and solutions that will be used by millions?You will find VentureDive to be a quick pace, high standards, fun and a rewarding place to work at. Not only will your work reach millions of users worldwide, you will also be rewarded with competitive salaries and benefits. If you think you have what it takes to be a VenDian, come join us ... were having a ball!

Keyskills :