Scope of Work/Responsibilities:

1. Data Pipeline Development:

· Design, implement, and optimize end-to-end data pipelines using Databricks and related technologies.

· Build workflows to handle large-scale data ingestion, transformation, and storage.

2. Data Preparation for LLMs:

· Preprocess, clean, and structure diverse datasets (text, structured, and unstructured) for LLM training and fine-tuning.

· Implement feature engineering, tokenization, and vectorization techniques to support NLP models.

3. Performance Optimization:

· Use Databricks features, including Delta Lake and MLflow, to streamline data workflows.

· Optimize data infrastructure for high availability, scalability, and cost-efficiency.

4. Collaboration with Teams:

· Work closely with data scientists, ML engineers, and other stakeholders to understand data requirements for LLM technology requirements.

· Ensure alignment between engineering pipelines and machine learning goals.

5. Data Quality & Governance:

· Implement processes to ensure data quality, consistency, and compliance with governance policies.

· Monitor and maintain data integrity throughout the pipeline lifecycle.

6. Emerging Technology Adoption:

· Stay updated on advancements in Databricks, generative AI, and LLM technologies.

· Contribute to the adoption of innovative tools and practices to improve workflows.

Requirement and Qualification (Education & Work Experience):

Experience:

· 7+ years of experience in data engineering roles, with at least 2 years in a leadership role and projects involving Databricks.

· Proven expertise in data pipelines, feature engineering, and dataset preparation for machine learning, specifically LLMs.

· Experience building enterprise-grade applications with GenAI or AI/ML integrations.

Technical Skills:

· Expertise in Databricks, Apache Spark, and Delta Lake.

· Strong programming skills in Python and SQL; knowledge of libraries like pandas, NumPy, or PyTorch is a plus

· Understanding of state management libraries like Redux, Recoil, or Zustand.Cypress), and version control (Git).

· Understanding of web security principles and compliance requirements for enterprise applications.

Soft Skills:

· Exceptional problem-solving and decision-making abilities.

· Excellent communication and leadership skills, with the ability to guide technical discussions and mentor team members.

· Strong focus on delivering quality

Please refer to job description.