Senior Data engineer Key Responsibilities • 7 to 10 years of experience in data engineering and minimum of 3+ years of experience working in Databricks • 4+ years Working in leveraging Pyspark for building reusable Data ingestion and Data quality frameworks • Lakehouse & Data Engineering: Design scalable batch and real-time ETL pipelines using PySpark, Structured Streaming, and Delta Lake built on a Medallion Architecture (Bronze, Silver, Gold). • ABAC and RBAC Security - Experience in setting up ABAC and RBAC security for Agentic AI Journey use cases • Agentic AI & Orchestration: Build autonomous, multi-agent workflows and reasoning loops using MCP Protocol and Serverless compute . • RAG & Vector Integration: Develop end-to-end Retrieval-Augmented Generation (RAG) pipelines and integrate vector databases for semantic retrieval. • Governance & Operations: Manage data lineage, access controls, and security using Unity Catalog alongside CI/CD workflows and LLM evaluation frameworks

Originally posted on Himalayas