Role- Application developer

Job Description - Data Engineer Consultant - Airflow / Astronomer Migration

Job Title: Data Engineer Consultant - Airflow / Astronomer Migration

Experience: 3 to 5 years

Location / Shift:

• Location: Pune, India / Remote / Hybrid as per project need
• Working Mode: Full-time assignment
• Client working hours may apply

Role Summary:

You will work as a Data Engineer Consultant.
You will support the migration of orchestration workflows from Azure Data Factory (ADF) to Astronomer / Apache Airflow.
You will design, build, test, and document data workflows.
You will work closely with the Principal Data Engineer and the Data Engineering team.
You will also suggest improvements for performance, cost, reliability, and maintainability.

Primary Skills:

Must Have

• Apache Airflow / Astronomer
• Airflow DAG design and development
• Python and SQL
• dbt
• Snowflake
• Azure Blob Storage
• Data pipeline orchestration
• Workflow testing and validation

Secondary Skills:

Good to Have

• Azure Data Factory (ADF)
• ADF to Airflow migration experience
• Data pipeline modernization experience
• Snowflake performance tuning
• dbt model and flow optimization
• Git / Azure DevOps
• CI/CD for data pipelines
• Monitoring and alerting for data workflows

Key Responsibilities:

1) Requirement Understanding

• Understand existing ADF pipelines and orchestration flows.
• Analyze current workflow schedules, triggers, parameters, and dependencies.
• Understand ingestion steps, transformation logic, and downstream impact.
• Work with the Principal Data Engineer and team to clarify open points.
• Identify risks, blockers, assumptions, and dependencies early.

2) Solution Design

• Design the migration approach from ADF to Astronomer / Airflow.
• Convert existing orchestration logic into clean Airflow DAG design.
• Define DAG structure, task dependencies, retry logic, and schedule patterns.
• Design workflows that are easy to maintain and support.
• Suggest improvements instead of only doing one-to-one migration.

3) Development / Implementation

• Design and build DAGs in Apache Airflow / Astronomer.
• Develop and adapt data pipelines as per migration requirement.
• Modify existing ingestion steps in Astronomer where required.
• Modify existing dbt flows and transformations when needed.
• Use Python and SQL for workflow logic, validation, and automation.
• Follow coding standards and project guidelines.

4) Integration / Configuration

• Configure schedules and dependencies for Airflow DAGs.
• Integrate Airflow workflows with dbt, Snowflake, and Azure Blob Storage.
• Ensure dbt jobs are properly triggered and monitored through Airflow.
• Validate integration between ingestion, transformation, and consumption layers.
• Configure workflow parameters, environment settings, and required connections.

5) Testing & Validation

• Perform unit testing for DAGs and workflow components.
• Perform integration testing for end-to-end data workflows.
• Validate migrated workflows against existing ADF output wherever applicable.
• Check data accuracy, completeness, and processing status.
• Fix defects and retest before production readiness.
• Prepare test evidence and validation notes.

6) Performance Optimization

• Review existing workflows and identify improvement areas.
• Optimize Airflow DAG performance and execution time.
• Improve reliability using proper retries, failure handling, and dependency management.
• Suggest cost-efficient execution patterns.
• Improve maintainability by creating reusable workflow components.
• Proactively recommend better design wherever useful.

7) Security, Compliance & Governance

• Follow client security and data handling guidelines.
• Use secure connection and access patterns as per project standards.
• Avoid hardcoding secrets, passwords, or sensitive values.
• Ensure access and workflow configurations are controlled and traceable.
• Follow required governance process for data pipeline changes.

8) Deployment & Release Management

• Support deployment of Airflow / Astronomer workflows across environments.
• Prepare deployment steps and release notes.
• Support production readiness checks before go-live.
• Coordinate with the Data Engineering team during release activities.
• Validate workflows after deployment.
• Support rollback or quick fix approach if any deployment issue occurs.

9) Production Support & RCA

• Monitor workflow execution and identify failures.
• Debug failures using Airflow logs, task history, dbt logs, and Snowflake queries.
• Perform root cause analysis for recurring issues.
• Implement preventive fixes to improve workflow stability.
• Support production issues during migration and stabilization phase.

10) Documentation & Knowledge Transfer

• Prepare technical documentation for developed DAGs and workflows.
• Document schedule, dependency, parameter, and configuration details.
• Prepare development and testing documentation.
• Maintain handover notes and support instructions.
• Provide knowledge transfer to the Data Engineering team.
• Explain design decisions, known issues, and support steps clearly.

11) Agile Delivery & Collaboration

• Work closely with the Principal Data Engineer and Data Engineering team.
• Provide regular status updates on progress, risks, and blockers.
• Collaborate with team members in a structured and transparent way.
• Take ownership of assigned tasks and deliver on time.
• Work as a hands-on team reinforcement for the migration project.

Tools / Technologies:

• Cloud / Platform: Azure
• Orchestration Tools: Apache Airflow, Astronomer, Azure Data Factory
• Data Transformation: dbt
• Storage: Azure Blob Storage
• Database / Warehouse: Snowflake
• Programming Languages: Python, SQL
• DevOps Tools: Git / Azure DevOps, if used in project
• Monitoring Tools: Airflow UI, Astronomer monitoring, dbt logs, Snowflake query history
• Documentation Tools: Confluence / SharePoint / Project documentation repository, as applicable

Qualification:

• BE / BTech / MCA / MSc / BSc / BCA or equivalent practical experience
• Relevant Data Engineering, Cloud, Snowflake, dbt, or Airflow certification is good to have

Soft Skills:

• Autonomy and ownership
• Curiosity to understand existing systems
• Proactive mindset
• Ability to suggest practical improvements
• Good communication
• Strong collaboration with team members
• Ability to integrate smoothly with existing team
• Good documentation discipline

Preferred Candidate Profile:

• 3 to 5 years of Data Engineering experience.
• Strong hands-on experience in Airflow / Astronomer.
• Good working knowledge of dbt, Snowflake, Azure Blob Storage, Python, and SQL.
• Prior experience in orchestration migration or data pipeline modernization.
• Ability to understand ADF workflows and redesign them in Airflow.
• Good delivery ownership and clear communication.
• Able to work independently with minimum supervision.