Technologies I've worked on. How were the experiences and biggest challenges. Issues addressing hard and soft skills
Senior Data Engineer Interview Questions
2,400 senior data engineer interview questions shared by candidates
1) 1. Python list tuple, set 2. homogenous and heterogenous 3. map and flatmap 4. skewness 5. repartitioning and coalesce 6. Cost optimisation optimisation repartition 7. SQL emp_id joining_date manager_id 101 2015-03-10 NULL 102 2017-06-15 101 103 2016-08-22 101 104 2016-02-05 102 105 2018-09-01 102 106 2015-11-30 103 107 2019-01-20 103 108 2021-05-12 102 109 2020-07-25 108 110 2022-02-18 108 111 2019-08-15 101
1. Mostly around the projects you do/mention on your resume. 2. SQL/Python.
you have to check if customer A has any records in a huge say 100+ billion records table, how will you do in spark without effecting cluster/performance.
Problem solving using Python arrays, dictionaries
What are some technologies you're familiar with
- Questions on SQL, Data Structures and Normal ETL in pyspark - Spark Optimization Techniques - Resume Discussion - How to handle data quality in Big Data - Apache AIrflow Infrastructure - Discussion on Role based policies
General questions about data streaming and event-driven architecture, when to use tabular vs columnar storage etc
how many years of python experience do you have?
There was a technical home work question to design a ETL process that loads csv files into a SQL system. The files's structures could change and so they had to be handled dynamically.
Viewing 1681 - 1690 interview questions