Data pipelines are sets of processes that move and transform data from different sources to a specific destination, where new value can be derived. Data pipelines are the foundation of data analytics, reporting, business intelligence, and machine learning capabilities. Data engineers specialize in building and maintaining the data pipelines that underpin the analytics ecosystem. Data pipelines often consist of multiple steps including data extraction, data preprocessing, data validation, and at times training or running a machine learning model before delivering data to its final destination.

Data engineers work with data scientists and data analysts to understand the purpose of the data and help bring their needs into a scalable production state. Data pipelines are normally built and maintained by data engineers, unless there is no data engineer to do the work and the pipeline is built by the data scientist or the data analyst.

The skills required for building data pipelines include SQL and data warehousing fundamentals, Python and/or Java programming, knowledge of distributed computing platforms, basic system administration, and a goal-oriented mentality.

In the data engineering lifecycle, the ingestion stage is where data engineers begin actively designing data pipeline activities.