Dagster Data Engineering Glossary:
Data Engineering Terms Explained
Terms and Definitions You Need to Know as a Data Engineer
Back to Glossary IndexACID Properties
The set of properties of database transactions intended to guarantee validity even in the event of errors or failures, encompassing Atomicity, Consistency, Isolation, and Durability.
Agile Methodology
An iterative approach to software development and project management that prioritizes flexibility and customer satisfaction, often used by data engineering teams to manage projects.
Alation
A machine learning data catalog that helps people find, understand, and trust the data.
Aligning
Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
Apache Atlas
A scalable and extensible set of core foundational governance services, enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop.
Argo
An open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
Association Rule Mining
A machine learning method aimed at identifying interesting relations between variables (items or events) in large databases, frequently used for market basket analysis.
Backend-as-a-Service (BaaS)
A cloud computing service model that serves as the middleware that provides developers with ways to connect their web and mobile applications to cloud services via application programming interfaces (APIs) and software developers' kits (SDKs).
CBOR (Concise Binary Object Representation)
A binary format encoding data in a more efficient and compact manner than JSON. It is designed to efficiently serialize and deserialize complex data structures without losing schema-free property of JSON.
Combinatorial Explosion
A phenomenon in computer science where the number of possible solutions or combinations in a problem grows exponentially with the size of the problem.
Crash Recovery
The process by which an operating system or application restarts operation after a crash, possibly recovering lost data.
CRON
A time-based job scheduler in Unix-like computer operating systems for scheduling periodic jobs at fixed times, dates, or intervals.
Cursor
A database object used to traverse the results of a SQL query, allowing individual rows to be accessed.
DAG (Directed Acyclic Graph)
A finite directed graph with no directed cycles, used extensively in representing data flow in data processing systems like Apache Airflow.
Data Degradation
The gradual loss or deterioration of data quality over time.
Data Dictionary
A collection of descriptions of the data objects or items in a data model for the benefit of programmers and others who need to refer to them.
Data Provenance
Information that helps to trace the origins, processing, and use of data, helping to determine the quality and reliability of the dataset.
Drift Detection
Identifying when the statistical properties of the target variable, which the model is trying to predict, change.
Elasticsearch
A search engine based on the Lucene library, providing a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents.
Feature Scaling
A method used to normalize the range of independent variables or features of data.
Feature Store
A centralized repository for storing, serving, and sharing machine learning features, allowing for the consistent use of features across different models.