Dagster Data Engineering Glossary:
Data Engineering Terms Explained
Terms and Definitions You Need to Know as a Data Engineer
Back to Glossary IndexA/B Testing
A statistical hypothesis testing for a randomized experiment with two variables, A and B, which are used to compare two models or strategies and determine which performs better.
See WikipediaACID Properties
The set of properties of database transactions intended to guarantee validity even in the event of errors or failures, encompassing Atomicity, Consistency, Isolation, and Durability.
See WikipediaAgile Methodology
An iterative approach to software development and project management that prioritizes flexibility and customer satisfaction, often used by data engineering teams to manage projects.
See WikipediaAlation
A machine learning data catalog that helps people find, understand, and trust the data.
Vendor WebsiteAligning
Aligning data can mean one of three things: aligning datasets, meeting business rules, or arranging data elements in memory.
See Glossary entryAmazon DynamoDB
A managed NoSQL database service provided by Amazon Web Services.
Vendor WebsiteAmazon Kinesis
A platform to stream data on AWS, offering powerful services to make it easy to load and analyze streaming data.
Vendor WebsiteAmazon Redshift
A fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL.
Vendor WebsiteAmazon Web Services (AWS)
Offers a broad set of global cloud-based products including compute, storage, databases, analytics, networking, mobile, developer tools, and more.
Vendor WebsiteAnnotation
The process of adding metadata or explanatory notes to data, often used in machine learning to create labeled data for training models.
Read moreApache Airflow
A platform to programmatically author, schedule, and monitor workflows of tasks.
Read moreApache Arrow
Apache Arrow is a cross-language development platform for in-memory data that specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware.
Project websiteApache Atlas
A scalable and extensible set of core foundational governance services, enabling enterprises to effectively and efficiently meet their compliance requirements within Hadoop.
Project websiteApache Camel
An open-source integration framework that empowers you to quickly and easily integrate various systems consuming or producing data.
Project websiteApache Flink
A framework and distributed processing engine for stateful computations over unbounded and bounded data streams.
Project websiteApache Hadoop
A framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.
Project websiteApache Kafka
A distributed streaming platform capable of handling trillions of events a day.
Project websiteApache Nifi
A tool designed to automate the flow of data between software systems.
Project websiteApache Pulsar
A highly scalable, low-latency messaging platform running on commodity hardware.
Project websiteApache Samza
A stream processing framework for running applications that process data as it is created.
Project websiteApache Spark
A fast and general-purpose cluster computing system, providing high-level APIs in Java, Scala, Python, and R.
Project websiteApache Storm
A free and open-source distributed real-time computation system.
Project websiteAPI (Application Programming Interface)
A set of rules and definitions that allow different software entities to communicate with each other.
Learn MoreArgo
An open-source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
Vendor WebsiteAssociation Rule Mining
A machine learning method aimed at identifying interesting relations between variables (items or events) in large databases, frequently used for market basket analysis.
Find out moreAugmented Data Management
The use of AI and ML technologies to optimize and enhance data management tasks, improving data quality and metadata development.
Find out moreAutomated Machine Learning (AutoML)
The process of automating the end-to-end process of applying machine learning to real-world problems, facilitating the development of ML models by experts and non-experts alike.
Learn moreAvro
A binary serialization format developed within the Apache Hadoop project, compact, fast, and suitable for serializing large amounts of data. It uses JSON for defining data types and protocols, and it serializes data in a compact binary format.
Visit the projectAWS Step Functions
Enables you to coordinate AWS components, applications and microservices using visual workflows.
See vendor siteBackend-as-a-Service (BaaS)
A cloud computing service model that serves as the middleware that provides developers with ways to connect their web and mobile applications to cloud services via application programming interfaces (APIs) and software developers' kits (SDKs).
See WikipediaBig Data
Refers to extremely large datasets that can be analyzed for patterns, trends, and associations, typically involving varied and complex structures. What constitutes 'big' is debated, but a rule of thumb is a volume of data that cannot be analyzed on a single machine.
See WikipediaBig O Notation
A mathematical notation used to describe the limiting behavior of a function when the argument tends towards a particular value or infinity, primarily used to classify algorithms by how they respond to changes in input size.
See WikipediaBinary Tree
A tree data structure in which each node has at most two children, referred to as the left child and the right child.
See WikipediaBitwise Operation
Operations that manipulate one or more bits at the level of their individual binary representation.
See WikipediaBlend
A term coined by data analytics vendors to describe the process of combining data from multiple sources to create a cohesive, unified dataset. Typically used in the context of data analysis and business intelligence.
Learn MoreBlockchain
A system of recording information in a way that makes it difficult or impossible to change, hack, or cheat the system. A blockchain is a digital ledger of transactions that is duplicated and distributed across the entire network of computer systems on the blockchain.
See WikipediaBroadcast
A method in parallel computing where data is sent from one point (a root node) to all other nodes in the topology.
Learn MoreBroadcasting
A method in distributed computing to send the same message to all nodes in a network.
Learn MoreBSON (Binary JSON)
A binary-encoded serialization of JSON-like documents used to store documents and make remote procedure calls in MongoDB. BSON supports embedded documents and arrays, offering additional data types not supported by JSON.
Learn MoreBucketing
A method for dividing a dataset into discrete buckets or bins to separate it into roughly equal parts based on some characteristic.
Learn MoreBulk Extract
The process of extracting large amounts of data from a database in a single transaction.
Learn MoreBusiness Intelligence (BI)
A set of strategies and technologies used by enterprises for the data analysis of business information, helping companies make more informed business decisions.
Learn MoreCache Invalidation
A process in a computing system where entries in a cache are replaced or removed due to change in the underlying data.
Learn MoreCaching
The process of storing copies of files in a cache, or temporary storage location, so that they can be accessed more quickly.
See Glossary entryCallback
A piece of executable code that is passed as an argument to other code and is expected to execute at a given time.
Learn MoreCapacity Planning
The process used to determine how much hardware and software is required to meet future workload demands.
Learn More