Data Fabric Abstract
The data fabric is an architecture that facilitates the end-to-end integration of various data work lines and cloud environments through the use of intelligent and automated systems. In the last decade, developments within hybrid cloud , artificial intelligence , the Internet of Things (IoT), and edge computing have led to the exponential growth of big data, creating even more management complexity for businesses. This has made unifying and managing data environments an increasing priority, as this growth has created significant challenges such as data silos, security risks, and overall decision-making bottlenecks. data management teamsare tackling these challenges head-on with data fabric solutions . They are leveraging them to unify their disparate data systems, integrate management, tighten security and privacy measures, and give workers, particularly business users, more access to data.
What is a Data Fabric?
The data fabric is an emerging design concept for data management that addresses the challenges of data complexity. Its goal is to provide an agile enterprise database to support a wide variety of business use cases. The notion of a data fabric is closely tied to DataOps and initiatives for data modernization and digital innovation in general.
A data fabric can be thought of as a fabric that connects data from multiple locations (edge, core, and cloud), data types, and data sources, with methods to access that data. For users consuming applications and systems alike, it abstracts away the complexities associated with underlying data storage, movement, transformation, security, and processing.
A data fabric is not a replacement for more traditional data management architectures such as data lakes, data warehouses, data concentrators, and databases. Instead, a data fabric includes those systems as active participants in a unified approach.
Data fabric architecture
Leveraging data services and APIs, data fabrics bring together data from legacy systems, data lakes , data warehouses , sql databases, and applications, providing a comprehensive view of business performance. Unlike these individual data warehouse systems, its goal is to create more fluidity in data environments, trying to counteract the problem of data gravity, that is, the idea that data becomes more difficult to move to as they grow in size. A data fabric abstracts away the technological complexities involved in moving, transforming, and integrating data, making all data available across the enterprise.
That said, this is just an example. There is no single data architecture for a data fabric, as different businesses have different needs. The diverse number of cloud providers and data infrastructure implementations ensure variation between companies. However, companies using this type of data structure exhibit similarities in their architectures that are unique to a data fabric. More specifically, they have six fundamental components, which Forrester (link external to ibm.com) describes in the "Enterprise Data Fabric Enables DataOps" report. These six layers include the following:
- Data management layer: This is responsible for data management and data security.
Data ingestion layer: This layer begins to piece together the data from the cloud, finding connections between the structured and unstructured data.
Data processing: The data processing layer refines the data to ensure that only data relevant to the data extraction is displayed.
Data Orchestration: This critical layer does some of the most important work for the data fabric: transforming, integrating, and cleansing data, making it usable for teams across the enterprise.
Data Discovery: This layer shows new opportunities to integrate disparate data sources. For example, you might find ways to connect data in a supply chain data mart and customer relationship management data system, enabling new opportunities for product offerings to customers or ways to improve customer satisfaction.
Data Access: This layer enables data consumption, ensuring the correct permissions for certain equipment to comply with government regulations. Additionally, this layer helps display relevant data through the use of dashboards and other data visualization tools
Data Fabric vs. Data Virtualization
Data virtualization is one of the technologies that enables a data fabric approach. Instead of physically moving data from various on-premises and cloud sources using the ETL (extract, transform, load) standard, a data virtualization tool connects to the different sources, integrating only the necessary metadata and creating a layer of virtual data. This allows users to tap into source data in real time.
Conclusion
Data fabrics are still in the early stages of adoption, but their data integration capabilities help businesses with data discovery, enabling them to take on a variety of use cases. While the use cases that a data fabric can handle may not be too different from other data products, it differs in the scope and scale that it can handle, as it eliminates data silos. By integrating multiple data sources, companies and their data scientists can create a comprehensive view of their customers, which has been particularly useful with banking customers.
Source Website:
No comments:
Post a Comment