Lorem ipsum dolor sit . Proin gravida nibh vel vealiquete sollicitudin, lorem quis bibendum auctonisilin sequat. Nam nec tellus a odio tincidunt auctor ornare.

Stay Connected & Follow us

What are you looking for?

Simply enter your keyword and we will help you find what you need.
PBT GroupPull vs. Push based source data extraction

Pull vs. Push based source data extraction

The Company

A leader in innovative banking, is continuously searching for better ways to optimize internal business processes to improve client experience and insights. The client growth is mostly attributed to a combination of continuous improvement and innovative product offerings. Maintaining such growth requires bleeding edge solutions to gain insights into external and internal information.

The Business Challenge

Defining the most appropriate extraction mechanism considering the company specific environment/s and circumstances.

The Solution

A standardised pattern, based on Microsoft SQL Server Integration Services (SSIS), with an Interface Control Document acting as contract / agreement between source and target system owners to govern extract frequency, scheduling, security, etc. The SSIS packages connect to source system database, extract source database tables (untransformed) and to the Data Warehouse Data Lake.  These patterns are being optimized to use SQL Server Bulk Copy to generate files to support an Enterprise Data Lake implementation, with future implementations intended to facilitate the streaming of data.

The Value Proposition

Facilitates improved sharing / democratisation of data:  Data belongs to the organisation, not the source system. Improves compliance with Data Lake principles:  All data is extracted in rawest form. Reduced time-to-value: Eliminates dependency on availability of source system resources to develop prepared extracts. Reduces impact/overhead on source systems:  Tables are extracted without expensive joins and/or transformations. Standardised implementation pattern improves consistency and reliability, as opposed to distinct implementations per source. Facilitates improved, more holistic understanding and visibility of source system data and data modification behaviour. Facilitates improved data management best practices, e.g. modelling standards applied in the source systems.

Technology

MS_SQL_Server
Top