By Rohit Kuruppath, Cox Communications Inc
Continuous Integration and Continuous Deployment (CI/CD) procedures have been extensively used across software development life cycles specifically around application development. The CI/CD implementation around data engineering tools was challenging within the UI centric tool stack. The delivery requirements were not agile without the need of paired programming and common code base. In most cases, the versioning capabilities of the data tools itself was utilized without utilizing the enterprise CI/CD capabilities.
This paper discusses how data engineers can take advantage of the new flexibility that comes with the deployment of software across different data and integration tools, more quickly and securely. It will provide an overview of CI/CD, with a specific focus on how it is being applied across on-premises, cloud and hybrid data environments.
Over the years, real-time data integration and automation have been highly critical for all organizations to meet their analytic needs and providing better insights and capabilities to customers and business leaders. With the huge amount of data to be collected and processed from different systems across the network, before building analytics models around it, there is a pressing need to adopt DevOps guidelines to deliver quality data outputs.
This paper will describe how Cox Communications leveraged the existing CI/CD tool capabilities and expanded upon the configuration tools to support Agile DevOps delivery for building a hybrid data lake. Data Engineers and analysts could leverage a stable pipeline flow and common set of automation tools for delivering the solutions without much onboarding and transitioning into these CI/CD processes and tools.