Compliance and business insight – delivered by data lineage

The term automation means many things to many people – Philip believed that its definitions in the market are misguided.

“Unfortunately, automation takes time, it takes understanding, and it takes a lot of effort. It really comes down to the ‘why’ – what’s the value that we’re generating? The state of automation that most people are looking to achieve is if a system changes, tell me about it. The problem in global investment banks and complex organisations is that if you’re hearing about a change after it’s happened in production, you’ve got a bigger problem in the organisation. We need to move away from this reactive approach to a more proactive lineage approach – we should be thinking about planning and execution, with the automation as the validation that what we’ve done is what we plan to do.”

The creation and retention of data has increased exponentially in recent years, with a growing remote workforce catalysing the production and addition of more cloud applications to companies’ data landscapes. Philip made the point that the journey to the cloud is like any other transformational change companies enact.

“You need to know where you’re starting from and where you want to end up – an organisation looks at their current state on-premise and plans to move these elements into the cloud, and typically, people don’t pick up their whole organisation and start moving all the data into the cloud at once; it’s this journey that takes time and part of that is designing a cloud migration strategy to ensure you don’t breach regulations like cross-border data sharing, or that you sufficiently structure the migration so you’re not moving huge volumes of data up to the cloud and then back down into on-prem, generating huge costs. Lineage allows us to simulate what that future state of an organisation looks like while remaining in the safety of a confined space.”

Before the session ended, the panel discussed the relevance on AI and Machine learning within metadata management – Philip observed that much of what the vendors are pushing in the market is overhyped mainly because we work in metadata, and not data. The record sets for metadata management are so shallow that you can never train an AI to really deal with it. When you’re working with the actual underlying data and trying to infer lineage based on that data, there’s a much better case for using an AI or an ML – it’s important to understand the distinction between the two.