Shift left to write data once, read as tables or streams

Hi, my name is Adam, and today I'm going to talk to you about Shift Left.

Shift Left is a rethink of how we circulate, share and manage data in our organizations.

It involves moving data preparation and cleaning work that organizations already do today deep in their analytics plane and we're moving that upstream.

We shift the work to the left closer to where the data is actually created to make it cheaper, easier and faster to use it all across your company.

But before we can talk about Shift Left in more detail, we need to look at how data tends to move around in a company today.

In particular, let's look at how data analysts, scientists and data engineers, also known as data practitioners, get and access data today.

To start with, the vast majority of data practitioners work in a dedicated analytics plane, often using something like a data lake or data warehouse.

But how do these practitioners get the data they need?

Since by and large it's upstream in the operational plane.

This is where the multi-hop architecture comes in.

In a multi-hop architecture, we extract, transform and load the data into the analytics plane.

This ETL data is loaded into what we call a data landing zone, and this is the start of the analytics journey, but it's the first hop in what we call a multi-hop architecture.

The medallion architecture is a popular example of the multi-hop architecture.

So I'll write that right there.

A medallion architecture has three levels of data quality represented by the medallions that you would expect to see in the Olympic Games.

Bronze, silver and gold.

Now the bronze layer is the landing zone.

This is where raw data and also data from our operational planes, Kafka topics, SAS endpoints or even just plain text files comes from.

It's predominantly unstructured or maybe semi-structured at best.

Data practitioners then add formal structure schemas, enrichment and filtering to fix that data, which is then written into the second medallion layer, which is the silver layer.

This is the job that will do the work to convert it from bronze into silver.

The silver layer is where you'll find well defined data.

Common data includes business entities, transactions and measurements.

For example, it may contain datasets representing in an e-commerce business, customers, sales, inventory, customer interactions, and all of these are well-formed schematized, standardized, and duplicated.

One way to think about the silver layer is that the silver layer provides these well-formed building blocks for you to then use to build more complex data sets.

Now this is the first hop in the multi-hop architecture where we create a copy of data.

This is the second hop between bronze and silver, and then we're going to move on to the third hop here, which is the gold layer.

The gold layer is where we deliver these business level and application aligned data sets.

These are purpose built to provide data for specific applications or use cases, projects, reports or even for data export.

Where the silver layer provides these building blocks, the gold layer provides the building where you take that data you cemented together with some additional logic and you create a data set for a specific use case.

It's also the third hop in our multi-hop architecture and the third copy of data.

So what are the problems with this architecture?

Well, the first is that it's slow and multi-hop architectures are most commonly implemented with periodically triggered batch processes.

So data needs to get from the source into the bronze before it can go from bronze to silver and silver to gold.

But each of these is typically run at a defined interval, say, every 15 minutes.

So if it takes 15 minutes to go from database to bronze, it's going to take another 15 to go from bronze to silver, because you can't run that job until you have the data in here.

So even if you dial this down to maybe one minute per hop, you're still going to have about 3 to 4 minutes before that data can end up in your gold layer.

The second problem is that it's expensive.

Each hop is yet another copy of data as well as the processing power to load it, process it and write it to the next stage in the multi-hop architecture.

Third, they tend to be brittle and they're brittle because different people, they're brittle because different people own the different stages of the workflow.

Source database devs here, data engineers here, maybe analytics engineers or data analysts here, maybe data scientist there depends on your company.

But very strong coordination is necessary to prevent breakages.

But in practice this tends to be really hard to do at scale.

A fourth problem is that you end up with similar yet different pipelines.

So I'm just going to put multi pipes and what causes this is it's really a consequence of the distributed ownership from end to end.

You have people down here that get burnt by breakages in their pipeline.

And what happens is that they, even though you really want to fix the pipelines that come earlier, sometimes it's easier just to build another pipeline, just build a pipeline that you own and load that data into your own slice of the medallion layer and you manage the whole thing end to end.

And this is a reaction to...this is the load.

This is a reaction to these breakages and these problems. This becomes more common when you have a larger company, especially when you don't necessarily know which teams own what and which data is coming from where.

So this leads to our fifth problem, which is, identifiable here, is you get these similar yet different data sets.

So I'm just going to write five in here, a similar yet different.

And the problem is that it becomes difficult to tell why are there similar yet different data sets in here.

Which is the source of truth, which one should I use?

Is this data set here still maintained, or is it a zombie data set that's still being regularly updated, but nobody's overseeing it any more?

And just like the multiple pipelines, similar yet different pipelines, you get similar yet different data sets as you get larger, as your company grows, as you have a larger organization, and it just becomes difficult to know what you should be doing and what you should be using.

The consequence of these similar yet different data sets can be difficult to detect, but often show up in the end results when a customer, for example, gets a report that says they owe, let's say, $1,000, but their dashboard shows that they only have $900 worth of engagement.

So now they're going to lose trust because they don't know which one to believe, and especially if we're providing them those numbers and they don't make sense, it can be very confusing and you run the risk of losing their business.

So even if you sort all of these out, all these five problems I mentioned, you still haven't provided anything from the analytics side that the operational plane can use.

All of the work you've done to extract data, make it available, restructure it, model it and so forth, this is all in the analytical plane.

So what we're looking to do is we're looking to Shift Left so that we can use this data for operations, analytics and everything in between.

At its core, Shift Left is about taking work that we're already doing, this work here, including bronze layer, all the way up to silver layer.

We'll talk a bit more about how this integrates in a sec.

We're going to take this work here and including this modeling and structuring, and we shift it to the left.

We're going to do the cleaning, the structuring and all of this extra work over here.

To address latency, we are going to take a stream first approach.

So I'm going to draw this across our dotted line barrier and this is where the stream goes.

And we're taking a stream first approach because it's fast.

We don't have to wait for a periodic job to trigger every 15 minutes, but we're using Kafka with sub second end to end latency.

Native event driven applications that are already in your operational plane, for example.

EDAs.

They already produce events to streams. All of the work that these EDAs do to structure and schematize the data, the events that they're writing to the stream, is already done entirely within this application.

It's already shifted to the left by default.

Now if you have data at rest in a database, so we have this application here, you can use connectors like a change data capture connector that tails the binary log or the right ahead log or whatever immutable log your database is using.

It tails that log and it creates well-structured events that you can then write to the stream.

These events will have a schema and that schema will represent the table in the database.

Now you can also choose to perform further operations on that using something like, say, Flink SQL.

Very common operations that you might do include de normalizing the relationships that you're getting from this database.

One of the reasons why you'd want to do that is to isolate the internal data model from the events that you're going to be sharing in your operational and your analytical plan.

If you'd like to learn more about some of these techniques as they go a bit beyond the scope of the details of this talk, check out the video description links below.

We'll have some more information there.

So we have a stream and we can be populating it perhaps by CDC with Flink or perhaps by event-driven architectures.

But what about tables?

Right.

This whole multi-hop architecture is predicated on tables.

So to bridge this stream to table gap, we're going to rely on something that we call Tableflow.

Tableflow automatically converts your stream into a table, in this case, an Apache Iceberg table.

Anything you write to the stream is automatically appended to the Iceberg table.

So you get that table representation for free.

And what's great about Iceberg tables is that they're readable by any processing engine that supports Iceberg.

This includes things like Trino, Flink, Presto, SPARK, and even SaaS solutions like Databricks or Snowflake, for example.

By shifting the data clean up and structuring that you're already doing downstream and to the left you unlock both streams and tables for operations, analytics and everything in between.

As a data practitioner, you can access the data here in the iceberg table with the same level of rigor, quality and service level agreements that operations depends on via their stream.

You simply plug the data into your silver layer here, and since the data is already clean, structured and well formatted, it slots right into our silver layer, ready for use for querying and generating these gold level data sets.

And while I'm only showing you this data plugged into the existing data lake, you can plug it in anywhere.

Like I mentioned, you can plug it into any of these processing engines here, but you could also plug it into other parallel data solutions.

Perhaps you have a series of different data warehouses, maybe you have different data lakes, for example, for different isolation purposes.

But this pattern becomes very powerful for larger businesses where there's different organizations who might be very deeply entrenched in competing tech stacks.

Instead of having to figure out how do we unify all these different tech stacks and how do we make sure everyone's using the same thing, what you can do is you can take the data and plug it into where you are today.

There's no ETLs, there's no data ingress or egress costs, no duplicated data sets, and no similar yet different data.

A major benefit of shifting left is that you get a simpler and more flexible data architecture.

It also unlocks some powerful capabilities that aren't present in your typical copy and paste multi-hop medallion architecture.

For one, we can manage streams and table evolution together in a single logical entity, validating that evolutions to our stream don't break the table guarantees that we're providing.

And because we have shifted left out of this data analytics space, we can integrate these tests into our deployment pipelines for our EDA processes or for our change data capture mechanisms. In short, we can prevent these breakages that occur, these brittle breakages before code goes into production, instead of detecting it somewhere downstream here,

after we've already copied it and maybe passed it on a few hops.

And since your table is derived from the stream, you only have to fix it in one place.

Whatever you write to the stream will propagate to the table.

Streaming applications, perhaps other EDA applications in here, these will automatically get the updates propagated to them, they'll of course have to figure out what those updates mean and how to correct based on that.

Now for tables, we've plugged it into our existing data lake.

Tables you're still going to have to identify which jobs have been affected and which ones to rerun.

And last but not least, Shift Left is not all or nothing.

In fact, it's incredibly modular and incremental.

You selectively choose which of these workloads, these silver layer data sets that you want to shift to the left.

And by shifting those to the left, you can then plug them back in, test your code in here, test your jobs, make sure they work, make sure you're getting the same expectations, and then you can swap over and deprecate the old ETL pipelines that you already have.

That's it for today.

If you like this video, I urge you to like and subscribe to our channel.

I'll be releasing more light board videos in the future, so I hope that you will stay tuned for more on event-driven architectures, streams, tables and how to build a healthy data ecosystem.