Surfing with Zapier, Part 2: The Integration Wave
Over the last 8 months, I have been schooled hard in the differences between a “data migration” system and an “integration” between two systems. I didn’t learn this in a bootcamp. I didn’t learn it in school. I didn’t even learn it from YouTube. I was given the opportunity to solve a problem at work: We need to use two different systems for different reasons, but we want the reality of what is happening in one system to be reflected in the other.
More specifically: we use a system called Arlo to let people know when our trainings are taking place and give participants the platform to register and pay for their participation in the training. We use Salesforce as our main database of information - a place from which we review the data we have about our trainings and use that data as a largely contributing factor in our decision-making. So we want the data that is collected and stored in Arlo - about the trainings we are offering and the people who register for those trainings - to ALSO be in Salesforce. But we want an exact representation of the data from the first system (the system of origin or initiation) reflected in the second.
This is a familiar story to many people. It can be as straightforward as having two spreadsheets and wanting them to have the same data reflected in them both, or in fact having two tabs within a single spreadsheet exchanging and matching data. Or it can be complex, as in the example I had at work, needing to get two distinct systems to talk to each other in the effort for some of the data in each to be mirror images in the other.
So what makes up a “data migration” system and what makes up an “integration”? Somehow before I started working on a solution to our needs, I didn’t do the simple task of Googling the difference. Having done that now just for this writing, my favorite breakdown came from the enjoyably titled Noisy Little Monkey blog. To summarize:
Data migration is to permanently move data from one source to another.
Data integration (or perhaps better named as Systems integration) is “to creat[e] links between data sources and then synchron[ize] the exchange of data between them.”
Another way to look at this is to say that data migration is a finite game - it has a beginning and an end. Whereas data integration is an infinite game - an ongoing exchange between systems that must iterate and improve as business processes and technological capabilities change. I’ve been reading James Carse’s Finite and Infinite Games and it is really influencing my outlook on work and life. In the book, Carse notes that we can play innumerable finite games within an infinite game - and that sounds about right to me for the relationship of data migration and data integration. With that in mind, we can view data integration as many, many data migrations happening over time - each migration works within a set of boundaries that we establish in order for our computers to consistently move data in the way that we want and expect them to. But the integration can and must play with these boundaries - finding ways to improve them so that data can keep moving, so that the game can continue on indefinitely, reacting and adapting to the changes in environment - such as our business processes, the people using the data and the systems, the technological advances in the world, etc.
OK enough philosophy, let’s get to the surfing.
So get this - Arlo already has an integration with Salesforce. Yeah, that’s right. Some folx with way more knowledge about this stuff and ability to use programming languages to write code have some stuff set up in the background already that creates a link between Arlo and Salesforce and facilitates the ongoing exchange of data (data migrations) between the two systems. UNFORTUNATELY, the rules of the individual data migrations within the integration don’t work for us. They just don’t work. So we had to DIY it. Some might call it re-inventing the wheel, but the existing wheels just don’t fit our car, so we needed to make our own.
We decided to use Zapier as a low-code option that we were familiar with and knew from some initial investigation would likely be able to solve for our needs. I say “low-code” because I did have to use some fancy code-like knowledge within Zapier to really get things to work for us. In Zapier we set up the boundaries and rules for the many, many, many data migrations that would happen to move data from Arlo to Salesforce so that the data we need in both systems would reflect each other. We turned it all on and BAM there was data moving at a mile a minute. Mini-migrations happen any time a participant registers for a training or their registration changes, which could be hundreds in a single day. Both systems are oceans of data, and we were creating waves between the two of them. But to really know if it was all working as expected, we needed to learn how to surf those waves.
And to surf, first we had to build a surfboard. Something that we could stand on and look around, see what waves were coming, see where waves were going - something to let us ride on top of all this movement of data. There are a lot of digital materials to choose from when building a data surfboard and we decided to start out with something we know really well how to build with - spreadsheets. This spreadsheet is a game of data migration in and of itself. At any given point in time, we run a report from Salesforce and get that data into the spreadsheet. At that same time we run a report from Arlo and get that data into the spreadsheet. There are tabs in the spreadsheet for data from each system, and then more tabs that help us compare the data between the two systems - the goal being to see a match across all data points in our two systems. This data surfboard allows us to stand on top of the movement of data - the wave - and ride it. Waves are always moving, always changing. Even though we are putting “static” or unmoving data from each system into the spreadsheet to compare against each other, when we see discrepancies or have questions, we ride around on the data, looking for patterns, openings, places to do cool tricks, and more. We change the data and the rules about the data movement in real-time to improve on the ongoing integration.
With the data surfboard in place, it was time to learn how to surf. As with actual surfing, it takes time, experience, and deliberate practice for some of the motions and reactions to become second nature - to get good enough at one level, then push yourself to go to the next level, where you suck at first and then get better with more time, experience, and deliberate practice. At first I got hit in the face with the waves of data and knocked off the board - TONS of things were not as expected. I thought the data would go this way, but it went that way. I thought I could turn my board and attention to one wave, but another came and knocked me down. Data wasn’t matching between the systems. We had to ride each one out, figuring out what needed to change in our data migration rules but also what we would need to do with our surfboard (how to analyze the data, recognize patterns, know where to turn and what to do when we found things didn’t match our expectations) in order to stay on top of the ongoing movement of data within a changing environment.
Was that all a little too metaphorical? If you want to see the actual surfboard and watch me do some surfing, I made a video showing it in action (below).
Something I learned from all this is a lesson I continue to be taught over and over and over again. While surfing can be a solitary sport (you are on that board by yourself), the work and the games are so much better when you do them with and around a crew. That’s an invitation to YOU to come and work and play with me about data migrations, data integrations, surfing, and anything else mission-driven data & tech!!
