Why point-to-point integrations suck

Our Bots do a lot of pulling and pushing data around, and there’s one rule we learned pretty quickly:

Point-to-point (P2P) integrations suck.

For a while we thought it was because we just weren’t smart enough, or good enough, or gosh darn it, people didn’t like us. After some Bot-assisted therapy (don’t ask), though, we realized that it really wasn’t us at all. So, here’s a cheat sheet about why your current P2P integration sucks, and some things to make them not do so:

Integrations don’t scale

If you’re integrating two systems (A+B), you have one mapping to think about — between A and B.

If you’re integrating three systems, you now have (A+B)(A+C)(B+C) … three mappings to think about.

If you’re integrating four systems, you have (A+B)(A+C)(A+D)(B+C)(B+D)(C+D) … six mappings to think about.

And not just think about, but execute and implement. Each mapping has its own order of operations, its own idiosyncrasies, its own kludges and workarounds.  You get the picture. The more things you want to use, the harder it is to actually get it all working together.  And, sorry to crush your dreams, but there is no such thing as a one-size-fits-all systems these days … you’re going to be using a bunch as you grow.

Technology changes

Four weeks and thousands of dollars into a custom P2P integration, you may start to realize that one of the technologies you’re trying to integrate was old 10 years ago. And that the integration you’re building will be completely irrelevant if you ever want to upgrade to what kids these days are using. Even if the company is still around, if they’re good they’ll be upgrading how you’re supposed to integrate. Ever hear of code debt? For your clever solution today, you’ll be paying for the next five years if you want to do anything remotely special.

Staff changes

Got any old integrations sitting around?  Ones that your brilliant predecessor went through weeks or months of hard thinking to put together and execute on …. only to go on to an astounding acrobat career?  And that brilliance is buried somewhere in a 15-tab Excel document that was written in 1997?

Yeah, we’ve seen that a lot.  The reality is that if you connect two of your systems together and then leave, your successor is probably going to prefer a different system, or at the very least want to advance the company goals by upgrading the integration. Then they’ll have to start from scratch merely to figure out why you did what you did.

Costly in terms of time and money — and really kind of frustrating work.

Business Logic soup?

Where did I put that business rule again?

If you embody any logic in a P2P connection, now you’ve chained your firm to a rule that can’t be modified without the costly aid of IT. Heck, maybe it can’t even be found. Maybe it’ll be completely forgotten, and just lurk like a booby trap waiting to clobber some future mission-critical task whose authors had no idea that for some reason it made perfect sense to a Pearl Jam-era project team to exclude from the sync all AOL users in Vermont.

Got it … What’s the Alternative?

The alternative is, a warehouse — a database that sits at the center of the technology you use, but is COMPLETELY INDEPENDENT from them.

No, you can’t use Salesforce, or your CRM, or another of your technology products as the warehouse … because then you’ll not only invite all the same P2P problems, you’ll also be be assuring that you can never again change that CRM, not ever (ACK!!!).

When you warehouse, you connect your many platforms independently, in a standardized way, to avoid all the crap you get with P2P integrations.  The warehouse has to have flexible and standard schemas, and easy ways (like SQL) to get data in and out with zero drama. Otherwise it’s not a warehouse: it’s really just another platform you’re fighting.

Why does warehousing work?

  1. It Scales — Each new technology only requires a single integration point; if you have four technologies you only need to connect them to the warehouse, and not to all the other systems you’re using. AND, psst, some people have already done the heavy lifting there.
  2. Technology can be hot-swapped — Because each technology is now only connecting to the warehouse, you can upgrade or change each system you use, INDEPENDENT of the other systems.  The warehouse takes care of the connections and business logic you have, in a transparent, centralized, easy-to-change way.  And if your new staff wants to use a new set of technology, all you have to do is figure out how to connect it to the warehouse, NOT to everything else your organization uses.
  3. Central Business Logic — Please stop hunting in old configuration files for how a mapping is set up.  That hurts all of us.  Keeping that information OUT of your technology configurations will make it easier to upgrade, easier to change, and easier to understand what’s going on with your data.

So, in the immortal words of one of my teammates — don’t suck.  Warehouse.