Lesson 1: Every API is Different
While there are many best practices for designing APIs, there’s no real standard that everyone follows. Each provider makes different choices: how to handle authentication, how to structure responses, how to deal with errors or limits.
Some APIs return neat JSON objects, others provide CSV files, and some just a link to another endpoint. Authentication is just as fragmented: one API might use OAuth, another an API key, and another something entirely custom.
Even the OpenAPI specification doesn’t solve this problem. For authentication, for example, it doesn’t enforce a standard, it simply lists all the possible authentication methods a provider might choose from. Thanks, but no thanks.
This means that even if you’ve worked with dozens of integrations before, every new API still feels like learning a new dialect.
Lesson 2: Docs Don’t Tell the Whole Story
Documentation is supposed to make integrations easier, but in practice it rarely gives you the full picture. Often, it’s written by people who know the system so well that they skip over things they consider obvious because mentioned in another page, or don’t even realize what an outsider would find confusing. That means missing examples, unclear explanations, or assumptions that only make sense if you built the API yourself.
And even when the docs look solid, surprises still happen. Some are just gaps in the documentation, but others are actual bugs: endpoints that behave differently than described, fields that don’t return what you expect, or rate limits that appear out of nowhere.
Lesson 3: Business Views Aren’t API Endpoints
From the outside, it sounds simple: “Can you just pull the same data we see in the dashboard?” The problem is that APIs are almost never a 1:1 mirror of the product interface. Dashboards are designed for humans, APIs are designed for machines; the way they structure, group, and expose data can be completely different.
That means the person doing the integration has to play translator. They need to understand what the business is asking for in terms of the UI, then figure out which API endpoints hold the right pieces of information, how they relate to one another, and how to stitch them together to get to the same business view.
The real challenge isn’t just technical, it’s about bridging perspectives. Businesses think in dashboards, filters, and metrics; APIs think in endpoints, objects, and payloads. Making the two meet requires patience, communication, and a lot of detective work.
Lesson 4: Transparency Beats Black Boxes
Many off‑the‑shelf integration tools look appealing: they promise quick setup and pre‑built connectors. But when something breaks—and with APIs, something always breaks—you quickly discover the limits of a black box. Errors appear without context, data goes missing, and you’re left waiting on vendor support to dig into issues you can’t see.
With open‑source tools and custom integrations, we had full visibility. We could inspect every request, log raw responses, and adapt logic immediately when an API changed its behavior. That transparency meant faster fixes, fewer surprises, and more control over the final outcome.
Lesson 5: Pipelines Change, Good Practices Last Forever
The real value of a data ingestion project isn’t just in getting a pipeline to work today, it’s in the reusable frameworks and practices we leave behind. While introducing dlt, we used each integration as a chance to implement version control, Docker, CI/CD pipelines, Infrastructure as Code, data quality tests, proper scheduling, and data lineage.
Those frameworks don’t stop at ingestion. The same foundations can support dbt transformations, power machine learning models, or run other critical workflows. A pipeline may solve today’s problem, but the practices built around it create long‑term resilience and adaptability.
List of pipelines
Over the past year, we’ve put these lessons into practice by building dlt sources and pipelines for more than 20 different tools. Each integration brought its own quirks, but also an opportunity to raise the maturity of their data setup beyond a single pipeline.
Here’s a snapshot of the systems we’ve worked with:
| Category | Tool | Description |
|---|---|---|
| Advertising | Amazon Ads | Advertising platform for promoting products across Amazon and partner sites. |
| Advertising | AdBook+ | Ad campaign management and reporting platform, often used in media agencies. |
| Channel Management | iasset | Platform for managing sales channels, renewals, and quoting in B2B environments. |
| Collaboration | Smartsheet | Collaborative work management platform based on spreadsheets. |
| Communication / Events | Zoom | Video conferencing and webinar platform for online meetings and virtual events. |
| CRM / Marketing | HubSpot | Marketing automation and CRM platform for inbound marketing and sales. |
| CRM / Marketing | Lytics | Customer Data Platform (CDP) for audience segmentation and personalization. |
| CRM / Sales | Salesforce | Leading CRM platform for managing customer relationships, sales, and marketing. |
| Data Collection / Forms | Gravity Forms | WordPress plugin for form creation and data collection. |
| Data Collection / Forms | Unbounce | Landing page and form builder focused on marketing conversion. |
| E-commerce | Shopify | Popular platform for building and managing online stores. |
| Email Marketing | PostUp | Email campaign platform focused on publishing and media companies. |
| ERP | Oracle Netsuite | Cloud-based ERP system for finance, operations, and business management. |
| Forms / CMS | Airtable | Flexible database and collaboration tool combining spreadsheet and database features. |
| Marketing Analytics | Adverity | Data integration and analytics platform for marketing teams. |
| Observability | Datadog | Monitoring and analytics platform for infrastructure and applications. |
| Payments / POS | Square | Point-of-sale and payment processing platform for small businesses. |
| POS | Revel (revelup) | iPad-based point-of-sale system for restaurants and retailers. |
| Project Management | ClickUp | All-in-one productivity tool for managing tasks, docs, and workflows. |
| Promotion / Loyalty | Talon.One | Rule-based engine for managing discounts, promotions, and loyalty programs. |
| Recruiting | Greenhouse | Recruiting platform offering ATS and onboarding tools. |
| Recruiting | Workable | Applicant tracking system (ATS) for hiring and recruitment. |
| SEO / Webmaster Tools | Bing Webmaster | SEO and indexing tool for managing website presence on Bing. |
| Support / Ticketing | Freshdesk | Customer support platform with ticketing, live chat, and automation. |
| Support / Ticketing | Zendesk | Customer service platform for ticketing, chat, and knowledge base. |
| Wealth Management | QPLIX | Platform to manage financial assets. |
| Webinar / Events | on24 | Webcast and webinar platform for virtual events and audience engagement. |
Leave a Reply