Airbyte Blog

I'l also give a shout-out to Airbyte (https://airbyte.com/), with which I've had some limited success with integrating Salesforce to a local database. The particular pull for Airbyte is that we can self-host the open source version, rather than pay Fivetran a significant sum to do this for us. It's an immature tool, so I don't yet know that I can claim we've spent _less_ than Fivetran on the additional engineering and ops time, but it feels like it has potential to do so once stabilized.

- https://hevodata.com/ - https://airbyte.com/ - https://www.stitchdata.com/ Unfortunately none of them are perfect as your data scales, each have up and downsides...

for what its worth airbyte's connectors themselves are MIT https://airbyte.com/blog/a-new-license-to-future-proof-the-commoditiza… arguably this is "much of it" rather than ELv2

#2. https://zapier.com #3. https://airbyte.com #4. https://usebracket.com

I love the article, and it gave me the essential pointers to start with. I loved it so much that it inspired me to write a piece focusing on python vs. rust in data engineering. I leave the article here, in case of interest: https://airbyte.com/blog/rust-for-data-engineering

Being GDPR compliant may require obfuscation or removal of some data before it gets loaded into a data warehouse. This can be achieved with an EtLT (Extract, “tweak”, Load, Transform) approach to data integration – in which a very light “tweak” (small “t”) transformation is done on the data before it reaches the destination. This is discussed in detail in the following article: https://airbyte.com/blog/etlt-gdpr-compliance

Looks super polished. Will have to give it a spin. what we use right now is Airbyte https://airbyte.com/ - super exhaustive in connectors but configuring some connectors like SFDC was very painful

I think it is great stuff you are building, especially for data integration of external sources that don’t offers webhooks or other pub/sub integration options. But is Airbyte really Open Source? I’m not questioning the reasons for moving to a source available license due to how AWS treated open source, but I think it is a bit of false marketing of Airbyte referring to it as open source. Sure, some pieces are MIT but the core is Elastic License 2.0 from what I can see. https://airbyte.io/blog/a-new-license-to-future-proof-the-commoditizat…

Airbyte | Engineering & Product | Remote | Full-Time | https://airbyte.io/ Airbyte is the new open-source data integration platform that runs in the safety of your cloud and syncs data from applications, APIs & databases to data warehouses, lakes and other destinations. In addition to offering a quickly increasing list of maintenance-free connectors, it makes building your own connectors trivial. Airbyte’s ambition is to make data integrations a commodity. Airbyte has multiple engineering and product openings across Platform & Connectors. We keep a very transparent handbook on culture, our roadmap, and much more! https://handbook.airbyte.io/ Jobs: https://jobs.lever.co/airbyte

I found another tool which uses Singer: https://airbyte.io Looks very promising, and was fast to get set up

They have the right idea, it's a real need and a major opportunity. Airbyte[1] is a few steps ahead. I also expect that the Airflow community will also wade into these waters at some point. 1. https://airbyte.io/

Sure! Those are actually great questions! Regarding monetization, you can see more details here: https://airbyte.io/pricing We consider 2 monetization approaches: Open core (connectors staying open-source forever) with premium features as: hosting & management (cloud-based control panel without access to your data plan), and enterprise features (privacy compliance, SSO, user access management, etc) What we call “Powered by Airbyte” where we enable you to offer integrations to your own customers using our API Regarding our runway, with the team as is, mid 2025. We intend to grow the team though, given the adoption growth we have. We’ve already been approached for a Series-A, but will consider it in mid 2022. Regarding self-sustainability, do you mean financially? Possibly at the end of 2023 or 2024. How does that sound to you? Genuinely curious.

Checkout Airbyte and singer.io for data movement. They're tailored for moving offsite data local, but also have support for moving data between different database vendors. I'd recommend starting with Airbyte. For transformations I'm a sucker for dbt-core, which executes transformations as SQL against your data warehouse vs. a separate process. This keeps everything simple and (generally) snappy. It uses the Python Jinja2 template library to compose SQL templates, and the process that parses templates to SQL also implicitly creates an execution DAG so you could have hundreds of transformations running in the exact order they need to. There are also many commercial offerings (Fivetran, Dataform) that don't tick self-hosted, but could be easier to deploy if getting internal hardware support is an issue. https://github.com/airbytehq/airbyte https://airbyte.com/connectors https://www.singer.io/#taps https://github.com/dbt-labs/dbt-core

Hi there Michael! We continue to love nomad. SQL is the C ABI of querying for sure. BI tools will never adapt to use Convex directly, and nor should they. So... yes, Convex actually had a prototype SQL adapter for the read side of things back in the early few months when we were figuring things out. Convex's read semantics are very compatible with SQL. We've kept this adapter on ice in part because of point #3 in the article -- we don't want to overpromise things which are a bad idea. Meaning, if we exposed SQL on the thing as-is, this would presumably be for more analytical type queries involving patterns normal Convex queries can't express. Right now that would be a Bad Idea because your website would slow down just like every other database system allows you to. So the current recommended practice is use our Airbyte Egress connector (https://airbyte.com/connectors/convex) and get yourself into an offline Clickhouse/MySQL/Snowflake whatever and jam SQL over there to your heart's content. That's basically what we do. We may one day abstract this away by embedding some sort of mirrored column store sql thing (maybe DuckDB based? who knows) so you can do your analytical work without impact on your OLTP or integrating 3 more systems. But the team hasn't invested in that yet and probably won't for some time.

It is of course true... it is well known that SQL Server scales to department level, but Oracle scales to company level. This is true inside Microsoft and Oracle as well. Inside Microsoft, they have a bug database per division but Oracle has a single database for the entire company. Ask people who work at those companies. See also scalability sections in these artcles: https://airbyte.com/data-engineering-resources/oracle-vs-sql-server https://futuramo.com/blog/oracle-vs-sql-server-head-to-head-comparison…

Since Pandas 2, Apache Arrow replaced NumPy as the backend for Pandas. Arrow is also used by Polars, DuckDB, Ibis, the list goes on. https://arrow.apache.org/overview/ Apache Arrow solves most discussed problems, such as improving speed, interoperability, and data types, especially for strings. For example, the new string[pyarrow] column type is around 3.5 times more efficient. [...] The significant achievement here is zero-copy data access, mapping complex tables to memory to make accessing one terabyte of data on disk as fast and easy as one megabyte. https://airbyte.com/blog/pandas-2-0-ecosystem-arrow-polars-duckdb

Neither, just an interesting factoid that highlights just how zany the VC landscape was in the MDS space just a few years ago. https://airbyte.com/blog/the-deck-we-used-to-raise-our-150m-series-b

https://fivetran.com https://airbyte.com https://hightouch.com https://usebracket.com https://retool.com

Makes sense, thanks for the reply! Pricing may be steep compared to diy but as a comparison, Airbyte Cloud seems to charge $10/GB or $15/1M rows transferred[0] which seems like it would be even more expensive. I'm doing some more research but will keep ya'll in mind and reach out if I have more questions! [0] https://airbyte.com/pricing

How do you know what the status of a connector is? I just see https://airbyte.com/connectors and have no obvious indication of status.

do you already have an ETL pipeline to BQ? You could use Airbyte[0] for pg to bq. It's pretty heavy if this is your only source, but should work well if you need to query the pg data in bq frequently. 0 - https://airbyte.com/connections/PostgreSQL-to-BigQuery

Yes, Airbyte is using Temporal. Here is a blog post they wrote a few weeks ago that goes into more detail about it: https://airbyte.com/blog/scale-workflow-orchestration-with-temporal

Sorry, wrong link, and I couldn't delete the post. I reposted with the correct link to: https://airbyte.com/blog/sql-vs-python-data-analysis

I think its still "coming soon" https://airbyte.com/blog/airbyte-strategy-to-commoditize-all-data-inte…

Also confused here. Slide 7 [0] says seed funding of $1.8M, but Airbyte is also calling the most recent round a $5M seed. Implies they will call the first round Pre-Seed retroactively, rather than calling the second round Seed+. It's all grey at this stage either way! [0]: https://airbyte.io/articles/our-story/the-deck-we-used-to-raise-our-se…

Interesting to see the competitive analysis with Fivetran in the article but then see almost identical copies of infographics used between their site and Fivetran's. Airbyte: https://airbyte.io/wp-content/uploads/2021/03/Airbyte-Seed-Deck-3-1024… Fivetran: https://images.cms.fivetran.com/mgtdf72hs0mx/6qYtmEEotXqScarsRwIFpA/ec…

As mentioned by cgardens in reply to another question, you can read through John's (one of the founders of Airbyte) article on the future of ETL/ELT: https://airbyte.io/articles/data-engineering-thoughts/why-the-future-o… It provides some solid arguments that should help your discussion with your managers.

If you are interested, John (one of the co-founders) wrote an article about how we are imagining ELT evolving. https://airbyte.io/articles/data-engineering-thoughts/why-the-future-o…

I saw this project a few months ago and I'm really surprised how fast they've been adding new connectors [1]. I was initially a little doubtful about them implementing their own protocol [2] instead of using Singer [3], already OOS, but it looks like it panned out for them. In fact I've been using Stitch Data for a couple of years and I've find myself "fighting" with their support team about bugs I think we're Singer related, mainly with DDL commands. [1] https://docs.airbyte.io/changelog/connectors [2] https://airbyte.io/articles/data-engineering-thoughts/why-you-should-n… [3] https://www.singer.io/

I am one of Airbyte's founder. The initial version of was actually fully based on Singer and this is when we realize it wouldn't be possible for us to depend on it. Amongst the main reasons: Singer seems to have been abandoned by StitchData (after they got acquired by Talend), the quality of the connectors is too unpredictable, Singer connectors are not usable outside the box. We would have preferred to use an existing standard if one already existed. It was a tough decision for us to create something from scratch but now we are very satisfied with the decision. It is way easier for the community and for us to build connectors that meet quality standards and we can make it MIT so the community can have control on the evolution of the protocol. We actually wrote a few articles about it: https://docs.airbyte.io/faq/differences-with.../singer-vs-airbyte https://airbyte.io/articles/data-engineering-thoughts/airbyte-vs-singe…