I
Insight Data Engineering Blog
insightdataengineering.com · Company blog · on HN since 2014
Insight Data Engineering's blog comes from a fellowship program, with posts on Spark, Flink, Elasticsearch, Cassandra and Kafka projects.
What HN saysHN commenters discuss the fellowship's admissions, such as who can apply, and one advises moving into data engineering before machine learning.
21HN people shared it
6linked it in comments
33stories submitted
706story points
9 years agolast shared
2014people per year2026
Most-shared posts
- 1Insight Data Engineering Fellows Program3 people · 53 points · first shared 2014 · 20 comments on HN
- 2Working with Apache Spark: Highlights from projects built in three weeks2 people · 26 points · first shared 2015 · 6 comments on HN
- 3MapMyCab: How I Chose a Data Engineering Project2 people · 13 points · first shared 2015 · discussion on HN
- 4Planting Quadtrees for Apache Flink2 people · 8 points · first shared 2016 · discussion on HN
- 5Anatomy of an Elasticsearch Cluster: Part III2 people · 4 points · first shared 2016 · discussion on HN
- 6Anatomy of an Elasticsearch Cluster: Part II2 people · 4 points · first shared 2016 · discussion on HN
- 7Building a Forex Trading Platform Using Kafka, Storm and Cassandra1 people · 139 points · first shared 2014 · 39 comments on HN
- 8Cassandra: Daughter of Dynamo and BigTable1 people · 81 points · first shared 2015 · 47 comments on HN
- 9Anatomy of an Elasticsearch Cluster: Part I1 people · 67 points · first shared 2016 · 9 comments on HN
- 10Streaming Search on Tweets: Storm, Elasticsearch, and Redis1 people · 59 points · first shared 2016 · 7 comments on HN
- 11The New Data Engineering Ecosystem: Trends and Rising Stars1 people · 50 points · first shared 2015 · 8 comments on HN
- 12The Data Engineering Ecosystem: An Interactive Map1 people · 47 points · first shared 2015 · 8 comments on HN
- 13Spinning Up a Spark Cluster on Spot Instances: Step by Step1 people · 37 points · first shared 2015 · 10 comments on HN
- 14Computing shortest distances incrementally with Spark1 people · 26 points · first shared 2015 · 10 comments on HN
- 15Insight Data Engineering Fellows Program applications open for September 20141 people · 22 points · first shared 2014 · discussion on HN
What HN people wrote when they linked it
vishalzone2002 · 2014-09-12
Consider doing a soft move rather than a hard move. I will explain what I mean by that.
Instead of spending your time learning ML I would suggest first make a move to data engineering where you use more of the big data technologies like hadoop, spark, python, pandas, etc. You can also look at some kind of a bootcamp or at the least the topics they cover is obviously helpful: http://insightdataengineering.com/
Some other tips :
- Follow data engineering and data science topics on quora.
- Once you feel confident try to rewrite your linkedin profile from data warehousing developer role to a data engineer role. Add big data technologies to your skill set.
- Try to get into an auction on hired.com
- It is false that MS does not use Hadoop. I have interviewed with teams that do but they are all in seattle. Try to move internally to teams like Azure, HDInsight.
- ETL, data modeling, SQL, designing data warehouses and pipelines are actually still the most useful skill to have. So hold on to that.
- One thing that I haven't done yet but really mean to is to get on Kaggle. You should too at some point.
ronaknnathani · 2016-10-07
Sorry about the broken links on the Data Engineering page. Here is the correct one to subscribe to the notification list. http://insightdataengineering.com/data-engineering-notification-list
javadi82 · 2015-06-24
I know this beats the very premise of your product - "personalized" guidance but thought I would ask anyway.
How open are you to sharing general guidelines (that are insightful and not already available everywhere) ?
It would also help if you could ask the people who are successfully placed to write their learnings along the process. Similar to how the Insight data engineering fellow and others at coding schools share what they learnt/built.
Eg: http://insightdataengineering.com/blog/mapmycab.html
jakek · 2014-03-28
The reason for this is that the Insight Data Science Fellows Program is designed for (and only accept applications from) PhDs/postdocs. The new stand-alone Data Engineering program (http://insightdataengineering.com) is open to all applicants, as long as they have good engineering/CS fundamentals, regardless of level of education or discipline.
syntaxing · 2016-10-07
I actually pretty interested in joining this program since I am near one of the locations! Do you guys accept other type of engineer (in my particular case Mechanical Engineering) with a background in programming?
Also side note, I think you guys probably just migrated to new domains but there seems to be a couple 404 links/pages on the site? The ones I was looking in particular are these two http://insightdataengineering.com/insightdataengineering.com/data-engi… http://insightdataengineering.com/insightdataengineering.com/data-engi…
Also, there's an cert error that pops up in Firefox for http://insightdata.ai
ddrum001 · 2016-07-01
We don't have a fixed syllabus since we don't operate like a school per se. Instead, the Fellows choose a project and have to make engineering tradeoffs about which tools to use. With that said, most of the past participants focused on distributed systems like Hadoop, Spark, Flink, Kafka, and NoSQL DBs. You can check out past projects on the blog (http://insightdataengineering.com/blog/) and the past Fellows are here (http://www.insightdataengineering.com/fellows.html)