Ayasdi Blog

AYASDI | http://www.ayasdi.com | Menlo Park, CA Ayasdi is a hot late stage startup in AI/machine learning/machine intelligence. Our core technology is Topological Data Analysis. Our Healthcare vertical app is making a big difference in care path management. Senior UI Engineer opening: http://app.jobvite.com/m?3ce6diww

Do you recommend any good primers on topology? I thought this (https://colah.github.io/posts/2014-03-NN-Manifolds-Topology/) was an interesting article and I see what looks like some great papers and videos available at http://www.ayasdi.com/approach/data-scientist/, but I don't know the difference between homotopy and homology (yet) :) . What kinds of infrastructure/tech do you think will have the most utility for topological data analysis in the near future? E.g., GPUs, Apache Spark, FPGAs, etc. Any thoughts on an Ayasdi public offering? I'd like to consider investing but I don't have millions of dollars (yet) :) . Thanks for your time.

Ayasdi | Menlo Park, CA | Full Time | REST API Engineer -- the pitch -- Ayasdi uses a pioneering approach to automatically discover insights from complex data. Founded in 2008 after a decade of research at Stanford, DARPA, and NSF, Ayasdi uses Topological Data Analysis (TDA) to unify best-of-breed machine learning approaches into a common framework without the need to write algorithms, queries or models. We’re building a platform to solve some of the world’s most complex data problems, are well funded by top-tier VCs and have a passionate team with a great culture. -- position info -- As a REST API engineer on the frontend team you will help architect one of the most important products Ayasdi has to offer. The API allows customers to automate common processes and integrate the core Ayasdi functionality with their existing tools, in addition to being the data layer that all of our web applications use. You would be the second person on our API team, so you will have the freedom and flexibility to make large architectural decisions as the company grows. -- requirements -- • Experience building and maintaining APIs used by demanding customers • Ability to write comprehensive unit and integration tests • Proficiency with the modern Java platform and patterns, including dependency injection, advanced annotation usage, and concurrency facilities • Experience navigating the Java library ecosystem, in particular outside of JEE • Familiarity with Jersey, Mockito, and code…

The most well-known practitioners of this sort of 'topological' approach are Ayasdi, they have some slick demos [1]. The general name for this idea is topological data analysis [2]. I replicated this particular experiment in WL, of course, because it's a 5-minute thing to do [3], and I could actually do the community detection the author alluded to. But I noticed that the correlation matrix itself is much more suggestive than the graph ends up being, with or without community detection. Take a look at the correlation matrix (note that MatrixPlot does some clever combination of rank and absolute value to get high dynamic range): http://imgur.com/WKn029o The tri-diagonal structure is because the original dataset is derived from the pixel counts from successive 4x4 tiles on NIST written-digit images [4]. Those 8x8 matrix of tiles is flattened onto the 64 random variables, so the large correlation with tiles on the left and right explain the 1-off-diagonal orange lines, the other two diagonals are offset by 8 and correspond the high correlation with the tiles above and below. That's the 'connectivity kernel' of a 2D manifold, so to speak. The curious squiggles in all the other blocks of this matrix are unusual. I don't know what's going on there. Maybe something interesting. [1] http://www.ayasdi.com/ [2] http://en.wikipedia.org/wiki/Topological_data_analysis [3] https://www.wolframcloud.com/objects/c7927909-448d-4502-9c1a-ff65243c1… [4]…

This is a really fascinating subject, and as of last year I had read through essentially all the major papers in the area, implemented the central algorithms, and given a few more-or-less casual talks on it. There are two things that strike me as very different about topological data analysis. The first and primary thing is that it's not a silver bullet by any means. It's not like most machine learning and data mining, where you pick some parameters and wham-bam-thank-you-ma'am you have 80% accuracy. No, this kind of analysis gives you qualitative features of your data, and after all the topology is done there's still years of work before you arrive at a mathematical model that admits an algorithm that has accuracy comparable to the state of the art. An interesting case study in this is Carlsson's work on texture classification. They ran their 3x3 image-patch database through their topological analysis algorithms and it said (essentially) "your data looks like a Klein bottle!" That sounds interesting and fun, but how can you actually use that to do anything? This followup paper [1] then gave an actual model of image patches as a Klein bottle, but even then there is still a ton of math to trudge through (specifically, functional analysis and differential geometry) before they actually got to an algorithm, and it still wasn't strictly better than the leading methods. The real benefit seems to be entirely in the science part of everything. They have a novel hypothesis for how…

The key here is that TDA is packaged into an application that is designed explicitly for use by practitioners. All of the underlying math (and you know there is lots of it in TDA) is abstracted. What is shown is the groups and the atomic level explains (this group is here for these reasons e.g. they received albuterol upon admittance). Your instinct is correct, but that is what is interesting about this case - the hospital, without a single data scientist, was able to to achieve this with only slick SQL skills and engaged doctors. Screenshots for the app and videos can be found here: https://www.ayasdi.com/solutions/clinical-variation-management/

Papers are available on Ayasdi's site: http://www.ayasdi.com/approach/data-scientist/

I'm a frontend developer at Ayasdi and we're hiring!!! http://www.ayasdi.com/company/careers/ Come be a part of future breakthroughs.

I'm a data scientist and mathematician at Ayasdi. For a variety of reasons we don't participate in competitive data science - however - we do a lot of actual science that you can use to get some idea on the power of the technique: http://www.ayasdi.com/resources/#publications Work using related techniques but not directly Ayasdi's implementations of TDA can be found here (although for some reason the page hasn't been updated in the last year): http://appliedtopology.org/preprints/ Although you'll have to take my word for it - we are frequently brought in when people have tried the standard techniques and have run out of ideas. In those cases we have to show that we can improve on those models. We do that by discovering how some outcome of interest is spread (localized) in the data. You can think of this as finding different local descriptions of a phenomena. Many methods will fine the dominant description but miss more subtle variations - we find a more complete picture. For example, if you're looking at fraud (credit card, login etc) velocity-based measures may catch machine-assisted fraud but miss manual and more sophisticated fraud. Different kinds of fraud have different (local) descriptions. From a data scientist workflow this means that you (1) select features. (2) create topological summaries.(3) find regions where fraud is localized (4) find underlying reasons why the fraud is being localized in these groups (5) use this information to build better…

If you want to read up on ayasdi go straight to the Source: http://www.ayasdi.com/resources/ (click on publications) There are probably some talks by Gunnar Carlsson around on the web too. There is also a great overview from Larry Wassermann on his outstanding blog: http://normaldeviate.wordpress.com/2012/07/01/topological-data-analysi… Other examples of manifold inspired math in data analysis that I find more useful as they describe ways to solve complex problems with simple systems: http://web.stanford.edu/~yyye/hodgeRank2011.pdf http://www.cs.jhu.edu/~misha/Fall09/Hirani03.pdf A really interesting method for learning manifolds from heterogeneous data (decision trees can be extended for categorical data, text data, almost anything) that doesn't boil down to just choosing a metric: http://research.microsoft.com/apps/pubs/default.aspx?id=155552 Edit: Also they do have a really slick UI and I like the math involved. I wanted to like their product however I worry that the math just serves to lend legitimacy when the results aren't as exciting/promising as advances being made in other areas of data analysis on similar datasets. Also them keeping it proprietary makes it hard for people to do legitimate comparisons. We implemented some of our own versions of their stuff to tory on other cancer datasets but it just didn't work very well because of numerous issues with lack normalization, batch effect, bias and noise.