Blaze Blog

Unfortunate name overlap with an under-loved PyData project: https://blaze.pydata.org

They've been great supporters of this effort, as well as Blaze[1] and Numba[2]. [1] http://blaze.pydata.org [2] http://numba.pydata.org

> how does this compare to theano? it seems like some of the ideas are similar? It's quite similar, we just take some of the ideas farther and try to generalize the data storage to include storage backends that data scientists use more frequently ( i.e. SQL, CSV, S3, etc ). We're very friendly with the Theano developers and hope to bridge the projects with a compatibility layer at some point. > (it gives the illusion of a python solution, but in practice you're off in some other world that only looks like python). I would argue that's what make Python a great numeric language, and NumPy so succesfull. You get this high level language where you can express domain knowledge but also this 1:1 mapping between fast code execution at the C level. Blaze is the continuation of that vision > i guess the AST is fed to something that does the work. and that something will have an API and be replaceable. Precisely, we build up a intermediate form called ATerm out of the construction expression objects, do type inference, graph rewriting, and then pattern match our layout, metadata, and type information against a number of backends to find the most optimal one to perform execution. Or if needed we build a custom kernel with Numba informed by all this type and data layout information we've inferred from the graph. We don't aim to solve all the subproblems in this area ( expression optimization passes, distributed scheduling ) but I think we have a robust enough system that others…

Great comment. BTW, you can make interactive visualizations in pure python with bokeh: http://bokeh.pydata.org/en/latest/ Also with Blaze, you can use Pandas (or even Dplyr) syntax in python to query Hive, Spark and other large stores. http://blaze.pydata.org/en/latest/

Well, we'll be able to do that without sending the data to the client, not sure if our implementation right now will work without loading the data into memory, though long term that is definitely the plan (we will leverage http://blaze.pydata.org/) If you want to discuss further, please email [email protected]

There seems to be a large overlap between several of the projects mentioned here and those developed by Continuum through a DARPA grant. For instance Seamless (Enthought) seems quite similar to Numba (Continuum), and ODIN (Enthought) looks a lot like Blaze (Continuum). http://numba.pydata.org/ http://blaze.pydata.org/

Some of the work that continuum have done on blaze [1] look to be tackling the problem of streaming in python too. In fact some of the ideas in this library come directly from haskell and one of the main developers is Stephen Diehl, who posted the very popular "what I wish I knew about haskell" slides recently. Looks like a match made in heaven :-) [1] http://blaze.pydata.org/

More info from Continuum Analytics: http://continuum.io/blog/continuum-wins-darpa-xdata-funding The money is going to develop Blaze and Bokeh: http://blaze.pydata.org/ https://github.com/ContinuumIO/Bokeh

Uh, no they are not. They lazy- scale pandas to on disk and distributed files. http://dask.pydata.org/en/latest/dataframe.html "Dask dataframes look and feel like pandas dataframes, but operate on datasets larger than memory using multiple threads." http://blaze.pydata.org/blog/2015/09/08/reddit-comments/

Spark has a python API and python has a bigger ecosystem for exploratory data analysis than spark. Also python has a big data interface with out of core matrices and tables as well as a compiler than can speed numerical code, with a just a function decorator, to C like speeds. The former is an interface to the latter. http://blaze.pydata.org/docs/dev/index.html https://github.com/numba/numba

Check out python blaze: Pandas and linq style frontend to lots of different backends. Soon there will be OOC array ops. http://blaze.pydata.org/docs/dev/index.html http://nbviewer.ipython.org/url/blaze.pydata.org/notebooks/timings-bco…

Check out python blaze: http://blaze.pydata.org/docs/v_0_6_5/index.html http://continuum.io/blog/blaze-datasets

You mean like blaze in python?: http://blaze.pydata.org/docs/v_0_6_5/index.html

Not OP, but I've also been look for a way to leave ggplot2 behind and make the jump to 100% python for data analysis. These look neat. Seaborn - http://web.stanford.edu/~mwaskom/software/seaborn/ Blaze - http://blaze.pydata.org/docs/v_0_6_5/index.html Bokeh - http://bokeh.pydata.org/

The author of numpy is making the next generation numpy, blaze (http://blaze.pydata.org/docs/index.html). There're also many python project, such like numexpr, blz, numpy aim to boost scientific computing in python. Having a strong community, I think python might dominate the data analysis in the nearly future.

Having a bit of a look at the docs^1, I don't really see it as "yet another arrays package". The API exposes generalized table/array operations, and a subset of those features can be implemented using NumPy's ndarrays. If the interface is clean, then I don't think adoption will be an issue: there simply isn't anything else that does this. You are absolutely right that both of these projects are extremely ambitious, but, given the $3 mil grant, I can't help but be optimistic. 1 http://blaze.pydata.org/docs/overview.html#blaze-is-a-generalization-o…