RISELab Blog

Interesting Framework. Seems to hail from : > https://rise.cs.berkeley.edu/ Quite the bee-line from the authors' previous papers. It only seems to have been preceded by : https://arxiv.org/abs/1703.03924 However, the language in the [Ray] paper is completely different. Different from the preceding paper but familiarly distinct. Any specification of outside groups/companies/individuals who were consulted/collaborated with on this project? And who more or less lead the development and design? Maybe I am jumping the gun, but it clearly wasn't of the listed names. Of all of the authors listed, none of their previous papers read like the Ray paper nor does the proposal paper (Real-Time Machine Learning : The missing pieces). http://arxiv.org/abs/1703.03924 reads like a corporate/industry grade requirements/proposal doc which is a huge departure from all of the author's prior papers... So, out of the blue, corporate level infrastructure project proposal and completion within the span of a year? Can any of the paper's authors speak more clearly on who led this project over what span of time, under what direction, and with which industry groups? I see no background or papers from the individuals priority listed in the paper reflective of the sort that creates a formalized and industry grade Distributed Computational Framework such as this. > Robert Nishihara > Philipp Moritz Are priority listed yet have no prior papers leading to such a development. To what degree did :…

This is a project out of the riselab https://rise.cs.berkeley.edu/projects/ Most data processing and distributed systems have some sort of link back to the research this lab has done.

Oh, I have seen some of the lasp papers. And I guess https://dl.acm.org/doi/abs/10.1145/2578855.2535842 is related too. Some of the stuff coming out of the RISE lab too eg https://rise.cs.berkeley.edu/projects/anna/. > The only work I've seen on this is Honestly, my brain in the morning...

Since the original URL (https://rise.cs.berkeley.edu/blog/michael-i-jordan-artificial-intellig… is responding slowly and points to the medium.com URL as the original source anyhow, we've changed to the latter. Thanks!

I continue to want to dabble here-abouts, since at least the time of having seen "Anna: A Crazy Fast, Super-Scalable, Flexibly Consistent KVS"[1], which is now one[2] of the two major components of the Hydro project, alongside Cloudburst[3], a Python "stateful serverless platform". There's also some other pieces like a local Anna-cache[4] & a coordinated retry library, & kubernetes-centric cluster-manager. I have no idea how related the Hydro-Project work is to this recent publication on Calm, but I hope to find out! [1] https://rise.cs.berkeley.edu/blog/anna-kvs/ [2] https://github.com/hydro-project/anna [3] https://github.com/hydro-project/cloudburst [4] https://github.com/hydro-project/anna-cache

If you are interested in learning more about this, there is also a recent PyData talk: https://rise.cs.berkeley.edu/blog/modin-talk-at-pydata-nyc-2018/

I just learned about modin as well. Based on this blog post [1], it seems to be a bit slower for small datasets. But this was 6 months ago. Now the documentation claims that it is suitable for datasets from 1kB to 1TB. [1] https://rise.cs.berkeley.edu/blog/pandas-on-ray-early-lessons/

The primary author is Joseph Hellerstein, head of the Berkeley RISELab (the new Spark/AMPLap). Their primary project is called Fluent (which until recently was called Anna [4])...it's the all encompassing name changes that get ya! ;-) Similarly, see Matei Zaharia's (the original Spark lead) and his team at MIT (he's now at Stanford) SIGMOD 2018 paper called MISTIQUE [4]... [1] Joe Hellerstein https://github.com/jhellerstein [2] Berkeley RISELab https://rise.cs.berkeley.edu https://github.com/ucbrise [3] Fluent Compute Platform https://github.com/fluent-project/fluent [3] Anna: A Crazy Fast, Super-Scalable, Flexibly Consistent KVS https://databeta.wordpress.com/2018/03/09/anna-kvs/ Discussion: https://news.ycombinator.com/item?id=16551072 [4] MISTIQUE: A System to Store and Query Model Intermediates for Model Diagnosis https://cs.stanford.edu/~matei/papers/2018/sigmod_mistique.pdf

How does this compare to dask.distributed? Dask dataframes are also a wrapper on pandas API. edit- They explain differences in a section of this blog post: https://rise.cs.berkeley.edu/blog/pandas-on-ray/

Did some digging. Details are a little fuzzy, but I think I see why the Australian courts had a problem with this. Here's one blog post that expands on the topic from a couple years back: https://rise.cs.berkeley.edu/blog/the-right-to-not-be-tracked-a-spotli… Google isn't just making an educated guess about where you are based on what you searched for... It's making an educated guess based upon what its servers are allowed to know about your query. It's unclear to me whether all the data Google collates is published anywhere public, but this includes things like originating IP address and headers sent in the request that can be inferred to be locale-relevant. For example, looking at my own search history, Google's real certain it knows when I'm doing queries from "home" (perhaps because I told it my home location in Maps and it's correlated requests from a specific IP address with that location, perhaps because Google's fabric is huge and it can disambiguate origin of request based on where the request enters the network, perhaps because I registered Google appliances at "home" that are on the same network). Key idea is: none of those are technically "Ask the device for its GPS state and cache it," so none of them are controlled by disabling Location History. If the Court is arguing "you can't claim you aren't tracking location by making 'location' a technical term and then combining several other pieces of state to estimate…

I got interested in Arrow recently after reading this blog post showing that Arrow (and Ray) are much faster than Pickle: https://rise.cs.berkeley.edu/blog/fast-python-serialization-ray-apache… I have a question about whether it would fit this use-case: * I need a SUPER fast KV-store. * I'm on a single machine. * Keys are 10-bytes if you compress (or strings with 32 characters if you don't), unfortunately I can't store it as an 8-byte int. sqlite said it supports arbitrary precision numerics, but then I got burned finding out that casts integers to arbitrary precision floats and only keeps the first 14 digits of precision :\ * Values are 4-byte ints. Maybe 3 4-byte ints. * I have maybe 10B - 100B rows. * I need super fast lookup and depending upon my machine can't always cache this in memory, might need to work from disk. Would arrow be useful for this? Currently just using sqlite.

I'm not especially familiar with either MVars or STM, so you'll have to make do with my first impressions... MVars seem most useful for a token-passing / half-duplex form of communication between modules. I've implemented something very similar, in Java, when using threads for coroutines. (Alas, but Project Loom has not landed yet.) They don't seem to add a whole lot over a mutable cell paired with a binary semaphore. Probably the most valuable aspect is that you're forced to think about how you want your modules to coordinate, rather than starting with uncontrolled state and adding concurrency control after the fact. STM seems very ambitious, but I struggle to imagine how to build systems using STM as a primary tool. Despite its advantages, it still feels like a low-level primitive. Once I leave a transaction, if I read from the database, there's no guarantee that what I knew before is true anymore. I still have to think about what the scope of a transaction ought to be. Moreover, I get the impression that STM transactions are meant to be linearizable [1], which is a very strong consistency requirement. In particular, there are questions about determinism: if I have two simultaneous transactions, one of them must commit "first", before the other, and that choice is not only arbitrary, the program can evolve totally differently depending on that choice. There are some situations where this "competitive concurrency" is desirable, but I think most of the time, we want…