
AresDB: Uber's GPU-Powered Open Source, Real-Time Analytics Engine - Recovery2020
https://ubere.ng/2HzMPVK
======
uberemployee
When I worked at Uber, this project was openly mocked. One of the CTO’s
biggest failures was implementing a promotion scheme where you needed to
create a new service in order to be considered “innovative”. This promotion
scheme marked what I consider the end of Uber’s engineering excellence and the
start of what made Uber turn into a bureaucratic mess.

One of the VP’s of engineering called it “toil vs talent”. People who “toiled”
at work, meaning doing good maintenance work, would be rewarded with good
bonuses but those with “talent” would be rewarded with promotions. Of course
this drove people to come up with fake new services so that they could
demonstrate “talent”. This also lead to an explosion of new services that
overlapped or did nothing useful. Instead of working together, groups would
make new services instead of working with existing service-owners because they
needed to justify writing a new service. It was sickeningly transparent.

This project was one of those projects. It has no real use case because why
the fuck would we want to use GPUs except to look cool on your resume. The sad
thing is that the projects is overstating how well it’s being used internally.
Internally people use Pinot instead of this.

For all you future CTOs, consider your incentive schemes carefully and don’t
be so far removed from the action that you can’t see when your org is rotting.
This is what the CTO did, and like I said, it was one of his biggest failures
because it gutted the engineering org. Instead of working together, every team
was looking at get promotions at the expense of the company and it showed.

~~~
PragmaticPulp
I wish I had understood this earlier. My past company made a push to hire from
top companies like Uber for some key positions. Some of them were great people
who were relieved to be out of the FAANG rat race. Others were single-mindedly
focused on rewriting everything they touched with cleverly-branded project
names, regardless of whether or not it made business sense.

Early I on, I made a harmless comment in Slack about how one person’s pet
project wasn’t a good fit for our needs so our team would be using the older,
more proven solution. Later that evening the person pulled me aside, almost in
tears, begging me to never say anything critical about his project in a Slack
channel again. He explained that at his previous role, success or failure
depends entirely on the perception of one’s personal projects and that
seemingly innocent comments could tank someone’s promotion chances for years.

I felt bad for him because he had clearly come out of a toxic situation.
However, one of his teammates later warned me that he was keeping a journal of
potentially incriminating things that I had said in Slack and a detailed log
of every issue that he could find with our team’s project in case he “had to
use it against me later”.

I could never tell if this was a unique experience or the norm at some
companies like Uber.

~~~
uberemployee
What you describe was definitely not a common occurrence. Engineering wasn’t
toxic for many years until the last 9-12 months or so I would say. Pre-Susan
Fowler memo, it was the best company I had ever worked at. From 2017-2019 we
sort of stalled because of the internal drama and it didn’t get really bad
until the last 9 months, where attrition of our best engineers and vile
political maneuvering from the dregs made it too much for me to stick around.

The engineer you describe sounds like they have mental health issues. There
may be some teams with terrible managers but all companies have this, and I’ve
seen similar or worse situations at Amazon.

Most engineers I worked with were great but there were many engineers that
“played the game” in order to get a promotion and more money. It was sickening
but if that’s the way the CTO sets the incentive scheme, who can blame an
engineer for following it? It’s more on the CTO for setting the terrible
culture than the engineers.

~~~
kamaal
Being blind to politics rarely ever means politics isn't going on.

Pretty much every company out there has a concept of _' promotion packet'_,
its basically building a case for one's promotion. Of course in a company the
budgets are fixed, and so are promotion cycles(yearly in most places). You
miss out a turn, you could lose an year, or even risk losing two. In that case
its fairly common for managers to build a list of
accomplishments(file/packet), and rival managers to build a anti-case/defence
for the same. Stack ranking eventually is all about a combination of
merit+advocacy+lobbying+counter-lobbying at so many levels that I'd say the
engineer who cried wasn't wrong at all.

This is the case in nearly every company. We just wish to delude ourselves
that politics is absent at some places.

This sort of power play comes with the territory in a large people structure.

------
statictype
> Like Pinot, Elasticsearch is a JVM-based database, and as such, does not
> support joins

Uh. What does the JVM have to do with the data model’s ability to do handle
joins?

~~~
EdwardDiego
Yeah that's a very odd statement, I mean, PrestoDB, Impala, KSQL...

------
einpoklum
GPUs and analytic DBMSes / query engines are actually my own field of research
([https://eyalroz.github.io);](https://eyalroz.github.io\);) and it's
obviously beyond what a comment would encompass, but:

1\. There are very few analytic DBMSes which are actually fast (and compare
against reasonable baselines). Most claims of speed are bogus. Or rather,
might be better than what's otherwise available to use, but are still slow.

2\. Designing an analytic DBMS to properly utilize a massively-parallel
processing device is a monumental task, and I would claim that it has not yet
been undertaken. Existing research and production systems graft such use onto
a system whose fundamental design dates back to the 1980s in many ways.

3\. CPU-utilizing anallytic DBMSes are typically faster than GPU-based ones,
to a great extent due to the above - but also since we've had decades of work
on optimizing them.

4\. GPUs are artificially handicapped on Intel-architecture systems, because
they are placed "far" from main memory relative to the CPU. More literally -
the bandwidth you get t between your GPU and main memory is typically 0.25x
the bandwidth a CPU socket has with main memory. This is critical for analytic
query processing (as opposed to neural network simulation which is more
computation-heavy and can tolerate this handicap much better).

\--

PS - Always glad to discuss this further with whoever is interested.

~~~
bronxbomber92
How much would the trade offs change if GPUs shared the same main memory as
CPU?

~~~
einpoklum
Not sure I understand exactly which trade-off you're referring to, but on
systems without the GPU-handicapping (e.g. IBM Power; and also when you put
link up many GPUs together with NVLink) - there is still a significant design
and implementation challenge to produce a full-fledged analytic DBMS,
competitive vis-a-vis the state-of-the-art CPU-based systems.

There are also other considerations such as: The desire to combine analytics
and transactions; performance-per-Watt rather than per-processor; performance-
per-cubic-meter; existing deployed cluster hardware; vendor lock-in risk; etc.

------
social_quotient
Curious - what’s the use case for an organization like Uber needing real time
analytics at high frame rates? I noticed the emphasis on dashboards but was
curious what a real-time dashboard at this scale actually ends up being used
for.

Maybe my question is more around, what business decision would be impacted by
not having real-time instantly reserved dashboards.

Honest question here not trolling.

~~~
Donald
All sorts of departments at Uber use real-time queries (operations,
marketplace, eats, new mobility, their data science and growth group, finance,
communications, legal.) Marketplace in particular has a demand for real-time
prediction, matching and dispatching, and pricing queries.

~~~
bogomipz
I understand why any marketplace-based system would need real time data but
why would any of "growth group, finance, communications, legal" require real-
time data to do their jobs?

------
ed25519FUUU
It must have been fun to be at Uber 2017-2019. They seemed to have an
unlimited appetite (and funding) for “invent it here”, and a lot of those
projects made it into open source.

~~~
tyingq
I'm sure it was fun, but it does seem to point at a lack of focus. I suppose
that's hard to resist when there's an endless pipeline of money.

------
uberOG
I worked at Uber before. The team and the project is pretty much gutted after
last couple layoffs. Check the commit history/contributors and go figure.

It was some amazing tech, but it falls into the category of "when all you have
a hammer, everything looks like nail". sometimes you really need a company
culture to reward people for creating values instead of deliverable for
promotion

------
DevKoala
Does anybody with Clickhouse experience at scale know if AresDB is better on
some use cases?

~~~
hodgesrm
It's hard to say, though I think the UPSERT capability looks useful because it
simplifies handling duplicates. On the other hand it does not appear that Ares
offers clustering, which is critical for large datasets.

(I work on ClickHouse and enjoyed this article when it came out.)

~~~
einpoklum
It should be mentioned, to ClickHouse' credit, that they made an effort to
publish relatively detailed benchmark results for a some data sets and
queries, when they first came out. They even got in contact with my research
group at the time (the MonetDB group at CWI) to make an effort to present the
MonetDB results in a fair manner.

------
kuharich
Past comments:
[https://news.ycombinator.com/item?id=19028860](https://news.ycombinator.com/item?id=19028860)

~~~
ramoz
> GPU databases are brilliant for cases where the working set can live
> entirely within the GPU's memory. For most applications with much larger (or
> more dynamic) working sets, the PCIe bus becomes a significant performance
> bottleneck. This is their traditional niche.

> That said, I've heard anecdotes from people I trust that heavily optimized
> use of CPU vector instructions is competitive with GPUs for database use
> cases.

This comment is important imo. Also related to applied ML inference in
applications... the memory needs can grow quite a bit and this data transfer
cost, including the memory size limitations vs RAM, becomes very real very
fast.

Not sure I understand the scale of the use case or where it's mentioned as
well as in comparison to the big data tools mentioned.

~~~
kanwisher
Next gen Nvidia 30x0 series, can have direct access to SSD without hitting the
CPU. In that case would they be any worse than cpus on any workloads? I guess
you could still have larger ram amounts on the cpu, albeit slower ram usually

~~~
WanderPanda
Wow nvidia is selling SBCs now?

~~~
ReactiveJelly
No, there's a new thing about giving GPUs some kind of DMA to storage. And
it's pointless on HDDs, so it's only discussed in terms of SSDs.

Microsoft is bringing the DirectStorage API from XBox to Windows, Nvidia calls
theirs RTX IO. I think they're the same class of idea, like Vulken vs. Metal.

They do have SBCs, I think, but other than being the basis for the Nintendo
Switch I haven't heard much about them.

------
briandilley
> Like Pinot, Elasticsearch is a JVM-based database, and as such, does not
> support joins and its query execution runs at a higher memory cost.

What does the JVM have to do with joins?

------
kevsim
> In the past, we have utilized many third-party database solutions for real-
> time analytics, but none were able to simultaneously address all of our
> functional, scalability, performance, cost, and operational requirements.

Completely original excuse for an over-staffed engineering organization to
justify doing some crazy stuff.

------
grej
Does anyone know how this compares to the RapidsAI project called BlazingSql?

~~~
roaramburu
Howdy, full disclosure I'm the CEO at BlazingSQL (BSQL).

I'm not incredibly familiar with Ares save the linked article, but we aren't a
DBMS or manage data in any way.

BlazingSQL is a SQL engine, it's easier to think of it similar to SparkSQL,
Presto, Drill, etc.

We're core contributors to RAPIDS cuDF (CUDA DataFrame), which is a Pyhton and
C++ library for Apache Arrow in-GPU memory. The Python library follows a
pandas-like API, and the compute kernels are in C/C++.

BSQL binds to the same C++ as the pandas-like cuDF. What this enables users to
do is interact with a DataFrame with either SQL or pandas depending on their
needs or preferences. This interoperability means that the rest of the RAPIDS
stack can be applied to a variety of different use cases (data viz, ML, Graph,
Signal Processing, DL, etc), with the same DataFrame.

The DataFrame also has performant libraries for IO, Joins, Aggregations, Math
operations, and more.

Here is an example of running a query on ~1TB on a single GPU in under 9
minutes. The data was stored on AWS S3 in Apache Parquet.
[https://twitter.com/blazingsql/status/1303370102348361729](https://twitter.com/blazingsql/status/1303370102348361729)

Here is an example of scaling that same query up to 32 GPUs and running it in
16 seconds.
[https://twitter.com/blazingsql/status/1304450203030880257](https://twitter.com/blazingsql/status/1304450203030880257)

Again, think of BSQL as a query engine, that runs queries on data wherever and
however you have it. Here is a BSQL user running 1-2 minute queries on 1.5TB
of CSV files using 2 GPUs.
[https://twitter.com/tomekdrabas/status/1303824164273270789](https://twitter.com/tomekdrabas/status/1303824164273270789)

Let me know if that helps at all (or not).

------
shaklee3
This should have the 2019 tag in the title

