
Improving Stack Overflow jobs search with machine learning and R - slashdotdash
https://medium.com/@aurelien.gasser/a-dive-into-stack-overflow-jobs-search-62bc6e628f83
======
rattray
I found it got off to a bit of a slow start, but was a fun, rewarding, and
very real-world read.

In particular, I appreciated admissions like this:

> Writing our own genetic algorithm in C# was a bad idea. It took us weeks to
> implement, test, and optimize. Not to mention all the time spent waiting for
> results. There was a better solution available all along (the optim function
> in R). Because we didn’t do proper research, we overlooked it and lost time.
> Sigh.

... which far too few eng blogs overlook / fail to mention.

------
gaius
R is a very well supported language on Windows (since MS bought Revolution
Analytics), very well integrated with SQL Server and Azure, and R Server
solves a pain point that a lot of real-world users have. And of course it's
open source.

------
stevenwu
Question to spark discussion and for me to fill potential gaps in my
knowledge, not to criticize the article as I very much appreciate the
transparency and build-up from the simple naive initial approach to the final
approach used in production:

Is anyone else bothered by the claim that there "is a 100% chance that the new
version is better than the current one" shown by using bootstrap? Maybe I've
just never come across such a use of bootstrap through my encounters with
statistics. I know it as a tool for resampling from a population to build up
properties of your estimator (mean, variance, what have you) when all you have
is a dataset and no clue about the actual distribution. When I saw bootstrap
with that probabilistic claim, I thought the author would calculate a
bootstrapped (100-x)% confidence interval for both the current and the new
weights: and if the intervals didn't overlap with one another then you can
claim with (100-x)% certainty that one is better than the other. But the
author creates a new statistic that is a function of both datasets; Z_i = 1 if
new is better than current on iteration i (on a random subset of data) else 0,
and for all N=10000 iterations Z_i = 1. The chance/probabilistic claim made of
new being better than current is based on the fact that no variation was seen
on Z_i (I'm also kind of skeptical that out of so many iterations with random
subsets that each time the new weights were better than the current). I think
at most you can say that you simulated subsets of the data and 100% of the
time new > current; the current claim leads me to believe there's inference
that isn't there.

Maybe I should just ask one of my past stats profs. Open to someone
enlightening me.

~~~
brockf
A quantile-based confidence interval from bootstrapping can yield a 100%
confidence interval that does not contain 0, i.e., with 100% of cases
positive/negative. But that does not (necessarily) mean that there is a 100%
chance that the new version is better than the old one. Confidence intervals
are not Bayesian credible intervals and cannot be treated as such. (That said,
making some certain assumptions about the underlying model can in some times
allow one to treat nonparametric bootstraps in such a way.)

~~~
stevenwu
Right. The author finds 100% of the time for his current dataset but makes a
statement that implies some certainty or inference on future cases. Like
taking 100 men, 100 women and finding that 100 randomly matched pairs had the
man taller than the woman 100 times, and making the claim that there is a 100%
chance that men are taller than women.

The more I type the more I realize how pedantic this is, but we're emphasized
in stats to pay extra attention to the conclusions we draw from the data we
analyze.

------
EternalData
Great read. I loved the honest balance between engineering features and how
much they ultimately ended up mattering to users. Oftentimes, the features we
select can be quite arbitrary -- it's good to do gut checks by running real-
time validation of results as often as possible. Fortunately, at Stack, you've
got the userbase to do just that :)

------
amenod
> Genetic algorithm running on 56-core machine

Wow, just... wow. I wonder why they didn't utilize (multiple) GPUs instead? I
would guess it would be far more efficient in all aspects. Especially now that
there is TensorFlow & co.

~~~
gaius
$150k on GPGPUs seems a bit steep if you're going to throw that approach away
anyway...

~~~
annnnd
Why $150k? Ordinary GPU (<$1k) would probably be equivalent to 56 CPU cores,
if not faster.

I imagine the reason for using CPU was that they would throw it out anyway, so
there was no need to make an ideal solution - and they already had those 56
CPU cores.

~~~
gaius
That was just the price of some nVidia kit I was looking at, roughly.

~~~
annnnd
I had no idea they have this, but it would probably be an overkill for SO. An
ordinary (nVidia) GPU which starts at $150 is roughly >10 times faster at NN
problems than 4 core CPU.

~~~
gaius
This: [https://www.nvidia.com/en-us/data-
center/dgx-1/](https://www.nvidia.com/en-us/data-center/dgx-1/)

