So, last Friday, after a frantic few months of searching, I scored an
onsite interview at last - my second one this year. I had reasons to be
hopeful about it. I knew the recruiter who contacted me, from my
preceding job when she was a social media herder, and the VP for the
data department turned out to be a well-known name outside of work whom
I knew at least by sight. I was aware I'd done very well on the coding
challenge which featured a nontrivial SQL puzzle and a Python part
where I corrected a design anti-pattern in the requirements (I don't know
if it was an intentional trick question).
But it was all for nought. Right from the first session, it started
surfacing that they are in fact running full speed away from SQL, except
perhaps as a playground for data scientists. They were looking for
someone to help building a data pipeline, starting with CSV data
obtained from partners, transforming it into JSON, rotating the JSON in
various ways, and finally loading the results into a huge Redis
cluster. I tried to keep positive but my distaste must have been
noticed - I had used Redis in the previous job and I know it is to SQL
as Perl is to ML.
The rotating part was going to be implemented as microservices, and I
was asked if I had written any. My reaction was, what does the term
mean? Sure, I have written a couple small daemons talking lightweight
protocols. But I have done that in C (before I even knew Python), and
the protocols were line oriented, not JSON over HTTP. Does this
disqualify me, especially when the JSON and HTTP part will all be done
by mature 3rd party code?
And then there is Docker. By the time I was asked if I'd had any
experience with it, I only managed a straight "no" answer, and didn't
bother to mention my time working with VMware and Virtualbox.
Why is all this happening, where are the nice tools going to which I
have a sort of emotional or aesthetic attachment, and which are backed
by some theory? To the extent this is a technical question (and I want
to only write about the technical part in this post), it's quite clear
to me that the answer is size, otherwise known as scale. I believe
this team (and others who tell or write the same) that under their growth
assumptions a PostgreSQL database or even a cluster of them, running in
the cloud, couldn't manage.
What's worse for me is that this is now the only kind of shop there is.
No companies are looking for engineers to take care of their computing
in house, they're all happy to let Google or Amazon handle that, or at
any rate a startup whose goal is to become a Google or Amazon, if needed
by getting acquired by one of them.
Oops, I'm veering into the non-technical. I want to reserve that for my
other post or posts about my current adventures, perhaps in another
space, which you may read soon. But please, if you know of an opening
for someone with my old-fashioned skills, let me know soon. You can
see my resume online.