Knowledge science is overspecialized (or me underspecialized)?
I am at the moment a knowledge scientist working in monetary trade. I am self-taught, do not have grasp’s/PhD in stats/math (have grasp’s in unrelated filed, would not assist me with DS). İn my job market, sadly, most knowledge scientists are self-taught or bootcamp graduates, so, employers do not require levels actually. However, the shortage of formal related training actually makes me uncertain, insecure and annoyed truly about my future. As I might prefer to work remotely or relocate (to Europe), I am feeling actually insecure. DS in US/EU appears very overspecialized, with everybody within the trade having PhDs, being wizards of stats, and many others. On this subreddit I’ve additionally seen lots of people speaking concerning the want of strong math/stats basis, or simply overlook about DS.
Is it true? Do I must spend 1-2 years determining math/stats by myself someway (nonetheless with out diploma) to get a good job? It feels quiete overwhelming and laborious to be sincere. I actually love technical a part of the job (coding normally, deployment, constructing pipelines, and many others), however the theoretical half actually haunts me. Ought to I search for one thing extra software program associated like knowledge/mlops engineering as an alternative? I am actually feeling overwhelmed and misplaced right here, any recommendation could be nice.
P.S. I am not speaking about working in FAANG-level firms, I might prefer to have a good DS job with ample pay, not $300k+shares sort of stuff.
Comments ( 21 )
I think of the “science” as a real description of what the data scientist is supposed to do, “science”. Hypothesis tested, building on research., science things.
Don’t learn the skills to do these activities if you don’t want to, get hired by someone who will bestow DS title upon you BUT you then don’t question why you’re called a DS but just doing ETL.
Mlops seems like a good fit for you.
But to the bigger context. I myself am in BFSI but a s a PM in DS. In the credit space , we mostly work with tabular data for default models, product recommendations, prospect analysis ,RWA etc. As you can see almost none of these use cases require the transformer models which happen to be latest fad amongst people even though they have existed for quite a bit if time.
But I did manage to expedite a few sentiment analysis use cases based on them, for the team even though they weren’t top of lime or anything. The main point being, I never used transformers professionally before but neither did anyone in my team that gave me an opportunity to develop an use case which was outside my are of specialization.
Being overspecialised isnt a problem man. We still have COBOL developers who are making bank XD but it’s good to try out a few things outside your are of comfort within your org. They don’t have to be huge use cases ( eventually they might be) but thilese things add to your CV.
Regarding getting a degree, to learn stats/math. Given your competence level already , a degree is only going to add to the checklist a recruiter will look at. Personally it’s not going to help you a lot, you won’t believe the number of people in DS who don’t know basic stats like hypothesis testing ( a recent post in this sub about the same got surprising replies by people here about how that’s “old school” and “redundant”)
Most people you will meet in DS dint have stellar fundamentals , being just good enough is actually better than average. So don’t stress too much when people from other specializations tell you fancy buzzwords it’s almost always not nearly as difficult or out if reach as it seems
Why do you call a job market “overspecialized” where a proper educational background is required? I don’t feel that the market would be saturated. There are trillions of jobs for those who are properly educated. Btw. you can also be properly educated, if you feel the need to have a specialized degree.
No one in tech, nor any specific job, has a well-defined identify. Follow the skills that you have and enjoy at any given moment. Lateral transitions to new roles aren’t hard if you have a shallow but broad understanding of how computation/computers/networks work. DS does identify of course with theory/stats/math closely, so don’t force it if that’s not you. You’ll be exposed easier as a DS that doesn’t like/know the field than a DE that doesn’t have experience. Try something new. The canonical DS role is exactly what you described hating. DS and DE can be widely different and DE may just provide the SWE tasks you enjoy.
>In this subreddit I’ve also seen a lot of people talking about the need of solid math/stats foundation, or just forget about DS.
I agree with them. Data Science is basically applied maths, so you need to know and understand what’s behind the things you do. I guess this is the toughest part. Otherwise it’s just implementing stuff without knowing how and why they work.
These days models have the underlying math and stats built-in, so you really just need to understand what to use, when, and why.
If you haven’t, I’d recommend Intro to Statistical Learning which does an excellent job explaining the fundamentals of models without requiring much math or stats backgrounds. This will enable you to use models responsibly and under the right conditions.
I’m in the same boat with no math/minimal stats background and while I do get intimidated when I see formulas/proofs that some other DS’s reference, I just need to understand the rules behind it… not the mathematical concepts that led to its creation in the first place.
This is something I’ve suffered through for long time. Here are my two pages on this:
I am about to graduate from MSc DS program in UK. I graduated from bachelors in CSE in India.
I missed 4 major opportunities in my life despite being in better position than my peers. The very first one was in high school when i was in one of top program in country to crack IIT entrance exam. I ended up crashing and quitting because i fell behind in physics,chemistry and my math teacher won’t like my unusual approach to solve problems (a subject really close to my heart)….same people in the batch who were very poor in understanding went on get under 1000 rank in IIT and went to very good positions (through sheer hard work). Second time, I was in bachelors program, the quality of teaching was so poor…my advance C programming teacher didn’t know 2 is a prime number. I ended up mentally withdrawing from there as well. I would self study and cracked 2 rounds of google code jam with only one semester of programming. But stress caught up with me and I got diagnosed with drug resistant tuberculosis. For next three years i was eating meds, felt boxed and just barely survived while graduating, basically i graduated with blur core CS concepts in my head. Its like I know where everything is but dont know how to use it. For example, I know there is thing called byzantine protocol in distributed systems but idk how would i use it ever. Anyway, I went on to work as operations engineer in a web hosting company. A position of SRE in a web hosting company is one hell of a gold mine, there is so much infra, site management you could work your whole life in this field. But the way company was setup was very bad. Only system engineers in denmark had understanding of these systems and knew how to make changes. There was very little training to indian employees about how scale up. In addition, the team of “wizards” would look down upon us when we made mistakes. A lot of companies who outsource their work to india expect them to meet futile KPI while paying them dirt cheap and give them lower grade work because apparently that’s what we are only capable of. Anyway, I ended up withdrawing from there as well and quit my job, move to england. Now I came here I had an opportunity for redemption. I ended up choking again. One my program was very poor, a professor from Phd from cambridge university doesn’t know how to teach statistics (he literally taught us measures of centrality and variability for two weeks in Msc program; what a joke), I ended up withdrawing again and study on my own. I would grind calculus while i was supposed to be learning other things like data visualisation but health issues and social isolation caught up with me again.
Basically, I studied everything and coudn’t master anything. I am in forever prerequisities filling loop and tutorial hell.
But recently, after studying financial markets, the dots started to connect together. I started to see patterns in everything people do in their life. Finance is one hell of a field that ties every math, programming, systems, data science concept you studied ever. Now I started to see the big picture in almost everything.
So you can see how i know many things but have this brain fog on actually implementing those concepts. And my insecurity would kick in hard with every rejection i get from companies. My dopamine release is going complete sinusoidal.
Here are the takeaways :
1. Going to DS program will be a bad use of time until you have solid understanding of prerequisities. You certainly will add qualification to your list and make it through the door but you could end up insecure again.
2. Faculty matters not the program. Majority of people knew what matrices are and how to multiply them but it was Gilbert Strang whose taught what to do with them. You will have to decide for yourself whether school you are choosing has good faculty. I’ve met people from top universities without good intuition about some basic concepts.
3. The biggest thing of all is what you wanna do with it. For example, a Phd for me personally is waste of time now. But I love impact of ds/stats in social science. Take a course in business/pyschology/social science/economics/finance/ physics/chemistry and see how these basic statistical concepts permeates through them. A course in research methods is the best idea.
4. Coming to technical gaps, there will always be. And there will always be people who know those concepts better. You have your own journey. DE knuth once talked about how the book “Thomas’ calculus” impacted his career and he would grind every single problem in it compared to what was being taught at school. I believe in that philosphy. Try doing some projects, identify the gap, learn those concepts, implement and reiterate again. **Your research skills in DS matters more than statistical wizadry and influencing business decisions even more so.Statistical wizadry is byproduct of research. Statistics is full of academics who spent their lives in field only to realize their ideas doesn’t work in some niche industry.** when you will reiterate over different project again and again, refining your intution, that’s how you build technical skills and gain self confidence.
5. So going to masters program just to know more statistical methods without having something specific in mind about what you wanna do is truly waste of time. Figuring out what you wanna do is done in bachelors, masters program doesn’t have enough space or time to really gain good intuition. Doing good quality projects will give you more satisfaction and once you hit upper level on technical skills and possibly wanna gain more, maybe then go for PhD.
Looking back on my own life. I will never hit the level of someone who grinded throught the likes of IIT. But one thing I’ve learned from such wide experience (especially from operations) is that for the most role (unless you are on really high end things like rocket science) what matters is whether you can solve a problem after reading the documentation. This is SRE mindset (oh how do I remember those countless nights trying to debug a load-making service i’ve no clue about)
tldr: Dots don’t connect forward. They only connect backward. As for the technical gaps, explore a domain, do projects, fill the gaps, reiterate. Eventually you will see patterns. Surround yourself with people you can grow with, you dont wanna be self learning in isolation. As Ray Dalio says, surround yourself with people of different opinion. Go to totally unrelated conference and find out what people are doing. You will be amazed how quickly you can learn without grinding for stamp on paper.
In my experience, the “technical part of the job” as you call it has been the majority of the job. Knowing DS tools and workflows is of course important, but you can set yourself apart from other Data Scientists based on your knowledge of engineering tools and software practices.
If you can interact with the tools for accessing and combining information without other people needing to help you, and if you can implement your own solutions in software using best practices, you’re a star DS in my book, degree or not.
I’m one of those pesky (engineering) PhDs though, so take that for what you will 🙂
There are plenty of people with the title Data Scientist who have pretty poor mathematical and statistical backgrounds, who understand how to use various ML libraries, and if you ask them to make a forecast, they’ll show up right on schedule with a number and maybe even some metrics to support how good that number is. But without the core mathematical background, they usually have a low career ceiling, and they often end up adding little or even negative value to their organizations.
I am a Senior Data Analyst for government and I work with data scientists all the time. My experience is that a lot of them have Physics and Engineering PhDs and are super ‘educated’ but that is not because they needed those qualifications to be better at their job. Rather, they pursued the ‘academic’ path for a bit and then wanted to get a ‘real’ job. A lot of ‘techy’ roles are full of people like this. Not just data scientists but data engineers, devops engineers and developers. There is a correlation in data scientists – in my experience – being more ‘educated’ but that’s because data science has become really an umbrella term as a good job for someone with good analytical skills. However, my experience is also that – and I am someone self-taught with a background in Economics and Mathematics – a lot of these people are *very* difficult to work with. They are very educated but they need to develop people skills. Like it or not this is what matters. Because you can make the fanciest most advanced piece of code or tool that you can think of. But if you can’t explain how it works and communicate its value to people, even through inference and body language, building rapport then what’s the point? You’ve created something that might be clever but nobody will use it. And trust me I love nothing more than self-inflicting myself with thought experiments and mental masturbation. Such as who would win a fight between batman and spiderman? And yes I know what you’re gonna say: batman, obvs. 😝🤔
I think that now we are in an age where education as a creed doesn’t matter so much, particularly with a tech role. What employers care about is whether you can do the job and you are easy to work with. Simple. Having a degree or not is completely irrelevant. When I hire I don’t care about this unless the person has literally no demonstrated experience. Sometimes for reasons including those above it can even hinder you. I am faced with a similar dilemma in that I really want to a PhD or something but I want to do it because I just love learning. My university background is in economics and mathematics to MSc level. I taught myself much of what I know: Python, R, LaTeX, SQL, Java and now thinking of learning C/C++. Because I love coding. Simple as that. And I think it really comes down to whether you actually want to do it. It’s a big commitment, so be sure to ask yourself why it is valuable to you and you only. Not because you think it will make your CV better. In short yes having those qualifications can help you, but only if you want to apply to places like the European Central Bank, or many top EU/US universities. These places ask for minimum education requirements as a means of sifting through applications. And you probably wouldn’t enjoy working towards a job in many of these academic-oriented places because their culture is usually very archaic and competitive. Some people like that, and you might be like that. In which case, get that postgraduate qualification. But I worked for one if these such institutions for many years and I was miserable, and I was the straight-A type student etc. That was a big learning lesson for me. If you are miserable…life is too fucking short man.
So no you don’t need those things. If anything, do them only because you will get pleasure from doing it, or if by doing them you develop your cognitive and problem-solving skills. But you definitely don’t need that piece of paper. Know what your strengths are, where your interests lie, keep an open mind, be kind to yourself and others, learn how to connect with people; and I am sure opportunities will present themselves. Hope this helps and wish you the very best of luck 🙂
You don’t need anything more than the fundamentals in stats calc linalg etc
Ppl here be gatekeeping to make themselves seem better
As a self taught DS with unrelated business diploma working in Paris, you really don’t need the theoretical part, it’ll probably come naturally along the way, but the important part is that you get things done, there is lots of need for DS in the world, even more nowadays in NLP
Well, a degree is unparalleled. Self taught is good but it’s a path that’s difficult to walk – self motivation, actively researching good study material etc while ensuring that you are on right track always.
Bootcamps are faster way to land a job. It’s a safer investment both in terms of cost and time.
That said… If you really want to do actual Data science, please start studying math. Get solid in there. Stats, algebra and calculus. And a word of advice… the titles don’t matter. Like seriously. There are data scientists who just do pivots on Excel. There are also analysts who test hypothesis and validate-maintain models. There are data scientists who are just cloud engineers or backend developers using ML libraries. So it doesn’t matter. Just ensure the JD matches what you like.
I’m also someone who fell for the DS tap. I started as DS right out of college. I’m a SDS now in a mid size start up that makes B2B products ‘driven by data’. In terms of JD, i was an analyst for an year. Business analyst for 2 more years. 3 months of APM. 9 months of backend developer. Even though I have a solid math background (degree in electronics n communication), the jobs out there are just not real DS unless you don’t have a master’s or PhD.
Data science is ETL and AB testing. ETL=data, AB testing=science
Bra not knowing stats in DS is like not knowing maths as an accountant. You don’t need a masters in stats but you absolutely have to know applied statistics.
I’m also in data science but I came from a com sci background. You can do a stats course at your local uni and get yourself sorted.
Self taught is close to worthless imho. The only thing you are qualified for is being a data wrangler. Which is good. Real data scientists need code monkeys that implement the pipelines and perform the sql
With poor math and stat skills, you can never be a top data scientist. Complex coding and modeling are 100% based on math.
Following
No you don’t, not event at FAANG (source: I’ve worked at multiple FAANG with only a bachelor’s), unless you want to be a research scientist/ML eng you don’t need masters/PhD etc. I’ve also gotten offers in Europe wehn I’ve been thinking about relocating there (never took them though)
Just a comment on you wanting to move to Europe; I live in Denmark and normally most companies would very much prefer (almost always choose) someone with formal education. Tech is one of the very few fields here, where you can get by with a relevant bachelor’s degree only. Most of the time tho, a Master’s is required. Self taught is not really a big thing here but Denmark is such a small country, so you probably don’t want to move here – most people want to go to Europe to live in warmer (and friendlier) countries.
I’ve done quant internship analytics final round interviewing for two of the largest banks in the United States. I can tell you in my space, we wouldn’t even look at a candidate without a masters degree and in these banks most AI roles come through our internship program.
The modal candidate either has a Ph.D from a reasonable university (i.e. large flagship state school) or a Masters Degree from an Ivy League school. I can’t imagine leading tech companies taking much less than that. Even people on our india/polish teams have masters degrees from good schools in their countries.