I wonder if they will improve in a specific direction - frameworks and libraries built to be easier for LLMs to use.
This could even happen through accidental evolution - a framework that is easier on an LLMs context window results in more successful projects, which results in more training data, which results in LLMs being even better at it.
I came into a CS and math background without CE or EE, and took two dedicated optimization courses (one happened to be in a EE department, but had no EE prereqs), as well as the optimization introduced in machine learning classes. To be honest a lot of the older school optimization is barely even useful, second-order methods are a bit passe for large scale ML, largely because they don't work, not because people aren't aware (Adam and Muon can be seen as approximations to second-order methods, though, so it is useful to be aware of that structure).
Isn't Nagle usually introduced in a networking class typically taken by CS (non-CE/EE) undergrads?
Just because EEs are exposed to some mathematical concepts during their training doesn't mean that non-EEs are not exposed through a different path.
> Isn't Nagle usually introduced in a networking class typically taken by CS (non-CE/EE) undergrads
Networking, OS, and Distributed Systems is increasingly treated as CompE or even EE nowadays in the US.
> Just because EEs are exposed...
That's the thing - I truly do not believe that EE and CS should be decoupled, and I believe ECE as a stopgap is doing a disservice to the talent pipeline we need for my verticals to remain in the US, especially when comparing target American CS and EECS programs to peer CEE, Indian, and Israeli CS programs [0].
There is no reason that a CS major should not be required to take a summary circuits, DSP, computer architecture, and OS fundamentals course when this is the norm in most CS programs abroad. Additionally, I do not see any reason for EEs and ECEs to not take Algorithms, Data Structures, and Compilers as well.
> Just because EEs are exposed to some mathematical concepts during their training doesn't mean that non-EEs are not exposed through a different path
Mind you, I'm primarily in Cybersecurity, AI/ML infra, DefenseTech, and DeepTech adjacent spaces - basically, anything aligned with the "American Dynamism" or Cyberstarts thesis.
From what I've seen, the most successful founders are those who are able to adeptly reason and problem solve, but are also able to communicate to technical buyers because you are selling a technical product where those people make the decision.
Just because an approach isn't useful today doesn't necessarily imply it isn't in the future and being exposed to those kinds of knowledge and foundational principles makes it easier for one to evaluate and reason through problem spaces that are similar but not necessarily the same - for example, going to the Nagle's example - this was a bog standard networking concept that has now become critical in foundation model training because interconnect performance is a critical problem which can impact margins.
A lot of foundational knowledge is useful no matter what, and is why we fund founders and hire talent at competitive salaries.
I know multiple people who went to CMU who dropped out of CS to CE because they couldn't handle the rigor of the OS fundamentals course required for the CS major, as the OS course where you had to implement large parts of a kernel was required for CS, but not for CE.
Yes, you need solid fundamentals. I am saying CE/EE is not where you get them unless you actually are designing circuits. If you're doing the hard tech like you're describing, take the more academically rigorous program of CS instead of the easy CE/EE program where you learn irrelevant skills like circuit layout (that is now done by an AI anyway!) rather than very relevant skills like OS and networking, and algorithms which is not required by EE.
The issue is most CS programs in the US don't require a 15-410 style class anymore, or removed much of the more complex content within it.
There's a reason why a CS@CMU degree and a handful of other similar programs that are its peers are well regarded and continue to attract early career recruiters.
That said, I do still feel that the EE/CE/CS shouldn't be treated as different majors and should instead be treated as tracks within the same degree (yes ik what I'm describing is EECS but it ain't wrong), but that shouldn't distract from your overarching point which hits the nail on the head.
Muon is much more sophisticated than Newton's method. Neural networks have started to borrow techniques from statistical mechanics, and various branches of maths like invariant theory that were previously rarely used in engineering. CS is not dumbing down; its needs and focus are changing.
I've never needed or benefited from most of the EE curriculum. There is an opportunity cost in learning things you don't need.
I doubt it explains any reasonable fraction of this, but github moving from early adopter techies to general population "normies" would be a reason for the shift. I would expect it explains at least some increase in the use of em-dashes.
There is the pre-training, where you passively read stuff from the web.
From there you go to RL training, where humans are grading model responses, or the AI is writing code to try to pass tests and learning how to get the tests to pass, etc. The RL phase is pretty important because it's not passive, and it can focus on the weaker areas of the model too, so you can actually train on a larger dataset than the sum of recorded human knowledge.
This could even happen through accidental evolution - a framework that is easier on an LLMs context window results in more successful projects, which results in more training data, which results in LLMs being even better at it.
reply