Mikolov & Boháček: On the future of neural networks and search

In the first part of the interview, the Neuron Award laureate Tomáš Mikolov asked the Neuron Foundation scholarship holder Matyáš Boháček about his experience of research in the United States and about artificial intelligence as the phenomenon of the moment. Now they return to the subject, but their roles are reversed — the student of the Jan Kepler Grammar School asks the questions and the internationally recognised expert on artificial intelligence and machine learning answers.

Matyáš Boháček: Let me ask about basic research. It seems to me that science is heading down a path that leads to attractive, applicable results. Basic research, by contrast, is not so sexy.

Tomáš Mikolov: Exactly. Basic research is harder, you have more work, less reward and the future of the research is complicated. What is interesting about basic research, though, is that we can think up something that will be revolutionary elsewhere. It is great to discover something like that; I have managed it a few times in my career. As for neural language models, which are so popular today, I was in fact the first to start training really large language models and generating text from them. It felt like being the first person on the Moon. Anyone who has not experienced it probably will not understand. I was the first to see something that changes the future. And that is the engine of basic research — people do it because they enjoy it, not because they will make money from it. Basic research really is risky in that a lot of ambitious things never work out. But I believe the fun of unexpected discoveries really is the main driving force.

Matyáš Boháček: Do you also focus on areas of artificial intelligence outside neural networks, evolutionary approaches for instance?

Tomáš Mikolov: I think the success of artificial intelligence over the past ten years has been driven by machine learning. What made a splash were neural networks, which fall under machine learning, and people like me managed to get neural networks running on large datasets. The more data they had, the better they did at beating the old techniques. Suddenly it seemed we would soon be able to solve things that had previously looked unsolvable. That was very optimistic. For machine learning to do more on its own and not merely copy human decisions, a different approach is needed; if we want a computer to be creative, we need techniques that go beyond supervised learning. And the question is which direction to take. There is reinforcement learning, for instance, but even after ten years it has no results with commercial potential. We are nowhere near algorithms being able to learn the way people do.

Matyáš Boháček: What is the way forward?

Tomáš Mikolov: An interesting challenge for the future, in my view, is one of the evolutionary directions, the so-called emergent one. If models in such a system could develop for an unlimited time and to arbitrary complexity, they would be models of general artificial intelligence, which ought to have the potential to solve any task. That does not mean it would solve every task immediately. It would learn much as children do at school — reading, writing, working with fractions, programming… We have that plasticity, that we are able to learn new things without any clear limit. It looks as though the evolution of our minds could continue indefinitely. Getting that into machine learning strikes me as a terribly interesting goal. But even here nobody has yet managed to get it working.

Matyáš Boháček: And yet the scientific community keeps working on neural networks.

Tomáš Mikolov: I would even say most of the community works on things that have been solved. When I worked with language models in neural networks, it was not mainstream. The community did not want to accept that there was a potential paradigm shift; nobody was thrilled at the prospect of throwing away everything they had done all their lives because I could solve their problem twice as fast. Now everyone knows neural networks are great. But working on something else that might be better than neural networks? Nobody much wants to.

Matyáš Boháček: I can imagine it must then be hard to get grants for something completely new, when you cannot support the application with attractive demonstration applications as with neural networks.

Tomáš Mikolov: Of course, it is much easier to get grants for what has already been solved.

Matyáš Boháček: That is why I am interested in how to go about it.

Tomáš Mikolov: I was in industry for a long time and only returned to science two years ago, and I can see that the people around me know far more about getting grants than I do. It is terrible bureaucracy in which the most important thing is to find out how applications are scored.

Matyáš Boháček: I also wonder what research into neural networks was like twenty years ago, say. And how fast does scientific knowledge change?

Tomáš Mikolov: Back then every textbook said neural networks were interesting models but did not really work in practice… In science our understanding can change markedly within, say, two years; if we listed ten things we know to be definitely true, two or three of them might no longer be true ten years from now.

Matyáš Boháček: That was very much reflected in my placement at Berkeley. The summer seminars were not textbook exposition but a discussion of current papers. At one conference, for instance, I saw four different posters saying that such and such a paper had been beaten by so many per cent. It is then hard to recognise genuinely valuable improvements. On the other hand, the public's approach is changing. Lots of secondary school teachers wrote to me asking whether I, as a computer scientist, would come and show students the basics of machine learning so that they have a basic idea of what models can and cannot do.

Tomáš Mikolov: I hear you also work on recognising deep fake videos; tell me a bit more.

Matyáš Boháček: Current models for recognising deep fakes are made mainly for prominent figures, politicians or celebrities. For them we have a large amount of data available, so we can easily train models to recognise identity. In future, though, deep fakes will affect private individuals too; artificially created pornography is appearing, for instance. And one of the things I would like to continue with is detecting deep fakes of people for whom we do not have so much data.

Tomáš Boháček: And other plans?

Matyáš Boháček: The second thing that interests me is the translation and recognition of sign language. In cooperation with the University of West Bohemia in Pilsen we have created a model that is markedly smaller than its competitors, runs in a browser and does not need as much data for training. We are now working out how to connect it with online translators and sign language dictionaries. I can imagine that with the right adjustments it could also help in other applications (such as Zoom), where it would help signing people take part more fully in online meetings and conferences. And as for my practical goals, I have my school-leaving exams, choosing a university and the admissions process ahead of me.

Tomáš Mikolov: Which universities are you applying to?

Matyáš Boháček: In the USA I am applying to several. My dream would be to study at Stanford or Berkeley. But studying in America carries high financial costs, so I will see what the options are. In any case I will also apply to the Faculty of Mathematics and Physics, the Czech Technical University and other universities in Europe.

Tomáš Mikolov: I think something will certainly work out. I wish you the best of luck.

Matěj Boháček: Thank you very much. And what are your plans for the future?

Tomáš Mikolov: Essentially I am interested in anything that could lead to some kind of surprise. Finding a new direction, for instance through the emergent evolution I mentioned. And I still see potential in machine learning. An internet search engine, for instance, is old technology from the nineties, and if it were redone using modern models it could work far better — there is an explosion of information on the internet and it would be good if we used machine learning to summarise it to measure. That means developing a generative search engine model that can create for me the pages that best answer what I am looking for and convey information in an interesting, comprehensible way tailored to me. (Editor's note: the article was written before ChatGPT was released.)