The History and Future of NeurIPS

Article by Hengtee Lim | July 30, 2020

The Conference and Workshop on Neural Information Processing Systems (NeurIPS) has become an integral part of the machine learning community since its establishment in 1986. While the conference began as an interdisciplinary gathering for researchers exploring biological and artificial neural networks, many now see it as one of the most important conferences for machine learning and artificial intelligence.

With the announcement of the first ever entirely virtual NeurIPS conference in 2020, we reached out to the current President of the NeurIPS Foundation, Terrence Sejnowski, and Secretary Michael Mozer, to get their thoughts on the development of the conference over the last 30+ years. In this interview, we touch on both past and current trends, the move to a virtual setting, and the responsibility of the conference in the future.

 

The Early Years of NeurIPS

Sejnowski and Mozer both attended the conference in its early years. Sejnowski was the general chair in 1988 and became President in 1993. Meanwhile, Mozer happened to start a faculty position at the University of Colorado in Boulder, which was not far from the location of the early conferences. Both published papers at many of the early conferences.

“Those were exciting times,” said Sejnowski. “Most of the learning algorithms currently used in deep learning were invented in the 1980s. Though the meeting has grown, it has retained its scientific and engineering diversity, and its energy.”

Mozer remembers the old conferences as being more interdisciplinary, featuring more neuroscientists and cognitive scientists. The scale was smaller, too, meaning people could get a general sampling of what was happening in the field, and see all the posters in the display sessions.

“The [early conferences] were far more intimate and spontaneous,” he said. “Although there was excitement in the field, there wasn’t the big bucks, so the folks at the meeting were doing it for the love of the field.”

 

Trends: Past and Present

As long-time attendees, Sejnowski and Mozer have both had the chance to see the development of trends in the papers and presentations that are a hallmark of the conference. Mozer sees a circularity to research topics, a trend he hopes is moving in a positive direction.

“Research circles back to topics studied previously, and although ideas are often reinvented, this reinvention happens on a grander scale,” said Mozer. “Neural nets are a good example. There have been three surges of interest in the topic, from Perceptrons in the 1960s, to back propagation and connectionism in the 1980s, to deep learning in the 2010s.”

Sejnowski has also noticed the trend towards deep learning, and said, “Deep learning has put the Neur back into NeurIPS, primarily because learning algorithms for neural networks have scaled well and hardware has greatly advanced.”

However, these are not the only areas growing in popularity, and Sejnowski said that as research expands, the conference is seeing a diverse menu of algorithms for analyzing data and solving difficult computational problems.

However, as the field has grown, research areas have become more specific. Mozer noted that whereas the field was once small enough to spot trends easily, it’s becoming more difficult as the conference grows. “The trends seem to be happening within subfields, which makes it hard for nonspecialists to notice,” he said. “For example, in vision models, key-value attention mechanisms have become very popular, sometimes replacing local connectivity and convolutions.”

More recently though, Sejnowski has observed a deeper convergence between deep learning and neuroscience over the last decade. He commented, “Machine learning is being used to analyze from a million neurons recorded simultaneously, leading to discoveries that will inspire the next generation of large-scale ML systems.”

 

NeurIPS Going Virtual in 2020

Although the COVID-19 pandemic has forced events worldwide to consider virtual and online alternatives, Sejnowski and Mozer both said plans were already in motion for NeurIPS to move towards a hybrid online/in-person conference.

One of the reasons for this was the conference’s exponential growth. The number of registered participants went from 5,000 in 2016 to 13,000 in 2019 (with another 6,000 on a waiting list). Tickets for NeurIPS 2018 reportedly sold out in under 12 minutes, prompting a move to a lottery system. Many of the conference’s main presentations are also already live streamed.

“Before the pandemic,” said Sejnowski, “the Board of Trustees was concerned that the exponential growth of the in-person meeting would not be sustainable. Virtual meetings solved that problem.”

Mozer believes there are many benefits to having virtual conferences. The conference itself will be more accessible and the registration cap can be lifted. Travel and accommodation costs will no longer be an issue and registration becomes significantly cheaper. However, he also laments the lack of engagement sometimes apparent in virtual conferences. “It’s really hard to engage with a virtual meeting as one does with an in-person conference. When you travel to a conference, you’re physically removed from your ordinary life, so it’s much easier to break the routine.”

For this reason, he sees future conferences being a mix of virtual and in-person. “Although the meeting is purely virtual this year, everyone expects a hybrid virtual/real meeting moving forward. In addition, the plan is for the meeting to become more geographically distributed. Rather than having everyone in the world converge on one location, we are moving towards a model in which the conference may occur in multiple locations at once.”

This model would allow for individuals to catch live streaming talks and presentations closer to home, while preserving the ability to connect and network with other people in the field.

 

Challenges for the ML Community

Perhaps unsurprisingly, both Sejnowski and Mozer see the broad social issues of AI application as the most pressing challenge for the machine learning community at present. 2020 has already shown issues regarding bias in AI technology, data privacy as it relates to facial recognition, and identity theft in deepfake technology.

“Social issues are evolving rapidly as AI applications make their way into society,” said Sejnowski. “NeurIPS has been a leader in increasing awareness of ethics in AI and in promoting diversity and inclusion. We have co-hosted Women in Machine Learning for over 15 years and are supporting several other affinity groups.”

Mozer also sees similar issues in the future, adding, “The technology is so readily accessible that the bar for deploying models is pretty low and the potential for deploying bad or biased models is high.”

To help address some of these issues, the conference now requires that researchers submitting papers state the impact of their work on society. This equates to statements regarding the ethical and future societal consequences of work, and any conflicts of interest. It also has dedicated Diversity & Inclusion chairs whose work aims to expand the grassroots efforts of groups including Women in Machine Learning, Black in AI, Queer in AI, and LatinX in AI.

 

The Future of NeurIPS

As the NeurIPS conference grows with each passing year, I wondered how Sejnowski and Mozer saw its role in the future. The last few years have seen growing interest in ML and AI research and development, and the wider application of AI technology means more media coverage, and more scrutiny from within the community itself.

Mozer pointed to a wider support net, and said, “Recent conference chairs have considered the broader societal role of the meeting. They’ve focused on inclusion and diversity issues, identifying and supporting the next generation of researching. We’re also becoming more proactive with the press, showcasing top technical work in tech companies, and attempting to support smaller, less-established academic exchanges.”

Sejnowski envisions the conference as a platform for building the future. “I am optimistic that NeurIPS will continue to evolve along with the larger ML community that we support,” he said. “A new generation of young ML researchers is entering the field, and they are building the next level of AI on solid foundations.”


The 2020 NeurIPS conference will be entirely virtual and is scheduled for Dec. 5-12. More information on the registration process, talks and workshops, and the event schedule will be updated through the official event website. For more information on upcoming AI conferences in 2020 and 2021, please check out our conferences page.

 

Michael Mozer (left) and Terrence Sejnowski (right)

About the Interviewees

Terrence Sejnowski is a pioneer in the fields of neural networks and computational neuroscience. He demonstrated the ability of neural networks to learn tasks in the early 80s, and is also known for pioneering the application of learning algorithms to difficult problems in speech and vision. At present, his work focuses on architectures for lifelong learning, computing with spiking units, and analyzing large-scale neural recordings.

Michael Mozer’s research walks a line between cognitive science and machine learning. His work is in the field of “human optimization,” which researches the leveraging of machine learning models to improve how humans learn, perceive, and reason. Some of his work utilizes theories and data from the literature on human cognition to inspire novel machine learning methods. He refers to this work as “cognitively informed AI.”

 

About Lionbridge

With over 20 years of experience as a trusted training data source, Lionbridge AI helps businesses large and small build, test, and improve machine learning models. Our community of 1,000,000+ qualified contributors is located across the globe and available 24/7, providing access to a huge volume of data across all languages and file types. Get in touch today.

Subscribe to our newsletter for for more news and interviews:
The Author
Hengtee Lim

Hengtee is a writer with the Lionbridge marketing team. An Australian who now calls Tokyo home, you will often find him crafting short stories in cafes and coffee shops around the city.

    Welcome!

    Sign up to our newsletter for fresh developments from the world of training data. Lionbridge brings you interviews with industry experts, dataset collections and more.