What is a neural network?

A neural network is an in­for­ma­tion tech­nol­o­gy system that is based on the structure of the human brain and equips computers with ar­ti­fi­cial in­tel­li­gence features. Neural networks are a central component and one of many methods of modern AI ap­pli­ca­tions, for example chatbots such as ChatGPT.

There are different types of ar­ti­fi­cial neural networks, each of which offers different pos­si­bil­i­ties for in­for­ma­tion pro­cess­ing when it comes to deep learning. Research in this area has made huge leaps in recent years. Neural networks are therefore a key tech­nol­o­gy for teaching machines to think for them­selves, enabling computers to solve problems in­de­pen­dent­ly and improve their skills. Neural networks are now part of mul­ti­modal systems that can combine text, images, audio and video.

How do neural networks work?

Neural networks are based on the structure of the human brain, which processes in­for­ma­tion via a network of neurons.

Ar­ti­fi­cial neural networks can be described as models con­sist­ing of at least two layers — an input layer and an output layer — and usually ad­di­tion­al layers in between (hidden layers). Modern networks such as Con­vo­lu­tion­al Neural Networks (CNNs) or Trans­former models often require many layers, even for simple tasks, because the depth con­tributes to their ef­fi­cien­cy. Each layer of the network contains a large number of spe­cial­ized ar­ti­fi­cial neurons.

Pro­cess­ing in­for­ma­tion in the neural network

In­for­ma­tion pro­cess­ing in the neural network always follows the same process: In­for­ma­tion in the form of patterns or signals is received by the input layer neurons which process it. Each neuron has a weight assigned to it so that neurons receive in­for­ma­tion of different levels of im­por­tance. The weight and the transfer function determine the input which is then forwarded.

In the next step, an ac­ti­va­tion function and threshold value calculate and weight the output value of the neuron. Depending on the value and weight of the in­for­ma­tion, ad­di­tion­al neurons are linked and activated to a greater or lesser extent.

This linking and weighting creates an algorithm which generates a result for each input. With each training, the weighting and thus the algorithm are modified so that the network will deliver even more accurate and better results.

Example of neural network ap­pli­ca­tions

Neural networks can be used for image recog­ni­tion. Unlike humans, a computer cannot tell at a glance whether an image shows a person, plant or object. It has to examine the image for in­di­vid­ual char­ac­ter­is­tics. The computer de­ter­mines which char­ac­ter­is­tics are relevant using the im­ple­ment­ed algorithm or through data analysis.

In each network layer, the system checks the input signals (i.e. the images) for specific criteria such as colors, corners and shapes. With each check, the computer becomes better at being able to determine what is shown in the image.

Initially, the results will be prone to error. When a neural network receives feedback from a human trainer and thus is able to adapt its algorithm, this is called machine learning. Deep learning does not require training from humans. In this case, the system learns from its own ex­pe­ri­ence and improves according to the amount of image materials it has available to it.

Ideally, the final algorithm can ac­cu­rate­ly identify the content of images, re­gard­less of whether they are black and white, the subject’s position, or the per­spec­tive from which the subject is viewed.

Types of neural networks

Different struc­tures of neural networks are used depending on the learning method used and what the intended ap­pli­ca­tion is.

Per­cep­tron

This is the simplest form of neural network and orig­i­nal­ly referred to a network formed from a single neuron which is modified by weight­ings and a threshold value. Now, the term “per­cep­tron” is also used for single-pass feed­for­ward neural networks.

Feed­for­ward neural networks

These ar­ti­fi­cial neural networks can only process in­for­ma­tion in one direction. These can be single-layer networks (i.e. con­sist­ing only of input and output layers) or multi-layer networks with multiple hidden layers.

Recurrent neural networks

In recurrent neural networks, in­for­ma­tion can also go through feedback loops and thus return to previous layers. This feedback allows the system to develop a memory. Recurrent neural networks are used for purposes such as voice recog­ni­tion, trans­la­tion and hand­writ­ing recog­ni­tion.

Con­vo­lu­tion­al neural networks

These networks are a subtype of multi-layer networks. They consist of a minimum of five layers. Pattern recog­ni­tion is conducted on each layer, and the results from one layer are trans­ferred to the next. This type of neural network is used for image recog­ni­tion.

Learning methods

To properly establish con­nec­tions in ar­ti­fi­cial neural networks so that they can perform tasks, the neural networks must first be trained. There are two basic methods:

Su­per­vised learning

In su­per­vised learning, specific outcomes are assigned to each input option. For instance, if the system is designed to recognize cat images, humans oversee the system’s cat­e­go­riza­tion process and provide feedback on which images were correctly iden­ti­fied and which were not. This feedback adjusts the network’s weight­ings, leading to algorithm op­ti­miza­tion.

Un­su­per­vised learning

During un­su­per­vised learning, the result for the task is not pre­de­fined. Instead, the system learns ex­clu­sive­ly from the input in­for­ma­tion. Hebb’s rule and adaptive resonance theory are used in this learning method. Nowadays, practice focuses on al­go­rithms such as Sto­chas­tic Gradient Descent (SGD).

Ap­pli­ca­tions for neural networks

Neural networks are es­pe­cial­ly effective when there is a large amount of data to be evaluated and only limited sys­tem­at­ic problem-solving knowledge. Classic ap­pli­ca­tion uses include text, image and voice recog­ni­tion (i.e. sit­u­a­tions in which computers examine data for specific char­ac­ter­is­tics in order to cat­e­go­rize it).

Neural networks such as con­vo­lu­tion­al neural networks (CNNs) enable computers to recognize content in images. This tech­nol­o­gy is used in medical image analysis or in automated quality control in industry. There, neural networks are sometimes used in control en­gi­neer­ing, in which they monitor target values and au­to­mat­i­cal­ly take coun­ter­mea­sures in the event of de­vi­a­tions or in which they in­de­pen­dent­ly specify target values based on their data eval­u­a­tion.

Language models such as ChatGPT, which are based on neural networks, generate realistic-sounding texts, answer questions or analyze large amounts of text data.

Ar­ti­fi­cial neural networks can also be used for pre­dic­tions and sim­u­la­tions (e.g. weather forecasts or medical diagnoses). Con­vo­lu­tion­al neural networks (CNNs), for example, enable computers to recognize content in images. This tech­nol­o­gy is used in medical image analysis, for example to identify tumors on X-ray images.

De­vel­op­ments in the area of un­su­per­vised learning of ar­ti­fi­cial neural networks are in the process of massively expanding the ap­pli­ca­tion area and per­for­mance of the networks. One of the most prominent ap­pli­ca­tions of self-learning neural networks is speech synthesis of voice as­sis­tants. Systems such as Alexa, Siri or Google Assistant use neural networks to convert spoken language into text and react to it. Trans­former models such as GPT or BERT have rev­o­lu­tion­ized the quality of machine trans­la­tions.

History and future outlook

In the past ten years, neural networks have moved into the public con­scious­ness due to the dis­cus­sions around ar­ti­fi­cial in­tel­li­gence. However, the basic tech­nol­o­gy has already been around for many decades.

Talk of ar­ti­fi­cial neural networks can be dated back to the early 1940s. Back then, Warren McCulloch and Walter Pitts described a model that linked together el­e­men­tary units and was based on the structure of the human brain. It was supposed to be able to calculate almost any arith­metic function. In 1949, Donald Hebb developed Hebb’s rule which we explained about earlier. It is still used in many neural networks today.

In 1960, a neural network was developed which had worldwide com­mer­cial use for echo filtering in analog tele­phones. After that, research in the field ground to a halt. One reason for this was that leading sci­en­tists had come to the con­clu­sion that the neural network model could not solve important problems. Another was that effective machine learning requires large amounts of digital data which was not available at the time.

This only changed with the advent of big data. With the in­tro­duc­tion of the back­prop­a­ga­tion algorithm, the training of multi-layer networks became possible. This laid the foun­da­tion for modern deep learning models. The com­bi­na­tion of enormous amounts of data and the computing power of modern graphics pro­cess­ing units (GPUs) led to a break­through in the 2010s. Frame­works such as Ten­sor­Flow and PyTorch made the de­vel­op­ment of neural networks more ac­ces­si­ble.

Interest in ar­ti­fi­cial in­tel­li­gence and neural networks resur­faced, and the victory of a CNN in the ImageNet com­pe­ti­tion in 2012 marked the beginning of modern deep learning. Since then, this tech­nol­o­gy has rapidly gained sig­nif­i­cance and is shaping nearly every field of computer science.

Since then, de­vel­op­ment in this field has continued at a rapid pace. As promising as the results are, neural networks are not the only tech­nol­o­gy to implement ar­ti­fi­cial in­tel­li­gence in computers. They are only one option, even if they are often presented in the public debate as the only prac­ti­ca­ble way. Today, research goes beyond classical neural networks. The focus is on mul­ti­modal models that combine text, image and speech, and ap­proach­es to reduce energy con­sump­tion. At the same time, neural networks are being more and more in­te­grat­ed into everyday ap­pli­ca­tions, from smart­phones to in­tel­li­gent household ap­pli­ances.