An AI Revolution in Sports Scoring Technology

Never Done Before

Sports can be categorized into competitions like swimming and athletics in which athletes are ranked based on their times, heights, or distances, competitions like jūdō and basketball in which athletes or teams compete against each other, and judged sports, in which athletes’ performance is scored by judges. Gymnastics, the archetypal judged sport, has undergone a revolutionary development in recent years in the form of a system that uses artificial intelligence to assess gymnasts’ performance.

The system made its official debut at the 2019 World Artistic Gymnastic Championships, where it was used to score the men’s pommel horse and still rings and the women’s vault events. This year, at the World Gymnastics Championships, held in Kitakyūshū from October 18 to 24, the system was used to score half of the 10 (men’s and women’s) events, as well as being subjected to final testing on two additional events.

Roughly speaking, the system comprises a scanning unit and a database. It works by first shining two million laser pulses per second on the gymnast to obtain three-dimensional coordinates corresponding to the position of his or her body, before using artificial intelligence to determine the positions of the gymnast’s frame and joints and create a three-dimensional rendering of the routine. This data is then compared with a database of gymnastic routines to determine what kind of routine was performed. Then, in accordance with the relevant code, the system identifies elements subject to additions or deductions and displays the result as a score.

Overview of the judging support system: A laser scans the gymnast to obtain three-dimensional coordinates (top left); the system determines the position of the gymnast’s body and creates a three-dimensional rendering of the routine (top right); the system identifies the elements used in the routine (two frames at bottom left); and the routine is identified by comparing it with database records (bottom right).

While scores awarded by the system are reviewed by judges before being declared final, the system has proven so accurate that the process is often referred to as “automated scoring.” In recent years, electronic line judges have come into use in tennis and soccer, but the automation of events as complex as gymnastics had not been attempted, making the new system not only highly sophisticated but also a world first.

Judged sports like gymnastics have long faced challenges. Because these sports are judged with the human eye, competitors sometimes dispute the judge’s assessment of whether a routine was successful or whether points were awarded or deducted appropriately, which occasionally leads to allegations of scoring errors. The new system, which uses artificial intelligence to objectively assess gymnasts’ routines, received a phenomenal response for its ability to solve problems like these.

Testing the Limits of the Human Eye

The system was developed by the IT manufacturer Fujitsū. Fujiwara Hidenori, director of Fujitsu’s sports business division, says the process of getting the system to a place where it could be put in practice was extremely difficult

While motion capture technology, in which markers are attached to actors’ bodies to record their movements, is already used in film production, engineers had yet to design a system that was able to capture movement without making contact with or affixing markers to its subject. The development team therefore had to start out by building such a system.

However, gymnastics is a sport that requires athletes to perform a series of rapid, complex movements. In the course of a routine, gymnasts stretch, curl, and move their limbs and joints in various ways. The team decided that they needed a brand-new technology to build a device capable of capturing this information.

The revolutionary 3D sensor unit is able to record gymnasts’ movements without the use of motion capture markers.

Enabling the system to differentiate between gymnasts and apparatus was also a challenge.

“Take the rings,” says Fujiwara. “Because the rings and the cables holding them move during a routine, the system could perceive them as an extension of the gymnast’s arms, resulting in images that looked more like crustaceans than humans.”

In order to convert captured movements into scores, the system needed to be able to not only identify the type of routine being performed, but also to rank the routines of different gymnasts. The team performed matching by feeding a digitized version of the code of points into an AI-based system. There are some 1,400 men’s and women’s routines in all, all of which needed to be learned by the system.

Fujiwara explains: “First we tried the pommel horse, which we thought would be easy because the gymnast performs on a stationary platform. But gymnasts on the pommel horse actually perform several different moves as they are twirling around, so it was difficult to determine where one move started and the next began. Getting the system to make this distinction was really hard work. Of course, those in the gymnastics community said they couldn’t see anything difficult about it.”

Data captured with 3D sensors takes the form of a constellation of points.

In the course of learning about the world of gymnastics, the team learned just how difficult a process traditional scoring is. During an event, judges use shorthand to note down the elements used in the routine, before tallying up the points. There is nothing at all hi-tech about the approach. Recently, the rising standard of competition has meant that events that used to be scored by a single judge now require something in the vicinity of 10. In fact, one recent event required a total of 120 judges to score all the men’s and women’s events.

Fujiwara was keenly aware of the fact that you could only go so far using human judges to score gymnastics. A case in point is this excerpt from the code of points for the pommel horse: “If the feet become un-pointed as the gymnast is rising to the handstand, 0.1 of a point is deducted for a deviation of up to 15°, 0.3 of a point is deducted for a deviation of between 16 and 30°, and 0.5 of a point is deducted for a deviation of between 31 and 45°.”

Therefore a difference of just 1 degree affects the number of points are deducted. The question is whether the human eye is able to discern such a subtle difference. Elsewhere, the code mentions a deduction for bent limbs, with “0.1 point off for slight bending” leaving it down to the judges to make a judgement call on what constitutes a straight arm or leg as opposed to slightly bent one. Amid claims that scoring differed from competition to competition, these issues were frequently a cause of criticism and dissatisfaction from gymnasts.

In 2017, in partnership with Fédération Internationale de Gymnastique, Fujitsū gathered a large volume of data on gymnastics events as it worked to create concrete definitions for what had been ambiguous rules. In addition to the difficulty of the development process, Fujitsū was also initially the subject of criticism from judges.

“Like workers in any other industry, judges were concerned that artificial intelligence would take their jobs away,” says Fujiwara.

The team responded by doggedly extolling the system’s benefits to gymnastics, such as more accurate and fairer scoring. In fact, the system’s biggest advocates were the gymnasts themselves, and this helped keep the project moving forward.

It Began with a Misunderstanding

The AI judging support system, whose development faced the hurdles described above, has been used in many major gymnastics events, beginning with the 2019 World Championships. In the World Gymnastics Championships set to take place in Liverpool in 2022, the system will be subjected to final tests on the men’s and women’s floor events and men’s parallel bars, as Fujitsū moves towards its aim of creating a system that can be used for all 10 events.

Fujiwara Hidenori, head of Fujitsu’s second Sports Business Development Division, overcame many obstacles to develop the revolutionary system.

With the labors of the development process now bearing fruit, the system has now become part of major gymnastics events. However, the project was in fact the result of a misunderstanding. When Tokyo won the bid to host the 2020 Olympics and Paralympics, Fujiwara, who had been looking for sporting applications for IT technology, began talking to representatives from various sporting bodies. One of these was Watanabe Morinari, the current president of the FIG, who told Fujiwara that he would like to have robotic judges to score the events.

“When I asked Watanabe about his comment later, he said he was joking,” recalls Fujiwara with a laugh. Nonetheless, he took up the challenge and decided to build a system to do the job. Around six months after his conversation with Watanabe, Fujiwara presented a prototype of the system at meeting of the FIG in Japan.

“We set up a real pommel horse, much to the vexation of the hotel where we met their team,” he says.

The team had gymnasts perform routines and displayed computerized images on a monitor behind them.

“It resembled a kamishibai,” says Fujiwara—a traditional Japanese storytelling approach accompanied by picture cards.

At that stage, the team had yet to perfect AI-based scoring. They therefore created a manual rendering of what the system would eventually do. Promoting the system on the basis of a nonexistent technology was risky, but Fujiwara was not afraid of risk.

“In many cases, you start off with a proprietary technology and then build a device around that technology,” notes Fujiwara. “We took the approach of starting out with a futuristic idea, though. Even though the technology is not here yet, we show what the future will be like when it finally arrives. We then make the technology catch up. When considering how you can make the world a better place, I believe it’s important to be design-focused—that is, to have a design for society.”

Fujiwara believes that this kind of thinking is the seed of innovation. However, he says he was met with opposition from his colleagues: “Engineers told me that I was attempting the impossible.”

In fact, the opposition was not limited to engineers. Fujiwara failed to win the support of many of the divisions in his organization, and struggled to have his proposal approved. There was a feeling that even if the technology could be developed, it might not be commercially viable. Nonetheless, Fujiwara hung in there and kept advocating the project. He saw potential for applications beyond gymnastics scoring.

Possibilities Beyond Sport

Fujiwara explains: “This technology can be applied to other sports, and even to other fields, such as fitness. Instructors taking online fitness classes don’t monitor their students so can’t tell whether they are doing their exercises correctly. If artificial intelligence can be used to give participants appropriate feedback, they will be able to exercise properly while at home. One can also conceive of a system that ‘kills two birds with one stone’ by tracking the movement of elderly hospital outpatients from the hospital entrance to the reception desk, so they can be given advice on their posture and gait.”

Fujiwara also talks of potential industrial and artisanal applications. There is a “right way” of performing every artisanal technique. Attaching numeric values to actions that currently have to be picked up intuitively could help people acquire traditional artisanal skills.

“When you glance through the annals of history, you see how modern technological progress began with the steam engine, followed by the advent of electricity, then computers, and finally the Internet of things,” Fujiwara argues. “In the computer age, we were still asking how we could use technology to make tasks less labor intensive. However, since the advent of the internet of things, the trend has been for human operators to become completely redundant. These days, I believe technology should be harnessed to make people happier and healthier, and I believe our gymnastics scoring system is a first step toward that,” says Fujiwara.

A project that started as a joke has given rise to diverse possibilities that may have far reaching implications for the very way we live our lives.

(Originally published in Japanese. Banner photo: The judging support system is already in use at World Artistic Gymnastics Championships and other international sporting events. The judge’s monitor displays scores and statistics. All photos courtesy of Fujitsū.)