Fujitsu Laboratories, Ltd. has developed the world's first AI technology that accurately captures essential features, including the distribution and probability of high-dimensional data in order to improve the accuracy of AI detection and judgment.
High-dimensional data, which includes communications networks access data, types of medical data, and images, remains difficult to process due to its complexity, making it a challenge to obtain the characteristics of the target data. Until now, this made it necessary to use techniques to reduce the dimensions of the input data using deep learning, at times causing the AI to make incorrect judgments.
Fujitsu has combined deep learning technology with its expertise in image compression technology, cultivated over many years, to develop an AI technology that makes it possible to optimize the processing of high-dimensional data with deep learning technology, and to accurately extract data features. It combines information theory used in image compression with deep learning, optimizing the number of dimensions to be reduced in high-dimensional data and the distribution of the data after the dimension reduction by deep learning.
Akira Nakagawa, (Associate fellow) of Fujitsu Laboratories commented, "This represents an important step to addressing one of the key challenges in the AI field in recent years: capturing the probability and distribution of data. We believe that this technology will contribute to performance improvements for AI, and we're excited about the possibility of applying this knowledge to improve a variety of AI technologies."
Details of this technology will be presented at the International Conference on Machine Learning "ICML 2020 (International Conference on Machine Learning 2020)" on Sunday, July 12.
In recent years, there has been a surge in demand for AI-driven big data analysis in various business fields. AI is also expected to help support the detection of anomalies in data to reveal things like unauthorized attempts to access networks, or abnormalities in medical data for thyroid values or arrhythmia data.
Data used in many business operations is high-dimensional data. As the number of dimensions of data increases, the complexity of calculations required to accurately characterize the data increases exponentially, a phenomenon widely known as the "Curse of Dimensionality"(1). In recent years, a method of reducing the dimensions of input data using deep learning has been identified as a promising candidate for helping to avoid this problem. However, since the number of dimensions is reduced without considering the data distribution and probability of occurrence after the reduction, the characteristics of the data have not been accurately captured, and the recognition accuracy of the AI is limited and misjudgment can occur (Figure 1). Solving these problems and accurately acquiring the distribution and probability of high-dimensional data remain important issues in the AI field.
About the Newly Developed Technology
Fujitsu has developed the world's first AI technology that accurately captures the characteristics of high-dimensional data without labeled training data.
Fujitsu tested the new technology against benchmarks for detecting data abnormalities in different fields, including communication access data distributed by the International Society for Data Mining "Knowledge Discovery and Data Mining (KDD)", thyroid gland numerical data and arrhythmia data distributed by the University of California, Irvine. The newly developed technology successfully achieved the world's highest accuracy in all data with up to a 37% improvement over conventional deep-learning based error rates. Since this technology solves one of the fundamental challenges in the field of AI, which is how to accurately capture the characteristics of data, it is expected to prove an important development to unlocking a wide range of new applications.
Press release by Fujitsu
Publié le 21 juillet 2020