Fundamental Limits of Learning (Fun LoL) Request for Information (RFI)The Defense Advanced Research Projects Agency (DARPA) Defense Sciences Office (DSO) is requesting information on research related to the investigation and characterization of fundamental limits of machine learning with supportive theoretical foundations. Although the main focus is on machine learning, extensions and implications for human-machine systems are also of interest. The notion of fundamental limits here means that the conclusion about achievable performance limits should hold independent of specific learning methods or algorithms.Current machine learning techniques often rely on huge amounts of training data, take prolonged periods of time for learning, and follow a trial and error methodology. Resulting learned concepts have poor generalization capability. Much of the success of such methods to date has been enabled by the availability of large datasets and platforms that can provide vast amounts of computational resources to be used for training. Less understood is how performance of current methods fares compared to the limits of what could be achieved for a given learning problem or even how such limits might be determined. For example, how efficient is a given learning method and how close is its performance relative to what theoretically might be achieved? Methodologies for answering such questions would enable evaluation of the capabilities of learning system designs and guide practical implementations relative to the capability limit of what is ultimately possible. DARPA seeks to address general problems involving learning systems by asking a central scientific question which is the focus of this RFI: What are the fundamental limitations inherent in machine learning systems?Defense Advanced Research Projects AgencyDARPA_46ad30a4-ad5d-11df-9c96-10167a64ea2aDr. Reza GhanadanPrimary Point of Contact -- Program Manager, DARPA/DSO.Multi-Disciplinary TeamsDARPA encourages responses from multi-disciplinary teams with expertise that might inform novel approaches to this problem. Areas of expertise that may be relevant include Machine Learning, Information Theory, Computer Science Theory, Statistics, Control Theory, Artificial Intelligence, and Cognitive Science._7eb9669c-2c60-11e6-af3a-4288bea16470To develop, validate, and apply a theoretical framework for learning_7eb967c8-2c60-11e6-af3a-4288bea16470LearningDARPA seeks information regarding mathematical frameworks for learning systems with quantifiable and generalizable measures of learning and limits to achievable performance in machine learning systems, along with architectures and methods to achieve them.QuantificationGeneralizationPerformancePracticalityOf interest is information about frameworks that can ultimately answer a number of questions for a given problem that will be useful in practical applications, such as:ExamplesWhat are the number of examples necessary for training to achieve a given accuracy performance?TrainingTrade-OffsWhat are important trade-offs and their implications?EfficiencyHow "efficient" is a given learning algorithm for a given problem?AccuracyHow close is the expected achievable performance of a learning algorithm compared to what can be achieved at the limit?ResilienceWhat are the effects of noise and error in the training data?StatisticsWhat are the potential gains possible due to the statistical structure of the model generating the data, due to any prior knowledge, or due to the nature of the concept or task that is being learned?KnowledgeMetricsProvide quantifiable and generalizable measures of learning and fundamental limits across learning settings._7eb9685e-2c60-11e6-af3a-4288bea164701Foundational general theory: Articulation of a general mathematical framework that, independent of any particular machine learning method, provides quantifiable and generalizable measures of learning and fundamental limits across supervised, unsupervised, and reinforcement learning settings. Such a framework may need to consider a number of factors such as:SignalsConsider the type, quality, and relevance of the signals available from the data (e.g., labels, distance/similarity measures, rewards)._7eb968ea-2c60-11e6-af3a-4288bea164701.1Structure, Complexity & ObservabilityConsider the structure, complexity, and observability of the target task/concept space._7eb96976-2c60-11e6-af3a-4288bea164701.2TasksConsider task performance metrics (e.g., accuracy, speed, computational complexity, sample complexity) as well as the interactions and trade-offs among such factors._7eb96a02-2c60-11e6-af3a-4288bea164701.3ApplicationsCharacterize the capabilities or performance envelopes of current learning techniques._7eb96a98-2c60-11e6-af3a-4288bea164702Applications of theory: Application of a general framework to existing machine learning methods in order to characterize the capabilities or performance envelopes of current techniques in each of the following categories: * Supervised learning from example input/output pairs (e.g., deep neural networks, decision trees, support vector machines, random forests, etc.); * Unsupervised learning from only input data (e.g., clustering, topic models, principal component analysis, community detection, etc.); and * Reinforcement (policy) learning from reward/penalty signals (e.g., Q-learning, direct policy search, etc.). This Research Area should also address questions, such as: * What are the performance limits of methods/architectures that combine the algorithms from the above categories? * For what classes of problems are these algorithms optimal or near the performance limit?CombinationCombine learning algorithms_7eb96b2e-2c60-11e6-af3a-4288bea164702.1OptimizationDetermine for which classes of problems these algorithms are optimal or near the performance limit_7eb96bba-2c60-11e6-af3a-4288bea164702.22016-05-122016-06-072016-06-07https://www.fbo.gov/index?s=opportunity&mode=form&id=f1e29b723a39c00b7f32694e7fc5ac27&tab=core&_cview=0OwenAmburOwen.Ambur@verizon.net