The NexGen Algorithms

Modern evolutionary algorithms do not learn. They can be adaptive during the search of solution, but when the run is over, they forget everything and if we re-run the problem, the algorithm starts its work from a scratch. This is normal for algorithms, but if we look at all living beings we'll see that they do not forget everything that catastrophically! Yes, single experience can be resultless, but after trying to solve some problem for hundreds and thousands times a new knowledge and experience eventually emerges. This is the way people learn to walk, speak, handle spoon and many other things. As people get older their past experience helps them to learn things easier (if these new things correlate with their experience).
From the other hand there are incremental learning algorithms, which are being more or less actively developed for the last 2 decades. Those algorithms can learn in complexifying and/or dynamic environments, which makes them more adaptive and they do not suffer that much from 'afterun amnesia'. There are certain problems though, like 'catastrophic forgetting', when after several updates algorithm totally forgets what it learned initially and starts behave in a new unexpected way. Well, that's normal for humans, it's just like if some person learned to speak English in a childhood and after living in France for years speaking in French this person forgets its first language. I believe that if catastrophic forgetting is not very fast then it's not a major problem. The larger issue here is that incremental earning algorithms usually are applicable to a certain domain only. That is they do learn something and collect past experience, but this experience is almost useless for solving other problems. So 'increamental learning' can be called something like 'temporal adaptation' and is good in its way, but still far from true learning algorithm. And, yes, increamental learning is usually for machine learning, but not for optimization.
The ultimate goal here may be in creation of the algorithms, which simply learns from any problem and extracts some general abstract regularities and rules to improve its performance for future tasks. This is closely related to the meta-learning concept and I believe that this can be done if some external trainable control system is attached to the algorithm to guide this algorithm's behaviour. This is a very amibitious task, connected with creation of very adaptive algorithms and probably one of the milestones to overcome on the way to true AI, since some implicit method 'to learn how to learn' is already built-in by evolution inside living organisms (animals can improve their skills!) and is essential for survival and for emergence of highly-organized intelligence.
There's a well-known meta-systems transition theory by V. Turchin which states that system enters a new level of development when a meta-level control system appears. For the existing evolutionary algorithms some origins of such systems can be found, like rules for adaptation or some heuristics like: if we move in the same direction, then increase movement speed; or try to cluster variables to understand their relations for proper optimization. And the next big step here is possible unification of these heuristics and adding memory fo EA, so that the algorithm could remember what problems it already solved and what were useful and useless rules. Having this memory it could be possible to recognize some common parts between problems (like, 'hmm, I solved something like this before, and I probably should apply correpondent set of optimization rules'). That may be my fantasy, and there are a lot of open questions (how to store information about problems features and compare problems of different dimensionalities etc.), but this is a real task for scientists, which probably can boost development of algorithms. That's why NexGen :)    

Gradiention Day

This is a sketch for a review on meta-heuristic tricks for learning properties of a search space. More formally, a lot of advanced (and not so much) algorithms try to figure out an information about local gradient for more efficient evolutionary search. This is surely feasible, because (ideally) the gradient shows the best direction to move towards local optimum and most numeric optimization methods use this information. The question about how to get away from a local optimum concerns another (although associated) problem and is not considered.

It's worth noting, that getting gradient info for noisy functions can be difficult and requires some statistical processing, which may result in several alternatives for the best direction (not obligatory collinear). Probably fuzzy logic could be of help here. I also assume that objective function has local trend, i.e. that for each point *there is* the best movement direction, which is at least good for some not very small neighborhood, because otherwise finding gradient doesn't make sense. One can say that this means that the function can be approximated by some smooth function with the same global optimum.

We can find 2 groups of methods, which learn gradient information:
1. Explicit gradient learning. There are formulae for gradient approximation, which are used during recombination or variation operations.
2. Implicit gradient learning. These methods simply do not tell directly that they are learning gradient, but they are sort of 'I'll-know-it-when-I-see-it'.

Examples of the first group are: EA with direction crossover and Pseudo-Gradient, CMA-ES, many real-valued EDAs and (h)BOA algorithms.

For the second group examples are: Swarm Optimization Methods, Differential Evolution.

In my opinion, the meta-heuristic method can be aknowledged as 'gradient learning' if there is sort of memory for the best movement direction.

Where are conventional real-coded EAs? In my current understanding it should be placed into the 2nd group. Here is my motivation: many EA operators do not look for gradient, but simply search for better solutions in vicinity of parent individuals. Most operators do not 'remember' good searching directions. But there is a selection, which selects in which direction the population evolves and thus follows the gradient. Probably many EAs should even be treated as 'implicitly implicit' gradient learning methods :).

Up to now only continuous spaces were considered, because gradient is natural there. What about binary spaces? Or spaces of graphs, computer programs, agents policies or neural networks? I believe that the gradient notion can still be applied there as long as there is a proper metric over the set of representations.

Recall that there's a large class of finite-difference methods, which quintify the space making it non-continous, but can still be succesfully applied for continous modelling problems. It can be also argued that one of the most used variant of the Taylor expansion has linear form from which the approximate gradient can be calculated.

So the largest problem for defining gradient in non-continous spaces is setting a good metric. By this 'good' word I mean that it should not be just everything, which satisfies metric properties, but it also should reflect the problem's properties. For example, setting metric for directed weighted graphs for TSP problem is not the same as setting metric for directed weighted graphs for evolution of neural networks. That's the issue. An interesting question: if a kind of 'approximate' metrics exists, which is metrics, but without the whole pack of desired problem dependent properties?