There may be a couple of approach to visualize high-dimensional data. Here, we return within the history of AI to explore the evolution of those visualizations.

I submitted my first paper on AI in 1990 to a small, local conference — the “Midwest Artificial Intelligence and Cognitive Science Society.” In those days, the AI field was entirely defined by research into “symbols.” This approach was often known as “Good, Old-Fashion AI” or GOFAI (pronounced “go fi” as in “wifi”). Those of us working in what’s now often known as “Deep Learning” had to essentially argue that what we were researching should even be considered as AI.

Being excluded from AI was a double-edged sword. On the one hand, I didn’t agree with most of the essential tenets of what was defined as AI on the time. The essential assumption was that “symbols” and “symbol processing” should be the muse of all AI. So, I used to be joyful to be working in an area that wasn’t even considered to be AI. Then again, it was difficult to seek out people willing to take heed to your ideas in the event you didn’t package it as no less than related to AI.

This little conference accepted papers on “AI” and “Cognitive Science” — which I saw as an invite for ideas outside of just “symbolic processing.” So I submitted my first paper, and it was accepted! The paper featured a neural network approach to handling natural language. Lots of us on this area called the sort of neural network research “connectionism,” but now days the sort of research, as mentioned, could be labeled “Deep Learning” (DL) — although my initial research wasn’t very deep… only three layers! Modern DL systems could be composed of a whole lot of layers.

My paper was accepted on the conference, and I presented it in Carbondale, Illinois in 1990. Later, the organizer of the conference, John Dinsmore, invited me to submit a version of the paper for a book that he was putting together. I didn’t think I could get a paper together by myself, so I asked two of my graduate school friends (Lisa Meeden and Jim Marshall) to hitch me. They did, and we ended up with a chapter within the book. The book was titled “The Symbolic and Connectionist Paradigms: Closing the Gap.” Our paper slot in nicely with the theme of the book. We titled our paper “Exploring the symbolic/subsymbolic continuum: A case study of RAAM.” To my delight, the book focused on this split between these two approaches to AI. I believe the sector remains to be wrestling with this divide to this present day.

I’ll say more about that initial research of mine later. For now I would like to speak about how the sector was coping with easy methods to visualization “embeddings.” First, we didn’t call these vectors “embeddings” on the time. Most research used a phrase reminiscent of “hidden-layer representations.” That included any internal representation that a connectionist system had learned with a purpose to solve an issue. As we defined them back then, there have been three kinds of layers: “input” (where you plugged within the dataset), “output” (where you place the specified outputs, or “targets”), and every part else — the “hidden” layers. The hidden layers are where the activations of the network flow between the input and the output. The hidden-layer activations are sometimes high-dimensional, and are the representations of the “concepts” learned by the network.

Like today, visualizing these high-dimension vectors was seen to assist in giving insight into understanding how these systems work, and oftentimes fail. In our chapter within the book, we used three kinds of visualizations:

1. So-called “Hinton Diagrams”
2. Cluster Diagrams, or Dendograms
3. Projection into 2D space

The primary method was a newly-created idea utilized by Hinton and Shallice in 1991. (That is identical Geoffrey Hinton that we all know today. More on him in a future article). This diagram is an easy idea with limited utility. The essential idea is that activations, weights, or any style of numeric data, could be represented by boxes: white boxes (typically representing positive numbers), and black boxes (typically representing negative numbers). As well as, the scale of the box represents a price’s magnitude in relation to the utmost and minimum values within the simulated neuron.

Here is the representation from our paper showing the typical “embeddings” on the hidden layer of the network as a representation of words were presented to the network:

The Hinton diagram does help to visualise patterns in the info. But they don’t really assist in understanding the relationships between the representations, nor does it help when the variety of dimensions gets much larger. Modern embeddings can have many 1000’s of dimensions.

To assist with those issues, we turn to the second method: cluster diagrams or dendograms. These are diagrams that show the gap (nevertheless defined) between any two patterns as a hierarchical tree. Here is an example from our paper using euclidean distance:

This is identical kind of knowledge shown within the Hinton Diagram, but in a far more useful format. Here we are able to see the interior relationships between individual patterns, and overall patterns. Note that the vertical ordering is irrelevant: the horizontal position of the branch points is the meaningful aspect of the diagram.

Within the above dendogram, we constructed the general image by hand, given the tree cluster computed by a program. Today, there are methods for constructing such a tree and image routinely. Nevertheless, the diagram can change into hard to be meaningful when the variety of patterns is far more than a number of dozen. Here is an example made by matplotlib today. You may read more in regards to the API here: matplotlib dendogram.

Finally, we come to the last method, and the one which is used predominantly today: the Projection method. This methods uses an algorithm to seek out a technique of reducing the variety of dimensions of the embedding right into a number that may more easily be understood by humans (e.g., 2 or 3 dimensions) and plotting as a scatter plot.

On the time in 1990, the fundamental method for projecting high-dimensional data right into a smaller set of dimensions was Principal Component Evaluation (or PCA for brief). Dimensional reduction is an lively research area, with recent methods still being developed.

Perhaps the most-used algorithms of dimension reduction today are:

1. PCA
2. t-SNE
3. UMAP

Which is the most effective? It really depends of the main points of the info, and in your goals for creating the reduction in dimensions.

PCA might be the most effective method overall, because it is deterministic and permits you to create a mapping from the high-dimensional space to the reduced space. That is beneficial for training on one dataset, after which examining where a test dataset is projected into the learned space. Nevertheless, PCA could be influenced by unscaled data, and may result in a “ball of points” giving little insight into structural patterns.

t-SNE, which stands for t-distributed Stochastic Neighbor Embedding, was created by Roweis and Hinton (yes, that Hinton) in 2002. It is a learned projection, and may exploit unscaled data. Nevertheless, one downside to t-SNE is that it doesn’t create a mapping, but is merely a learning method itself to seek out a clustering. That’s, unlike other algorithms which have Projection.fit() and Projection.transform() methods, t-SNE can only perform a fit. (There are some implementations, reminiscent of openTSNE, that provide a transform mapping. Nevertheless, openTSNE appears to be very different than other algorithms, is slow, and is less supported than other forms.)

Finally, there’s UMAP, Uniform Manifold Approximation and Projection. This method was created in 2018 by McInnes and Healy. This will be the best compromise for a lot of high-dimensional spaces because it fairly computationally inexpensive, and yet is able to preserving necessary representational structures within the reduced dimensions.

Here is an example of the dimension reduction algorithms applied to the unscaled Breast Cancer data available in sklearn:

You may test out the dimension reduction algorithms yourself with a purpose to find the most effective on your use-case, and create images just like the above, using Kangas DataGrid.

As mentioned, dimensional reduction remains to be an lively research area. I fully expect to see continued improvements on this area, including visualizing the flow of knowledge because it moves throughout a Deep Learning network. Here’s a final example from our book chapter showing how activations flow within the representational space of our model:

Keen on where ideas in Artificial Intelligence, Machine Learning, and Data Science come from? Consider a clap and a subscribe. Let me know what you might be fascinated by!