Shape perception via a high-channel-count neuroprosthesis in monkey visual cortex

An implant with 1024 electrodes in the visual cortex

Most [although not all (3, 8)] previous studies in humans used electrodes that were positioned on the surface of the brain and required delivery of currents in the milliampere range, limiting the number of electrodes that could safely be stimulated at the same time. Surface electrodes activate several millimeters of cortex and produce large phosphenes, and this results in low spatial resolution. Furthermore, previous studies have observed interference between the percepts elicited by simultaneous stimulation on nearby electrodes (2, 3, 11, 22). By contrast, microstimulation by intracortical electrodes requires currents that are two orders of magnitude lower (3, 8, 15, 16, 19), activates neurons located within a few hundred micrometers of the electrode tip, and potentially yields higher-resolution phosphene percepts (25) (supplementary text).

To investigate the generation of artificial visual percepts using a large number of intracortical electrodes, we created a 1024-channel implant consisting of a titanium pedestal connected to 16 Utah arrays (26), each with eight rows and columns of 1.5-mm-long shanks (fig. S1A). We tested the system in two macaque monkeys, L and A. In both monkeys, 14 arrays were tiled across the left primary visual cortex (area V1) (Fig. 1, A and B). Two arrays were placed in V4 (fig. S2), allowing us to monitor the effects of V1 stimulation on neuronal activity in a higher cortical area.

We presented moving bars (27) and obtained multiunit receptive fields (RFs) on 820 of 896 and 616 of 896 V1 electrodes and on 120 of 128 and 92 of 128 V4 electrodes in monkeys L and A, respectively (movies S1 and S2). In accordance with known retinotopy (28), the RFs covered the foveal and parafoveal regions of the lower right visual field, with V1 RF eccentricities of ≤9° and ≤5° (Fig. 1, C and D) and V4 RF eccentricities of ≤5° and ≤15° in monkeys L and A, respectively (fig. S2).

Saccade-to-phosphene task

We used a saccade-to-phosphene task (Fig. 2A and movie S3) to investigate phosphene locations and determine current thresholds for phosphene perception (N = 184 electrodes in monkey L, and N = 164 in monkey A). The monkeys had previously been trained to make eye movements to visually presented dots on a computer monitor, and we replaced the visual stimuli by electrical microstimulation on individual V1 electrodes. Figure 2B shows the probability of saccade execution as a function of current amplitude during stimulation of an example V1 electrode in monkey L. The current threshold, C₅₀, defined as the level at which the monkey reported the phosphene 50% of the time, was 12.6 μA. Across electrodes, the median threshold in monkey L was 23 μA [interquartile range (IQR), 6 to 40 μA] and in monkey A it was 50 μA (IQR, 43 to 58 μA) (Fig. 2C).

We recorded neuronal activity in V4 during V1 microstimulation (Fig. 2D) and removed the electrical artifacts caused by stimulation from the V4 signal (28). Figure 2E shows the mean response across 17 V4 electrodes, during the same session as that in Fig. 2B. The V4 activity increased as a function of V1 current amplitude, with a particularly steep relationship around the C₅₀ of the electrode (Fig. 2F). We calculated the neurometric threshold, R₅₀, as the current amplitude at which V4 activity reached 50% of its maximum. The R₅₀ was 12.4 μA, similar to the C₅₀ of 12.6 μA. Across the currents tested on this V1 electrode, the correlation between the V4 activity level and hit rate was significant [correlation coefficient (r) = 0.94, p < 0.001 (Pearson’s correlation)].

It would be advantageous if V4 activity could be used more generally to estimate V1 current thresholds, because prospective prosthesis users might experience fatigue during the determination of thresholds on hundreds of electrodes based on visibility reports. The R₅₀ could be measured while the user is engaged in an unrelated activity. We observed a high correlation between the R₅₀ and C₅₀ in both animals [monkey L: r = 0.98, N = 169, p < 0.001, 104 unique V1 electrodes for which stimulation artifacts were removed successfully; monkey A: r = 0.96, N = 64, p < 0.001, 44 V1 electrodes (Pearson’s correlation)] (Fig. 2G). To further test how well V4 activity relates to phosphene visibility, we divided the V1 electrodes into three equally large groups based on the C₅₀ and determined the average V4 response as a function of V1 current amplitude within each group (Fig. 2H). The shape of V4 response as a function of current amplitude resembled the psychometric function within each group, confirming that V4 activity can be used to estimate V1 current thresholds.

Although we used a large target window (see methods), the monkeys’ eye movements were generally directed toward the RFs of the stimulated neurons (14) (compare Fig. 1, E and F to C and D; movie S3). To quantify this relationship, we computed the correlation between the polar angle of the mean saccadic end point and that of the RF (Fig. 2, I and J), which was significant in both monkeys (monkey L: r = 0.77, p < 0.001, N = 184; monkey A: r = 0.67, p < 0.001, N = 164). The correlation between the eccentricity of the saccade end point and that of the RF was also significant [monkey L: r = 0.90, p < 0.001; monkey A, r = 0.72, p < 0.001 (Pearson’s correlation)], although there was a consistent undershoot of the saccade relative to the RF [saccade amplitude as a fraction of RF eccentricity in monkey L: 0.77 ± 0.18 (mean ± SD), N = 184; monkey A: 0.71 ± 0.21, N = 164], which was more pronounced for weaker stimulation currents (fig. S3). These findings confirm that phosphenes are perceived at the RF of the stimulated neurons (9, 10, 12, 14, 17, 20). The next experiments addressed whether multielectrode stimulation elicits interpretable artificial percepts.

Generation of composite visual percepts

First, we tested whether the monkeys could report the spatial alignment of two phosphenes (vertical or horizontal) by requiring them to saccade to a target above or below the fixation spot, respectively (Fig. 3A and movie S4). Before array implantation, the animals were trained extensively on a visual version of the task, in which they reported the relative position of two small dots on a screen. In every session of the microstimulation task, we selected unfamiliar pairs of electrodes for simultaneous stimulation, such that the RFs of the stimulated neurons were vertically or horizontally aligned.

Figure 3B shows the results for an example session, with mean accuracies of 91 and 88% for the vertical and horizontal conditions, respectively. Performance was above chance level from the start of the session even though the electrode set was novel, which suggests that the monkey generalized its understanding from the visual to the phosphene version of the task. The average accuracy across sessions with novel electrode pairs was significantly above chance [monkey L: 70 ± 13%, N = 19 electrode sets, t₁₈ = 6.6, p < 0.001; monkey A: 64 ± 18%, N = 11, t₁₀ = 2.6, p = 0.026 (one-sample t test)] (Fig. 3C). We also ran a visual version of the task in which we presented dots on the screen at the RF locations. Monkey L reached an accuracy of 83 ± 10% (t₁₈ = 15, p < 0.001), and monkey A 79 ± 15% (t₁₀ = 6.2, p < 0.001). These visually presented dot configurations were not always novel, that is, the monkeys had seen some of these dot pairs in previous sessions.

We rewarded the monkeys after correct saccades; hence, they might have learned to associate arbitrary phosphene percepts with specific targets by trial and error. We therefore computed the mean accuracy across electrode sets during the first 30 trials of the microstimulation task (Fig. 3D). The animals’ accuracies remained relatively stable across time. For both monkeys, we determined the critical trial, c, for which the accuracy was higher than chance (p < 0.05, binomial test) when trials 1 through c were included, across electrode sets (28). In monkey L, c was the first trial [responses were correct on 17 of 19 electrode sets; p < 0.001 (binomial test)], and in monkey A, c was the fourth trial (30 of 44 electrode sets and trials; p = 0.011). Furthermore, no significant improvement in performance occurred between trials 1 to 10 and trials 21 to 30 [monkey L: t₁₈ = 0.0, p = 1.0; monkey A: t₁₀ = 0.17, p = 0.87 (paired t test)]. We conclude that the monkeys generalized from their experience on the visual task to judge whether novel phosphene pairs were horizontally or vertically aligned.

Direction-of-motion task

The monkeys had also been trained to report the apparent motion direction of a sequence of visually presented dots. We investigated whether the monkeys could interpret the direction of successively generated phosphenes (Fig. 3E and movie S5). We selected three electrodes with a vertical or horizontal offset and stimulated them sequentially such that their RFs formed a sequence, going from top to bottom or bottom to top for “vertical” sets of electrodes and left to right or right to left for “horizontal” sets of electrodes. Figure 3F illustrates an example session in monkey A, in which the RFs had a vertical offset and the monkey reached high accuracy from the first trial onward. Across all electrode sets of the microstimulation task (Fig. 3G), the average accuracies of monkeys L and A were 76 ± 17% [N = 13, t₁₂ = 5.5, p < 0.001 (t test)] and 69 ± 16% (N = 19, t₁₈ = 5.2, p < 0.001), respectively. The monkeys’ accuracies for novel electrode combinations were higher than chance from the first trials onward. The critical trial c was trial 1 in monkey L (correct on the first trial in 11 of 13 sessions; p = 0.011) and trial 4 in monkey A (trials 1 to 4, 48 of 76 correct; p = 0.014) (Fig. 3H), indicating that they were able to generalize their understanding of apparent motion from the visual to the phosphene task. No significant improvement in performance occurred between trials 1 to 10 and trials 21 to 30 [monkey L: t₁₂ = 1.35, p = 0.2; monkey A: t₁₈ = −1.85, p = 0.08 (paired t test)]. The accuracies of monkeys L and A in the task with a sequence of visually presented dots were 74 ± 11% (t₁₂ = 8.2, p < 0.001) and 89 ± 11% (t₁₈ = 15.6, p < 0.001), respectively, although these dot combinations were not always novel.

Letter task

Next, we examined the possibility of creating more complex shape percepts. We delivered stimulation on 8, 10, or 15 electrodes in V1 simultaneously, selecting the electrode sets such that the RFs collectively formed the shape of one of two letters. The number of stimulated electrodes was always the same between the two conditions in a given session. Before electrode implantation, the monkeys had learned to report the identity of 16 visually presented letters on a screen by making an eye movement to a target. The training program also included letters presented as sparse dot patterns, designed to simulate phosphene vision (fig. S4) (29). After array implantation, the training continued, and we initially alternated visual and microstimulation blocks. Thereafter, sessions consisted solely of visual or microstimulation trials. During visual trials, one of two letters was presented as a dot pattern on the screen. During microstimulation trials, we delivered simultaneous electrical stimulation to a novel set of electrodes for 167 ms (50 pulses at 300 Hz) (Fig. 4A and movie S6).

Figure 4B shows the accuracy of monkey A for an example set of electrodes on the microstimulation version of the task. In one condition, the RFs of the stimulated electrodes formed the letter T (orange dots) and in the other condition they formed the letter L (purple dots). In this example session, accuracy was 0.89 and 0.96 for the T and L conditions, respectively, and it was relatively high from the first trial onward. Across all electrode sets (Fig. 4C), the average accuracy of monkey L was 81 ± 18% [N = 9, t₈ = 5.4, p < 0.001 (t test)] and the accuracy of monkey A was 71 ± 18% (N = 10, t₉ = 3.6, p = 0.006) on the microstimulation version of the task. Accuracy on the visual version of the task was 88 ± 5% for monkey L (t₈ = 23.1, p < 0.001) and 93 ± 4% for monkey A (t₉ = 36.7, p < 0.001), although some of these dot displays were not novel.

To control for the possibility that the monkeys learned to map arbitrary percepts onto saccade targets, we examined the accuracy time course (Fig. 4D). The critical trial c (accuracy above chance level, binomial test) was trial 3 in monkey L (20 of 27 correct, p = 0.0096), and trial 7 in monkey A (43 of 70 correct, p = 0.036). No significant improvement in performance occurred between trials 1 to 10 and trials 21 to 30 [monkey L: t₈ = −0.7, p = 0.5; monkey A: t₉ = 0.7, p = 0.5 (paired t test)].

As a further control, we examined how long it took the monkeys to associate arbitrary microstimulation patterns with eye movements. We selected novel, nonoverlapping combinations of electrodes, such that their RFs did not form any recognizable letter (N = 10 electrode sets in both monkeys). The mean accuracy of monkeys L and A during the first 100 trials was close to chance level (48.2 and 46.9%; p > 0.2 in both monkeys) (gray data points in Fig. 4D) and lower than during the first 100 trials of the microstimulation task with familiar letters (monkey L, t₁₇ = 5.7, p < 0.001; monkey A, t₁₈ = 2.4, p = 0.03). We also ran a visual version of the control task, in which the monkeys saw two novel visual dot stimuli. The mean accuracies of monkeys L and A were 50.5 and 49.9%, respectively (p > 0.9), indicating that they also failed to rapidly learn to map unfamiliar dot stimuli onto saccade targets (Fig. 4D and fig. S5).

These results indicate that the monkeys’ abilities to perform the letter task were not due to rapid learning of new associations between stimuli and eye movement targets. Rather, the phosphene percepts must have borne a close resemblance to the visually presented letters, allowing the monkeys to benefit from their extensive prior experience.