Drowning in data - complexity's threat to terror investigations

John Lettice is right on the money again with his latest piece in The Rgister, Drowning in data - complexity's threat to terror investigations

"The core calculation is essentially the one put forward by the police and accepted by the Government - technology has been an enabler for international terrorism, with email, the Internet and mobile telephony producing wide, diffuse, international networks. The data on hard drives and mobile phones needs to be examined, contacts need to be investigated and their data examined, and in the case of an incident, vast amounts of CCTV records need to be gone through. As more and more of this needs to be done, the time taken to do it will obviously climb, and as it's 'necessary' to detain the new breed of terrorist early in the investigation before he can strike, more time will be needed between arrest and charge in order to build a case.

All of which is, as far as it goes, logical. But take it a little further and the inherent futility of the route becomes apparent - ultimately, probably quite soon, the volume of data overwhelms the investigators and infinite time is needed to analyse all of it. And the less developed the plot is at the time the suspects are pulled in, the greater the number of possible outcomes (things they 'might' be planning) that will need to be chased-up. Short of the tech industry making the breakthrough into machine intelligence that will effectively do the analysis for them (which is a breakthrough the snake-oil salesmen suggest, and dopes in Government believe, has been achieved already), the approach itself is doomed. Essentially, as far as data is concerned police try to 'collar the lot' and then through analysis, attempt to build the most complete picture of a case that is possible. Use of initiative, experience and acting on probabilities will tend to be pressured out of such systems, and as the data volumes grow the result will tend to be teams of disempowered machine minders chained to a system that has ground to a halt. This effect is manifesting itself visibly across UK Government systems in general, we humbly submit. But how long will it take them to figure this out...

The question of whether or not an action is illegal is however important. A legal but suspicious action requires investigation of context in order to determine intent, and to identify the actual crime, whereas an illegal action (which nevertheless might have a perfectly innocent explanation) allows prosecution without reference to or investigation of context.

And the wrong people sometimes get sent down. That however is not the immediate problem from the point of view of the system. Widespread prosecution for trivial and low-level offences will tend to overload the system and reduce focus on potentially more serious offences, while choking processes with low-level and irrelevant data, and directing resources down blind alleys. Police will frequently find themselves failing to find the conspiracy in cases where there really wasn't one...

There is clearly a major problem for the security services in distinguishing disaffected talk from serious planning, and in deciding when an identified group constitutes a real threat. But the current technology-heavy approach to the threat doesn't make a great deal of sense, because it produces very large numbers of suspects who are not and never will be a serious threat... Mischaracterising the threat by inflating early, inexpert efforts as 'major plots' meanwhile fosters a climate of fear and ultimately undermines public confidence in the security services...

we need a long-term survival/endurance strategy that doesn't drown the security services in a swamp of data, doesn't turn us into a police state, but does whatever is feasible to minimise risk."