Big Brother up in your grill?

Or so we were lead to believe by the vapid sky-is-falling Chicken Littles that told us The USA Patriot Act and the Defense Advanced Research Project Agency's Total Information Awareness initiative were the precursors to a "Department of Precrime."

The civil liberty-stealing TIA was hatched in the wake of 9/11 to indentify potential terrorists through the conduct of data-mining activities across a myriad of public and private databases. Data mining is essentially a process of identifying patterns and relationships among vast quantities of data. A classic example from the world of marketing is a discovery which showed an unexpectedly high correlation between male purchasers of beer and baby diapers. This relationship was ultimately explained by a high incidence of fathers being sent out to purchase diapers during the halftime of football games. Thus, the goal of the TIA was to use such sophisticated techniques to identify, track and classify potential terrorists through data collection and the generation of computer algorithms and human analysis.

The initiative was greeted with significant hue and cry from civil libertarians like William Safire who argued that

If the Homeland Security Act is not amended before passage, here is what will happen to you: Every purchase you make with a credit card, every magazine subscription you buy and medical prescription you fill, every Web site you visit and e-mail you send or receive, every academic grade you receive, every bank deposit you make, every trip you book and every event you attend — all these transactions and communications will go into what the Defense Department describes as "a virtual, centralized grand database."To this computerized dossier on your private life from commercial sources, add every piece of information that government has about you — passport application, driver's license and bridge toll records, judicial and divorce records, complaints from nosy neighbors to the F.B.I., your lifetime paper trail plus the latest hidden camera surveillance — and you have the supersnoop's dream: a "Total Information Awareness" about every U.S. citizen. This is not some far-out Orwellian scenario. It is what will happen to your personal freedom in the next few weeks if John Poindexter gets the unprecedented power he seeks.

And of course no story on TIA was complete without a mention of Poindexter and his conviction in the Iran-Contra scandal. Such mentions of course lent credence to the notion that the Bush Administration was "shredding the Constitution." After all, who better to abscond with our consitutional feedoms than a pipe-smoking, Reaganite, beady-eyed convicted felon?

Meanwhile, discussions of DARPA's TIA project contained predictably little meaningful analysis. While reasonable concerns about possible impacts and applications were raised and pushed to the hilt, precious little examination of the practicalities took place.

For example, no one asked whether the 2003 TIA budget of $10M was even sufficient to build such such an Orwellian nightmare database and the necessary algorithms. As someone who has built marketing databases for some of the world's best known corporations, I can say unequivocally that $10M is nowhere near sufficient to build the infrastructure or analytic capabilities required for such an unprecedented database. For example, Nestle spent close to $5M on a 20 million record marketing database in 2000. And that's just for the infrastructure. That doesn't include the hundreds (or even thousands) of man-hours needed to conduct the analytics needed to build-out profiles, models and algorithms. Let's see, there are nearly 300M citizens of this country. And how many legal/illegal aliens? And how many private and public databases? So you are talking about one seriously mammoth data processing problem. And when it comes to building out the algorithims, one could assume that the variables associated with terrorist behavior are at least, what? 50 or 100% more complicated than predicting responses to marketing campaigns? Sounds about right since we have a hard enough time explaining terrrorists' pre-attack activities ex post facto. And who would be doing all of these analyses? The database marketing, financial services and e-commerce sectors currently cannot find enough qualified statisticians to man their ostensibly more lucrative analytics departments. So how was a cash-strapped TIA supposed to staff the world's biggest data-mining project? Bottom line: incredibly complicated technical problem combined with a requirement for unprecedented data-mining capabilities all for a mere $10M. I don't think so.

But it wasn't just implementation questions that went unasked. The ACLU and others seemed wholly disinterested in the fact that database marketers have access to much of the same data that was to be earmarked for the DARPA project. With a Direct Marketing Association certification and a thick stack of bills I could go out on the market today and purchase individual records sliced and diced by your address, geodemographic index, FICO score, auto ownership, magazine subscriptions, online clickstream data, credit card purchase history, travel patterns, number of children, organization membership, etc. So can any advertising agency or company wishing to sell you XYZ. And yet, the hysterics were swinging from the chandeliers in outrage over the fact that the government was considering using such information in the name of stopping terrorists from taking advantages of operating in a free society. So what is good enough for Nestle, AMEX, Ford, Microsoft, Delta and Amazon is clearly not good enough for the Federal government. While we trust our personal data in the hands of Evil Corporations (TM) that frivolously market to us, we definitely don't want to entrust this same data to a government which might be able to actually protect us from terrorist attack. Mickey Kaus elaborates on this aspect.

And so whatever happened to Poindexter and his Orwellian project best known for its creepy logo? Well, in September of 2003 the Congress shuttered the operation, and the ACLU, the hysterics and terrorists all breathed a huge sigh of relief. But thankfully, The Transportation Security Administration, FBI and other intelligence groups are still working on TIA-type projects.

I say thankfully because it's an incredibly smart idea which capitalizes on some of our natural advantages in the Global War on Terrorism: our possession and mastery of technology resources that far outstrip those of our enemy. That is not to say that we should trade threats to our physical security for threats to our personal liberties, but to dismiss such initiatives out of hand would be both hopelessly cynical and foolishly irresponsible. And yet, that's precisely what we have done. The ACLU, EPIC and other opponents were so successful in coloring the public's understanding of these projects with a patina of Enabling Act, constitution-shredding rhetoric that TIA was shut-down. But fortunately, these less visible efforts continue.

Again, I say fortunately because revelations publicized in Tuesday's New York Times have again raised the issue in the public's conciousness. Although one has to dig a little in the NYT article to find it, the connection is perhaps at the center of a nascent controversy which might make Rovegate pale in comparison.

More than a year before the Sept. 11 attacks, a small, highly classified military intelligence unit identified Mohammed Atta and three other future hijackers as likely members of a cell of Al Qaeda operating in the United States, according to a former defense intelligence official and a Republican member of Congress. In the summer of 2000, the military team, known as Able Danger, prepared a chart that included visa photographs of the four men and recommended to the military's Special Operations Command that the information be shared with the Federal Bureau of Investigation, the congressman, Representative Curt Weldon of Pennsylvania, and the former intelligence official said Monday.

Yes, 2000. A full year prior to the attacks and during the final months of a Clinton Administration that, remember, was so terribly pre-occupied with the activities of al Qaeda.

And just how were the hijackers identified? Buried deep in the piece (paragraph nine to be exact), we learn that the answer is....data mining.

Mr. Weldon has long been a champion of the kind of data-mining analysis that was the basis for the work of the Able Danger team. The former intelligence official spoke on the condition of anonymity, saying he did not want to jeopardize political support and the possible financing for future data-mining operations by speaking publicly. He said the team had been established by the Special Operations Command in 1999, under a classified directive issued by Gen. Hugh Shelton, then the chairman of the Joint Chiefs of Staff, to assemble information about Al Qaeda networks around the world.

And just what did the data mining exercise reveal?

He said the original chart, like the new one, had included the names and photographs of Mr. Atta and Marwan al-Shehhi, as well as Mr. Mihdhar and Mr. Hazmi, who were identified as members of what was described as an American-based "Brooklyn" cell, as one of five such Al Qaeda cells around the world. The official said the link to Brooklyn was meant as a term of art rather than to be interpreted literally, saying that the unit had produced no firm evidence linking the men to the borough of New York City but that a computer analysis seeking to establish patterns in links between the four men had found that "the software put them all together in Brooklyn."

And ostensibly put the government in a position to do something to disrupt the plot. But the information was dismissed for legal, procedural and likely, priority reasons. My point here is not to make political hay at the expense of the Clinton Administration. That's better served in a post all its own. 9/11 was a systemic failure and ascribing blame to any one political party, group or department is just as pointless and destructive as the 1950s Who lost China? debate. It's not the mistake that should be the focus. It's what we have learned from the mistake that should be at issue. And that is the purview of the heavily lauded 9/11 Commission. A commission whose staff apparently shielded the commissioners from vital information and a report which shockingly included no mention of Able Danger.

A former spokesman for the Sept. 11 commission, Al Felzenberg, confirmed that members of its staff, including Philip Zelikow, the executive director, were told about the program on an overseas trip in October 2003 that included stops in Afghanistan and Pakistan. But Mr. Felzenberg said the briefers did not mention Mr. Atta's name. The report produced by the commission last year does not mention the episode.

A commission which failed to make any specific recommendation regarding the use of data-mining to prevent a follow-on attack. One wonders whether the TIA project would have been cancelled if the 9/11 Commission's report had included an explicit recommendation to pursue data mining techniques on the basis of Able Danger's sucess. And one wonders whether the existing projects might not be higher priorities and thus, more heavily funded.

Additionally, how do Tuesday's revelations now color our views of former Clinton National Security Advisor Samuel Berger's pilfering of highly classified documents from the National Archives? And whither Michael Moore's favorite quote-machine Richard Clarke? In his incendiary testimony and best-selling book, Clarke never once mentioned Able Danger. Why not? I have no evidence to suggest wrongdoing, nor do I seek to defame either man. But in light of the Able Danger revelations, it would seem proper to revisit both of these topics.

In an era where partisan politics portrayed TIA as a civil liberties-constricting python shortly after it had demonstrated a significant degree of success in identifying 20% of the 9/11 hijackers, is it any wonder that Rovegate is Vesuvius to Able Danger 's pimple? I sincerely look forward to explanations from Berger, Clarke, Zelikow and others in the coming days. Again, not for gotchas but to glean greater insight regarding the proper emphasis to be placed on a potentially valuable tool in fighting terrorism. TIA and other such efforts have been blackballed as Big Brother initiatives and though there isn't the funding, technology or personnel in place to empower such a global threat to our freedoms, the tools have apparently shown significant promise in identifying potential terrorists. Enough promise that they might have encouraged the disruption of the most devastating terrorist plot in the history of the world.