Thursday, July 04, 2019

The Open University Summer Schools of Old

I have just entered my 25th year at The Open University and that venerable institution is also celebrating its 50th anniversary in 2019.

The period between 2015 and 2018 was the mostly intensely self-destructive in the history of the OU. The most harmful single act, in that period, was the closure of the regional centres but there were a multitude of other calamitous top-down decisions, actions and schemes.

It is the greatest failure of my professional career that I was unable to prevent the worst of the damage and I remain angry and depressed about that to this day. There may come a time when I am able to write about it dispassionately but that time is not yet and not close.

Today I want to talk about the more gradual decline of a key structural element of the OU's provision for our first 30 years, summer schools.

For a large chunk of those decades the OU would take over conventional university campuses, all over the country, for at least a couple of months in the summer. We installed our staff, students and equipment and ran labs, lectures, tutorials, workshops, field trips and all manner of conceivable conventional educational activities; to complement all of the other supported open learning at a distance activities run through the rest of the year. At the time I spent two weeks every year, teaching, lecturing, assessing, testing, experimenting, running tutorials, labs and field trips at and from UEA, Keele, Stirling and Bath universities.

At the time most of our undergraduate courses ran from February to September. Summer schools would happen just as students were beginning to flag, from the heavy workload of studying degree level subject matter at home. The educational, psychological and social boost they would then get from spending a week or two, in a conventional university setting, with their peers and teachers, got the motivational juices flowing again and saw them successfully through to the end of the academic year. They were an enormously important factor in student retention and success.

When I mention summer schools and lament their much diminished role in our current provision, I often get accused of being a dinosaur, nostalgic for and hankering after a non-existent golden age and anyway (and this line is always delivered with the supreme confidence of those who know they have the clinching argument), "students voted with their feet". What's interesting is that even OU staff who were not with us at the time, have bought into this story and repeat it with complete conviction.

The assertion, essentially, is that
  • students are customers (a notion I reject - educational institutions are students' partners in facilitating their learning and growth)
  • customers should get what they want
  • these customer students were declining to go to summer school in increasing numbers
As with all the best false narratives, this one has an element of truth. Students were declining to go to summer school in increasing numbers.

The real reason for that was not, however, lack of 'customer satisfaction.' Rather is was a step by step, often well intentioned and, in each particular corner of the OU where the decisions were made, internally justifiable, bureaucratic dismantling of the institutional processes underpinning summer schools.

When I started at the OU in the mid 1990s, summer school attendance was compulsory. If you signed up to a course with a summer school you had to attend to pass. Excusal was possible in exceptional circumstances but tiny numbers of students got excusals.

Many students hated the idea of being away from home, family and friends for a week and it was often a wrench to get to the summer school. This was especially the case for foundation students scheduled to attend their first summer school.  The vast majority, however, derived enormous benefit from that attendance. I too hated leaving my family to decamp to summer school but got to meet and work with some incredible people and treasure the experience - OU students and staff are amazing.

Summer schools, though, were huge, logistically complicated and resource intensive operations. Even when not in full flight supplies and equipment had to be warehoused for the 10 months of the year they were not in use.

For a big bureaucracy like the OU, costs are nominally easier to measure than esoteric things like the educational, motivational and goodwill value provided by things like summer schools. Staff time, travel expenses, equipment costs, transport, logistics, campus venues hiring costs etc all appeared on the balance sheet and added up to large sums.

The senior management and their bureaucrats realised that courses with summer schools were more costly for the university than those that did not have summer schools.

At the same time there was a partially formal and partially informal loosening of the processes through which students were able to get excusal from summer school. I know several of those directly involved in this enablement of excusal and their motivation was entirely altruistic. It was all about being nice to students. In some instances students were being actively encouraged by the OU to apply for excusal from summer school. With this development the numbers of students getting excused summer school went up. I don't recall the exact figures but I think it reached about 20% of students getting excused from summer school by the late 1990s.

Management also decided students taking courses with summer schools should be charged a significantly higher fee than those without. In keeping with our institutional values, the OU were open about this. They declared, in multiple communications to students, that we were explicitly charging the higher fee due to the summer school element of the course.

This led to courses with summer schools seeing a significant drop in student registrations.

Then academic course teams producing and presenting courses were reluctant to include summer schools in their courses, since it directly affected student numbers. Hence the number of courses with a summer school component was drastically reduced.

At that point it became easy to claim "students were voting with their feet" and summer schools shouldn't be a compulsory part of the OU degree experience any more, unless external accrediting bodies, like the engineering institutions demanded it. Then we could fulfill that obligation by splitting summer schools away from core courses and running a few of them as stand alone courses that students would sign up for a week to attend e.g. to meet the lab requirements of their engineering degree.

So though students did 'vote with their feet,' the bureaucracy, intentionally or not, orchestrated that situation through undermining summer schools value from the perspective of the students over a period of years, through -
  • Loosening of summer school excusal processes
  • Explicit campaigning to and encouragement of students that they could get out of summer school if they wanted to
  • Consequent increase in numbers skipping summer schools (a core part of the learning experience of the courses concerned)
  • Management and bureaucracy then claimed that summer schools could not be that important if so many students were getting excusals; so there was increasing internal pressure to reduce the numbers of summer schools
  • There were some detrimental effects on success of students on the courses where summer school excusals were up but I'm not aware of any explicit empirical assessment of that link
  • Resource intensive summer schools targeted for efficiency savings by senior management 
  • Explicit differential fee pricing was introduced where courses with summer schools charged students a significantly higher sum
  • Relentless communications to students that the higher fee was due to the summer school, including detailing the specific element of the fee charged against the summer school 
  • Consequent decrease in the number of students taking courses with a summer school element
  • Further reinforcement of the internal belief that students were 'voting with their feet' and summer schools were not that important
  • Parallel decrease in the number of academics willing to produce courses with summer schools as we were constantly under pressure to produce popular courses
  • Big cost savings for senior management as the OU no longer had to spend the summer running summer schools on conventional university campuses nationwide; or sustain associated warehouse costs for the remainder of the year
Meanwhile the key motivational and educational value of the summer schools got lost in the noise.

Nowadays none of our courses have an integrated summer school, though some of our qualifications require students to take stand-alone summer school courses as part of their qualification e.g. engineering degrees.

My own Technology Faculty, in 1999, introduced the first entirely online undergraduate course anywhere in the world, T171 You, Your Computer and the Net, chaired by Martin Weller. We piloted it with 1000 students in 1999 and then ran it with 13,500 students in the year 2000. That itself was a massive institutional challenge for the OU and John Naughton, who along with Gary Alexander and Martin wrote T171, likes to describe it as a success disaster. In the space of 9 months the OU had to put in place the institutional infrastructure to support T171 and anything that might come along like it.  It was a massive organisational shock and one to some degree we have been suffering from ever since. Processes and bureaucracy were locked in place that have been restrictive and difficult to change to this day. Again that's a story for another day.

Along with T172 Working with our environment: technology for a sustainable future, T171 then replaced the old foundation course, T102 Living with Technology and its summer school. I was deeply implicated in both T171 and TM172 but by the time they came along the battle for summer schools was lost. I still believe they have enormous value and could feature in a future sustainable and successful OU but they won't be returning in the short or medium term.

Why am I discussing summer schools now? Well I was prompted partly by Martin Weller's and Stephen Downes's recent exchanges on connectivism and scale. I have a high regard for both and both are committed, in the educational context, to putting people in touch with people. Where I side with Martin is that it is expensive to do this at scale successfully. Martin might not agree, as one of his early T171 mantras in the late 1990s and early 2000s was that face to face tuition was old school dated and unnecessary, but summer schools were (an expensive but) incredibly important and successful component of the OUs supported open learning at a distance. By the same token they gave life to Stephen Downes's justifiable conviction in the power of peer student support/teaching/learning.

The second reason I'm writing about summer schools, now, is that every time someone in the OU churns out the "way-da-minute, students voted with their feet" tune when I talk about summer schools, I have threatened to put the record straight. My old friend Steve Walker did just that, when we were both at the 9th Tensions of Europe conference in Luxembourg last week. (I posted my talk on Turing and mass surveillance earlier this week) To the embarrassment of our travelling companions, we got into a loud discussion about it on the train from the university back into Luxembourg city. Maybe now I've finally made good on the threat to write the story down, I won't be so loud the next time someone brands me a dinosaur, pining for a non-existent gilded age.

The final reason is that the last session I was able to attend at the 9th Tensions of Europe conference was on manufacturing industries in the late l980s and early 1990s. Now I spent a fair chunk of that time working in the aerospace industry. And depressing though it was to have academics and PhD students studying that time as history, two of the panelists were describing story somewhat at odds with my experience. Several questions were raised about research methods by the audience and Steve Walker asked specifically how they were going to take multiple perspectives into account. The third panelist picked up Steve's point immediately and the importance of reality in relation to discourse. Yet as with the summer school story, when a narrative takes hold it can be difficult to derail. Yet some of the 'accepted' narratives around computer integrated manufacturing (CIM) are not what I would accept as an accurate portrayal of my reality of the time. Apparently CIM is back in vogue but it's now labelled 'Industry 4.0.' Unfortunately I didn't get the chance to talk to the panelists after their session, as I was hastily departing to the airport.

In any case, it seems all too easy for history to get revised, even by those genuinely studying it (or repeating a believed narrative about it) for enlightenment. So summer school catharis this was.

Monday, July 01, 2019

From Alan Turing to the mass surveillance machine that is the internet

I gave a talk, on Saturday last, at the 9th Tensions of Europe Conference in Luxembourg. Draft of my remarks below. They are relatively brief and as a result somewhat over-simplified.


From Alan Turing to the mass surveillance machine that is the internet

In November 1942, English code breaker, Alan Turing, arrived in the US on a four-month intelligence sharing visit. He was met by three immigration officers and very nearly denied entry and dispatched to Ellis Island immigration detention centre, due to anomalies in his documentation. Two of the three eventually agreed he should be admitted.

There followed an intense period of work with the US Navy’s intelligence service in Washington DC and Bell Labs in New York. This was partially hampered by bureaucratic issues with his security clearances, and Turing’s unofficial instructions from British intelligence to reveal as little as possible to their US counterparts. The British were distrustful of the US government and vice versa.[i]

It became clear that the absence of trust and cooperation was impairing the war effort and shortly after Turning’s visit, US Intelligence officials Colonel Alfred McCormack, Lieutenant Colonel Telford Taylor, and Lieutenant Colonel William Friedman travelled to Britain to work with the head of Bletchley Park, Edward Travis. Friedman played a key role in the cracking of the Japanese Purple code and Taylor went on to become the chief US prosecutor at the Nuremberg trials. The parlay between Travis and the US delegation led to the 1943 BRUSA Agreement (Britain–United States of America agreement) to share intelligence.

BRUSA in turn spawned the UKUSA Agreement in 1946 to share signals intelligence (sigint) and communications security (comsec, the security of the processes, infrastructure and products of that sigint).

By 1956 Canada, Australia and New Zealand became parties to the agreement and it became known as the Five Eyes (FVEY). This FVEY agreement (or now collection of agreements) forms the basis of intelligence cooperation between these countries to this day.



[Norway, Denmark, and West Germany became secondary associates in the 1950s.]

At this point I’m going to have to fast forward through decades and significant parts of the story.*
To the FYEV intelligence & security services in the 1990s. The Cold war was supposedly over. They were suffering what they considered to be underinvestment and lack of appreciation. The Internet & WWW were going global and there were serious concerns in the agencies about keeping up.

ECHELON, the satellite centred surveillance system, developed by FVEY in the late 60s to early 70s, intended to collect communications of Soviet leaders, military personnel and diplomats, had already been turned to spying on FVEY allies like Germany and France; and the surveillance of individuals and commerce, facilitating industrial espionage.

That’s not my conclusion btw, that comes from a 2001European Parliament report on ECHELON at a time when the FVEY alliance was refusing to confirm or deny ECHELON officially existed. The report said there was no longer any doubt – it exists but all the EU could do is ask the FYEY, nicely, to stop spying on us. Some members of the committee disavowed the report as too soft and declaring that the deployment of ECHELON constituted a blatant breach of European law and the EU Charter of Fundamental Rights. It did conclude, however,

“However extensive the resources and capabilities for the interception of communications may be, the extremely high volume of traffic makes exhaustive, detailed monitoring of all communications impossible in practice.”

So even those who were deeply critical of the surveillance activities of FVEY accepted that these organisations were being snowed under with electronic data.

9/11 to 7/7

When the September 11, 2001 attacks on the US happened with the tragic loss of thousands of lives, everything changed. The US & UK now had a new demon to replace the Soviet Union – terrorism. So began the US orchestrated war on terror and huge resources were poured into recruitment and mass surveillance technology. Much of it was wasted e.g. Trailblazer and, if we take the word of NSA whistleblowers such as Thomas Drake or William Binney, fraudulently so.

Military action in Afghanistan began within weeks and followed in Iraq about 18 months later.[ii]

The action in Iraq & Afghanistan stretched GCHQ operationally.

On 11/3/2004 the Madrid train bombings, the biggest terrorist attack in Spain in history, killed 191 and injured more than 2000 people.

The following year. The 7/7/2005 London attacks led to 56 deaths and nearly 800 were injured.

The Data retention directive, an intimate part of the mass surveillance story in Europe

In the wake of the 2005 London attacks there was a reinforced urgency in government about doing something about terrorism. In the UK the Blair government obsessively pursued mass data retention and all manner of other privacy decimating policies, regulations and processes, culminating in the EU Data Retention Directive 2006. Government ministers were drilled to chant the poisonous & deceitful but powerful ‘nothing to hide, nothing to fear’ sound bite, at every conceivable opportunity.  One of the things, incidentally, UK governments are going to miss after Brexit is the policy laundering they pursued so successfully through the EU.

Mass communications data retention was later found unlawful in multiple high courts around Europe - Romania (2009), Germany (2010), Bulgaria (2010), the Czech Republic (2011) and Cyprus (2011) have all declared the data retention directive unconstitutional and/or a disproportionate unjustified interference with the fundamental right to privacy, free speech and confidentiality of communications.

In 2006 GCHQ began their ‘SIGMod Initiative’ (signals intelligence modernisation programme) on gathering, processing, analysing, assessing, storing, distributing and sharing communications data. The government proposed an Intercept Modernisation Programme (IMP) 2008 involving the spending of £12 Billion + passing a proposed new law, the Communications Data Bill. A small number of NGOs, notably the Open Rights Group, Liberty and Privacy International, managed to get the attention of the media and a few politicians, noting the proposals were a terrible idea and labelling the whole thing a‘Snoopers’ charter.’ And with the financial crash of 2007/’08 and an election imminent it was officially shelved but the government and security & intelligence services implemented it in secret anyway.
 
Snowden 2013

Meanwhile stateside, an insider at the NSA, Edward Snowden, decided that the activities of the FVEY had reached the point of unchecked intrusion into the lives of ordinary people to a degree that was unconscionable and indefensible. In June 2013 Snowden chose to smuggledocumentary evidence of these activities to Hong Kong where he handed them over to journalists Glenn Greenwald, Laura Poitras and Ewan MacAskill.[iii]

What was revealed was a spectacular array of FVEY resources, technical capabilities and activities, with a very limited degree of legal or political oversight, checks or balances. Mass surveillance was not only being conducted by the commercial behemoths of Silicon Valley and every economic actor with a Web presence but by governments of the FVEY alliance. And these security services, like Silicon Valley, had their processes and technologies[iv] targeted at entire populations.

One of the surprises for informed security and intelligence analysts that came out of the Snowden revelations was that GCHQ and the NSA had got these large-scale systems working. The history of government deployments of large-scale information age IT projects had not previously been promising.

Circumventing & breaking law

According to the Snowden documents, one of the effects of the FVEY agreement was that NSA shared intelligence with GCHQ to circumvent UK law and vice versa. The documents quote US intelligence services staff considering that their UK equivalents had no real legal restrictions to abide by. The UK end of the operation likewise talked of their light regulatory regime as being a ‘selling point’ in soliciting funds from the NSA, amounting to $100 million between 2010 and 2013. So, if there were technical legal restrictions on the NSA’s activities – e.g. not being permitted to target US citizens, they could just get the British to do the surveillance for them. Officially this was denied. 

Even where to request the information would be a technical legal breach, it could be circumvented by the transatlantic sharing of information, under FVEY, without the need for a formal request.

Snowden changed things in Europe, if not the UK. EU allies were angry at the scale and reach of FVEY surveillance resources, targeted at their populations, policymakers (including tapping Angela Merkel’s phone) and economic actors. The European Court of Human Rights and the Court of Justice of the European Union became sensitised to mass surveillance and issued a series of decisions declaring the activities unlawful.

The European Court of Justice in the Digital Rights Ireland case in 2014 declared the data retention directive so bad it should never have existed and abolished it.

DRIPA 2014 – the UK’s let’s pretend the data retention directive didn’t get abolished Act.

The UK government decided to ignore the ruling. UK chief police officers issued an edict to their police forces to continue retaining data. When the government couldn't ignore it any more because they were being sued and the press were about to start paying attention to it, they passed a new law, the Data Retention and Investigatory Powers Act 2014.

This contained 8 sections and was rushed through parliament in record time with no scrutiny, by means of a very rarely used parliamentary process, just as MPs were about to go on their summer holidays. [The party briefings instructing MPs what to say about this law in public were longer than the law and both the parties of the coalition government - the Tories and Lib Dems - and the Labour party were all in favour.]

UK Investigatory Powers Act 2016 [v]

Far from reigning in surveillance and other activities revealed by Snowden in 2013, and those previously known and found by high courts all round Europe to be in breach of fundamental human rights, the UK passed the Investigatory Powers Act 2016, to legalise them. Whereas the US made some effort to be seen to be engaging in at least cosmetic reforms to that nation’s surveillance laws, the UK government denied there was an issue, trotted out tropes about national security and “nothing to hide, nothing to fear”, issued gagging orders, ritually destroyed the Guardian’s computers and reinforced and expanded the scope of intelligence gathering activities permitted.  Providing this legal infrastructure, with extraterritorial reach, to enable and facilitate the exploitation of modern digital technologies and networks, nominally for security and intelligence purposes and, with arguably limited checks and balances, has profound implications for democracy, all around Europe.

It remains also, however, the long standing FYEY intelligence sharing operation between the US, UK, Canada, Australia and New Zealand, that now deploys the considerable resources made available by the respective governments to exploit the infrastructure of the internet to engage in mass surveillance around the globe. This is not about FYEY being old and dated. The UN Declaration of Human Rights and the European Convention on Human Rights both stem from the same period and stand strong; as do multiple other historic documents like the US Constitution and Bill of Rights. However, the FVEY sigint agreement, as an arrangement emerging from the devastation of WWII and the ‘Second Red Scare’ and designed primarily to facilitate the collection of intelligence on the Soviet Union, China and their allies, in the modern context now reaches deeply into the lives and homes of ordinary people.

Liberty and others have taken the battle over the Investigatory Powers Act 2016 bulk surveillance provisions back to the courts. In April 2018 the UK High Court ruled that the data retention elements of the Act were unlawful.[vi] On 11 June 2019 it emerged that, even with the extra permissions of the Act, MI5 had been acting so far outside the scope of the legislation, in relation to their data management practices, that documents compelled to be revealed to the court showed that the independent ‘Investigatory Powers Commissioner’ (IPC) declared the agency’s bulk surveillance data management practices “undoubtedly unlawful”.[vii] [The Investigatory Powers Commissioner was a new office, set up under the Investigatory Powers Act, charged with dual oversight, along with the relevant Secretary of State, of the activities subject to the Act.] 

MI5 had effectively been caught out unlawfully retaining innocent people’s data for years, failing to give the IPCO (IPC's Office) accurate information about repeated breaches of its duty to delete bulk surveillance data, and mishandling sensitive legally privileged material. Even if this can be chalked up to normal bureaucratic failings on the part of a government service, this must be concerning.

The reality of FVEY is significantly more complex than I have the time to cover here. It has not, in practice, facilitated blanket, open, totally frictionless sharing of intelligence between the US, UK and other FYEY partners. Just because they agreed to share intelligence and not spy on each other, did not mean they stuck to that agreement or collection of agreements. Intelligence and security services, even within national boundaries, tend to be complex Faustian ecologies of competing institutions, individuals, agendas, bureaucracy and politics, wrapped up in an evolutionary internecine game of the survival of the fittest, surfing on the cause of protecting national security.

We should take infinitely more care in building and continuing to expand the legal, technical & organisational infrastructure of mass surveillance. Such complex systems fail naturally - systems fail, people make mistakes, staff under pressure circumvent the systems to get the job done and the temptation to hide those failures is organisationally irresistible. It will always be so & that's before you start factoring in malign actors because complex systems can also be made to fail by internal and external attackers with nefarious intent. Create these systems and the failures will come. We know this because they have failed and there is not a computer scientist or security specialist anywhere in the world who can secure them and make them water-tightly safe in practice.


The internet has become a huge surveillance machine.

It is possible, as the Net is an entirely artificially designed and constructed entity, to wrestle/retrofit it into something useful that is not a mass surveillance machine. However, it will be difficult to do, in practice, as all the most powerful governmental and commercial economic actors, as well as us the masses of the bread & circuses distracted unwashed users, caught in the headlights of seductive surveillance, are addicted to that architecture of surveillance. 

The critical question is how. How do we cultivate, energise, harness, direct and sustain sufficiently powerful socio-economic, political, commercial, cultural, environmental, social and technical forces to transforming the internet into something with a human rights respecting architecture, at an individual, community, district, national, transnational and global level?


As Carl Sagan said, science and technology heap a new and awesome responsibility on the shoulders of scientists, technologists, policymakers and Jo Public, to pay more attention to the hazards and long-term consequences of advances, from individual, communities, regional, global & multi-generational perspectives, avoiding appeals to simplistic claptrap and the nationalism, chauvinism and hate mongering so prevalent in modern politics & media.


[i] The British worried about the rivalry between US navy and army potentially leading to leaks. The US were equally distrustful of the British and frustrated, given the 500+ US ships sunk by U-boats in the previous year, that they were so unwilling to share information.
[ii] {Katherine Gun GCHQ whistleblower case – UN second resolution, NSA memo 31 Jan 2003 requiring UK to spy on world leaders in the hope of blackmailing them into supporting war. This came about a week after GCHQ staff, deeply concerned about the legitimacy of the impending conflict, had been officially assured they would not be required to engage in illegal activity. Gun, a 28-year-old analyst, admitted passing the NSA memo to the Observer newspaper which printed it in full on its front page in early March, having spent a month confirming its provenance. AG equivocal legal advice on war led to Gun’s prosecution being dropped in February 2004}
 [iii] Unlike Wikileaks who tended to put everything openly on the internet, Snowden decided the documents should be curated by respected news organisations, like The Guardian and The Washington Post newspapers, with revelations to be made public selected purely based on the public interest and the avoidance of exposure of intelligence services personnel to risk.
[iv] [of what the UK end of the business now calls “bulk” interception, acquisition, equipment interference and personal dataset warrants]
My evidence to Joint Committee on Investigatory Powers Bill https://b2fxxx.blogspot.com/2016/01/evidence-to-joint-committee-on.html
‘S253 Technical capability notices
(1)     The Secretary of State may give a relevant operator a technical capability notice…’
Operators have multiple dutes to assist with implementation of IPAct measures.
[vi] [Since ministers were empowered by the Act to issue data retention orders without independent review and authorisation – and for reasons which have nothing to do with investigating serious crime – it was a breach of fundamental rights.] 
[vii] [He also said that he has effectively put them in special measures after discovering they were misleading the Investigatory Powers Commissioner’s Office (IPCO).
“Without seeking to be emotive, I consider that MI5’s use of warranted data... is currently, in effect, in ‘special measures’ and the historical lack of compliance... is of such gravity that IPCO will need to be satisfied to a greater degree than usual that it is ‘fit for purpose'".]
*Including the cold war & evolution of sigint processes and technology, establishment of Menwith Hill and other sigint infrastructure, the Korean war, the development of the ARPANET, ECHELON, the emergence of the WWII sigint story, the Pentagon papers, Watergate, Nixon, the FISA court, the ABC trials, the accidental but happy coincidence of technology and regulation that enabled the early internet to be built on the back of telephone networks, with an end to end architecture – the ‘intelligence’ was not built into the network but rather the devices that connected to it – enabling anyone to innovate, Reagan’s Executive Order 12333, Duncan Campbell’s 1988 revelation of ECHELON (it was an extension of the UKUSA Agreement; He also detailed how Echelon worked), Tim Berners Lee’s creation of the WWW protocols, the WWW & Net going mainstream, the cryptowars, the internet’s midwifery of today’s big 5 tech giants, the West’s military adventures, RIPA, 9/11, the US Patriot Act, the ‘war on terror’, Total Information Awareness, Trailblazer, NSA whistleblowers Bill Binney (ThinThread) & Thomas Drake, National Security Letters, Blair government architects of the data retention directive 2006 and national identity cards and a blizzard of serious crime and anti-terrorism regulations expanding powers of law enforcement, intelligence & security services, US FISA Amendment Act 2008 Act – guilty of being a foreigner – Caspar Bowden & Microsoft, NSA violation of FISA court orders, Bush & Blair establishment and Obama and Con-Dem coalition consolidation and expansion of architecture and resources of mass surveillance conducted by FYEY. Some of Snowden's revelations
PRISM – targeted intelligence, this had some justification and defensible due process overseen by the FISA Court
Tempora – GCHQ hardwire tap of UK backbone cables (UK connected to 57 countries by fibre optic cables; US is connected to 63)
Upstream - BLARNEY, FAIRVIEW, OAKSTAR and STORMBREW NSA interception tools
Boundless Informant – metadata engine, data analysis and data visualisation tool
Blanket open-ended court orders for Verizon phone records
XKeyscore – the NSA’s Google, for collection of "almost anything done on the internet" (Snowden claimed he could wiretap anyone anywhere with it and indeed Angela Merkle’s and other world leaders’ phones were tapped; Angela Merkel's phone communications were monitored by the Special Collection Service, part of the STATEROOM program)
OpticNerve
Mainway - NSA mass phone tapping
Bullrun (NSA) & EdgeHill (GCHQ) to crack encryption
MUSCULAR (mainly GCHQ run) secretly tapped Yahoo! & Google data centres
NSA black budget to pay commercial organisations for secret access to their networks
Spied on gaming sites, charities, commercial enterprises like Brazil’s biggest oil company, dozens of world leaders including Merkle
TURBINE – malware
Tailored Access Operations (TAO) – NSA’s cyberwar sigint operation
QUANTUM suite of attacking facilities e.g. compromising routers, interception, duplication & compromising of traffic
Tapping phones of world leaders including Germany’s Angela Merkel
GCHQ’s Smurf Suite for hacking mobile phones
NSA & GCHQ tapping fibre optic cables to Google and Yahoo data hubs
NSA allowed to surveillance connections three hops from identified targets
UK operating a surveillance system where “anything goes”
If you want to know how some of this data collection and processing works one of the single most useful Snowden documents is the “HIMR Data Mining Research Problem Book
And even that lot is a wholly incomplete OTTOMH list but then there has been a lot of activity in this arena since WWII.