As the upstart voter-profiling company Cambridge Analytica prepared to wade into the 2014 American midterm elections, it had a problem.

The firm had secured a $15 million investment from Robert Mercer, the wealthy Republican donor, and wooed his political adviser, Stephen Bannon, with the promise of tools that could identify the personalities of American voters and influence their behaviour. But it did not have the data to make its new products work.

So the firm harvested private information from the Facebook profiles of more than 50 million users without their permission, according to former Cambridge employees, associates and documents, making it one of the largest data leaks in the social network’s history. The breach allowed the company to exploit the private social media activity of a huge swath of the American electorate, developing techniques that underpinned its work on President Donald Trump’s campaign in 2016.

An examination by The New York Times and The Observer of London reveals how Cambridge Analytica’s drive to bring to market a potentially powerful new weapon put the firm – and wealthy conservative investors seeking to reshape politics – under scrutiny from investigators and lawmakers on both sides of the Atlantic.

Christopher Wylie, who helped found Cambridge and worked there until late 2014, said of its leaders: “Rules don’t matter for them. For them, this is a war, and it’s all fair.”

“They want to fight a culture war in America,” he added. “Cambridge Analytica was supposed to be the arsenal of weapons to fight that culture war.”

Details of Cambridge’s acquisition and use of Facebook data have surfaced in several accounts since the business began working on the 2016 campaign, setting off a furious debate about the merits of the firm’s so-called psychographic modelling techniques.

But the full scale of the data leak involving Americans has not been previously disclosed – and Facebook, until now, has not acknowledged it. Interviews with a half-dozen former employees and contractors, and a review of the firm’s emails and documents, have revealed that Cambridge not only relied on the private Facebook data but still possesses most or all of the trove.

Cambridge paid to acquire the personal information through an outside researcher who, Facebook says, claimed to be collecting it for academic purposes.

During a week of inquiries from The Times, Facebook downplayed the scope of the leak and questioned whether any of the data still remained out of its control. But on Friday (US time), the company posted a statement expressing alarm and promising to take action.

“This was a scam – and a fraud,” Paul Grewal, a vice-president and deputy general counsel at the social network, said in a statement to The Times earlier on Friday (US time).

He added that the company was suspending Cambridge Analytica, Mr Wylie and the researcher, Aleksandr Kogan, a Russian-American academic, from Facebook.

“We will take whatever steps are required to see that the data in question is deleted once and for all – and take action against all offending parties,” Mr Grewal said.

Alexander Nix, the chief executive of Cambridge Analytica, and other officials had repeatedly denied obtaining or using Facebook data, most recently during a parliamentary hearing last month. But in a statement to The Times, the company acknowledged that it had acquired the data, though it blamed Mr Kogan for violating Facebook’s rules and said it had deleted the information as soon as it learned of the problem two years ago.

In Britain, Cambridge Analytica is facing intertwined investigations by Parliament and government regulators into allegations that it performed illegal work on the Brexit campaign. The country has strict privacy laws, and its information commissioner announced on Saturday (US time) that she was looking into whether the Facebook data was “illegally acquired and used”.

In the United States, Mr Mercer’s daughter Rebekah, a board member, Mr Bannon and Mr Nix received warnings from their lawyer that it was illegal to employ foreigners in political campaigns, according to company documents and former employees.

Congressional investigators have questioned Mr Nix about the company’s role in the Trump campaign. And the Justice Department’s special counsel, Robert Mueller, has demanded the emails of Cambridge Analytica employees who worked for the Trump team as part of his investigation into Russian interference in the election.

While the substance of Mr Mueller’s interest is a closely guarded secret, documents viewed by The Times indicate that the firm’s British affiliate claims to have worked in Russia and Ukraine. And WikiLeaks founder Julian Assange disclosed in October that Mr Nix had reached out to him during the campaign in hopes of obtaining private emails belonging to Mr Trump’s Democratic opponent, Hillary Clinton.

The documents also raise new questions about Facebook, which is already grappling with intense criticism over the spread of Russian propaganda and fake news. The data Cambridge collected from profiles, a portion of which was viewed by The Times, included details on users’ identities, friend networks and “likes”. Only a tiny fraction of the users had agreed to release their information to a third party.

“Protecting people’s information is at the heart of everything we do,” Mr Grewal said. “No systems were infiltrated, and no passwords or sensitive pieces of information were stolen or hacked.”

Still, he added, “it’s a serious abuse of our rules”.

Reading voters’ minds

The Bordeaux flowed freely as Mr Nix and several colleagues sat down for dinner at the Palace Hotel in Manhattan in late 2013, Mr Wylie recalled in an interview. They had much to celebrate.

Mr Nix, a brash salesman, led the small elections division at SCL Group, a political and defence contractor. He had spent much of the year trying to break into the lucrative new world of political data, recruiting Mr Wylie, then a 24-year-old political operative with ties to veterans of President Barack Obama’s campaigns. Mr Wylie was interested in using inherent psychological traits to affect voters’ behaviour and had assembled a team of psychologists and data scientists, some of them affiliated with Cambridge University.

The group experimented abroad, including in the Caribbean and Africa, where privacy rules were lax or nonexistent and politicians employing SCL were happy to provide government-held data, former employees said.

Then a chance meeting bought Mr Nix into contact with Mr Bannon, the Breitbart News firebrand who would later become a Trump campaign and White House adviser, and with Mr Mercer, one of the richest men on earth.

Mr Nix and his colleagues courted Mr Mercer, who believed a sophisticated data company could make him a kingmaker in Republican politics, and his daughter Rebekah, who shared his conservative views.

Mr Bannon was intrigued by the possibility of using personality profiling to shift America’s culture and rewire its politics, recalled Mr Wylie and other former employees, who spoke on the condition of anonymity because they had signed nondisclosure agreements. Mr Bannon and the Mercers declined to comment.

Mr Mercer agreed to help finance a $1.5 million pilot project to poll voters and test psychographic messaging in Virginia’s gubernatorial race in November 2013, where the Republican attorney general, Ken Cuccinelli, ran against Terry McAuliffe, the Democratic fund-raiser. Though Mr Cuccinelli lost, Mr Mercer committed to moving forward.

The Mercers wanted results quickly, and more business beckoned. In early 2014, the investor Toby Neugebauer and other wealthy conservatives were preparing to put tens of millions of dollars behind a presidential campaign for Senator Ted Cruz of Texas, work that Mr Nix was eager to win.

When Mr Wylie’s colleagues failed to produce a memo explaining their work to Mr Neugebauer, Mr Nix castigated them over email.

“ITS 2 PAGES!! 4 hours work max (or an hour each). What have you all been doing??” he wrote.

Mr Wylie’s team had a bigger problem. Building psychographic profiles on a national scale required data the company could not gather without huge expense. Traditional analytics firms used voting records and consumer purchase histories to try to predict political beliefs and voting behaviour.

But those kinds of records were useless for figuring out whether a particular voter was, say, a neurotic introvert, a religious extrovert, a fair-minded liberal or a fan of the occult. Those were among the psychological traits the firm claimed would provide a uniquely powerful means of designing political messages.

Mr Wylie found a solution at Cambridge University’s Psychometrics Centre. Researchers there had developed a technique to map personality traits based on what people had liked on Facebook. The researchers paid users small sums to take a personality quiz and download an app, which would scrape some private information from their profiles and those of their friends, activity that Facebook permitted at the time. The approach, the scientists said, could reveal more about a person than their parents or romantic partners knew — a claim that has been disputed.

When the Psychometrics Centre declined to work with the firm, Mr Wylie found someone who would: Dr Kogan, who was then a psychology professor at the university and knew of the techniques. Dr Kogan built his own app and in June 2014 began harvesting data for Cambridge Analytica. The business covered the costs — more than $US800,000 — and allowed him to keep a copy for his own research, according to company emails and financial records.

All he divulged to Facebook, and to users in fine print, was that he was collecting information for academic purposes, the social network said. It did not verify his claim. Dr Kogan declined to provide details of what happened, citing nondisclosure agreements with Facebook and Cambridge Analytica, though he maintained that his program was “a very standard vanilla Facebook app”.

He ultimately provided over 50 million raw profiles to the firm, Mr Wylie said, a number confirmed by a company email and a former colleague. Of those, roughly 30 million — a number previously reported by The Intercept — contained enough information, including places of residence, that the company could match users to other records and build psychographic profiles. Only about 270,000 users — those who participated in the survey — had consented to having their data harvested.

Mr Wylie said the Facebook data was “the saving grace” that let his team deliver the models it had promised the Mercers.

“We wanted as much as we could get,” he acknowledged. “Where it came from, who said we could have it – we weren’t really asking.”

Mr Nix tells a different story. Appearing before a parliamentary committee last month, he described Dr Kogan’s contributions as “fruitless”.

An international effort

Just as Dr Kogan’s efforts were getting underway, Mr Mercer agreed to invest $15 million in a joint venture with SCL’s elections division. The partners devised a convoluted corporate structure, forming a new American company, owned almost entirely by Mr Mercer, with a license to the psychographics platform developed by Mr Wylie’s team, according to company documents. Mr Bannon, who became a board member and investor, chose the name: Cambridge Analytica.

The firm was effectively a shell. According to the documents and former employees, any contracts won by Cambridge, originally incorporated in Delaware, would be serviced by London-based SCL and overseen by Mr Nix, a British citizen who held dual appointments at Cambridge Analytica and SCL. Most SCL employees and contractors were Canadian, like Mr Wylie, or European.

But in July 2014, an American election lawyer advising the company, Laurence Levy, warned that the arrangement could violate laws limiting the involvement of foreign nationals in American elections.

In a memo to Mr Bannon, Ms Mercer and Mr Nix, the lawyer, then at the firm Bracewell & Giuliani, warned that Mr Nix would have to recuse himself “from substantive management” of any clients involved in United States elections. The data firm would also have to find American citizens or green card holders, Mr Levy wrote, “to manage the work and decision making functions, relative to campaign messaging and expenditures”.

In summer and fall 2014, Cambridge Analytica dived into the American midterm elections, mobilising SCL contractors and employees around the country. Few Americans were involved in the work, which included polling, focus groups and message development for the John Bolton Super PAC, conservative groups in Colorado and the campaign of Senator Thom Tillis, the North Carolina Republican.

Cambridge Analytica, in its statement to The Times, said that all “personnel in strategic roles were US nationals or green card holders”. Mr Nix “never had any strategic or operational role” in an American election campaign, the company said.

Whether the company’s American ventures violated election laws would depend on foreign employees’ roles in each campaign, and on whether their work counted as strategic advice under Federal Election Commission rules.

Cambridge Analytica appears to have exhibited a similar pattern in the 2016 election cycle, when the company worked for the campaigns of Mr Cruz and then Mr Trump. While Cambridge hired more Americans to work on the races that year, most of its data scientists were citizens of the United Kingdom or other European countries, according to two former employees.

Under the guidance of Brad Parscale, Mr Trump’s digital director in 2016 and now the campaign manager for his 2020 re-election effort, Cambridge performed a variety of services, former campaign officials said. That included designing target audiences for digital ads and fund-raising appeals, modelling voter turnout, buying $5 million in television ads and determining where Mr Trump should travel to best drum up support.

Cambridge executives have offered conflicting accounts about the use of psychographic data on the campaign. Mr Nix has said that the firm’s profiles helped shape Mr Trump’s strategy — statements disputed by other campaign officials — but also that Cambridge did not have enough time to comprehensively model Trump voters.

In a BBC interview last December, Mr Nix said that the Trump efforts drew on “legacy psychographics” built for the Cruz campaign.

After the leak

By early 2015, Mr Wylie and more than half his original team of about a dozen people had left the company. Most were liberal-leaning, and had grown disenchanted with working on behalf of the hard-right candidates the Mercer family favoured.

Cambridge Analytica, in its statement, said that Mr Wylie had left to start a rival firm, and that it later took legal action against him to enforce intellectual property claims. It characterised Mr Wylie and other former “contractors” as engaging in “what is clearly a malicious attempt to hurt the company”.

Near the end of that year, a report in The Guardian revealed that Cambridge Analytica was using private Facebook data on the Cruz campaign, sending Facebook scrambling. In a statement at the time, Facebook promised that it was “carefully investigating this situation” and would require any company misusing its data to destroy it.

Facebook verified the leak and — without publicly acknowledging it — sought to secure the information, efforts that continued as recently as August 2016. That month, lawyers for the social network reached out to Cambridge Analytica contractors. “This data was obtained and used without permission,” said a letter that was obtained by The Times. “It cannot be used legitimately in the future and must be deleted immediately.”

Mr Grewal, the Facebook deputy general counsel, said in a statement that both Dr Kogan and “SCL Group and Cambridge Analytica certified to us that they destroyed the data in question”.

But copies of the data still remain beyond Facebook’s control. The Times viewed a set of raw data from the profiles Cambridge Analytica obtained.

While Mr Nix has told lawmakers that the company does not have Facebook data, a former employee said that he had recently seen hundreds of gigabytes on Cambridge servers, and that the files were not encrypted.

Today, as Cambridge Analytica seeks to expand its business in the United States and overseas, Mr Nix has mentioned some questionable practices. This January, in undercover footage filmed by Channel 4 News in Britain and viewed by The Times, he boasted of employing front companies and former spies on behalf of political clients around the world, and even suggested ways to entrap politicians in compromising situations.

All the scrutiny appears to have damaged Cambridge Analytica’s political business. No American campaigns or “super PACs” have yet reported paying the company for work in the 2018 midterms, and it is unclear whether Cambridge will be asked to join Mr Trump’s re-election campaign.

In the meantime, Mr Nix is seeking to take psychographics to the commercial advertising market. He has repositioned himself as a guru for the digital ad age — a “Math Man”, he puts it. In the United States last year, a former employee said, Cambridge pitched Mercedes-Benz, MetLife and the brewer AB InBev, but has not signed them on.

-The New York Times