Hancock (programming language)

Hancock is a language for data stream mining programs. Data streams differ from traditional stored databases in that they experience very high volumes of data and allow analysts to act upon such data in near-real time. Stored databases, on the other hand, involve data being inputted for offline querying.[6] Data warehouses, which store intersectional data from different systems, can be costly to build and lengthy to implement. Simplified data warehouses can take months to build.[5]

The scale of massive data stream mining poses problems to data miners. For example, internet and telephone network data mining might be tasked with finding persistent items, which are items that regularly occur in the stream.[8] However, these items might be buried in a large amount of the network’s transactional data; while the items can eventually be found, data miners aim for increased time efficiency in their search.[8]

In database technology, users do not necessarily know where the data they are searching for is located. These users only have to issue queries for data, which the database management system returns. In a large data set, data can be contained in random-access memory (RAM), which is the primary storage, or disk storage, which is secondary storage. In 2000, Padharic Smyth estimated that, using the most recent technology, data located in RAM could be accessed relatively quickly, “on the order of 10⁻⁷-10⁻⁸ seconds,” while secondary storage data took significantly longer to access, “on the order of 10⁴-10⁵” seconds.[3]

Data mining can be broken down into the processes of input, analysis, and the reporting of results; it uses algorithms to find patterns and relationships among the subjects and has been used by commercial companies to find patterns in client behavior.[9] Data analysts are needed to collect and organize data and train algorithms.[4]

KianSing Ng and Huan Liu opine that even with straightforward data mining goals, the actual process is still complex. For example, they argue that real-world data mining can be challenged by data fluctuations, which would render prior patterns “partially invalid.” Another complication is that most databases in existence in 2000 were characterized by high dimensionality, which means that they contain data on many attributes. As Ng and Liu note, high dimensionality produces long computing times; this can be solved by data reduction in the pre-processing stage.[10]

Hancock's process is as follows:

• Hancock programs analyze data as it arrives, in real-time, into data warehouses.[2]
• Hancock programs computed the signatures, or behavioral profiles, of transactors in the stream.[1]
  • Data stream transactors include telephone numbers or IP addresses.[1]
  • Signatures enable analysts to discover patterns hidden in the data.[1]
• Telecommunications data streams consist of call-records, which include information on the locations of callers, time of calls, and sometimes include recordings of conversations.[11]
  • Hancock was used to process signatures based on data like the length of phone calls and the amount of calls to a particular area over a specified interval of time.[12]
• Hancock programs used link analysis to find “communities of interest," which connected signatures based on similarities in behavior. Link analysis require that linkages between data are continually updated, and are used to detect fraud networks.[12]
  • Link analysis, which can be considered a form of association data mining, aims to find connections between relationships.[13] One such relationship is call patterns in telecommunications.[13] Association data mining aims to find relationships between variables. For example, one research paper[13] suggested that a market could use association analysis to find the probability that a customer who purchases coffee also purchases bread; the market could then use that information to influence store layout and promotions.

Because Hancock code performed efficiently, even with large amounts of data, the AT&T researchers claimed that it allowed analysts to create applications "previously thought to be infeasible."[1]

Hancock programs assisted in AT&T's marketing efforts.[2] In the 1990s, large data stream mining and the increased automation of government public record systems allowed commercial corporations in the United States to personalize marketing.[15] Signature profiles were developed from both transaction records and public record sources.[15] Ng and Liu, for example, applied data mining to customer retention analysis, and found that mining of association rules allowed a firm to predict departures of influential customers and their associates. They argued that such knowledge subsequently empowers the company’s marketing team to target those customers, offering more attractive pitches.[10]

Data mining assisted telecommunications companies in viral marketing, also known as buzz marketing or word-of-mouth marketing, which uses consumer social networks to improve brand awareness and profit.[16] Viral marketing depends on connections between consumers to increase brand advocacy, which can either be explicit, such as friends recommending a product to other friends, or implicit, such as influential consumers purchasing a product.[16] For firms, one of the goals of viral marketing is to find influential consumers who have larger networks. Another method of viral marketing is to target the neighbors of prior consumers, known as “network targeting.”[16] Using Hancock programs, analysts at AT&T were able to find "communities of interest," or interconnected users who featured similar behavioral traits.[12]

One of the issues viral marketing promoters encountered was the large size of marketing data sets, which, in the case of telecommunication companies, can include information on transactors and their descriptive attributes and transactions.[16] Marketing data sets, when amounting in the hundreds of millions, can exceed the memory capacity of statistical analysis software.[16] Hancock programs addressed data scaling issues and allowed analysts to make decisions as the data flowed into the data warehouses.[2]

While the development of wireless communication devices allowed law enforcement to track the location of users, it also allowed companies to improve consumer marketing, such as by sending messages according to wireless user’s proximity to particular businesses.[15] Through cell site location data, Hancock programs were capable of tracking wireless users' movements.[2]

According to academic Alan Westin, the increase of telemarketing during this period also increased consumer annoyance.[15] Statisticians Murray Mackinnon and Ned Glick hypothesized in 1999 that firms hid their use of commercial data mining because of potential consumer backlash for mining customer records.[5] As an example, Mackinnon and Glick cited a June 1999 lawsuit in which the state of Minnesota sued US Bancorp for releasing customer information to a telemarketing firm; Bancorp promptly responded to the lawsuit by restricting its usage of customer data.[5]

AT&T researchers, including Cortes, showed that Hancock-related data mining programs could be used for finding telecommunications fraud.[14]

Telecommunications fraud detection includes subscription fraud, unauthorized calling card usage, and PBX fraud.[17] It is similar to mobile communications and credit card fraud: in all three, firms must process large amounts of data in order to obtain information; they must deal with the unpredictability of human behavior, which makes finding patterns in the data difficult; and their algorithms must be trained to spot the relatively rare cases of fraud among the many legitimate transactions.[17] According to Daskalaki et al., in 1998, telecommunications fraud incurred billions of dollars in annual losses globally.[17]

Because fraud cases were relatively few compared to the hundreds of millions of daily telephone transactions that occurred, algorithms for data mining of telecommunication records need to provide results quickly and efficiently.[12] The researchers showed that communities of interest could identify fraudsters since data nodes from fraudulent accounts are typically located closer to each other than to a node from a legitimate account.[14]

Through social network analyses and link analysis, they also found that the set of numbers that were targeted by fraudulent accounts, which were then disconnected, were often called on by fraudsters from different numbers; such connections could be used to identify fraudulent accounts. Link analysis methods are based on the assumption that fraudsters rarely deviate from their calling habits.[12]

Federal telecommunications surveillance is not a recent historical development in the United States. According to academic Colin Agur, telephone surveillance by law enforcement in the United States became more common in the 1920s.[18] Particularly, telephone wiretapping became a prevalent form of evidence collection by law enforcement officials, especially federal agents, during Prohibition.[18] Agur argues that the Communications Act of 1934, which established the Federal Communications Commission, reigned in law enforcement abuse of telephone surveillance.[18] Under the act, telecommunications companies could keep records and report to the FCC illegal telecommunications interception requests. After the Federal Wiretap Act of 1968 and the Supreme Court's decision in Katz v. United States, both of which extended Fourth Amendment protections to telephone communications, federal telecommunications surveillance required warrants.[18]

he FBI was first authorized to obtain national security letters (NSLs) for communication billings records, including those from telephone services, after Congress passed the Electronic Communications Privacy Act of 1986. The letters forced telephone companies to provide the FBI with customer information, such as names, addresses, and long-distance call records. Congress would eventually expand NSL authority to include warrants for local-distance call records as well.[19]

After the September 11, 2001 attacks, Congress passed the USA PATRIOT Act, which made it easier for investigators at the FBI to be issued national security letters for terrorism investigations (NSLs). Academics William Bendix and Paul Quirk contend that the PATRIOT Act allowed the FBI to access and collect the private data of many citizens, without the approval of a judge. The FBI was allowed to keep a collection of records, with no time limit for possession. It could also force NSL recipients to remain silent through the use of gag orders.[19]

The Wired article claimed that the FBI began making warrantless requests to telecommunication companies for "communities of interest" records of suspects under the USA PATRIOT Act. The article claimed that law enforcement discovered the existence of such records based on research by Hancock's creators.[2]

In 2005, government leaks revealed the FBI’s abuse of NSLs. In 2006, when the PATRIOT Act was renewed, it included provisions that required the Justice Department’s inspector general to annually review NSL usage. The first inspector general report found that 140,000 NSL requests, on nearly 24,000 U.S. persons, were granted to FBI agents from 2003 to 2005. The data was then added to databanks available to thousands of agents.[19]

The public-private relationship of telecommunication companies extends into the homeland security domain. Telecommunication companies, including AT&T, Verizon, and BellSouth, cooperated with NSA requests for access to transactional records.[20] Telecommunications companies, including AT&T, have maintained partnerships with government agencies, like the Department of Homeland Security, to collaborate on sharing information and solving national cybersecurity issues.[21] AT&T representatives sit on the board of the National Cyber Security Alliance (NCSA), which promotes cybersecurity awareness and computer user protection.[21]

Analysts at the NSA, under authority of the secret Terrorist Surveillance Program, also used data mining to find terrorist suspects and sympathizers. In this search, the NSA intercepted communications, including telephone calls, leaving and entering the United States. Agents screened the information for possible links to terrorism, such as the desire to learn to fly planes or specific locations of the communication’s recipients, like Pakistan.[20]

In 2005, the New York Times reported on the existence of the program, which the Bush administration defended as necessary in its counterterrorism efforts and limited to terrorist suspects and associates.[20]

However, in 2007, the Wired article noted how AT&T and Verizon were being sued in federal court for providing the NSA with access to billions of telephone records belonging to Americans for anti-terrorism activities, such as using data mining to locate suspected terrorists and warrantless wiretapping targets.[2]

In 2013, following the Snowden leaks, it was revealed that the program had also mined the communications of not just terrorist suspects, but also millions of American citizens. A 2014 independent audit by the Privacy and Civil Liberties Oversight Board found that the program had limited counterterrorism benefits.[20]