Data, simple. blog

David Tsukiyama
6/20/2015: Simple Polarity Scores with the New York Times API and TextBlob

Quick and Dirty Method to Search for all Relevant New York Times Article Comments and Classify Text. First I want to filter comments to only those articles that were the most popular articles for that week.

def emailed_results(data):
    for i, document in enumerate(data):
        return data['results']

def parsed_mailed(data):
    mailed = []
    for b in data:
        dic = {}
        dic['sub-title'] = b['abstract']
        dic['byline'] = b['byline']
        dic['column'] = b['column']
        dic['type'] = b['des_facet']
        dic['date'] =b['published_date']
        dic['section'] = b['section']
        dic['title'] = b['title']
        dic['url'] = b['url']
    return mailed

def title(data):
    for i, title in enumerate(d['title'] for d in data): 
            print i,title
def url(data):
    for i, url in enumerate(d['url'] for d in data): 
            print i,url

def most_mailed(days, api):
    import urllib
    import json
    bucket = ''
    string = bucket+days+api
    response_string = urllib.urlopen(string).read()
    response_dictionary = json.loads(response_string)
    results = emailed_results(response_dictionary)
    parsed_results = parsed_mailed(results)
    titles = title(parsed_results)
    urls = url(parsed_results)
    return titles, urls

#past seven days

#call the function
most_mailed(days, api)

Title url
To Lose Weight, Eating Less Is Far More Important Than Exercising More
How to Pick a Cellphone Plan for Traveling Abroad
Naomi Oreskes, a Lightning Rod in a Changing Climate
How to Make Online Dating Work
America’s Seniors Find Middle-Class ‘Sweet Spot’
Experts on Aging, Dying as They Lived
What It’s Like as a ‘Girl’ in the Lab
Three Simple Rules for Eating Seafood
Pope Francis, in Sweeping Encyclical, Calls for Swift Action on Climate Change
Stop Revering Magna Carta
Review: Pixar’s ‘Inside Out’ Finds the Joy in Sadness, and Vice Versa
Cardinals Investigated for Hacking Into Astros’ Database
In Tucson, an Unsung Architectural Oasis
Magna Carta, Still Posing a Challenge at 800
Democrats Being Democrats
Black Like Who? Rachel Dolezal’s Harmful Masquerade
A Sea Change in Treating Heart Attacks
In ‘Game of Thrones’ Finale, a Breakdown in Storytelling
My Choice for President? None of the Above
The Family Dog

Next we loop through the list of articles, and also get all the comments from each article. This takes several helper functions, the final function is called below.

df = []
for b in articles:
    initial_df = nytimes(b)
    df = df + initial_df 
    print 'Processing ' + str(b) + '...'

We have collected all the comments from all the most-emailed articles of the past seven days


{'comment': u'The only way to lose weight permanently and arrive at a normal weight (117 pounds for me) is to create a calorie deficit and then a calorie equilibrium once that goal is achieved. 

Thirty-five hundred calories eaten but not burned is equal to one pound of fat gained and 3,500 calories burned, but not consumed is equal to one pound of fat lost. Obviously eating 3,500 calories is a lot quicker than burning them. It would take 35 miles of walking to do that.

Fifteen years ago I was obese and about 70 pounds heavier, but I have maintained a normal weight since losing my excess weight in the year 2,000.

After studying the long-term research results, I started free weight loss groups of people who were suffering from my problem. We logged our daily calories, exercise and weight with each other. I still do it and you can join me if you are motivated and have less than 100 pounds to lose. You can see our results and my before and after pictures on

Let me know if this interests you by emailing me at See you lighter soon?', 'comment_type': u'comment', 'date': u'1434435437', 'editorsSelection': 0, 'email': u'', 'location': u'New York City', 'login': None, 'name': u'Roberta Russell', 'recommend': 23, 'replies': [], 'update_date': u'1434435486'}

I am kind of curious about recommended comments and those comments selected by the editor

import numpy as np
np.mean([b['editorsSelection'] for b in df])
np.mean([b['recommend'] for b in df])
time=np.asarray([b['date'] for b in df]).astype(float)
recommendations=np.asarray([b['recommend'] for b in df]).astype(float)

import matplotlib.pyplot as plt
import seaborn

model = zip( time, recommendations)
sliding = []

windowSize = 100

tSum = sum([x[0] for x in model[:windowSize]])
rSum = sum([x[1] for x in model[:windowSize]])

for i in range(windowSize,len(model)-1):
  tSum += model[i][0] - model[i-windowSize][0]
  rSum += model[i][1] - model[i-windowSize][1]

X = [x[0] for x in sliding]
Y = [x[1] for x in sliding]

plt.plot(X, Y)
plt.title("Recommendations over Time", fontsize=15)


One of the quickest ways to get a read on text sentiment without having to build your own classifier is to use TextBlob

from textblob import TextBlob

text=[b['comment'] for b in df]
text=[word.strip("!\"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~").lower() for word in text]

for b in text:


4/19/2015: Getting comments from specific articles with the New York Times API

While there is a Python wrapper for the New York Times Article Search API, getting article comments necessitates the use of the New York Time API directly. Compared to the Article Search wrapper, there are additional differences in parsing the results. In this example we are querying for the term 'misconduct' in an article's body, byline, and headline and filter this through those articles that contain Facebook in the headline and where the source is either Reuters, The AP, or the New York Times.

from nytimesarticle import articleAPI

articles = q = 'misconduct', 
	fq = {'headline':'Facebook', 'source':['Reuters','AP', 'The New York Times']}, 
	begin_date = 20140101 )

{u'copyright': u'Copyright (c) 2013 The New York Times Company.  All Rights Reserved.',
 u'response': {u'docs': [{u'_id': u'542d8c8638f0d87d7534ce9e',
    u'abstract': u'Facebook pledges that future research on its 1.3 billion users will be subjected to greater internal scrutiny from top managers, especially if it is focused on personal topics; pledge follows public backlash after company undertook study that used its newsfeed to manipulate the emotions of some users without telling them; declines to disclose particulars of new research guidelines.',
    u'blog': [],
    u'byline': {u'contributor': u'',
     u'original': u'By VINDU GOEL',
     u'person': [{u'firstname': u'Vindu',
       u'lastname': u'GOEL',
       u'organization': u'',
       u'rank': 1,
       u'role': u'reported'}]},
    u'document_type': u'article',
    u'headline': {u'main': u'Facebook Promises Deeper Review of User Research, but Is Short on the Particulars',
     u'print_headline': u'Facebook Vow on Research Is Short on the Particulars'},
    u'keywords': [{u'is_major': u'Y',
      u'name': u'organizations',
      u'rank': u'1',
      u'value': u'Facebook Inc'}

Parse the output, for example if you want just the deadline and publish date from the output

def parse(articles):
    news = []
    for i in articles['response']['docs']:
        dic = {}
        dic['headline'] = i['headline']
        dic['date'] = i['pub_date'][0:10]
    return news

{'date': u'2014-10-03',
 'headline': {u'main': u'Facebook Promises Deeper Review of User Research, but Is Short on the Particulars',
  u'print_headline': u'Facebook Vow on Research Is Short on the Particulars'}}

Now we will use the New York Times Community API to get comments from specific articles. Here we will get comments for David Brook's Sunday Op-Ed column, "The Moral Bucket List." The API retrieves only 25 comments per call; therefore a loop may be needed to retrieve all comments. I just happened to pick this article because it was one of the most e-mailed articles this past week. However you should definitely check out the documentation for a better understanding. To retrieve comments associated with a specific URL, use the following URI structure:


response = urllib.urlopen(bucket).read()
response_dictB = json.loads(response)
print response_dictionary.keys()

We write a function to extract only information we are interested in and call the function and view the first entry:

def parse(mail):
    brooks = []
    for b in mail:
        dic = {}
        dic['comment'] = b['commentBody']
        dic['date'] = b['createDate']
        dic['comment_type'] = b['commentType']
        dic['editorsSelection'] =b['editorsSelection']
        dic['email'] = b['email']
        dic['recommend'] = b['recommendationCount']
        dic['replies'] = b['replies']
        dic['name'] = b['userDisplayName']
        dic['location'] = b['userLocation']
        dic['login'] = b['login']
    return brooks


{'comment': u'The lament of the hollow man who sees but does not understand.',
 'comment_type': u'comment',
 'date': u'1428853503',
 'editorsSelection': 0,
 'email': u'',
 'location': u'Maine',
 'login': None,
 'name': u'Eileen Wilkinson',
 'recommend': 4,
 'replies': []}	

For more insight press here

New York Times API