Methods

Community Media Workshop constructed the NEW News report for 2012 in four stages:

1. Compiling a master list of sites to consider;
2. Narrowing down the list based on certain criteria;
3. Undertaking a broad quantitative review of outlets’ news output, social media reach and web reach; and
4. Conducting a qualitative assessment of 51 sites’ news and community engagement.

At the outset of the project, the Workshop compiled a list of more than 1,000 URLs to review from several sources:

• “Getting On Air, Online & Into Print,” the media guide published annually by Community Media Workshop.
• The 2010 NEW News list, collected via a survey of news site operators.
• A list Rich Gordon of Northwestern University compiled as part of a separate project funded by The Chicago Community Trust to understand the network connections among Chicago news and information sites.
• A list maintained by Workshop staff of sites overlooked in the 2010 report.
• An open call posted on the Workshop’s NP Communicator blog.

The Workshop combined all of the lists, removing both duplicates and sites that do not primarily serve the City of Chicago (as opposed to a primarily suburban audience), reducing the list to 476 websites. The Workshop then reviewed each of those sites to see if they published a minimum amount of news production, which we defined as publishing 12 or more items in April 2012. We borrowed our criteria here from Michele McLellan’s prior research on hyperlocal news.

Our review uncovered 191 sites that met the criteria. For the 191 sites, we measured the following sources of data, assigning scores between 1 and 5 as the chart below indicates. In the end, each site had a quantitative score of between 5 and 25, based on the factors listed in Chart 1 below.

*Note: The lower the rank, the larger the site’s traffic. For comparison: Facebook is ranked #1.

The Workshop also categorized sites into one of five categories:

• Citywide news: news outlets that cover the entire city.
• Neighborhood news: news outlets that cover one or more neighborhoods (such as hyde Park or Austin) in the city.
• Specialty news: news outlets that cover one or more topics or themes (such as schools or crime) or that focus solely on investigative or watchdog journalism.
• Arts, entertainment and culture: sites that cover the arts, culture, entertainment or dining.
• Aggregation: sites that primarily, though not exclusively, bring together the latest news from multiple sources.

Many sites could have been reviewed in more than one category; in the end, final category selection reflected the overall judgment of the Workshop.

Determining which sites to score for news quality and community engagement

Our goal was to rank approximately 50 sites. Initially, our plan was to score 10 sites in each category. This turned out to be a problematic way to proceed. By the nature of the project, sites in each category have different levels of audience reach (citywide news and arts and entertainment having the largest potential audiences; neighborhood and specialty news having smaller potential audiences) so the scoring cutoff for each category varied. Additionally, a very large number of sites had very similar scores in a few categories, especially citywide news, making it difficult to limit those categories to 10. here is our breakdown of sites we scored in each category, and the scoring cutoffs we chose.

• Given the large number of citywide news sites scoring within a few points of each other, we ended up ranking 17 sites that scored at least 20 out of 25 points on our quantitative scoring. As a result, we had to trim the number of sites we ranked in other categories.
• It remained important to us to rank at least 10 sites in the categories of neighborhood news and specialty news. We ranked 11 sites scoring 16 or higher (out of 25) for neighborhood news; and 11 sites scoring a 17 or higher (out of 25) for specialty news.
• To accommodate the large number of citywide news sites, neighborhood news sites and specialty news sites, we ended up reviewing a relatively small number of arts and culture websites, reviewing five sites rating 21 or higher (out of 25) for arts, culture and entertainment.
• We had a small number of aggregators within the study. We ultimately reviewed seven aggregators that scored 13 or higher (out of 25).
• In all cases, Workshop members reviewed all sites in a category that had the same score.

We brought our proposed list of sites to score, as well as our categories, to our Advisory Committee, which provided feedback on our choices and our organization of data.

Reviewing sites for news quality and community engagement

After broad discussions within the research team and the Advisory Committee, we determined criteria to score sites for news quality and community engagement. The Workshop staff then ranked sites in two qualitative areas: news quality and community engagement. Each site could score between 1 and 5 in these two categories. As a result, the total score possible for each site increased from 25 to 35.

Workshop staff scored the following elements equally to come up with a 1 to 5 rating.

For news quality:

• Website covers a range of topics or geographic areas within its area of focus.
• Website’s coverage is thorough.
• Website coverage attempts to reach and cover multiple points of view in storytelling, especially those whose viewpoints are not necessarily heard from.
• News writing is strong and engaging (not scored for aggregators).
• Aggregation practices are fair (with clear links to sources and not excessive reuse of content) (for aggregators only).
• The source of aggregated content is clear and understood (for aggregators only).
• Website is well designed and takes advantage of all the tools of online/multimedia to tell stories.

For community engagement:

• It’s easy to find out who is publishing the website and to get a glimpse of the people reporting and editing the news.
• It is easy to share content and it is easy to find out if content is being shared.
• The voices of site visitors play a well-managed role on the site and/or on Facebook and Twitter.
• It is easy and prominent for users to email tips, photos and other information to the website for consideration for coverage.
• Site or its members holds offline events.

Workshop staffers conducted the reviews, with each site receiving at least two reviews. Workshop team members not involved with the rankings compared the reviews between reviewers and requested additional reviews for sites where rankings in one or more categories were widely divergent.

Once scoring was complete, the Workshop added the two averaged scores to the existing scores to complete a final ranking. Partial scores were discarded. We have awarded honorable Mention to several additional sites in each category at the judgment and discretion of the Workshop.

The report’s total scores can add up to a maximum of 35 points. Twenty-five of the 35 points reflect the size and reach of a site’s audience; 5 points reflect the Workshop’s assessment of its news quality; and 5 points reflect the Workshop’s assessment of its community engagement.

By definition, sites that made the rankings by definition had the largest audiences and reach among their peer sites that the Workshop’s methodology was capable of measuring. (Some sites are structured in a way that makes measuring their reach using our methodology difficult; we discuss this in the limitations.) Sites that are at the

top of the rankings often, but not always, have higher scores for news quality and community engagement than other sites. In some instances, sites with high scores in audience and reach develop a points lead that cannot be overcome by stellar news quality and community engagement scores. We have provided summaries of the scores for each category to satisfy reader interest as to the components that make up each site’s score.

Limitations

The Workshop continues to both build on prior research studies (NEW News 2009 and 2010) and the peer research in the field. These methods of research are evolving though, and people may not agree on any single combination of qualitative and quantitative methods to evaluate the quality and relevance of online local news.

Here are some of the limitations that we are aware of as we worked on this project:

• Due to the nature of our project, we did not have time or resources to evaluate the online news landscape serving suburban Chicago. We have written about trends in online news in suburban Chicago in a separate sidebar.
• We did not evaluate ethnic and multicultural news in a separate category. It is likely ethnic media is underrepresented in the final rankings because we suspect our measures of reach undercount or understate the online reach of many ethnic news sites, especially those not published in English. We evaluated sites geared toward a particular ethnic or racial audience based on the area of the city they covered. Therefore, only two ethnic news outlets are in the final rankings. We have discussed the challenges facing ethnic media, and additional ethnic media standouts, in a separate sidebar.
• We did not include Workshop sites such as the long-standing news site Newstips or sites run by Advisory Committee members in order to keep the ranking process fair and unbiased.
• We rely upon public measures of site traffic, such as Compete.com, rather than analytics from site publishers, which means that in many instances, traffic to smaller sites is likely undercounted. This is a choice we made based on our prior experiences with self-reported analytics. The 2009 NEW News study’s reliance on self-reported data turned out to be controversial because we could not audit data provided by site operators. In the 2010 report, we used Compete.com data to measure the reach of online news in Chicago but were stymied by the complete lack of data for most
• Chicago publishers. Site publishers had recom- mended we use Quantcast instead, although Quantcast data is sometimes not available and varies in accuracy based on whether the site operator has installed Quantcast on their site. We found more sites in our study had some Compete. com data than in our 2010 review, although Compete.com is still skewed toward larger sites with a national reach.
• Our focus this year is on news outlets that operate websites. As a result, a few email newsletters that broadly serve a similar purpose could not be considered. In a few instances, those e-newsletters receive honorable Mentions or are mentioned in “What I Read.”
• Our reach measurements favor freestanding websites. This methodology undercounts the reach of sites that are built as part of a larger site structure (for example, huffington Post Chicago).
• We borrowed heavily from the 2009 report’s scoring bands for web reach and readership. It is possible that this scoring does not make fine enough distinctions between large sites with large readership and smaller sites with different levels of readership. hopefully in both cases, this means that the Workshop’s assessment of news quality and engagement can help ensure that the largest sites aren’t the only sites with the possibility of a high ranking.

NEXT: Suburban Online News