Survey Says...

What's wrong with the way District 113 is seeking community input?

Over the last few weeks, District 113 conducted a survey that solicits community input about the future of our high schools

The survey questions covered roughly two broad areas: a post-mortem on the , and areas of need for the 113 schools. While I expect the results of the survey to start appearing in district communications soon, I am concerned about the way it was conducted and also the content of the survey, worries shared by many other District 113 residents.

The District 113 survey was widely publicized. District residents received a postcard soliciting survey responses, news stories provided links, and emails went home to booster club parents. My wife first heard about the survey from a city commissioner; within hours, both of us had taken the survey. That's when my spidey sense went off. 

In my day job, I run a product management organization and am quite familiar with market research survey design and output. Several things about the District 113 survey immediately worried me.

First, the survey has no validation. Anyone can take the survey unlimited times, which can lead to sample bias. The lack of validation also means survey responses could be completed by people who are not stakeholders in District 113, or students who aren't voting age. Few of the questions in the survey required answers, so you could go through and selectively provide input. Thus, there simply is no way to know how many times the ballot box has been stuffed, and whether the stuffers were eligible to do so at all. I asked District 113 about this concern. Their reply was that "Incorporating additional methodologies [for validation] would be very costly and in the opinion of the market research study group not worth the additional expense."

My second concern was the way in which some of the questions were asked. Who wouldn't indicate that they are in favor of "trustworthy elected officials" or "safe facilities" or "Adequate air ventilation"? Who would want to say that "enough quality lighting" is unimportant? Question seven asks readers to trade off these obviously-necessary components of any school. It seems like most answers will say that every possible aspect of a "quality high school" is needed; there is no weighting or reason for anyone filling out the survey to do anything other than indicate that every desire is important.

Third, I was anxious about the tone of some of the questions, and of District 113's response to criticism of the survey. The survey asked respondents to agree or disagree with statements like "I did not trust the intentions/expertise of the administrators who developed the plan"and "I was skeptical as to how well thought out the plan was." These questions wouldn't be asked in typical market research studies, because they assign a human, emotional element tied to people rather than process. Relating to these statements will not help District 113 design the next referendum, they only serve to assuage or impact the egos of those involved.

District 113, as is true in the survey itself, by asserting the qualifications of the volunteers assisting in this process. As with some of the survey questions, the tone of their response is "we know what we're doing, trust us."

This topic is too important to go with that approach. Simple techniques like IP address capture and cross-checking, use of authenticated Internet services for age verification, or serial number assignment are inexpensive and would have avoided any questions of propriety. 

I've personally come a long way over the last year on the issue of the referendum and the need to invest in both District 113 and District 112 schools. Like many voters, I am anxiously awaiting a new plan from District 113 on what the real needs are for our two area high schools. I've been persuaded by current and former families that the time is now to embark on some of the key improvements already identified. 

What I don't appreciate is a "lies, damn lies, and statistics" approach to decision-making. The district can assert that they will normalize the data from the survey, but without proper upfront controls, it is potentially garbage in, garbage out. Once District 113 starts using these statistics to influence the debate, how the numbers were arrived at will no longer be part of the discussion. It will be important for the District to prove that somehow the survey hasn't been unduly influenced by repeat voters, students, or partisans, or the survey was pointless.

Read District 113 Superintendant George Fornero's response to survey criticism .

For more news and updates from Highland Park Patch, "like" us on Facebook or follow us on Twitter.

Bob Levi January 28, 2012 at 04:33 PM
Did I say anything about anyone being dishonest? If you interpret my words that way, it's hard for me to imaging how you and the committee will interpret the survey results. Just wondering. BTW - My initial consulting work for the sugar industry was in 1969; however, I did follow-up work for them over subsequent years. I also tend to track developments in an industry after I've worked for that indsutry, particularly trade associations. As far as the New Coke introduction, didn't any of the many case studies ever give the reason why Coca-Cola reformulated. Since sugar was the main component of their original formula, perhaps the switch to lower-priced HFCS was one of the incentives they had to change. I felt I didn't need "proof" to make my "guess."
Ken Robertson January 28, 2012 at 06:44 PM
Bob - you said "Ask the professional people on your market research committtee the same question. It would be intersting to hear their responses, particularly if they'd be candid with their answers.". Here is the definition of "candid" from Merriam-Webster: "free from bias, prejudice, or malice;marked by honest sincere expression;indicating or suggesting sincere honesty and absence of deception". I am of course assuming that you are not referring to photography in your statement. Now, given your statement and the actual definition of the words you use, it seems that, yes, you are reserving the right to call them dishonest or deceptive (i.e. not candid). FYI - I am not on the committee, therefore how I interpret the results will be of no consequence to them. However, based on your pre-disposed opinion on the validity from having seen 1/3 of the survey, along with your experience in the sugar industry making you an expert in the analysis of market research re:New Coke (by the way - there were many case studies, reports etc about why Coke was reformulated, and it had nothing to do with HFCS), it's hard for me to imagine how YOU will interpret the results. My "guess" is that you will see them as a basis for powering Japanese buses using New Coke, but, candidly, I have no "proof" for that.
Bob Levi January 30, 2012 at 10:48 PM
I received a Patch folloe-up posting email on Sunday evening that the following comment had been posted to this thread: Richard Heineman commented on the article Survey Says... "Coke allowed its bottlers to switch to HFCS in 1980 and everyone had already converted before “New Coke” was introduced." Yet today, Monday, the comment doesn't appear on the thread. Did someone flag it as inappropriate by accident?
Richard H Heineman Jr January 30, 2012 at 11:02 PM
I deleted it because I decided that it was not important
Walter (Tripp) Hainsfurther January 31, 2012 at 02:52 PM
For everyone's informatiion, there will be a meeting of the 1914 Buildings/PE and Athletic Facilities Study Group this evening at 7 PM. The meeting will be held at Deerfield High School and we will be discussing needs for Deerfield High School that should be accounted for in the Master Plan. Everyone is welcome to attend.


More »
Got a question? Something on your mind? Talk to your community, directly.
Note Article
Just a short thought to get the word out quickly about anything in your neighborhood.
Share something with your neighbors.What's on your mind?What's on your mind?Make an announcement, speak your mind, or sell somethingPost something
See more »